2025 was a significant year for the SecureBio AI team – reflected by the fact that we tripled our headcount, adding to our roster of world-class research scientists and engineers. By deepening our interdisciplinary talent pool across virology, AI/ML, software engineering, and policy, we expanded our capacity for running multiple large projects in parallel. This allowed us to make major strides in turning AI-bio risk evaluation from a set of bespoke projects into something closer to an ecosystem.

A central focus of our work last year was building evaluation tools that move beyond “does the model know biology,” towards “does the model meaningfully expand harmful biological capabilities?” In practice, this meant building more rigorous benchmarks, developing deeper agentic evaluations, and more direct integrations with real-world safety pipelines. We increased the reach of our technical outputs by briefing senior decision-makers on how third-party evaluations are run and interpreted. This year we will intensify our work on mitigations, push the envelope on understanding frontier capabilities across agents, biological AI models, and work to increase our understanding of how advances in AI translate to actual risk.

If you’re working on adjacent problems (especially evaluation standards, mitigation tools, or safety audit readiness), we’re always keen to to compare notes and collaborate.

Benchmarks and Evaluations

Virology Capabilities Test (VCT) was our flagship effort in 2025. We designed and executed a large-scale benchmark and published the primary research paper, helping establish VCT as the leading reference point for AI-bio risk discussions and enabling large-scale expert/model comparison.1 In addition we expanded coverage across different parts of the biological landscape with non-public benchmarks like the World Class Bio benchmark (WCB), the Molecular Biology Capabilities Test (MBCT), and the Human Pathogen Capabilities Test (HPCT), each aimed at capturing distinct slices of capability that matter for real-world misuse, not just textbook knowledge.

We also pushed further into agentic and long-form evaluation of biological AI models (BAIMs). The Agentic Bio-Capabilities Benchmark (ABC-Bench) and Agentic BAIM-LLM Evaluation Benchmark (ABLE) were designed to test whether agentic systems can complete key components of dual-use workflows, such as using biological AI models to redesign viral proteins. ABC-Bench shows that AI agents can increasingly undertake biosecurity-relevant tasks across both in-silico design and wet-lab experiments, while ABLE shows that agents can effectively utilize AI protein design tools, but remain inconsistent at applying their knowledge across a multi-step computational design workflow. Several of these efforts were presented at NeurIPS and used in multiple real assessments, helping inform discussions about how agentic systems change the risk landscape.2

Our benchmarks and evaluations have been cited in model cards or risk management frameworks for major releases from all the frontier labs, including Anthropic, Google DeepMind, Meta, OpenAI, and xAI. VCT was also referenced throughout the House Energy and Commerce Committee hearing on Examining Biosecurity at the Intersection of AI and Biology3, and received coverage in Time. It remains an open question how model performance on benchmarks translates to changes in the real-world risk landscape; addressing this uncertainty is a key focus of our 2026 efforts.

Mitigations, Cross-Team Collaborations, and Safety Pipeline Deliverables

A major milestone for us this year was not just researching the capabilities of models once they are released, but actually working with frontier labs to make models safer. We delivered training datasets and lists of dangerous biological topics for pretraining data filtering that were directly fed into the design of several frontier models. This work sits at the interface of research and operational safety: it is empirically grounded, hard to game, and compatible with how frontier labs actually train and deploy safeguards. It reduces models’ capabilities for catastrophic bio, misusewhile preserving their beneficial capabilities.

We also contributed to work on AI-bio jailbreaking mitigations, helping to characterize how safety systems fail under pressure and what kinds of mitigations appear most promising. Complementing this, we secured dedicated funding for operational security research and follow-on methods development. This reflects a broader shift in the field toward treating misuse prevention as an end-to-end systems problem rather than a single refusal metric.

The team also conducted an exciting collaboration with the NAO, helping to build an AI tool that triages metagenomic sequences flagged by the NAO’s detection system for further investigation. The tool analyzes concerning sequences, enriches them with relevant facts and context, and surfaces the most important for human-expert review. We’re excited to undertake further such work that leverages each team’s strength.

Funding, Delivering, and Scaling our Work

On the funding side, we secured a multi-year Coefficient Giving grant that enables longer-horizon planning, deeper technical investment, and hiring. We also received multiple grants from the Foundation Model Forum, including support for research into agentic AI, operational security work, and follow-on research from earlier pilots.

We delivered several major evaluation projects with frontier labs, including expert baselining, quality control for evals, and holistic prerelease assessments. We also moved toward a more durable model by licensing evaluations to multiple frontier labs, creating a durable pathway for our tools to be used in real decision-making contexts rather than remaining purely academic artifacts.

Finally, we broadened our government-facing portfolio with a US CAISI Bio R&D contract and participation in an EU AI Office contract to deliver bio-evals, both steps toward institutionalizing evaluation as part of emerging governance and standards ecosystems.

Strategy and Policy Developments

As our technical evaluation capacity grows, the question of “what should decision-makers do with these results” becomes more pressing. We spent substantial effort to ensure our work aligned with the institutions that shape audit expectations and safety norms.

We delivered a national security briefing on frontier model capabilities, helping bring empirical evaluation results into senior biosecurity decision-making contexts. The team also presented to export-control policymakers through the BIS Technology Advisory Committee and briefed US CAISI staff working on bio-related standards, both efforts to translate technical work into governance-ready inputs.

Looking Ahead

The through-line of 2025 was a shift from one-off evaluations to a mature ecosystem posture: credible benchmarks, agentic evaluation methods, mitigation artifacts that plug into real safety pipelines, and growing institutional relevance with governments and standards bodies.

In 2026, we plan to keep pushing in four directions:

1. Mitigation strategies that measurably reduce risk in deployed systems and in contexts where malicious actors can use multiple models in combination.

2. Deeper work on measuring and understanding the “hard cases”: agentic systems, integrated toolchains, frontier BAIM capabilities, and most challenging of all, super-expert capability uplift.

3. Systematic and routine evaluations that are fast, reliable, and decision-relevant.

4. Better understanding of how increases in AI model performance translate to changes in real-world risks.

If you’re building adjacent infrastructure or want to pressure-test your own evaluation/mitigation approach, please reach out.

Ben Mueller, Executive Director and Seth Donoughe, Director of AI

We’re excited to share that the European Commission’s AI Office has chosen SecureBio to develop biological threat evaluations (‘evals’) to support the implementation of the EU’s landmark Artificial Intelligence Act. This is SecureBio’s second major engagement by a government to build such evals, following the award of a contract by the US government’s Center for AI Standards and Innovation.

As part of a consortium led by FAR.AI, SecureBio and other leading AI research organizations successfully completed a bid for Lot 1 of the EU AI Act’s “Technical Assistance for AI Safety” tender. In line with the protections set out in the EU AI Act, the consortium will monitor how AI might pose risks by expanding access to chemical, biological, radiological, and nuclear (CBRN) threats. Other members of the consortium include SaferAI, GovAI, Nemesys Insights, and Equistamp.

Over the next three years, SecureBio will be focused on:

• Delivering pre-made biological evaluations: We’ll integrate and deliver established, publicly available biological evaluations like our Virology Capabilities Test into the Commission’s assessment framework.

• Developing custom evaluations: We’ll design and build new AI evaluations to address gaps in current coverage of biological threat scenarios.

• Performing quality assurance and human baselining: We’ll establish rigorous quality standards for biological evaluations, including human baseline studies to calibrate AI performance against experts.

• Building evaluation infrastructure: We’ll help streamline and simplify the biological evaluation process for the EU’s AI Office, enabling consistent assessment of frontier models as they emerge.

Ben Mueller, Executive Director of SecureBio, said: “AI is poised to bring about tremendous progress in the medical and life sciences. At the same time, the technology generates risks that need to be better understood. We are proud that SecureBio’s AI team has been selected by the European Commission to support its efforts to understand risks posed by advanced AI. Our staff of scientists, researchers, and software engineers has a strong track record of producing rigorous, balanced evaluations to understand the capabilities and risks of frontier models in biology and associated fields, and we are pleased to contribute to this important undertaking.”

These evaluations help shape our understanding of biorisks associated with advanced AI. While much of our work remains confidential, you can see references to our evals in public system cards, including Claude 3.7 Sonnet, Claude 4, GPT-4.5, o3-mini, o4-mini, and Gemini 2.5 Pro. Our work has also informed emerging AI risk management frameworks. One public-facing example is xAI’s risk management documentation.

We also developed and ran the Virology Capabilities Test (VCT), a first-of-its-kind benchmark that measures the ability of a model to provide expert-level practical assistance in work with viruses. It was covered in Time.

The recent CBRN risk assessment of the Claude 4 model family used not only VCT, but also two additional SecureBio benchmarks: A DNA synthesis screening evasion task set for LLM agents and a set of creative biology scenarios that serve as proxies for novel biology abilities. Anthropic’s CBRN evaluation also included the long-form virology tasks; a suite of evals, co-developed with Deloitte, Signature Science, and SecureBio, which tests end-to-end pathogen acquisition.

More is coming: we are developing further targeted evaluations, testing how AIs uplift human abilities on dual-use biology, researching promising mitigations, and conducting holistic safety assessments. We will publish our findings; we strongly support greater transparency and accountability in AI deployment.

None of this would be possible without our team, an extraordinary group of engineers and biologists working with diligence, precision, and purpose. They conduct world-class research with minimal resources and are some of the finest minds to work in this space.

Their motivation is simple but vital: to help ensure AI contributes to a future of profound scientific and human progress, while reducing the risk of catastrophic misuse of these models.

System cards featuring SecureBio evals and benchmarks

🔗 Claude 3.7 Sonnet
🔗 Claude 4
🔗 GPT-4.5
🔗 o3-mini
🔗 o4-mini
🔗 Gemini 2.5 Pro

The Challenge of Measuring Scientific Expertise

Large language models (LLMs) have been at the forefront of AI progress over the past years. They help people write and edit text, assist with coding, and are great partners for brainstorming. They will patiently accompany you down rabbit holes, explaining and clarifying new things along the way. What has been less obvious is whether this general usefulness extends into the sciences, where success often relies on hard-to-find tacit knowledge, hard-won practical experience, or interpreting and connecting disparate pieces of information.

For our pandemic prevention work at SecureBio, we are especially keen to understand how AI progress impacts virology research. Virology is intrinsically dual-use. Powerful AI assistance could accelerate beneficial virology research on vaccines and antivirals. It might also enable malicious actors to more easily misuse viruses to cause harm. We needed a way to robustly measure an AI’s ability to assist in virology work.

Unfortunately, helpfulness for a practical science like virology is hard to measure. Many traditional AI benchmarks test knowledge retrieval using exam-style multiple choice questions. They ask models to answer academic facts, or perhaps write analysis code. But this approach misses a major aspect of successful laboratory work: troubleshooting experiments and protocols. Practical lab work often relies on the ability to interpret ambiguous results—often done visually—and then determine next steps, drawing from tacit knowledge residing not in textbooks but in lab meetings and hallway conversations. These abilities aren't easily articulated, let alone quantified.

Quantifying the Tacit

We developed the Virology Capabilities Test (VCT) to attempt exactly this quantification. We created a benchmark that measures an AI’s ability to provide the contextualized, visual troubleshooting assistance that researchers require in actual labs. VCT targets virology methods with dual-use potential as well as other closely related methods. It excludes general molecular and cellular biology methods, as well as a small portion of virology material that we judged excessively hazardous.

VCT comprises 322 multimodal questions covering practical virology problems. Each presents an experimental scenario—often with an image—and asks what went wrong or what to do next. The questions are designed to be:

• Important: testing knowledge essential for competent and successful lab work

• Google-proof: answers cannot be easily found through web searches

• Validated: answers are verified through expert peer review

To ensure these qualities, we designed a rigorous question-creation process:

1. We recruited virologists with at least one year of graduate-level research experience (the average ended up being just shy of 6 years)

2. Each question underwent double-blind peer review and editing by other experts

3. Questions that were answerable by non-experts using web search were eliminated

For the recruitment, we realized what a boon academic conferences are. Cramming a lot of project-context into an interesting cold email is difficult (though not impossible, we did recruit many participants through email outreach!), but talking to dozens of virologists about the project at the American Society of Virology’s 43rd annual meeting generated a lot of interest and follow-up participation.

After the submission phase concluded, 68 experts contributed over 500 questions drawn from their actual lab experiences. After review, editing, and non-expert testing, 322 questions remained: 221 containing an image, and 101 text-only questions. These questions can be answered in open-ended or multiple-choice format.

Finally, to establish a human baseline, we had expert virologists answer question subsets that were specifically tailored to their self-declared areas of expertise. That way, we could measure how well virologists fare when asked about methods they consider among their top competencies, rather than asking virologists about any virology-related method.

AIs Began Exceeding Human Virologists in February…2024

To our surprise, the performance gap between humans and LLMs is stark:

• Our expert virologists averaged 22.1% on question subsets individually tailored to their own areas of expertise

• The leading LLM, OpenAI's o3, achieved 43.8% accuracy on the whole benchmarks, and outperformed 94% of virologists on the matched question subsets.

• Google's Gemini 2.5 Pro scored 37.6%, placing in the 81st percentile.

• Anthropic's Claude 3.5 Sonnet (Oct ‘24 version) reached 33.6%, ranking in the 75th percentile.

• The first LLM to beat the median expert virologist, Gemini 1.5 Pro, was released in February 2024.

It was not the advent of reasoning models in the fall of 2024 that gave LLMs the edge over expert virologists; frontier models like Gemini 1.5 Pro or Claude 3.5 Sonnet have been able to match or exceed the ability of human experts to provide practical troubleshooting assistance for over a year, and the disparity between humans and models is widening.

Every VCT scenario represents a few virologists’ consensus on the right way to solve a problem. Thus, the results indicate that individual experts are less effective than we anticipated at identifying the expert consensus—whereas leading models are surprisingly good at identifying the expert consensus. We interpret this result to mean that the training corpus of leading models has a strong representation of expert human consensus in this domain, and we effectively are seeing “the wisdom of the crowd” at work, mediated by LLMs.

Dealing with the Downsides of Democratized Expertise

As a virologist myself, I find these results simultaneously impressive and familiar. During my PhD, many of my or my fellow researchers’ wet lab excursions entailed seeking advice from multiple colleagues—sometimes still PhD candidates themselves—who had spent the last two to twenty years working with many variations of a specific technique: specialization and collaboration. You would show them pictures of your cells or gels, share your lab notebook sections, and tell them next steps you’re considering (and you obviously don’t approach your PI with something that 2 minutes of Googling would solve!).

This is precisely the experience VCT simulates. Since LLMs have a broad exposure across the whole field (or rather, any field), they've synthesized the equivalent of conversations with hundreds of specialists from scattered pieces of information hidden in papers, online forums, and patents—knowledge previously considered tacit for individual human experts. Combined with the extensive reasoning and web search that frontier models employ, this creates a formidable troubleshooting assistant.

It is important to point out that VCT does not measure hazardousness per se. All techniques covered are standard methods that are used daily for beneficial research. What VCT shows is that AI systems can provide the kind of specialized troubleshooting advice that typically requires years of training—and this applies equally well to methods that are benign and those that would be particularly concerning for causing harm.

How accessible do these models make virology to non-experts now? One might object that asking such detailed troubleshooting questions already requires considerable expertise and familiarity with the subject. To some degree, this is correct. But existing resources—tutorials, manuals, and the same endlessly patient AI models that also excel in expert assistance—can help you reach the point where you get stuck without expert consultation. VCT covers precisely those problems on which actual virologists think you’re most likely to fail without experienced guidance.

Follow-up studies performed by SecureBio and others will soon examine whether AI assistance improves experimental outcomes in actual labs. During our evaluations, we also observed a few consistent cases where AIs disagree with expert-provided answer keys, prompting us to think about how to reliably measure AI progress on topics in which expert knowledge stops being a reliable yardstick.

What's clear is that the conversation in science labs will inevitably change. A first-year graduate or undergraduate student can now describe a failed experiment, show an image, and receive guidance comparable to consulting a senior colleague. The boundary between novice and expert—always porous in practice—is becoming even less defined.

Anecdotally, however, the diffusion of this technology into the lives of practicing researchers is still slow. After completing the benchmark, we asked some of our participants whether an LLM matching their performance on VCT-like problems would increase their productivity in the lab. Almost all of them envisioned large productivity gains and significant acceleration of their research. Yet when asked whether they currently use LLMs in their work, we received a unanimous ‘no’.

Experts will and should leverage AI assistance for dual-use research, but we must treat the ability to provide expert assistance itself as dual-use: requiring additional oversight, but accessible to legitimate researchers and institutions. We suggest that publicly available models should not offer expert guidance on methods that would be most conducive for causing mass harm, such as detailed culturing protocols for organisms falling under biosafety levels 3 or 4.

We would love to hear your thoughts and questions about VCT and the intersection of AI and biology at benchmarks@securebio.org.