Well-calibrated safeguards for biological capabilities of AI should balance refusal of unsafe or malicious queries with permission of safe requests. The vast amount of AI usage in biology is legitimate, helping scientists do research or students learn new concepts. However, a narrow slice of biohazardous information and capabilities (e.g. how to weaponize a biological agent) should be secured.

Our new benchmark, BioTIER (Biological Targeted Information for Exclusion and Refusal), measures how well AI models manage the balance between safety and utility in biological domains via refusal behavior. This dual evaluation relies on three graduated risk domains. Each domain is paired with a taxonomy and recommendations toward a standardized definition of “high-risk” biology.

We highlight three findings:

Huge variation in refusal behavior exists across the ecosystem, but under- and over-refusal are not a one-for-one trade: in short, a model can improve safety without over-refusing.
The strongest guardrails are found in a handful of highly-capable closed-weights models – the most robust safeguards offer little practical security if highly-capable and permissive open-weights models are freely available.
Model scores on BioTIER change from one day to the next – this is evidence that system-level safeguards are being modified over time, and underscores the need for frequent evaluation.

The BioTIER refusal tracker is live, and we will update it regularly as new models are released. Further details can be found in the associated publication.

The Safety versus Utility Problem

General-purpose AI systems know a lot about biology. They can explain CRISPR to a high schooler who dreams of becoming a scientist, can help a graduate student troubleshoot a stubborn PCR, and contribute to the design of better therapies for some of the worst diseases we know.

But that same knowledge could also contribute to biological misuse. Methods for making vaccines can be used to engineer and produce pathogenic viruses. Mapping pathogen evolution to provide predictions for enhanced preparedness could also reveal blind spots in our immune system. Tools to create new treatments can be applied to study dangerous toxins. The list of so-called “dual-use” applications goes on.

So, as well as helping to progress the frontier of biology, AI could lower the technical bar for bad actors trying to cause mass harm with biological approaches by providing easy access to, and in-depth guidance through, dangerous and dual-use knowledge.

How are AI developers gating this knowledge? At the level of refusal behavior, two approaches have emerged:

Refuse a lot. When in doubt, decline. Safer, but this approach could impact clinicians, frustrate students and educators, and hinder legitimate beneficial research.
Refuse very little. Trust users to act in good faith, but run the risk of providing dangerous knowledge to anyone who asks.

To reach that “Goldilocks” zone, where a model refuses on biohazardous prompts and answers benign ones, we need reliable measurements. And before we can measure, we need to understand what we’re measuring in the first place.

Can models tell the difference between safe and hazardous biological queries?

The vast majority of biological knowledge is beneficial, and only the tiny fraction that’s genuinely dangerous should be locked down. To aid in this, we present BioTIER, a benchmark and taxonomic framework to measure both the under- AND over-refusal behavior of AI models.

\"\" — **Figure 1.** Conceptual representation of the relative prevalence and overlap of high-risk and malicious knowledge that should be restricted, compared to the vast wealth of knowledge with beneficial use cases. “Close-to-boundary” knowledge represents information related to high-risk domains that should be accessible, and is essential for better defining the threshold of risk which remains a gradient, rather than a solid distinction.

Developing a taxonomy for biological topics

BioTIER is a benchmark of 542 expert-written prompts. Every prompt was written by hand by one of 15 PhD-level subject matter experts, then validated by consensus approval across three rounds. These prompts are sorted into three sets, represented by the schematic below.

CA (Catastrophe Avoidance) — 249 prompts covering the most high-risk information. Models should refuse these for everyone.
BD (Biomedical Dual Use Research of Concern) — 149 prompts. Dual-use research knowledge that could be misused. Models should refuse for the general public but permit for controlled-access verified researchers to enable beneficial application.
RB (Related Biology) — 144 prompts. Benign biology and “close-to-boundary” biosecurity adjacent content that brushes up against risky topics but is itself harmless. Models should always answer these.

\"\" — **Figure 2.** BioTIER Risk Taxonomy and Theoretical Framework. SA tag = a secondary tag to specifically highlight prompts about Select Agents on the HHS/USDA/Australia Group list of regulated agents. *CA access for the biosecurity community would require additional in-depth vetting and would be intended to enable evaluation of capabilities remaining post-mitigation.

We split these sets into two evaluation components: BioTIER-refuse (CA + BD: which models should decline) and BioTIER-permit (RB: which models should answer).

Results from 52 AI models

We looked at a wide variety of models from 10 major developers, going back as far as 2022 all the way to recent 2026 releases. Our analysis covers small models like Claude Haiku 4.5 and large models like GPT-5.5 Pro. And perhaps most importantly, we run BioTIER on open and closed models.

Measuring over- and under-refusals

On BioTIER-refuse, model behavior spanned more than 90 percentage points, with the least cautious model refusing under 10% of dangerous queries, while the most cautious refused almost 100%. Many of the models that refused infrequently were older or smaller, and no open-weight model came close to the frontier. Even the best-scoring open-weight model refused less than half of our prompts querying dangerous biological topics.

The strongest guardrails remain concentrated in a few highly-capable closed models. Importantly, the capabilities of closed-weight models are only ahead of those of open-weights by around 4 months. This means that even the most robust closed-model safeguards offer little practical security if highly capable and permissive open-weight alternatives are freely available.

We also identified specific and stark topical gaps within current mitigations using BioTIER’s detailed taxonomy. This taxonomy enables rapid development of targeted sub-evaluations to probe these specific vulnerabilities and inform developers for actionable patching.

On BioTIER-permit, the picture was far more uniform: almost every model answered almost every benign query, with many sitting at the 100% ceiling, and the lowest still scoring 75%. For now, over-refusal of legitimate science is a minority failure mode for most general access models.

Balancing the trade-off

Plotting these results against each other reveals a pattern: the models with the highest safety on BioTIER-refuse have a reduced performance on BioTIER-permit.

\"\" — **Figure 3.** Comparing safety on BioTIER-refuse (y-axis) with utility on BioTIER-permit (x-axis). Each point represents one model, colored by developer. The top-performing model from each developer and most recent frontier models are labeled.

But there’s an important nuance: when we looked at which BioTIER-permit prompts were being incorrectly refused, they most often weren’t queries about truly benign biology. Instead, overly-cautious models were refusing questions that lay at the risk boundary, sitting right next to genuinely dangerous areas of knowledge. These findings suggest that over-refusal by these models is a result of judgment-call errors at the edge of risk, rather than untargeted blanket refusals. So, the under- versus over-refusal tradeoff is real but bounded, and can likely be addressed by better definition of the line between dangerous and benign knowledge.

\"\" — **Figure 4.** Example of over-refusal at the risk-threshold. The user question is a representative “close-to-boundary” prompt from BioTIER-permit. An AI system with well-calibrated safeguards would answer this query. Model 1 (left, green) responds, which is the intended behavior. Model 2 (right, orange) refuses to answer, providing an example of over-refusal.

Biosecurity safeguards change over time

\"\" — **Figure 5.** Changing behavior of frontier model safeguards over time. BioTIER-refuse (correctly refusing hazardous prompts) score is shown for four models over two time points: May and July. Model scores change for three out of four of the models shown. BioTIER-permit (correctly answering benign prompts) is shown in the right panel. There is comparatively little change in the permit rates between May and July runs.

With an ever changing scientific and risk landscape, BioTIER measures a moving target. This was perfectly demonstrated during the three month period of finalizing the evaluation and writing our paper, in which we detected staggering changes to model performance. Google’s update to their mental health related safeguards made headlines, but we also identified changes to their biology safeguards, with an almost 30% increase in the refusal behavior of Gemini 3.1 Pro on BioTIER-refuse due to introduction of API-level refusals. Importantly, this increase in BioTIER-refuse performance had no impact on BioTIER-permit compliance. At the same time, we saw GPT-5.5 Pro and Grok 4.20 drop in performance on BioTIER-refuse, while Claude Opus 4.8 showed little change at all. Four models from four developers over 3 months exhibited completely different safeguard shifts, demonstrating how essential longitudinal analysis of safeguards will be.

\"\" — **Figure 6.** The safeguard layer responsible for refusing varies among firms and over time. Refusals on BioTIER-refuse can be triggered by API content filters (shown in blue) or model-classifiers (gold). Within model changes can be dramatic, as shown by Gemini 3.1 Pro.

Building on BioTIER

Advancements in biology and technology are constantly reshaping biological risks, so we will keep expanding BioTIER as the risk landscape changes, including:

Updating the BioTIER taxonomy to include novel and emerging risks.
Developing sub-evaluations based on topical gaps in mitigations.
Testing jailbreak robustness.
Expanding the prompt-set to include non-English languages.
Adding granularity to scoring to assess the quality of the responses and model level refusals.
Expanding topic areas to chemical, radiological, and nuclear domains.

We present a BioTIER tracker to complement our recently released Benchmarks Dashboard. This tracker will provide a dynamic picture of refusal-behavior across the ecosystem over time, and encourage developers to prioritize more nuanced mitigation approaches.

BioTIER provides the framework needed to help pave a path toward a biosecure AI ecosystem in which access to the tiny fraction of genuinely dangerous biological knowledge is restricted, while access to the vast majority, essential to science and medicine, is maintained.

BioTIER is made available to AI developers and verified biosecurity researchers via a gated request process. Contact: ai@securebio.org

","has_dynamic_content":false,"truncated_body_text":"Well-calibrated safeguards for biological capabilities of AI should balance refusal of unsafe or malicious queries with permission of safe requests. The vast amount of AI usage in biology is legitimate, helping scientists do research or students learn new concepts. However, a narrow slice of biohazardous information and capabilities (","wordcount":1458,"post_preview_limit":null,"language":"en","postTags":[{"id":"c07b5b35-35a3-4320-aaab-93579170f94a","publication_id":4680018,"name":"evaluations","slug":"evaluations","hidden":false},{"id":"d86764c5-76a6-4ea5-ad10-071da9ad6334","publication_id":4680018,"name":"AI","slug":"ai","hidden":false},{"id":"ed49ba5b-4c8f-47c3-9961-758f14bde88e","publication_id":4680018,"name":"biosecurity","slug":"biosecurity","hidden":false}],"postCountryBlocks":[],"headlineTest":null,"coverImagePalette":{"Vibrant":{"rgb":[206,107,18],"population":192},"DarkVibrant":{"rgb":[20,116,52],"population":2307},"LightVibrant":{"rgb":[220,196,140],"population":4},"Muted":{"rgb":[76,156,100],"population":15},"DarkMuted":{"rgb":[64,64,64],"population":2},"LightMuted":{"rgb":[217,198,171],"population":57}},"publishedBylines":[{"id":494480600,"name":"Eleanor Marshall","handle":"eleanormarshall3","photo_url":"https://substack-post-media.s3.amazonaws.com/public/images/c4204b13-c8ec-4885-b090-e4a9a29e87bc_800x800.jpeg","bio":"Do good, learn stuff. AIxBiosecurity.","profile_set_up_at":"2026-07-15T17:39:01.214Z","reader_installed_at":null,"is_guest":true,"bestseller_tier":null,"status":{"bestsellerTier":null,"subscriberTier":null,"leaderboard":null,"vip":false,"badge":null,"subscriber":null},"primaryPublicationId":10044162,"primaryPublicationName":"Eleanor's Substack","primaryPublicationUrl":"https://eleanorsb.substack.com","primaryPublicationSubscribeUrl":"https://eleanorsb.substack.com/subscribe?"}],"reaction":null,"reaction_count":17,"comment_count":0,"child_comment_count":0,"audio_items":[{"post_id":207343746,"voice_id":"en-US-OnyxTurboMultilingualNeural","audio_url":"https://substack-video.s3.amazonaws.com/video_upload/post/207343746/tts/bc7c6f5c-1abb-4b71-a528-769f5d35cfab/en-US-OnyxTurboMultilingualNeural.mp3","type":"tts","status":"completed"}],"is_geoblocked":false,"is_editor_preview":false,"hasCashtag":false},"initialComments":[],"initialAutomodHiddenComments":[],"bannedFromNotes":false,"canonicalUrl":"https://securebio.substack.com/p/measuring-biosecurity-safeguard-effectiveness/comments","ogUrl":"https://securebio.substack.com/p/measuring-biosecurity-safeguard-effectiveness/comments","activeLiveStream":null,"freeTrialCoupon":null,"defaultCoupon":null,"saleCoupon":null,"isChatActive":false,"publicationPageAlert":null,"features":{},"browser":{},"showCookieBanner":false,"disabledCookies":[],"dd_env":"prod","dd_ti":true,"sessionSampleRate":1,"designTemplate":null}

SecureBio

Measuring Biosecurity Safeguard Effectiveness…