When AI outgrows our best scientists

Seth Donoughe’s parting thoughts as he hands SecureBio AI to Jasper Götting

I am stepping down as Director of the AI group at SecureBio. The decision is mine, and it is not an easy one: SecureBio AI (SBAI) has never been stronger, and I would happily have stayed. I am leaving only because complementary work needs doing, and I feel compelled to take it on.

I am pleased to pass the baton to Dr Jasper Götting, who has led SecureBio’s AI Research since 2024. In that time he has been both a leading thinker and a tireless doer, driving forward the young science of evaluation and helping to reimagine biosecurity for an era of rapid technological change. He has been a close partner to Ben Mueller and me on every technical and strategic decision the group has made over the past two years. In addition to his many contributions on the science of AI evaluation, he is a virologist by training, with deep biosecurity experience. SecureBio’s AI work will flourish under his direction.

During my time at SecureBio, general-purpose AI has gone from curiosity to geopolitical headline, and will likely wind up being the most consequential technology that humanity has created. Against this backdrop, my attention has been on how AI will change the science and practice of biology. In particular, I have tried to develop robust ways to measure an AI’s impact on life sciences work, with the ultimate goal of maximizing the upside for beneficial research while reducing the possibility of hazardous misuse. We have made rapid progress, yet there are several challenges that strike me as especially pressing and underaddressed.

[1] Understanding scientific capabilities of AI when it exceeds all human experts. So far, the field has assessed the scientific abilities of AIs by running them through benchmarks constructed out of quantized chunks of human expert knowledge. The expert scientists write the challenges and (ideally) other scientists carefully check that the assessment is well-grounded. But once AI systems begin to outperform our most experienced and capable scientists, the experts cannot produce the needed ground truths, and we will no longer have any dynamic range remaining in evaluations built following such a process.

What to do next? One avenue is to assess AIs on tasks that are too challenging for expert humans to perform, so long as they remain human verifiable (e.g. discovering a novel mathematical proof or predicting the outcome of original empirical experiments). Biology is especially apt here: living systems are extraordinarily complex, providing a limitless well of prediction tasks for which nature itself can serve as the adjudicator. SBAI is investing a good deal of effort in this direction, and so far it is promising. However, there are important limitations to this approach: it is not suitable for cases in which the output is a novel idea, insight, judgment, research direction, or theory. For all of these, our test of their value is typically far off and diffuse. Unfortunately, this is also the territory of the most consequential science.

On this topic, there is a form of skepticism we often encounter: that an AI could simply never exceed the general capabilities of an experienced biologist, because so much of the expert’s knowledge is either embodied or a matter of intuition that cannot be written down. How much of a scientist’s expertise is truly and irreducibly tacit is an important empirical question, one that SBAI is actively working to answer. So far, we’ve found that under careful study, we scientists generally tend to overestimate our own abilities, overestimate the level of consensus between scientists on putatively uncontroversial topics, and overestimate the difficulty of converting scientists’ knowledge into legible form. Collectively, these imply that the importance of expert tacit knowledge is less important than many think.

Thus, although it is still not a foregone conclusion, we should prepare for a possible near future in which ~all the cognitive work of scientists could be done by AIs without our input. This prospect does not bring me joy. I was a research biologist studying evolutionary and cell biology for 16 years, a period of my life that was rich with delight and discovery. However, we scientists cannot usefully contribute to shaping the future if we dismiss the crucial technological changes out of hand.

[2] Studying the hybrid human-AI scientist. The producer of scientific knowledge is increasingly neither the scientist nor the AI tool, but the two together, with ongoing feedback in both directions. This means that measuring the model alone (as in [1], above) can mislead us, and that whatever guardrails we hope to impose on the most dangerous work (the problem I turn to next, in [3]) must be built around how the pair actually behaves, rather than how the AI itself behaves in the context of high-throughput, automated evaluation. What is happening when scientists use these systems well? What patterns emerge in the messy, complex ‘natural habitat’ of the lab? Controlled uplift studies will be essential – another area in which SBAI is doing ongoing research – but they will overestimate performance in some ways, and badly underestimate it in others. It will be increasingly important to study the human-AI pair with the same seriousness we currently bring to its parts.

[3] Establishing a safe process for AI-enabled discovery. Over millennia, we have followed a tried-and-true approach for handling the downsides of new technological discoveries. We try, fail, learn, and revise on repeat, publishing all findings, until a field (and a society) converges on norms for a technology’s safe use. This iterative procedure has served us well, but a key condition was that the cost of each mistake was one we could safely absorb. Now, with AI-enabled science, we could be uncovering transformative and unprecedentedly disruptive scientific capabilities. For instance, it is plausible for us to engineer pathogens more harmful than anything nature is likely to supply. In a world that has not yet solved the problem of activating a robust response to infectious disease, learning by doing is a price we should now be unwilling to pay. The core problem, then, is to establish a workable process for science: one in which we use AI to empower scientists to discover and engineer beneficial things, while minimizing their capacity to cause catastrophic harm. I do not expect us to reach complete consensus on how this should work; there are enough unknowns that consensus is too high of a bar. But we need a good-enough and functional process very soon.

I leave all three of the aforementioned challenges in good hands: Jasper and the whole SBAI team are exceedingly talented and mission-oriented researchers. SBAI will continue to grow in capacity and scope, and it will address the aforementioned challenges head-on. They work alongside SecureBio’s Detection group, which is animated by the same conviction: that our state-of-the-art AI and biotechnology can be fruitfully deployed for protecting people.

There is one important thing SBAI is neither institutionally positioned nor resourced to do on its own: to coordinate, and to prepare for the wide range of future paths that lay before us as technological change accelerates, including major disruptions to how our society functions. I am moving to RAND to contribute on that front. My work will center on readiness, and on the strengthening of ties among those who will have to act when it matters. I will remain at SecureBio as a Senior Advisor on Research & Policy, where I expect to stay engaged with our future-oriented work on scientific capabilities of super-expert AIs that are on the horizon.

It has been a privilege to have a front row seat to observe how much has changed in such a short span, including the field of biosecurity, the young science of evaluation, and the community trying to make AI go right. Most encouraging of all has been the caliber and dedication of the people now drawn to this work, many of them here at SecureBio. I am deeply grateful to Ben and the rest of SecureBio for welcoming me to this extraordinary organization, and for being such excellent and supportive partners. I eagerly look forward to what they will do next.

A New Chapter for SecureBio AI

Ben Mueller on the future of SecureBio AI under new leadership

In February of 2024 I was introduced to Seth, a post-doc based in Chicago. His research – building tools and computational models to understand how cells signal, form tissues, and evolve – sounded interesting and relevant to what we were trying to do: namely, develop ways to assess, evaluate, and understand the changing capabilities of advanced LLMs in the domain of biology.

SecureBio’s AI programme emerged as a bootstrapped initiative in response to demand from frontier AI firms to bolster their ability to make sense of what the models they were building could do in biology – for good and for ill. This was new ground for all parties involved. We had to build the plane as it was taking off, and turn it into a flying machine that was robust to all kinds of external turbulence, and with enough fuel in the tank to reach a destination unknown.

I knew after a few meetings with Seth that he was the right man to lead this programme. To prepare for what lay ahead, we needed to build measurement tools using a combination of rigorous empiricism, attention to detail, and aggressive methodological innovation. The missing ingredient was the personal dynamism needed to rally people behind a new discipline: that of measuring the capabilities of computational tools far in advance of anything humans had ever built before. Seth had that in spades.

And so we got to work. Seth assembled an incredible team of dedicated biologists and engineers at the top of their game, all of whom resisted the temptation to join a biotech or AI firm in pursuit of riches, and instead devoted themselves to painstakingly advancing the frontier of model measurement and evaluation in order to make the AI-bio convergence safe. Our first breakthrough was building the Virology Capabilities Test, with generous support from the Frontier Model Forum, which to this day remains the gold standard to assess models’ capabilities to assist with dangerous virology relative to human experts (its successor, VCT-2, is soon to be released – watch this space). On it went, as we built more and more evals to keep pace with the ever-faster progress of AI’s skills and knowhow. We also built out a large network of subject matter experts who contribute to our benchmarks and ensure the content thereof is world-class, which is the measurement standard we need to apply against AI.

Three of SBAI’s recent outputs make me particularly proud:

WCB: World Class Biology. A static benchmark of creative and exceptionally hard biology problems spanning non-hazardous and dual-use material, built to detect when an AI model can stand in for a PhD scientist with rare, specialized expertise. It’s the biology analogue of FrontierMath: experts score under 15%, because success often depends on insights that are unpublished or not easily searchable.
ABC-Bench: Agentic Bio-Capabilities. A suite of three agentic evaluations (which was accepted to ICML 2026) that measures not what a model knows but what it can do, testing whether AI agents can carry out discrete technical steps along the pathway to assembling viral DNA. Remarkably, frontier models matched or outperformed PhD biologists (who averaged 24%), and in a real-lab validation an AI-generated protocol run on a liquid-handling robot successfully assembled DNA confirmed by whole-plasmid sequencing. This provides concrete evidence that agentic bio-capabilities are advancing, and it’s now used by major AI developers for pre-release safety testing.
ABLE: Agentic BAIM–LLM Evaluation. A suite of agent evaluations measuring whether a model can design and execute a workflow that uses specialized biological AI models (BAIMs) to enhance a human-infecting pathogen’s properties. It targets one of the newest and least-understood frontier risks: the intersection of general-purpose agents and purpose-built biological AI tools. The evaluation is deliberately held for independent assessment rather than shared via open licensing.

Now, SecureBio is at the point where we build and ship best-in-class mitigations against AI/bio misuse: tools that excise the most dangerous features of biology while preserving all the good it promises to bring humanity. All the frontier AI labs, as well as all major government AI agencies, work with SecureBio’s AI team. Thanks to Seth’s leadership, we are at the forefront of developing evaluations and artefacts to make the convergence of AI and bio safe.

While SecureBio AI’s efforts continue to grow, we are all mindful that we cannot tackle it alone. The age of superhuman AI scientists is around the corner: the world needs to prepare for it, and it must be a joint effort, involving many organizations beyond SecureBio, working towards a common purpose. Therefore, Seth is moving to RAND to continue working on AI safety and biosecurity, with a particular focus on emergency preparedness. He grappled hard with the decision, before reaching the conclusion that the team he built at SecureBio is more than capable to take it from here. Thanks to Seth’s ability to attract and retain superbly talented researchers, we have a group of outstanding scientists and engineers in place. There is no greater testament to Seth’s leadership than that, rather than turning to outside hires, we instead could simply promote our incredible in-house talent to take over.

I am extraordinarily excited about our ongoing and upcoming projects – including our work to understand agents, study AI wet-lab uplift, and measuring and dealing with AI’s soon-to-be superhuman scientific capabilities. Our talent bench is deep, and it has been heartening to see them step up and assume more responsibility in the wake of this leadership change. We will soon write a post introducing the team in more detail, their incredible accomplishments, and what they are currently working on. For now, I will give a brief overview of the incoming SBAI leadership team.

Our new Director of AI is Dr Jasper Götting. Jasper joined SecureBio in the spring of 2024 to co-lead the Virology Capabilities Test, has subsequently been the Head of AI Research, and has been a close partner to Seth and myself on all technical and strategic work we’ve done in the AI group since its inception. Jasper’s career as a biosecurity researcher has an unusually coherent arc. He trained as a bench virologist, moved into pandemic-prevention hardware, and now sits at the frontier of evaluating how AI models could raise biological risk. The work of SecureBio AI will continue to flourish under his guidance. Jasper is assisted by three senior leaders:

Dr Andrew Liu, Head of Applied Research. Andrew is a computational scientist whose work on genetic-engineering attribution and cryptographically secure DNA-synthesis screening sits at the frontier of keeping biotechnology safe from misuse.
Dr Jon Sanders, Head of Foundational Research. Jon is a biologist and engineer who cut their teeth building custom biological sampling hardware tough enough for the crushing dark of the deep ocean and sequencing the microbiomes of a vast array of exotic animals, and now brings that same first-principles hardware instinct to our wetlab and superhuman science projects.
Samira Nedungadi, Head of Engineering. None of our work would be possible without our incredible engineering team, led by the remarkable Samira, who has single-handedly steered the professionalization and streamlining of our infra and pipeline, and without whom none of our evals or pre-release assessments would be possible.

I am more optimistic than ever that SecureBio’s AI group is on track to succeed in its mission: to ensure that AI cannot be misused to create catastrophic outcomes in biology, such as unleashing pandemics or building bioweapons, and instead focus laser-like on promoting and realising its incredible upside. We have an opportunity to consign entire swathes of disease to history, to make common ailments a thing of the past. If we manage to make AI for biology safe, with reasonable, prudent and targeted safeguards in place, we can avoid the mistakes the nuclear energy industry made in the 1950s – when a careless approach to this nascent technology led to preventable accidents, setting the sector back decades and preventing humankind from realising the potential of cheap, abundant, emission-free energy. Then, the 21st century will be marked by advances in the biological, medical and physiological sciences that even our parents’ generation would have considered impossible.

This is our task, and we will rise to it. Thank you, Seth, for your service: in the domain of AI/bio, you’ve done more than anyone else I know to make the world safer and prepare humanity for what is coming our way. We will continue this work in your absence. And in the meantime: good luck at RAND!

","has_dynamic_content":false,"truncated_body_text":"When AI outgrows our best scientists","wordcount":2836,"post_preview_limit":null,"language":"en","postTags":[],"postCountryBlocks":[],"headlineTest":null,"publishedBylines":[{"id":5563935,"name":"Seth Donoughe","handle":"sethdonoughe","photo_url":"https://substack-post-media.s3.amazonaws.com/public/images/48042a58-d6c7-4b86-85ca-9ca46c4e9e7f_1024x1024.jpeg","bio":"Researcher, AI and biology","profile_set_up_at":"2026-07-06T04:46:00.596Z","reader_installed_at":null,"is_guest":true,"bestseller_tier":null,"status":null,"primaryPublicationId":10085166,"primaryPublicationName":"Seth Donoughe","primaryPublicationUrl":"https://sethdonoughe.substack.com","primaryPublicationSubscribeUrl":"https://sethdonoughe.substack.com/subscribe?"},{"id":117317091,"name":"Ben Mueller","handle":"benmueller3","photo_url":"https://substack-post-media.s3.amazonaws.com/public/images/e8075a1e-6c33-4d6c-8b71-96c5299f86de_353x353.jpeg","bio":"Executive Director, SecureBio ","profile_set_up_at":"2026-02-25T16:57:57.971Z","reader_installed_at":null,"is_guest":true,"bestseller_tier":null,"status":{"bestsellerTier":null,"subscriberTier":null,"leaderboard":null,"vip":false,"badge":null,"subscriber":null}}],"reaction":null,"reaction_count":18,"comment_count":0,"child_comment_count":0,"audio_items":[{"post_id":205504439,"voice_id":"en-US-OnyxTurboMultilingualNeural","audio_url":"https://substack-video.s3.amazonaws.com/video_upload/post/205504439/tts/fa50959d-4885-4c4d-b766-d6b47eb75125/en-US-OnyxTurboMultilingualNeural.mp3","type":"tts","status":"completed"}],"is_geoblocked":false,"is_editor_preview":false,"hasCashtag":false},"initialComments":[],"initialAutomodHiddenComments":[],"bannedFromNotes":false,"canonicalUrl":"https://securebio.substack.com/p/a-new-chapter-for-securebio-ai/comments","ogUrl":"https://securebio.substack.com/p/a-new-chapter-for-securebio-ai/comments","activeLiveStream":null,"freeTrialCoupon":null,"defaultCoupon":null,"saleCoupon":null,"isChatActive":false,"publicationPageAlert":null,"features":{},"browser":{},"showCookieBanner":false,"disabledCookies":[],"dd_env":"prod","dd_ti":true,"sessionSampleRate":1,"designTemplate":null}

SecureBio

Leaving SecureBio AI in Good Hands