SecureBio’s Principles and Practices for Model Assessment

Our approach to model assessment to ensure transparency and integrity as an independent evaluation organization.

Independent assessment of AI, including testing biosecurity-relevant capabilities and safeguard effectiveness, helps to ensure decision-makers have the most comprehensive and reliable information about a model’s performance possible. As an independent evaluator, SecureBio follows a set of principles and practices, which expand upon the AI Evaluator Forum's AEF-1 Standard. Our organization follows these principles and practices, described below, for every model assessment we conduct.

SecureBio has performed several biosecurity model pre-release assessments (for our most recent example, see GPT-5.6 Sol). We undertake such work on the basis of a set of principles and practices (both from the AI Evaluator Forum and our own) to ensure that our assessments are independent and transparent. We have previously released public reports independently from model developers on our pre-release assessments (e.g. GPT-5.5) and plan to continue to do so moving forward when necessary and helpful for transparency and integrity.

AEF-1 Standard

As members of the AI Evaluator Forum (AEF), we co-developed and have adopted the AEF-1 Standard. Below, we highlight key examples from that standard and how we specifically fulfill them.

1. Obtaining Sufficient Access and Resources.

   1. We assess pre-release model snapshots that are as close to their deployment configuration as possible, including configured tools, reasoning settings, and safeguards.

   2. We request access to model snapshots without safety filters or safety fine-tuning, so that we can assess the model’s underlying capabilities if safeguards are circumvented.

   3. We disclose which model configurations to which we had access for each evaluation included in our assessment.

   4. If we believe our model access was insufficient to adequately assess any relevant aspects of model capability or safety, we state this in our public-facing report.

2. Minimizing Conflicts of Interest.

   1. For some engagements with for-profit firms, we request funding to cover the costs of running the assessments. For other engagements, we do not accept funding. We disclose funding or the lack thereof in all public-facing reports.

   2. We never enter into collaborations or engagements in which any funding is contingent on the results of our work.

   3. We do not allow any AI developers to exercise organizational or financial control over SecureBio’s research or assessment work.

   4. In addition to funding, we disclose other potential conflicts of interest relevant to the assessment. See our full Conflicts of Interest Policy.

3. Ensuring Analytic Autonomy.

   1. We retain autonomy in deciding the methods of the assessment.

   2. We retain control of the analysis, interpretation, and writeup of evidence when we write our public-facing reports.

   3. We share our reports with AI developers in advance of our publication, and they have an opportunity to identify any confidential business information, so that it can be redacted for the public report (see Section 5).

4. Describing Transparent Methods and Results.

   1. We disclose sufficient detail to allow the reader to appropriately interpret assessment results. We include:

      1. Which evaluations were run, with which frameworks, agent scaffolds, tools, and other settings.

      2. The evaluation results we attained.

      3. The model configurations and access window we had for the assessment, and the likely impact on our risk judgment if such resources were not sufficient.

      4. Moving forward, which of our evaluations we hold privately, which ones we share with collaborators and which ones are publicly available.

      5. The above are subject to exceptions from ‘Protection of Sensitive Information’.

5. Protecting Sensitive Information.

   1. We redact, anonymize, restrict, or otherwise withhold material if it is infohazardous—as determined by inclusion in the BioTIER refusal set and/or consultation with representatives from academia, civil society, and governments—or constitutes protected intellectual property (see Section 3C).

   2. AI developers have no authority to redact our results to conceal performance or unfavorable findings.

   3. If we redact, we indicate whether the redactions materially affect our assessment’s conclusions.

Additional Practices

In addition to following AEF-1 Standard, we follow additional practices to best transparency and integrity in our assessments.

1. Guaranteeing Evaluation Integrity.

   1. We maintain holdout sets for our evaluations (i.e. an independent set of tasks that are held privately and not shared with any external entities).

   2. We evaluate models on the holdout sets regularly to check for mismatches between non-holdout and holdout model performance.

   3. We add canary strings to our evaluations and check for tasks that may have leaked into the public domain.

   4. For non-holdout tasks of private evaluations (i.e. ones we hold and run on behalf of the model developers and other relevant stakeholders), we do not share solutions and other scoring information with firms.

   5. We report performance as overall summary statistics, so developers cannot pinpoint and optimize on specific tasks.

   6. Firms are not allowed to train on any parts of our evaluations. Note that our procedures are robust to harmful misuse even if firms violate the terms of our contractual arrangements with them.

   7. We reserve the right to modify or decline an evaluation where protections against potential harmful use of our testing are inadequate.

2. Publishing Independent Reports.

   1. For results where public disclosure is appropriate, we publish our own independent assessment report or assessment statement alongside the company’s report, to avoid our findings being taken out of context.

   2. We publish errata to our reports if we discover errors or changes in opinion.

3. Compliance With Laws and Regulations.

   1. We undertake every effort to follow all applicable laws (for instance, export control regulations) under advice from counsel.

Stating these practices openly is valuable for maintaining the credibility of independent assessment, and we hope other independent evaluators will do the same. We welcome feedback on the practices above, and as they evolve we will update this page and maintain a time-stamped changelog so changes remain transparent.