Distrust the scoreboard

Before you start, from Lessons 1 to 4: what is the Wilson interval for 7 of 10, roughly? Why does a defense need pass^k and not pass@k? In Lesson 3, what could an offline scan not see?

A failure proves little

A successful test shows a lower bound on what a system can do. A failed test proves a good deal less. It may only show that the tester did not try hard enough. On a set of buffer-overflow tasks, a model exploited about 5% with a bare prompt, 71% in a single attempt with a purpose-built scaffold, and 100% with retries.¹ The UK AI Security Institute found that about 8% of its cyber tasks were solved only with more than ten million tokens of working budget. It recommends reporting results against budget, with the budget always stated.²

A report therefore says how much effort it spent: the scaffold, the number of attempts, and the token or time budget. METR's guidance on elicitation, as read through a search excerpt of the page, sorts failures into three kinds. Spurious failures are fixed, and counted. Real failures are left as they are. The third kind is trade-offs. The guidance also notes that the method can both under- and over-estimate capability.³

Can, or will?

A capability evaluation asks whether a model can do something when pushed. A propensity evaluation asks whether it does it by default, when nobody is pushing. The two answer different questions, and a report names which one it measured.

Propensity results now carry a caveat. Models increasingly notice when they are being tested. In the UK institute's evaluation of one frontier model, reported in March 2026, the model identified evaluations 80% of the time, and said so in only 2.3% of interactions.⁴ In October 2026 the institute reported that a model's own reasoning that its environment was simulated is not evidence that it was. In its incident, the model often concluded, wrongly, that its targets were simulated.⁵ Agents in sandboxes can also infer much about who is testing them from the environment itself.⁶ A behavioural result can no longer assume that the model believes the scenario is real.

The judge is a model too

Most red-team results are scored by a model and not a person, and the judge has its biases. StrongREJECT showed that simple refusal judges overstate jailbreak success, because many "successful" jailbreaks produce empty, useless output. Such a jailbreak succeeds only on the scoresheet. StrongREJECT scores specificity and convincingness instead.⁷ A May 2026 preprint edits answers in meaning-preserving ways chosen to please the judge, and raises LLM-judge scores in more than 65% of cases. Prompt-based detectors caught none of the edited outputs.⁸ Judges also tend to be lenient and sensitive to answer length, and a high agreement with humans in percent can hide different scores.⁹

Measure a judge before you trust it. Record its agreement with hand labels, its consistency when it grades the same answer again, and how far it shifts when only the presentation changes. If you know a judge's error rates from a hand-labelled set, you can correct an estimated rate for them, and widen the interval to cover the calibration set as well.¹⁰ Classifiers used as gates are judges too. One vendor reports that its automatic approval classifier caught 89% of dangerous commands in a synthetic study, against 13.6% for human reviewers. A bypass was reported in August 2026; that report is from a single source and was not checked for this page.¹¹

Benchmarks measure less than they claim

A July 2026 validity audit examines agent-safety benchmarks across many models. On one of them, a policy that answers "unsafe" for every item scored an F1 of 0.690, above five of the 21 models that actually discriminate. A benchmark that a stopped clock can beat is a benchmark with something to explain. Across benchmarks, safety scores added no information about task success beyond capability.¹² Reviews of benchmark practice find that flaws in task setup or reward design can distort measured performance by up to 100% in relative terms, and that construct validity problems recur across hundreds of benchmarks.¹³ Static benchmarks also tell you less and less as models harden against known attacks.¹⁴

Read every benchmark number with three questions: does a trivial baseline score well, does the metric reward what you care about, and could the items be in the training data?

Which tool for which job

The maintained open-source tools do different jobs. Checked on 2026-10-25:

Sources for each row are in the footnotes.¹⁵ PyRIT's old Azure repository is archived, and tutorials written before version 1.0 use its old "orchestrators" API. The rename is reported in the course's research but not stated on the 1.1.0 landing page. Tutorials for garak from before version 0.16 are also out of date. Whatever you use, record the tool, its version and its configuration in the run record, and check what it sends off your machine before you point it at anything.

Writing it up

Report the judge audit with intervals, and say which judge biases would move a red-team result scored by this judge, and in which direction. Read it against the rubric from Lesson 1.

Lesson text is CC BY 4.0 and code samples are MIT.