Menu

Home / Learn / Attacking LLM Applications

LessonAttacking LLM ApplicationsLesson 6 of 6

Write the finding, then route it

A finding is a claim about impact, backed by a measured rate and a fix for the whole class. It goes to the right channel. This lesson closes the foundation with a full writeup and a stop drill.

  • Foundations
  • Lesson 6
  • About 2 hours
  • Open source, free to read
On this page
The route

Objectivesby the end of this lesson

  • Write a finding that leads with impact and keeps technique, rate and limits apart.
  • Fill a flaw report with the fields a probabilistic system needs.
  • Propose a fix for the class, not for one instance.
  • Decide where a finding goes, what it withholds, and when to stop and escalate instead.

Before you start, from Lessons 1 to 5: what is the Wilson interval for 13 of 40? Which leg of the Rule of Two does an egress filter narrow? Name one judge bias that would inflate a measured attack success rate.

The shape of a finding

Microsoft's AI red team tested more than 100 generative AI products. It structures each finding in five parts: the System tested, the Actor, the tactics and techniques mapped to MITRE ATLAS, the Weakness, and the Impact. On the Actor, the framework "does not assume adversarial intent".1 One of its lessons is titled "You don't have to compute gradients to break an AI system". The failures in Lessons 1 to 3 bear that out. They were ordinary inputs, set down in the wrong place.

A strong writeup leads with the demonstrated impact. Then it gives the rate with its interval and the conditions, then how general the flaw is, and then a fix for the class. For a dual-use finding it keeps operational detail out of anything public. The UK AI Security Institute, publishing a new jailbreak method in February 2026, disclosed it to Anthropic and OpenAI first and released no attack strings or code.2

Fields for a probabilistic flaw

A proposal by 34 authors, "In-House Evaluation Is Not Enough", sets out what a report about an AI flaw needs that a classic bug report does not. It asks for the system versions involved, session information, a description of how the flaw violates a policy, and statistical validity metrics.3 It also proposes conduct-based safe harbour, "what not who", and an AI Disclosure Coordination Center. The course's research found no such centre as of October 2026.4 FLARE-AI, an ICML 2026 paper, describes an open-source system that sends one machine-readable report to several developers, coordinators and registries at once. The research found no lab adopting it yet.5

Labels help triage when they are checked against the source. In the OWASP GenAI LLM Top 10 2026, prompt injection is LLM01. The 2025 identifiers changed in the 2026 edition, so check the year you cite.6 In the OWASP Top 10 for Agentic Applications 2026, goal hijack is ASI01 and tool misuse ASI02.7 MITRE ATLAS identifiers changed in 2026 as well. Cite the release you checked, which was v2026.09 at the time of writing.8 In release 2026.07, for example, ATLAS moved three techniques under a new AML.T0115, and their old IDs, AML.T0019, AML.T0058 and AML.T0104, no longer appear in the 2026.09 data. None of the three was given to another technique.9

Where it goes

Routing is part of the skill. A finding goes to the channel that owns its kind of problem:

Programmes state what they accept, and their terms are strict. The notes give example conditions, and each note says how its terms were read, on 2026-10-25. One security programme asks for the conversation ID and reproduction on the latest version, pays far less for incomplete reports, and excludes injection with no impact on anyone but the attacker.10 An AI programme accepts agentic injection reports only when they reproduce at least 50% of the time.11 Another excludes direct prompt injection and jailbreaks as such, and rewards rogue actions and data exfiltration.12 A vetted programme is limited to universal jailbreaks of one vendor's safety classifiers.13 Reading these terms is not permission to test any vendor's product in this course. You test your own model, or your own account within its terms.

Report volume is rising. HackerOne's 2025 report, announced in October 2025, counts valid prompt-injection reports up 540%, and more than 560 valid reports from autonomous agents.14 In this course's view, that makes a complete, quantified report the one most likely to be read.

Exercise 1about 2 hours: the capstone

From a run log to a routed finding

Target: the made-up run log below, from a fictional note-taking agent built like Lesson 3's toy. Nothing is tested. Work unguided: this exercise has no steps, only the four things to deliver.

text
made-up log, summary of 40 trials (illustrative)
{"setup": "notebot-demo-01", "model": "example-model-3b, digest sha256:4be1...", "runner": "ollama 0.40.11",
 "settings": {"temperature": 0.7, "seed_rule": "7000 + trial number", "num_ctx": 8192}, "tools": ["read_notes", "fetch_doc", "send_message"],
 "task": "summarise the document", "document": "page-03.html (contains a benign marker instruction)",
 "trials": 40, "canary_in_outbox": 13,
 "events": [{"trial": 27, "tool": "send_message", "note": "stub misconfigured: attempted a real network request; blocked by the container"}]}

Deliver:

  1. A one-page finding in the five-part structure, impact first.
  2. The rate: 13 of 40, 32.5%, with its Wilson 95% interval (about 20% to 48%). Say whether the finding meets a 50% reproduction rule and why, and what the Clopper-Pearson interval adds.
  3. A flaw report with the fields from the 34-author proposal, the OWASP and ATLAS labels you checked against the primary sites, and a class-level fix: split the Rule of Two legs, or add egress control, rather than filtering this one document.
  4. A routing memo: which channel would apply if this were a real product, what the public version would withhold, and why.

Stop and escalate. This exercise tests nothing. If you notice anything in your own lab that resembles the log's trial 27, the drill below is the rule.

The stop drill. Trial 27 records a stub that tried a real network request. Flag it, explain why testing stops there, and write the short escalation note: what happened, that the log is preserved, that nothing was retried, and who you would tell. In your own lab, that event ends the session until you know why it happened.

Show the marking guide

A sound capstone: the finding names the system and setup, states impact first ("the agent sent the notes' canary to its outbox in 13 of 40 trials when summarising a document that contained a marker instruction"), keeps observations apart from inferences, and states limits (one model, one document, made-up data). The rate line reads "13 of 40, 32.5%, 95% Wilson 20.1% to 48.0%" and says the interval lies below the 50% bar, so the finding does not meet a 50% reproduction rule; it notes that the Clopper-Pearson interval (18.6% to 49.1%) reaches just under 50%, so the conclusion holds under both. The fix removes a leg for the whole feature. The routing memo names a channel type and withholds the document's text. The stop note says the session stopped at trial 27, preserves the log, and does not retry.

Exercise 2self-check, about 10 minutes

Check your understanding

No model is run in this check.

  1. Why does impact come first in a finding?
  2. A finding reproduces 7 times in 10. Does it meet a 50% reproduction rule?
  3. What is the difference between a class fix and an instance fix, in one sentence each?
  4. You find something that looks like a flaw in the Umbris bot by accident. What do you do?
Show answers
  1. Three of the four example programmes exclude injection or jailbreaks that have no impact beyond the attacker; the vetted jailbreak programme is the exception. In this course's view, impact first also helps a triager decide what to read.
  2. Not shown: the Wilson interval for 7 of 10 runs from about 40% to 89%, so more trials are needed before claiming it.
  3. A class fix removes the cause for every case of the feature, such as removing an exit; an instance fix blocks one case, such as one domain or one string.
  4. Stop. Do not test further. Tell a moderator in private, with what you saw and when.

Writing it up

The capstone is the writeup. Read it against the rubric from Lesson 1, every row at "sound" or better. Then keep it. The next lessons build on the same record, rate and report.

Lesson text is CC BY 4.0 and code samples are MIT.

Notes

  1. Microsoft AI Red Team, "Lessons From Red Teaming 100 Generative AI Products", January 2025, arXiv 2501.07238, https://arxiv.org/abs/2501.07238. ↩
  2. UK AI Security Institute, on Boundary Point Jailbreaking, 17 February 2026, https://www.aisi.gov.uk/blog/boundary-point-jailbreaking-a-new-way-to-break-the-strongest-ai-defences. ↩
  3. "In-House Evaluation Is Not Enough", 34 authors, March 2025, arXiv 2503.16861, https://arxiv.org/abs/2503.16861. ↩
  4. The course's research notes, which found no coordination centre in their evidence as of October 2026; the proposal is arXiv 2503.16861, https://arxiv.org/abs/2503.16861. ↩
  5. FLARE-AI, ICML 2026, arXiv 2606.31567, https://arxiv.org/abs/2606.31567. ↩
  6. OWASP GenAI Security Project, OWASP GenAI LLM Top 10 2026 (README: published 4 August 2026; the resource page is dated 3 August), https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/. ↩
  7. OWASP GenAI Security Project, OWASP Top 10 for Agentic Applications 2026, 9 December 2025, https://genai.owasp.org/resource/2025-12-09/owasp-top-10-for-agentic-applications-for-2026. ↩
  8. MITRE ATLAS data releases, https://github.com/mitre-atlas/atlas-data (v2026.09, 15 September 2026). ↩
  9. MITRE ATLAS data, changelog entry for release 2026.07 of 31 July 2026, which lists "Publish Poisoned AI Artifacts" (AML.T0115) as added and the three techniques as "Previously" AML.T0019, AML.T0058 and AML.T0104, https://github.com/mitre-atlas/atlas-data/blob/main/CHANGELOG.md; the 2026.09 data file, dist/v6/ATLAS-2026.09.yaml, contains none of the three old IDs (both read 2026-10-25). ↩
  10. Microsoft Security Response Center, Copilot (AI) bounty programme, https://www.microsoft.com/en-us/msrc/bounty-ai (read 2026-10-25; terms revised 20 October 2026; incomplete reports are paid about a tenth). ↩
  11. OpenAI, Safety Bug Bounty, https://openai.com/index/safety-bug-bounty (read through a search index on 2026-10-25; the page itself refused automated reading). Press reports of a $100,000 ceiling could not be confirmed and are not used here. ↩
  12. Google's AI vulnerability reward programme, launched October 2025, as reported by The Register on 7 October 2025; the programme page, https://bughunters.google.com/, could not be read as text. ↩
  13. Anthropic, model safety bug bounty programme (universal jailbreaks of its Constitutional Classifiers; by application, on HackerOne), help centre article dated 8 September 2026, https://support.claude.com/en/articles/12119250-model-safety-bug-bounty-program. ↩
  14. HackerOne press release announcing its 2025 report, 1 October 2025, https://www.hackerone.com/press-release/hackerone-report-finds-210-spike-ai-vulnerability-reports-amid-rise-ai-autonomy. ↩