Menu

Home / Learn / Attacking LLM Applications

LessonAttacking LLM ApplicationsLesson 2 of 6

Untrusted data meets private data

A better prompt or a filter does not fix prompt injection. Design contains it, by deciding which untrusted text, which private data and which ways out may share one session.

  • Foundations
  • Lesson 2
  • About 100 minutes, no model needed
  • Open source, free to read
On this page
The route

Objectivesby the end of this lesson

  • State the lethal trifecta and the Agents Rule of Two, and say what each adds.
  • Take a published injection incident apart into its untrusted input, private data and exit.
  • Audit your own AI tools for sessions that hold all three legs at once.
  • Write a test plan for a toy agent before anything runs.

Before you start, from Lesson 1: why does a model act on text inside a document it was only asked to summarise? What would one reply that followed the instruction tell you, and what would thirty?

In Lesson 1 you measured that text from an untrusted part of the context gets followed some of the time. Some of the time is a good deal more often than never. This lesson asks what that costs once the same session also holds private data and a way to send something out. It is an analysis lesson. Nothing is sent to a model.

Why filters lost

The hope was that a better system prompt, or a detector, would stop injected instructions. The measurements went the other way. In "The Attacker Moves Second", 14 authors from OpenAI, Anthropic, Google DeepMind and other institutions took 12 published defenses that had reported near-zero attack success, and broke them with adaptive attacks, most at over 90% success. In their human study, more than 500 participants defeated every challenge.1 In a competition run by NIST's Center for AI Standards and Innovation with Gray Swan and the UK AI Security Institute, more than 400 participants made more than 250,000 attack attempts. At least one attack succeeded against every one of 13 frontier models. Attacks that beat the more resistant models tended to transfer to the less resistant ones, and not the other way round.2

This course takes one reading from these results. Injection cannot be filtered out of a model's input reliably, so it has to be contained by the way the system is built.

The lethal trifecta

Simon Willison named the dangerous combination in June 2025. An AI agent is exposed when it has all three of these: access to private data, exposure to untrusted content, and the ability to communicate externally. Anyone who can put text in front of such an agent can then ask it, quite politely, to send the private data out. His rule is never to combine all three.3

The Agents Rule of Two

Meta generalised the idea in October 2025. Within one session, an agent should satisfy no more than two of these three properties:4

Fig. 1 · The Agents Rule of Two

What Meta adds to the trifecta is leg C. It counts changing state, and not only sending data out. Take, as an illustration, an agent that reads a poisoned issue and can also merge code, delete files or move money. It is exposed even if it never sends a byte anywhere. If a task truly needs all three, Meta's guidance is to add supervision or to start a fresh session.5 Meta also notes that the rule does not cover attacker uplift, hallucination or excessive privilege, and that it supplements least privilege rather than replacing it. That caveat comes from the course's research summary of Meta's post.6

Designs that hold

"Design Patterns for Securing LLM Agents" states the principle behind the containment approach. Once an agent has ingested untrusted input, "it must be constrained so that it is impossible for that input to trigger any consequential actions."7 The paper catalogues six patterns: action-selector, plan-then-execute, LLM map-reduce, dual LLM, code-then-execute and context-minimisation. It predates the Rule of Two. Read through the Rule, as this course reads it, each pattern removes or isolates a leg. None of them relies on the model choosing to resist.

Whether these out-of-band designs survive determined, adaptive attackers at scale is still an open research question in 2026.8 In this course's view, a tester's job is to find out which kind the product has: a design that removes a leg, or one that only makes an attack harder.

Exits hide on allow-lists

Many incidents published in 2025 and 2026 share one shape. Injected text arrives in untrusted content, the agent finds an exit through a service the product already trusted, and the data leaves the system by that exit. As Simon Willison reports them, the cases include a coding assistant whose web guide led it to read a credentials file and send it out through an allow-listed domain, an email assistant that leaked other emails through a form service, and an agent that uploaded files to an attacker's account through the vendor's own allow-listed API.9 The control "block unknown domains" is therefore incomplete. You have to list every write-capable endpoint reachable from the allow-list.

The class is filed as LLM01 Prompt Injection in the OWASP GenAI LLM Top 10 2026, and as ASI01 Agent Goal Hijack in the OWASP Top 10 for Agentic Applications 2026.10

Exercise 1about 90 minutes in three parts

Take incidents apart, audit your setup, plan a test

Target: public incident write-ups and your own tool settings. Nothing is sent to a model, and nothing is probed.

Part A, five incidents (about 40 minutes). Pick five disclosures from 2025 or 2026, from Willison's prompt-injection tag or the vendors' own advisories. Read them; do not copy or run any text a write-up quotes.11 For each, fill one row of this table.

Field What to write
System product, feature and date of the disclosure
Private data what the agent could read that the attacker could not
Untrusted content where outside text arrived: a web page, an email, a shared document, a tool description, a repository file
Exit how data or action left: a fetched URL, a rendered image, a form, a vendor API, a message, a commit
Exit allow-listed? was the exit a service the product already trusted?
Rule of Two which two legs the session needed for its job, and which third it should not have had
Fix what the vendor changed, and which leg it removed or narrowed
Class or instance did the fix close the class (a leg removed for the whole feature) or one instance (one domain, one string)?

Then answer in three sentences: how many of the five exits were allow-listed, which leg vendors most often removed, and how many fixes closed the class rather than the instance.

Part B, your own setup (about 30 minutes). Leave the Umbris bot off this list: it is out of scope. List every AI tool you use that can reach something private: a coding agent, a browser agent, an assistant connected to email or files, MCP servers. For each, write the three legs as they stand in one ordinary session, mark the sessions where all three coexist, and name the one change that would break the combination: a read-only mode, no network, a separate profile, confirmation before any send. This is an inventory of your configuration; do not probe the products.

Part C, a test plan (about 20 minutes). Lesson 3 builds a toy agent with three stub tools: one reads a local notes file that holds a made-up canary string, one returns a local "web page" file, and one "send" tool only appends to a local log. Write the plan, not the probe: the permitted target, the three stubs and what each logs, the canary, what counts as success (the canary appears in the send log), the number of trials and the settings, and the stop rule. The plan is read before anything runs.

Stop and escalate. If, while auditing your own setup, you find a session that already sent private data somewhere, or what looks like a flaw in a product: stop. Keep notes of what you saw, revoke any exposed key at its provider, and make no other changes. Do not retry or share the technique. Report it to the vendor through its own security channel. If it concerns anything on Umbris, tell a moderator in private; never test the bot to confirm.

Show a sample row

Illustrative; the incident is invented.

Field Sample
System "NoteBot", a meeting-notes assistant, disclosure dated 2026-03
Private data the user's calendar and past notes
Untrusted content the text of a shared meeting invite
Exit an image link rendered in the summary, on the vendor's own image proxy
Exit allow-listed? yes
Rule of Two the job needs untrusted input and sensitive access; rendering arbitrary links added an exit it did not need
Fix links in summaries render as plain text only
Class or instance class: the exit was removed for every summary, not one domain blocked

Exercise 2self-check, about 10 minutes

Check your understanding

No model is run in this check.

  1. What does the Rule of Two add to the lethal trifecta?
  2. A vendor fixes an exfiltration bug by blocking one domain. Class or instance fix? What would a class fix look like?
  3. Why is "we detect 95% of injections" weak evidence of safety?
  4. Your coding agent reads repository issues, has your cloud credentials, and can push commits. Which leg would you remove first, and what does that cost you?
Show answers
  1. State change as a risk leg, not only external communication, and a session-level rule: at most two of the three.
  2. Instance. A class fix removes or narrows a leg for the whole feature, for example no outbound requests from sessions that read untrusted content.
  3. An attacker retries and adapts; a defense has to hold on every attempt. The measured adaptive attacks broke defenses that reported near-zero success.
  4. Any one works; which is cheapest depends on the job. Read-only issue access in a session without credentials is one workable split.

Writing it up

Write one paragraph per incident from Part A, using the row's fields in order, then the three-sentence answer. Keep observations (what the disclosure says happened) apart from inferences (why you think the fix worked). Read it against the rubric from Lesson 1.

Lesson text is CC BY 4.0 and code samples are MIT.

Notes

  1. "The Attacker Moves Second", 14 authors including Nasr and Carlini, USENIX Security 2026, arXiv 2510.09023, https://arxiv.org/abs/2510.09023. ↩
  2. NIST Center for AI Standards and Innovation (CAISI), research blog post on its large-scale AI agent security red-teaming competition, 23 March 2026, https://www.nist.gov/blogs/caisi-research-blog/insights-ai-agent-security-large-scale-red-teaming-competition. ↩
  3. Simon Willison, post on the lethal trifecta, 16 June 2025, https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/. ↩
  4. Meta AI blog post introducing the Agents Rule of Two, 31 October 2025, https://ai.meta.com/blog/practical-ai-agent-security/. ↩
  5. Meta AI blog post introducing the Agents Rule of Two, 31 October 2025, https://ai.meta.com/blog/practical-ai-agent-security/. ↩
  6. Meta AI blog post introducing the Agents Rule of Two, 31 October 2025, https://ai.meta.com/blog/practical-ai-agent-security/. ↩
  7. "Design Patterns for Securing LLM Agents", 2025, arXiv 2506.08837, https://arxiv.org/abs/2506.08837. ↩
  8. An adaptive-evaluation study of out-of-band defenses on a small AgentDojo reproduction, June 2026, arXiv 2606.26479, https://arxiv.org/abs/2606.26479; its authors note it used a weak 7B model and did not test white-box attacks. ↩
  9. Simon Willison, prompt-injection tag, https://simonwillison.net/tag/prompt-injection/; the cases named are his posts of 25 November 2025, 12 January 2026 and 14 January 2026. They are reported as he describes them; read the vendors' own advisories where they exist. ↩
  10. OWASP GenAI Security Project, OWASP GenAI LLM Top 10 2026 (README: published 4 August 2026; the resource page is dated 3 August), https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/, and OWASP Top 10 for Agentic Applications 2026, 9 December 2025, https://genai.owasp.org/resource/2025-12-09/owasp-top-10-for-agentic-applications-for-2026. ↩
  11. Simon Willison, prompt-injection tag, https://simonwillison.net/tag/prompt-injection/; the cases named are his posts of 25 November 2025, 12 January 2026 and 14 January 2026. They are reported as he describes them; read the vendors' own advisories where they exist. ↩