Your prompt injection defenses may not compose

Here is a result that surprised me. Take a prompt injection classifier that works: ProtectAI’s DeBERTa v3, a small open model that reads a piece of text and scores how likely it is to contain an injected instruction. It cut attack success from 14.6% to 7.4% on the full 1,054-case tool-misuse suite with zero false positives. Take spotlighting, Microsoft’s technique of wrapping untrusted content in delimiters so the model knows not to trust it. Also fine on its own: it never blocked a single benign case.

Now run them together, spotlighting first, classifier second. Attack success drops to zero. So does usability: the stack now flags 70% of benign traffic on tool outputs, and 100% of benign traffic on retrieved documents. Every single clean case, blocked. The classifier had never seen spotlight delimiters in training, so the wrapping itself reads as an injection.

Nobody had measured this because our benchmarks grade components, not combinations. That is the gap pi-bench fills. The aim is to make choosing a defense stack an empirical decision instead of a vibes one: one command, four numbers, and the tradeoffs on the table.

The four findings

1. Defenses that pass alone can fail composed. Spotlighting plus the DeBERTa classifier reaches zero attack success by flagging 70% of benign tool outputs and 100% of benign retrieved documents. Alone, spotlighting never blocks a benign case and the classifier blocks almost none. The interaction is the failure.
2. Susceptibility is channel-specific. Every model follows more injections through retrieved documents than through tool outputs, and a model’s rank on one channel says nothing about the other. Mistral posts 0.000 on tool outputs, by never calling tools at all, and 0.723 on documents, the worst of the four.
3. Input detection is best effort. An output policy is a guarantee, but only for its channel. The capability policy is the only stack that stops tool misuse completely with zero false positives, and it barely moves the document number (0.463 to 0.351).
4. Louder attacks are easier to block. The “ignore all previous instructions” escalation buys almost nothing against undefended models and hands the classifier a signature: residual attack success falls to zero on all four models.

The rest of this post is the evidence.

What existing benchmarks measure

Quick context if the attack is new to you. Prompt injection is when text a model reads contains instructions, and the model follows them as if they came from the user. The indirect variant is the dangerous one: the instructions arrive through a side channel the user never sees, such as a retrieved document, an email, or a tool response. The model treats data as commands.

Lakera’s PINT grades detectors. InjecAgent and AgentDojo grade models under indirect injection. Microsoft’s BIPIA grades models on poisoned documents. CyberSecEval grades models across attack classes. All useful. But none of them grade the thing a team actually ships, which is a stack: a detector on the input, some prompt hardening, and a policy on the output, all in front of a specific model.

Stacks fail in ways components cannot. The spotlighting result above is an interaction effect. You will never see it if you evaluate each defense alone.

So I built pi-bench. It grades whole stacks, and it reports four numbers per row, all lower is better:

• ASR, attack success rate on attack cases
• FPR, how much benign traffic the stack wrongly blocks
• p95 latency added per request
• cost per thousand requests

FPR is the honesty column. Any defense can reach zero ASR by blocking everything. The interesting question is what it costs you to get there.

The setup

Four open-weight models (Llama 3.1 8B, Qwen3 8B, Qwen2.5 7B, Mistral 7B), six defense stacks (none, spotlight, deberta, policy, and the compositions spotlight-deberta and spotlight-deberta-policy), three suites:

• injecagent-full: all 1,054 attack cases from InjecAgent, plus ten benign probes so the FPR column has a denominator. The injection arrives through a tool output and success means the model invokes the attacker’s tool.
• indirectrag-bench: 500 cases I built where the injection arrives through retrieved documents instead. 350 attacks, 150 benign. Each attack carries a unique canary token, so scoring is deterministic. No LLM judge. The dataset is on Hugging Face.
• A 20-case seed suite for fast iteration.

Everything is pinned to a seed, and every model response is cached to disk, so any committed row replays byte for byte. The whole GPU matrix ran on rented Modal A10Gs. The repo has the exact script.

What the models do with no defense

Two things jump out.

First, retrieved context is the softer channel for every model. Qwen2.5 follows 46% of RAG injections, three times its tool-output rate.

Second, the Mistral row. Zero attack success through tool outputs looks like robustness. It is not. Mistral emitted exactly one tool call in 3,123 responses; it narrates in text instead of calling tools, so it cannot invoke an attacker’s tool either. The same model follows 72% of injections that arrive through retrieved documents, the worst of the four. A model’s rank on one injection surface tells you nothing about its rank on another. If your threat model only covers tool misuse, you can ship the most injectable model in the lineup and never notice.

What the defenses do

On the retrieval suite with Qwen2.5, the frontier looks like this:

No stack wins outright. The classifier gets you to 0.100 but starts blocking 9% of clean traffic. The composed stack gets you to zero by blocking everything. Bolting the output policy on top (spotlight-deberta-policy) changes nothing: the classifier has already blocked every case before the policy gets to act. On this channel, no single defense is both safe and usable.

The output side tells the opposite story. A capability policy is an allowlist on outgoing tool calls: if the model tries to call a tool the stack has not been granted, the response is blocked. On the tool-misuse suite it is the only stack that reaches zero ASR with zero FPR. It does not inspect inputs at all, so nothing benign gets flagged, and it closes the tool channel even when detection misses. But it cannot see a canary leaving in plain response text, which is why it barely moves the RAG number (0.463 to 0.351).

That asymmetry is the design argument. Input detection is best effort. Output policy is a guarantee, but only for the channel it governs. You need both, and you need to measure them together, because the composition is where the surprises live.

The other two columns are quieter but worth a word, since the whole pitch is four numbers. Cost is effectively zero across the board here: every stack is open weights on rented GPUs. Latency is where the stacks separate. The capability policy adds nothing measurable, because it only inspects a response the model was already producing. The classifier costs a few dozen milliseconds per call, yet its aggregate p95 often comes out lower than undefended. That is not because it is free; it is because every case it blocks never reaches the model at all. The full columns are in the leaderboard.

Louder attacks are easier to block

InjecAgent ships an “enhanced” setting that prepends IGNORE ALL PREVIOUS INSTRUCTIONS to every payload. The obvious hypothesis is that a louder attack is a stronger one. On the tool-output suite, the numbers say otherwise. Here is attack success on the full suite versus its enhanced variant, undefended and then with the DeBERTa classifier in front:

Shouting barely moves the undefended model: the loud prefix buys the attacker a point or two on Qwen, nothing on Llama or Mistral. It is a windfall for the defender. The classifier was trained on exactly this pattern, so the enhanced payloads it could not catch before now trip it every time, driving residual ASR to zero on all four models with false positives still at zero. The louder attack is the easier one to block: the same signal that is supposed to strengthen the injection is what gives it away.

Limitations, stated plainly

• The Mistral tool-channel number measures non-engagement, not robustness. It stays in the table because that is what the criterion produces, but the caveat travels with it everywhere.
• The InjecAgent suites carry only ten benign probes, so their FPR column moves in coarse 0.1 steps. indirectrag-bench, with 150 benign cases, is the finer-grained usability read.
• indirectrag-bench is synthetic and templated. Real poisoned documents are messier.
• Canary scoring captures whether the model acted on the injected task. It does not capture partial compliance or a refusal that still leaks.
• One seed, quantized weights, one serving stack. The point is honest relative comparison, not leaderboard-grade absolutes.

What this means

Three practical rules fall out of the data, and they survive the caveats above.

Pick defenses per channel, not per reputation. Tool misuse and text exfiltration are different problems: a capability policy closes one completely and cannot see the other at all. Start from the channels your application actually exposes.

Measure the stack you ship, composed, not the parts in isolation. Every composition surprise in this post is invisible in a component benchmark, and component numbers are what vendors publish.

Treat any attack-success number that arrives without a false-positive rate, a latency, and a cost as marketing. Blocking everything is always available and always useless.

None of this needed a new defense or a new attack. It needed measuring the right unit.

Run it

pip install -e ".[dev,deberta]"
pibench bench --stack spotlight-deberta --model qwen2.5-7b --suite indirectrag-bench

The open-weight models serve through Ollama by default; --model mock runs the whole harness offline with no GPU and no keys. Every table in this post regenerates from the committed CSVs with pibench leaderboard and pibench report, and both charts with python docs/make_charts.py. Adding a defense is a single file; if you maintain a guardrail and want it graded, open a PR.

Repo: github.com/heisenberg-alt/pi-bench. Dataset: huggingface.co/datasets/heisenberg-88/indirectrag-bench. Full tables: leaderboard and ablation report.