pi-bench

Existing prompt-injection benchmarks grade single components. Lakera PINT grades detectors. InjecAgent and AgentDojo grade models under indirect injection. None of them grade the thing practitioners actually deploy: a stack — a detector in front of a defense in front of a capability policy in front of a model. pi-bench treats the composed defense stack as the unit of evaluation.

What it does

• Grades composed defense stacks: model × detectors × defenses × capability policy
• Measures attack success rate (ASR), false-positive rate (FPR), p95 latency, and $/1k requests together — because the interesting trade-offs only appear in composition
• Runs suites up to 1,054 attack cases (InjecAgent-derived) plus benign probes, and a 500-case retrieved-context suite (indirectrag-bench)
• Ships three live defenses today: a DeBERTa-v3 detector, spotlighting, and a capability policy — plus the composed stacks built from them
• Open-weight-first, so every leaderboard row is reproducible in one command without proprietary API access

pibench bench --stack spotlight-deberta-policy --model qwen3-8b --suite injecagent-full-enhanced

Why it exists

Real deployments layer defenses, and the trade-offs that matter — security vs. usability vs. latency vs. spend — never show up when you measure one layer at a time. A policy that blocks every attack is trivial (block everything) and useless; a detector with a great ASR number may double your p95. Grading the composition is the only way to see what a defense actually costs.

Repository

• heisenberg-alt/pi-bench
• Launch write-up with the findings: Your prompt injection defenses may not compose