AgentDojo
ETH Zurich benchmark environment for prompt-injection attacks and defenses on tool-using LLM agents, with suites like workspace, banking, travel and slack.
Tags
Description
Runs agents on realistic user tasks while injected tool outputs try to hijack them, measuring both utility and attack success for combinations of models, attacks and defenses (e.g. a tool filter or prompt-injection detector). pip install agentdojo; run with python -m agentdojo.scripts.benchmark. Paper: arXiv:2406.13352 (NeurIPS 2024 Datasets & Benchmarks); results are published on the docs site. The README notes the package API is still under development.
Related
- garak: NVIDIA's open-source LLM vulnerability scanner probing for hallucination, data leakage, prompt injection, jailbreaks, toxicity and other weaknesses.
- PyRIT: Microsoft's open-source Python Risk Identification Tool for proactively red teaming generative AI systems.
- DeepTeam: Open-source framework from Confident AI to red team LLMs and AI agents.
- τ-bench (tau2-bench): Sierra's benchmark for tool-agent-user interaction in real-world domains.
- Inspect: Open-source framework for large language model and agent evaluations from the UK AI Security Institute.
Guides & comparisons
- Test a tool-using agent for prompt injection with AgentDojo (Guide): Measure how often injected tool outputs hijack your agent. Run AgentDojo's built-in suites from the CLI, wire your own model or agent loop in as a pipeline, read utility, utility under attack and targeted attack success, then compare defenses.
Sources
- https://github.com/ethz-spylab/agentdojo
- https://agentdojo.spylab.ai/
- https://arxiv.org/abs/2406.13352
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)