AppWorld
Benchmark and sandbox of 9 simulated day-to-day apps operable via 457 APIs and populated by ~100 simulated people, for evaluating function-calling and interactive coding agents.
Tags
Description
Agents complete natural tasks by writing code that calls the apps' APIs on behalf of a supervisor; Evaluation uses database-state-based unit tests that also catch collateral damage. Includes a task explorer, API explorer, leaderboard and an AppWorld MCP server/client for evaluating terminal agents such as Codex and Gemini CLI. ACL 2024 Best Resource Paper (arXiv 2407.18901). Installs from PyPI as appworld.
Related
- τ-bench (tau2-bench): Sierra's benchmark for tool-agent-user interaction in real-world domains.
- Berkeley Function Calling Leaderboard (BFCL): UC Berkeley leaderboard evaluating how accurately LLMs call functions and tools; part of the Gorilla project.
- WebArena: A realistic web environment for building autonomous agents, with benchmark tasks used to evaluate web agents.
- OSWorld: NeurIPS 2024 benchmark of multimodal agents on open-ended tasks in real computer environments.
Guides & comparisons
- Agent benchmarks compared (Comparison): Ten widely cited agent benchmarks (SWE-bench, Terminal-Bench, tau-bench, BFCL, MCPMark, GAIA, BrowseComp, WebArena, Mind2Web and OSWorld) compared on what they measure, how many tasks they have, where the agent runs, how runs are scored, where results are published and how they are licensed.
Sources
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)