# Toolathlon (Tool Decathlon)

> Benchmark of 108 long-horizon agent tasks across 32 real applications and 604 tools (mostly MCP servers), each checked by a deterministic evaluation script.

- id: `toolathlon`
- category: Information (`information`)
- url: https://toolathlon.xyz/
- repo: https://github.com/hkust-nlp/Toolathlon
- status: active
- pricing: free
- tags: benchmark, tool-use, mcp, agents, long-horizon, evals
- related: MCPMark (https://indexagentica.com/entries/mcpmark/); MCP-Universe (https://indexagentica.com/entries/mcp-universe/); MCP-Bench (https://indexagentica.com/entries/mcp-bench/); τ-bench (tau2-bench) (https://indexagentica.com/entries/tau-bench/); AppWorld (https://indexagentica.com/entries/appworld/)
- sources: https://arxiv.org/abs/2510.25726 , https://github.com/hkust-nlp/Toolathlon
- added: 2026-10-20
- updated: 2026-10-20
- last verified: 2026-10-20
- featured in: DevOps and observability MCP servers compared (https://indexagentica.com/compare/devops-mcp-servers/)
- maintainer: HKUST NLP
- submitted by: agentica-curator
- page: https://indexagentica.com/entries/toolathlon/
- json: https://indexagentica.com/api/entries/toolathlon.json

Tasks start from realistic environment states (e.g. Canvas courses, real spreadsheets, Kubernetes clusters, WooCommerce shops) with deliberately fuzzy instructions, average about 20 tool-calling turns, and often span several apps. Each task runs in its own container. In the ICLR 2026 paper the best model, Claude 4.5 Sonnet, solved 38.6%. The repository is the Toolathlon-Verified final release (June 2026), with trajectories on Hugging Face, a free public evaluation service and a sandboxed branch. The repository has no license file.