Terminal-Bench
Continuously maintained benchmark and leaderboard (Stanford, Laude Institute) measuring how well AI agents complete hard, end-to-end tasks in real terminal environments.
Tags
Description
Each task has an English instruction, a test script and an oracle solution; the verifier runs in a separate container from the agent. Terminal-Bench 3.0 launched with 74 tasks across 7 domains. Terminal-Bench 4.0 (August 2026) calibrated CPU, memory and time (a flat 8-hour agent timeout), removed 8 tasks and fixed 19; versions are now semantic, so results across major versions are not comparable. Runs through the Harbor framework (harbor run -d terminal-bench/terminal-bench@4.0.0), which also evaluates agents such as Claude Code, OpenHands and Codex CLI; some tasks need a GPU-capable sandbox. The original beta (~100 tasks, tb CLI) remains at harbor-framework/terminal-bench-1.
Related
- SWE-bench: Benchmark and leaderboards (Verified, Lite, Multimodal, Multilingual) testing whether LLM agents can resolve real-world GitHub issues.
- mini-swe-agent: Minimal open-source software-engineering agent from the SWE-bench/SWE-agent team: about 100 lines of Python, bash as its only tool, a linear history, any model via LiteLLM.
- Modal: Serverless AI cloud for inference, training, sandboxes and functions, billed per second of use, with official SDKs and OpenAI-compatible Modal Endpoints.
- Daytona: Secure and elastic infrastructure (sandboxes) for running AI-generated code.
- Harbor: Framework from the Terminal-Bench creators for evaluating and optimizing agents and models in sandboxed environments, run in parallel across cloud sandbox providers.
Guides & comparisons
- Agent environment frameworks compared: Harbor vs AgentEnv Framework vs Seahaven (Comparison): Three open-source Python frameworks for building the environments agent evals and RL runs happen in: Harbor (container tasks), Scale AI's AgentEnv Framework and Kiln AI's Seahaven (synthetic SQLite worlds). Covers tasks, grading, compute, agent interface, interop and license.
- Agent benchmarks compared (Comparison): Ten widely cited agent benchmarks (SWE-bench, Terminal-Bench, tau-bench, BFCL, MCPMark, GAIA, BrowseComp, WebArena, Mind2Web and OSWorld) compared on what they measure, how many tasks they have, where the agent runs, how runs are scored, where results are published and how they are licensed.
Sources
- https://www.tbench.ai
- https://www.tbench.ai/news/terminal-bench-4-0
- https://www.tbench.ai/news/terminal-bench-3-0
- https://github.com/harbor-framework/terminal-bench
- https://github.com/harbor-framework/harbor
- https://github.com/harbor-framework/terminal-bench-1
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)