SWE-bench
Benchmark and leaderboards (Verified, Lite, Multimodal, Multilingual) testing whether LLM agents can resolve real-world GitHub issues.
Tags
Related
- Terminal-Bench: Continuously maintained benchmark and leaderboard (Stanford, Laude Institute) measuring how well AI agents complete hard, end-to-end tasks in real terminal environments.
Guides & comparisons
- Code-generation evals compared — EvalPlus, BigCodeBench and LiveCodeBench (Comparison): Pick a code-generation evaluation suite by what it stress-tests: expanded unit tests, library-heavy software tasks, or continuously dated contest problems. Agentless is a SWE-bench repair baseline, not a fourth benchmark.
- Agent environment frameworks compared: Harbor vs AgentEnv Framework vs Seahaven (Comparison): Three open-source Python frameworks for building the environments agent evals and RL runs happen in: Harbor (container tasks), Scale AI's AgentEnv Framework and Kiln AI's Seahaven (synthetic SQLite worlds). Covers tasks, grading, compute, agent interface, interop and license.
- Agent benchmarks compared (Comparison): Ten widely cited agent benchmarks (SWE-bench, Terminal-Bench, tau-bench, BFCL, MCPMark, GAIA, BrowseComp, WebArena, Mind2Web and OSWorld) compared on what they measure, how many tasks they have, where the agent runs, how runs are scored, where results are published and how they are licensed.
Sources
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)