BrowseComp
OpenAI's benchmark of 1,266 hard-to-find, short-answer questions that test how persistently and creatively browsing agents search the web.
Tags
Description
Answers are short and easy to verify against reference answers, so grading is simple; the paper frames it as analogous to programming competitions for coding agents: incomplete, but a useful measure of persistence and creativity in finding entangled information (arXiv 2504.12516). The reference implementation lives in openai/simple-evals, which stopped being updated for new models in July 2025 but still hosts BrowseComp, HealthBench and SimpleQA.
Related
- GAIA: Benchmark for general AI assistants ("Benchmarking General AI Agents"), published as a Hugging Face dataset under the gaia-benchmark org.
- WebArena: A realistic web environment for building autonomous agents, with benchmark tasks used to evaluate web agents.
- Mind2Web: Dataset and benchmark for generalist web agents: 2,000+ open-ended tasks on 137 real websites across 31 domains, with crowdsourced action sequences (NeurIPS 2023).
- Humanity's Last Exam: Multi-modal benchmark of 2,500 expert-written questions at the frontier of human knowledge, from the Center for AI Safety and Scale AI.
Guides & comparisons
- Agent benchmarks compared (Comparison): Ten widely cited agent benchmarks (SWE-bench, Terminal-Bench, tau-bench, BFCL, MCPMark, GAIA, BrowseComp, WebArena, Mind2Web and OSWorld) compared on what they measure, how many tasks they have, where the agent runs, how runs are scored, where results are published and how they are licensed.
Sources
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)