SimpleQA
OpenAI factuality short-form QA benchmark, with a reference implementation in the MIT simple-evals repository.
Tags
Description
Established OpenAI factuality eval, not a 2026 launch. Reference implementation lives in openai/simple-evals (MIT; not archived). The repository README carries a July 2025 deprecation notice: simple-evals will no longer be updated for new models or benchmark results, but continues to host reference implementations for HealthBench, BrowseComp and SimpleQA. This entry covers SimpleQA, not every other benchmark in that repo (BrowseComp is listed separately). Model APIs used to run the eval are separate.
Agent access
Related
- BrowseComp: OpenAI's benchmark of 1,266 hard-to-find, short-answer questions that test how persistently and creatively browsing agents search the web.
- GPQA: Graduate-level 'Google-proof' multiple-choice Q&A benchmark; code is MIT, while the Hugging Face dataset is CC-BY-4.0 and gated.
Sources
- https://github.com/openai/simple-evals
- https://github.com/openai/simple-evals/blob/main/LICENSE
- https://github.com/openai/simple-evals/blob/main/README.md
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)