OpenAI Frontier Evals
OpenAI's open-source code for frontier capability evals: PaperBench, SWE-Lancer and EVMbench.
Tags
Description
PaperBench measures end-to-end replication of state-of-the-art AI papers, SWE-Lancer uses real freelance software-engineering tasks with end-to-end tests, and EVMbench covers smart-contract security tasks. Each suite has its own README and a uv-managed environment (uv sync). MIT. Last commit October 2026.
Related
- OpenAI Evals: OpenAI's open-source framework and registry of benchmarks for evaluating LLMs and LLM systems, including custom private evals.
- SWE-Bench Pro: Scale AI benchmark of long-horizon software-engineering tasks for AI agents, with an open dataset on Hugging Face and a public leaderboard.
- MLE-bench: OpenAI benchmark measuring how well AI agents perform machine learning engineering tasks.
Sources
- https://github.com/openai/frontier-evals
- https://arxiv.org/abs/2504.01848
- https://arxiv.org/abs/2502.12115
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)