OpenAI Evals
OpenAI's open-source framework and registry of benchmarks for evaluating LLMs and LLM systems, including custom private evals.
Tags
Description
Evals provides a framework for writing and running evaluations of LLMs or systems built on them, plus an open registry of existing evals stored with Git LFS. You can write custom evals (including private ones on your own data) in Python 3.9+. Runs need an OpenAI API key. OpenAI now also offers hosted evals in the OpenAI Dashboard; the open-source repo's last commit was in April 2026.
Agent access
Related
- Inspect: Open-source framework for large language model and agent evaluations from the UK AI Security Institute.
- LM Evaluation Harness: EleutherAI's widely used framework for few-shot evaluation of language models across hundreds of benchmarks, with HF, vLLM and API backends.
- OpenEvals: LangChain's package of ready-made evaluators for LLM apps (sibling project agentevals targets agents).
- Promptfoo: Open-source tool to test and red-team prompts, agents and RAG: automated evals, vulnerability scanning and model comparison.
Sources
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)