BigCodeBench
Code-generation benchmark with 1,140 software-engineering-oriented Python tasks that exercise diverse library calls, plus a harder task subset.
Tags
Description
The GitHub repository is archived (read-only); PyPI 0.2.5 and the Hugging Face dataset remain available. Established benchmark released in 2024, not a new launch. Public bigcodebench Python package (PyPI 0.2.5) provides generation and execution-based evaluation, with completion and instruction-oriented task variants. Repository and Hugging Face dataset bigcode/bigcodebench declare Apache-2.0. Generated code requires isolated execution; model inference, sandbox infrastructure and external API use have separate requirements and costs. BigCodeBench-Hard is a 148-task subset described by the authors.
Agent access
Guides & comparisons
- Code-generation evals compared — EvalPlus, BigCodeBench and LiveCodeBench (Comparison): Pick a code-generation evaluation suite by what it stress-tests: expanded unit tests, library-heavy software tasks, or continuously dated contest problems. Agentless is a SWE-bench repair baseline, not a fourth benchmark.
Sources
- https://github.com/bigcode-project/bigcodebench
- https://github.com/bigcode-project/bigcodebench/blob/main/LICENSE
- https://huggingface.co/datasets/bigcode/bigcodebench
- https://pypi.org/project/bigcodebench/
- https://pypi.org/project/bigcodebench/0.2.5/
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)