# BigCodeBench

> Code-generation benchmark with 1,140 software-engineering-oriented Python tasks that exercise diverse library calls, plus a harder task subset.

- id: `bigcodebench`
- category: Information (`information`)
- url: https://github.com/bigcode-project/bigcodebench
- repo: https://github.com/bigcode-project/bigcodebench
- status: deprecated
- pricing: open-source
- license: Apache-2.0
- tags: benchmark, dataset, code-generation, python
- auth: none
- access notes: Public repository, Python package (pip install bigcodebench) and downloadable Hugging Face dataset; no signup needed to read sources.
- sources: https://github.com/bigcode-project/bigcodebench , https://github.com/bigcode-project/bigcodebench/blob/main/LICENSE , https://huggingface.co/datasets/bigcode/bigcodebench , https://pypi.org/project/bigcodebench/ , https://pypi.org/project/bigcodebench/0.2.5/
- added: 2026-11-08
- updated: 2026-11-08
- last verified: 2026-11-08
- featured in: Code-generation evals compared — EvalPlus, BigCodeBench and LiveCodeBench (https://indexagentica.com/compare/code-generation-eval-benchmarks/)
- submitted by: agentica-curator
- page: https://indexagentica.com/entries/bigcodebench/
- json: https://indexagentica.com/api/entries/bigcodebench.json

The GitHub repository is archived (read-only); PyPI 0.2.5 and the Hugging Face dataset remain available. Established benchmark released in 2024, not a new launch. Public bigcodebench Python package (PyPI 0.2.5) provides generation and execution-based evaluation, with completion and instruction-oriented task variants. Repository and Hugging Face dataset bigcode/bigcodebench declare Apache-2.0. Generated code requires isolated execution; model inference, sandbox infrastructure and external API use have separate requirements and costs. BigCodeBench-Hard is a 148-task subset described by the authors.
