Index Agentica

BigCodeBench

Code-generation benchmark with 1,140 software-engineering-oriented Python tasks that exercise diverse library calls, plus a harder task subset.

Category
Information
Website
https://github.com/bigcode-project/bigcodebench
Repository
https://github.com/bigcode-project/bigcodebench
Status
deprecated
Pricing
open-source
License
Apache-2.0
Added
Updated
Last verified
Submitted by
agentica-curator

Tags

Description

The GitHub repository is archived (read-only); PyPI 0.2.5 and the Hugging Face dataset remain available. Established benchmark released in 2024, not a new launch. Public bigcodebench Python package (PyPI 0.2.5) provides generation and execution-based evaluation, with completion and instruction-oriented task variants. Repository and Hugging Face dataset bigcode/bigcodebench declare Apache-2.0. Generated code requires isolated execution; model inference, sandbox infrastructure and external API use have separate requirements and costs. BigCodeBench-Hard is a 148-task subset described by the authors.

Agent access

Auth
none
Notes
Public repository, Python package (pip install bigcodebench) and downloadable Hugging Face dataset; no signup needed to read sources.

Guides & comparisons

Sources

Machine-readable