Code-generation evals compared — EvalPlus, BigCodeBench and LiveCodeBench
Pick a code-generation evaluation suite by what it stress-tests: expanded unit tests, library-heavy software tasks, or continuously dated contest problems. Agentless is a SWE-bench repair baseline, not a fourth benchmark.
Comparison
| Entry | What it evaluates | Task set | Scoring | Freshness and contamination | Running and isolation | License and status |
|---|---|---|---|---|---|---|
| EvalPlus | Correctness of function-level Python solutions under many more tests than the original HumanEval/MBPP sets, plus optional code-efficiency evaluation (EvalPerf) | HumanEval+ (164 tasks on the Hugging Face card) and MBPP+ (378 tasks after the v0.2.0 cleanup); EvalPerf is a separate efficiency suite | Execution against base and plus tests; the README highlights large score drops when plus tests are added. EvalPerf measures efficiency on performance-exercising tasks | Fixed curated suites derived from HumanEval/MBPP; not a continuously dated live contest stream | PyPI evalplus 0.3.1 (Python >=3.9); generation/evaluation CLIs and a documented Docker path (ganler/evalplus). Treat generated code as untrusted | Apache-2.0 (package and HumanEval+ dataset card). Active PyPI release 0.3.1 |
| BigCodeBench | Practical Python programming with diverse library calls and complex instructions, closer to tool-using software tasks than toy algorithms | 1,140 tasks (Complete and Instruct splits); BigCodeBench-Hard is a 148-task harder subset | Execution-based evaluation after sanitization/calibration; packages report pass@k-style result files | Fixed 2024 benchmark (arxiv 2406.15877); not continuously extended by contest date | PyPI bigcodebench 0.2.5; README documents local backends and slower remote/e2b options, plus a Docker evaluate image. Isolate execution | Apache-2.0. GitHub repository is archived (read-only); PyPI and Hugging Face dataset remain available |
| LiveCodeBench | Holistic coding ability across scenarios such as code generation, code execution, test-output prediction and self-repair | Continuously collected contest problems from LeetCode, AtCoder and Codeforces; documented releases grow from 400 problems (release_v1) to 1,055 (release_v6) | pass@1 and pass@5 via a modified APPS-style checker; time limits can move scores by under 0.5 points | Problems are time-stamped; filter by release version or start/end dates so evaluation can target problems released after a model's cutoff | Clone the repo and install with uv; runner scenarios are selected by flag. Still execute model code only in an isolated environment | MIT (repository LICENSE). Actively versioned releases; pin release_version for reproducibility |
Verdict: Use EvalPlus when you care whether a solution survives many more unit tests than HumanEval/MBPP. Use BigCodeBench when you need library-heavy, instruction-rich Python tasks. Use LiveCodeBench when you need time-windowed contest problems to reduce contamination. For repository issue repair, look at SWE-bench and baselines such as Agentless—not these three suites.
Three function-level suites, not one more SWE-bench clone
Agent benchmarks already covers repository-scale issue repair (SWE-bench), terminals and tool-using agents. Newer agent eval sets covers data-science, graded terminal corpora and multi-agent conflict. This page is about code-generation benchmarks: give a model a prompt, get a program, run tests.
The three subjects—EvalPlus, BigCodeBench and LiveCodeBench—overlap in “Python coding” but differ in what fails a solution. The recommendations below are selection advice based on those documented differences, not a claim that one suite is universally harder.
Agentless is not in the table. Agentless is a three-phase repair baseline for SWE-bench-style issues (localize, sample patches, validate with tests). It is a method for repository bugs, not a HumanEval-style generation suite. Confusing the two leads people to report Agentless “scores” as if they were another fixed prompt set.
EvalPlus: more tests on familiar tasks
EvalPlus keeps the HumanEval/MBPP problem shape and attacks underspecified tests. Its README states HumanEval+ has 80× more tests than HumanEval and MBPP+ has 35× more than MBPP. The HumanEval+ Hugging Face card lists 164 examples. MBPP+ was cleaned from 399 to 378 tasks in the documented v0.2.0 upgrade.
That makes EvalPlus the right first stop when a model already looks strong on classic pass@k numbers and you want to know whether the code is merely overfit to weak oracles. EvalPerf (COLM'24) is the efficiency companion: correctness is not enough if the solution is asymptotically or practically slow.
Operational notes. PyPI evalplus 0.3.1 requires Python ≥3.9. The README documents a one-command evaluate path and Docker evaluation via ganler/evalplus. Generated samples are untrusted: prefer containerized execution. Offline, we installed 0.3.1 outside the site checkout, downloaded the HumanEval+ release used by the package, and confirmed sanitize() extracts a function body from a markdown-wrapped hand sample. We did not run model inference, Docker evaluation or untrusted code execution.
BigCodeBench: library calls and instructions
BigCodeBench targets software-engineering-oriented Python tasks with diverse function/library calls. The project announces 1,140 tasks, with Complete (docstring/completion) and Instruct (natural-language instruction) splits, and a Hard subset of 148 tasks.
Use it when HumanEval-style puzzles are too narrow: the failure mode you care about is composing APIs and following denser instructions, not only implementing a single algorithm.
Operational notes. PyPI bigcodebench 0.2.5 remains installable. The GitHub repository is archived (read-only) as of this check; treat the package and Hugging Face dataset as the living distribution channels, and expect slower community fixes. The README documents sanitization/calibration artifacts and isolation options (including Docker and slower remote backends). As with EvalPlus, do not evaluate raw model output on a privileged host. We did not install BigCodeBench or run its evaluator in this draft.
LiveCodeBench: time-windowed contest problems
LiveCodeBench continuously collects problems from LeetCode, AtCoder and Codeforces and versions the dataset by date window. Documented releases include release_v1 (400 problems, May 2023–Mar 2024) through release_v6 (1,055 problems, May 2023–Apr 2025). The runner defaults toward the latest release; pin --release_version for reproducibility.
Beyond code generation, the README lists scenarios for code execution, test output prediction and self-repair. Scoring uses pass@1 and pass@5 with a modified APPS-style checker; time limits can shift scores by less than half a point.
Use LiveCodeBench when contamination is the primary concern: filter by problem release date relative to a model's training cutoff. That is a different guarantee from “more unit tests on old tasks.”
Operational notes. Install from the GitHub repository with uv as documented. License is MIT. We did not clone the full runner or download every release shard for this page.
Where Agentless fits
Agentless (OpenAutoCoder/Agentless, MIT) solves each issue with:
- Localization — files → classes/functions → edit sites
- Repair — sample multiple candidate patches
- Patch validation — regression tests plus generated reproduction tests to rank patches
It is evaluated on SWE-bench settings (Lite/Verified artifacts are linked from the README), not on HumanEval+. If your question is “can this system fix a real repository issue?”, start from SWE-bench and related agent baselines. If your question is “does this model emit a correct function for a prompt?”, use the three suites above.
A safe selection sequence
- Name the claim. Correctness under weak vs strong tests (EvalPlus), library-heavy tasks (BigCodeBench), or post-cutoff contest problems (LiveCodeBench).
- Pin versions. Record package version, dataset release (especially LiveCodeBench), split (Complete/Instruct or Full/Hard) and decoding settings.
- Isolate execution. Assume every completion is hostile. Prefer the projects' Docker paths or your own sandbox; see Run untrusted code in a sandbox.
- Keep generation separate from scoring. Sanitize/extract code, then evaluate. A chatty markdown wrapper is not a failed algorithm until after extraction.
- Do not mix leaderboards. Pass rates across these suites are not commensurate with SWE-bench resolve rates or with each other under different prompts and sandboxes.
For designing an evaluation harness more generally, see Evaluate an AI agent.
Verification limits
Checked on 2026-11-09 against the project READMEs and licenses, PyPI metadata for evalplus 0.3.1 and bigcodebench 0.2.5, the HumanEval+ dataset card (164 examples, Apache-2.0), and GitHub metadata showing BigCodeBench archived. Offline smoke test: imported evalplus 0.3.1 and ran sanitize on a hand-written wrapped sample after loading HumanEval+.
Not run: model inference, full EvalPlus/BigCodeBench/LiveCodeBench evaluation jobs, Docker evaluators, Agentless SWE-bench reproduction, or any paid API. No cross-suite accuracy ranking is claimed.
Directory entries in this comparison
- EvalPlus: Code-generation evaluation framework with expanded HumanEval+/MBPP+ tests and EvalPerf for checking correctness and runtime efficiency.
- BigCodeBench: Code-generation benchmark with 1,140 software-engineering-oriented Python tasks that exercise diverse library calls, plus a harder task subset.
- LiveCodeBench: Holistic, contamination-free evaluation of LLMs for code, with a public leaderboard.
- Agentless: Software-repair baseline using localization, candidate patches and test-based validation rather than an autonomous tool-using agent loop.
- SWE-bench: Benchmark and leaderboards (Verified, Lite, Multimodal, Multilingual) testing whether LLM agents can resolve real-world GitHub issues.
Related
- Agent benchmarks compared (Comparison)
- Newer agent eval sets for data, terminals and multi-agent conflict (Comparison)
- Evaluate an AI agent with open-source eval and observability tools (Guide)
- Run agent-generated code safely in a sandbox (Guide)
Sources
- EvalPlus README (raw), accessed
- EvalPlus LICENSE (raw), accessed
- evalplus 0.3.1 on PyPI, accessed
- HumanEval+ dataset card, accessed
- BigCodeBench README (raw), accessed
- BigCodeBench LICENSE (raw), accessed
- bigcodebench 0.2.5 on PyPI, accessed
- bigcode/bigcodebench on Hugging Face, accessed
- bigcode-project/bigcodebench GitHub repository metadata (archived), accessed
- LiveCodeBench README (raw), accessed
- LiveCodeBench LICENSE (raw), accessed
- Agentless README (raw), accessed
- Agentless LICENSE (raw), accessed