{
  "type": "comparison",
  "id": "code-generation-eval-benchmarks",
  "title": "Code-generation evals compared — EvalPlus, BigCodeBench and LiveCodeBench",
  "summary": "Pick a code-generation evaluation suite by what it stress-tests: expanded unit tests, library-heavy software tasks, or continuously dated contest problems. Agentless is a SWE-bench repair baseline, not a fourth benchmark.",
  "author": "Agentica Author",
  "tags": [
    "benchmarks",
    "code-generation",
    "evaluation",
    "python",
    "coding-agents"
  ],
  "published": "2026-11-09",
  "updated": "2026-11-09",
  "last_verified": "2026-11-09",
  "entries": [
    "evalplus",
    "bigcodebench",
    "livecodebench",
    "agentless",
    "swe-bench"
  ],
  "links": {
    "html": "https://indexagentica.com/compare/code-generation-eval-benchmarks/",
    "markdown": "https://indexagentica.com/compare/code-generation-eval-benchmarks.md",
    "json": "https://indexagentica.com/api/longform/compare/code-generation-eval-benchmarks.json",
    "source": "https://github.com/Drudley/indexagentica/blob/main/content-long/compare/code-generation-eval-benchmarks.md"
  },
  "status": "published",
  "comparison": {
    "subjects": [
      {
        "id": "evalplus",
        "name": "EvalPlus",
        "summary": "Code-generation evaluation framework with expanded HumanEval+/MBPP+ tests and EvalPerf for checking correctness and runtime efficiency.",
        "url": "https://indexagentica.com/entries/evalplus/",
        "json": "https://indexagentica.com/api/entries/evalplus.json"
      },
      {
        "id": "bigcodebench",
        "name": "BigCodeBench",
        "summary": "Code-generation benchmark with 1,140 software-engineering-oriented Python tasks that exercise diverse library calls, plus a harder task subset.",
        "url": "https://indexagentica.com/entries/bigcodebench/",
        "json": "https://indexagentica.com/api/entries/bigcodebench.json"
      },
      {
        "id": "livecodebench",
        "name": "LiveCodeBench",
        "summary": "Holistic, contamination-free evaluation of LLMs for code, with a public leaderboard.",
        "url": "https://indexagentica.com/entries/livecodebench/",
        "json": "https://indexagentica.com/api/entries/livecodebench.json"
      }
    ],
    "criteria": [
      "What it evaluates",
      "Task set",
      "Scoring",
      "Freshness and contamination",
      "Running and isolation",
      "License and status"
    ],
    "rows": {
      "evalplus": {
        "What it evaluates": "Correctness of function-level Python solutions under many more tests than the original HumanEval/MBPP sets, plus optional code-efficiency evaluation (EvalPerf)",
        "Task set": "HumanEval+ (164 tasks on the Hugging Face card) and MBPP+ (378 tasks after the v0.2.0 cleanup); EvalPerf is a separate efficiency suite",
        "Scoring": "Execution against base and plus tests; the README highlights large score drops when plus tests are added. EvalPerf measures efficiency on performance-exercising tasks",
        "Freshness and contamination": "Fixed curated suites derived from HumanEval/MBPP; not a continuously dated live contest stream",
        "Running and isolation": "PyPI evalplus 0.3.1 (Python >=3.9); generation/evaluation CLIs and a documented Docker path (ganler/evalplus). Treat generated code as untrusted",
        "License and status": "Apache-2.0 (package and HumanEval+ dataset card). Active PyPI release 0.3.1"
      },
      "bigcodebench": {
        "What it evaluates": "Practical Python programming with diverse library calls and complex instructions, closer to tool-using software tasks than toy algorithms",
        "Task set": "1,140 tasks (Complete and Instruct splits); BigCodeBench-Hard is a 148-task harder subset",
        "Scoring": "Execution-based evaluation after sanitization/calibration; packages report pass@k-style result files",
        "Freshness and contamination": "Fixed 2024 benchmark (arxiv 2406.15877); not continuously extended by contest date",
        "Running and isolation": "PyPI bigcodebench 0.2.5; README documents local backends and slower remote/e2b options, plus a Docker evaluate image. Isolate execution",
        "License and status": "Apache-2.0. GitHub repository is archived (read-only); PyPI and Hugging Face dataset remain available"
      },
      "livecodebench": {
        "What it evaluates": "Holistic coding ability across scenarios such as code generation, code execution, test-output prediction and self-repair",
        "Task set": "Continuously collected contest problems from LeetCode, AtCoder and Codeforces; documented releases grow from 400 problems (release_v1) to 1,055 (release_v6)",
        "Scoring": "pass@1 and pass@5 via a modified APPS-style checker; time limits can move scores by under 0.5 points",
        "Freshness and contamination": "Problems are time-stamped; filter by release version or start/end dates so evaluation can target problems released after a model's cutoff",
        "Running and isolation": "Clone the repo and install with uv; runner scenarios are selected by flag. Still execute model code only in an isolated environment",
        "License and status": "MIT (repository LICENSE). Actively versioned releases; pin release_version for reproducibility"
      }
    },
    "table": [
      {
        "entry": "evalplus",
        "name": "EvalPlus",
        "url": "https://indexagentica.com/entries/evalplus/",
        "values": {
          "What it evaluates": "Correctness of function-level Python solutions under many more tests than the original HumanEval/MBPP sets, plus optional code-efficiency evaluation (EvalPerf)",
          "Task set": "HumanEval+ (164 tasks on the Hugging Face card) and MBPP+ (378 tasks after the v0.2.0 cleanup); EvalPerf is a separate efficiency suite",
          "Scoring": "Execution against base and plus tests; the README highlights large score drops when plus tests are added. EvalPerf measures efficiency on performance-exercising tasks",
          "Freshness and contamination": "Fixed curated suites derived from HumanEval/MBPP; not a continuously dated live contest stream",
          "Running and isolation": "PyPI evalplus 0.3.1 (Python >=3.9); generation/evaluation CLIs and a documented Docker path (ganler/evalplus). Treat generated code as untrusted",
          "License and status": "Apache-2.0 (package and HumanEval+ dataset card). Active PyPI release 0.3.1"
        }
      },
      {
        "entry": "bigcodebench",
        "name": "BigCodeBench",
        "url": "https://indexagentica.com/entries/bigcodebench/",
        "values": {
          "What it evaluates": "Practical Python programming with diverse library calls and complex instructions, closer to tool-using software tasks than toy algorithms",
          "Task set": "1,140 tasks (Complete and Instruct splits); BigCodeBench-Hard is a 148-task harder subset",
          "Scoring": "Execution-based evaluation after sanitization/calibration; packages report pass@k-style result files",
          "Freshness and contamination": "Fixed 2024 benchmark (arxiv 2406.15877); not continuously extended by contest date",
          "Running and isolation": "PyPI bigcodebench 0.2.5; README documents local backends and slower remote/e2b options, plus a Docker evaluate image. Isolate execution",
          "License and status": "Apache-2.0. GitHub repository is archived (read-only); PyPI and Hugging Face dataset remain available"
        }
      },
      {
        "entry": "livecodebench",
        "name": "LiveCodeBench",
        "url": "https://indexagentica.com/entries/livecodebench/",
        "values": {
          "What it evaluates": "Holistic coding ability across scenarios such as code generation, code execution, test-output prediction and self-repair",
          "Task set": "Continuously collected contest problems from LeetCode, AtCoder and Codeforces; documented releases grow from 400 problems (release_v1) to 1,055 (release_v6)",
          "Scoring": "pass@1 and pass@5 via a modified APPS-style checker; time limits can move scores by under 0.5 points",
          "Freshness and contamination": "Problems are time-stamped; filter by release version or start/end dates so evaluation can target problems released after a model's cutoff",
          "Running and isolation": "Clone the repo and install with uv; runner scenarios are selected by flag. Still execute model code only in an isolated environment",
          "License and status": "MIT (repository LICENSE). Actively versioned releases; pin release_version for reproducibility"
        }
      }
    ],
    "verdict": "Use EvalPlus when you care whether a solution survives many more unit tests than HumanEval/MBPP. Use BigCodeBench when you need library-heavy, instruction-rich Python tasks. Use LiveCodeBench when you need time-windowed contest problems to reduce contamination. For repository issue repair, look at SWE-bench and baselines such as Agentless—not these three suites."
  },
  "entries_detail": [
    {
      "id": "evalplus",
      "name": "EvalPlus",
      "summary": "Code-generation evaluation framework with expanded HumanEval+/MBPP+ tests and EvalPerf for checking correctness and runtime efficiency.",
      "url": "https://indexagentica.com/entries/evalplus/",
      "json": "https://indexagentica.com/api/entries/evalplus.json"
    },
    {
      "id": "bigcodebench",
      "name": "BigCodeBench",
      "summary": "Code-generation benchmark with 1,140 software-engineering-oriented Python tasks that exercise diverse library calls, plus a harder task subset.",
      "url": "https://indexagentica.com/entries/bigcodebench/",
      "json": "https://indexagentica.com/api/entries/bigcodebench.json"
    },
    {
      "id": "livecodebench",
      "name": "LiveCodeBench",
      "summary": "Holistic, contamination-free evaluation of LLMs for code, with a public leaderboard.",
      "url": "https://indexagentica.com/entries/livecodebench/",
      "json": "https://indexagentica.com/api/entries/livecodebench.json"
    },
    {
      "id": "agentless",
      "name": "Agentless",
      "summary": "Software-repair baseline using localization, candidate patches and test-based validation rather than an autonomous tool-using agent loop.",
      "url": "https://indexagentica.com/entries/agentless/",
      "json": "https://indexagentica.com/api/entries/agentless.json"
    },
    {
      "id": "swe-bench",
      "name": "SWE-bench",
      "summary": "Benchmark and leaderboards (Verified, Lite, Multimodal, Multilingual) testing whether LLM agents can resolve real-world GitHub issues.",
      "url": "https://indexagentica.com/entries/swe-bench/",
      "json": "https://indexagentica.com/api/entries/swe-bench.json"
    }
  ],
  "related": [
    {
      "type": "comparison",
      "id": "agent-benchmarks",
      "title": "Agent benchmarks compared",
      "url": "https://indexagentica.com/compare/agent-benchmarks/",
      "json": "https://indexagentica.com/api/longform/compare/agent-benchmarks.json"
    },
    {
      "type": "comparison",
      "id": "newer-agent-eval-sets",
      "title": "Newer agent eval sets for data, terminals and multi-agent conflict",
      "url": "https://indexagentica.com/compare/newer-agent-eval-sets/",
      "json": "https://indexagentica.com/api/longform/compare/newer-agent-eval-sets.json"
    },
    {
      "type": "guide",
      "id": "evaluate-an-ai-agent",
      "title": "Evaluate an AI agent with open-source eval and observability tools",
      "url": "https://indexagentica.com/guides/evaluate-an-ai-agent/",
      "json": "https://indexagentica.com/api/longform/guides/evaluate-an-ai-agent.json"
    },
    {
      "type": "guide",
      "id": "run-untrusted-code-in-a-sandbox",
      "title": "Run agent-generated code safely in a sandbox",
      "url": "https://indexagentica.com/guides/run-untrusted-code-in-a-sandbox/",
      "json": "https://indexagentica.com/api/longform/guides/run-untrusted-code-in-a-sandbox.json"
    }
  ],
  "sources": [
    {
      "title": "EvalPlus README (raw)",
      "url": "https://raw.githubusercontent.com/evalplus/evalplus/master/README.md",
      "accessed": "2026-11-09"
    },
    {
      "title": "EvalPlus LICENSE (raw)",
      "url": "https://raw.githubusercontent.com/evalplus/evalplus/master/LICENSE",
      "accessed": "2026-11-09"
    },
    {
      "title": "evalplus 0.3.1 on PyPI",
      "url": "https://pypi.org/project/evalplus/0.3.1/",
      "accessed": "2026-11-09"
    },
    {
      "title": "HumanEval+ dataset card",
      "url": "https://huggingface.co/datasets/evalplus/humanevalplus",
      "accessed": "2026-11-09"
    },
    {
      "title": "BigCodeBench README (raw)",
      "url": "https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md",
      "accessed": "2026-11-09"
    },
    {
      "title": "BigCodeBench LICENSE (raw)",
      "url": "https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/LICENSE",
      "accessed": "2026-11-09"
    },
    {
      "title": "bigcodebench 0.2.5 on PyPI",
      "url": "https://pypi.org/project/bigcodebench/0.2.5/",
      "accessed": "2026-11-09"
    },
    {
      "title": "bigcode/bigcodebench on Hugging Face",
      "url": "https://huggingface.co/datasets/bigcode/bigcodebench",
      "accessed": "2026-11-09"
    },
    {
      "title": "bigcode-project/bigcodebench GitHub repository metadata (archived)",
      "url": "https://api.github.com/repos/bigcode-project/bigcodebench",
      "accessed": "2026-11-09"
    },
    {
      "title": "LiveCodeBench README (raw)",
      "url": "https://raw.githubusercontent.com/LiveCodeBench/LiveCodeBench/main/README.md",
      "accessed": "2026-11-09"
    },
    {
      "title": "LiveCodeBench LICENSE (raw)",
      "url": "https://raw.githubusercontent.com/LiveCodeBench/LiveCodeBench/main/LICENSE",
      "accessed": "2026-11-09"
    },
    {
      "title": "Agentless README (raw)",
      "url": "https://raw.githubusercontent.com/OpenAutoCoder/Agentless/main/README.md",
      "accessed": "2026-11-09"
    },
    {
      "title": "Agentless LICENSE (raw)",
      "url": "https://raw.githubusercontent.com/OpenAutoCoder/Agentless/main/LICENSE",
      "accessed": "2026-11-09"
    }
  ],
  "front_matter": {
    "id": "code-generation-eval-benchmarks",
    "type": "comparison",
    "title": "Code-generation evals compared — EvalPlus, BigCodeBench and LiveCodeBench",
    "summary": "Pick a code-generation evaluation suite by what it stress-tests: expanded unit tests, library-heavy software tasks, or continuously dated contest problems. Agentless is a SWE-bench repair baseline, not a fourth benchmark.",
    "description": "A source-checked comparison of EvalPlus (HumanEval+/MBPP+/EvalPerf), BigCodeBench and LiveCodeBench for LLM code generation. Covers task design, scoring, contamination handling, isolation guidance and licenses. Positions Agentless as a three-phase SWE-bench repair pipeline rather than a code-generation benchmark. No model leaderboard ranking; sanitizer tested offline on EvalPlus 0.3.1 without running untrusted generated code.",
    "author": "Agentica Author",
    "tags": [
      "benchmarks",
      "code-generation",
      "evaluation",
      "python",
      "coding-agents"
    ],
    "entries": [
      "evalplus",
      "bigcodebench",
      "livecodebench",
      "agentless",
      "swe-bench"
    ],
    "related": [
      "agent-benchmarks",
      "newer-agent-eval-sets",
      "evaluate-an-ai-agent",
      "run-untrusted-code-in-a-sandbox"
    ],
    "subjects": [
      "evalplus",
      "bigcodebench",
      "livecodebench"
    ],
    "criteria": [
      "What it evaluates",
      "Task set",
      "Scoring",
      "Freshness and contamination",
      "Running and isolation",
      "License and status"
    ],
    "rows": {
      "evalplus": {
        "What it evaluates": "Correctness of function-level Python solutions under many more tests than the original HumanEval/MBPP sets, plus optional code-efficiency evaluation (EvalPerf)",
        "Task set": "HumanEval+ (164 tasks on the Hugging Face card) and MBPP+ (378 tasks after the v0.2.0 cleanup); EvalPerf is a separate efficiency suite",
        "Scoring": "Execution against base and plus tests; the README highlights large score drops when plus tests are added. EvalPerf measures efficiency on performance-exercising tasks",
        "Freshness and contamination": "Fixed curated suites derived from HumanEval/MBPP; not a continuously dated live contest stream",
        "Running and isolation": "PyPI evalplus 0.3.1 (Python >=3.9); generation/evaluation CLIs and a documented Docker path (ganler/evalplus). Treat generated code as untrusted",
        "License and status": "Apache-2.0 (package and HumanEval+ dataset card). Active PyPI release 0.3.1"
      },
      "bigcodebench": {
        "What it evaluates": "Practical Python programming with diverse library calls and complex instructions, closer to tool-using software tasks than toy algorithms",
        "Task set": "1,140 tasks (Complete and Instruct splits); BigCodeBench-Hard is a 148-task harder subset",
        "Scoring": "Execution-based evaluation after sanitization/calibration; packages report pass@k-style result files",
        "Freshness and contamination": "Fixed 2024 benchmark (arxiv 2406.15877); not continuously extended by contest date",
        "Running and isolation": "PyPI bigcodebench 0.2.5; README documents local backends and slower remote/e2b options, plus a Docker evaluate image. Isolate execution",
        "License and status": "Apache-2.0. GitHub repository is archived (read-only); PyPI and Hugging Face dataset remain available"
      },
      "livecodebench": {
        "What it evaluates": "Holistic coding ability across scenarios such as code generation, code execution, test-output prediction and self-repair",
        "Task set": "Continuously collected contest problems from LeetCode, AtCoder and Codeforces; documented releases grow from 400 problems (release_v1) to 1,055 (release_v6)",
        "Scoring": "pass@1 and pass@5 via a modified APPS-style checker; time limits can move scores by under 0.5 points",
        "Freshness and contamination": "Problems are time-stamped; filter by release version or start/end dates so evaluation can target problems released after a model's cutoff",
        "Running and isolation": "Clone the repo and install with uv; runner scenarios are selected by flag. Still execute model code only in an isolated environment",
        "License and status": "MIT (repository LICENSE). Actively versioned releases; pin release_version for reproducibility"
      }
    },
    "verdict": "Use EvalPlus when you care whether a solution survives many more unit tests than HumanEval/MBPP. Use BigCodeBench when you need library-heavy, instruction-rich Python tasks. Use LiveCodeBench when you need time-windowed contest problems to reduce contamination. For repository issue repair, look at SWE-bench and baselines such as Agentless—not these three suites.",
    "sources": [
      {
        "title": "EvalPlus README (raw)",
        "url": "https://raw.githubusercontent.com/evalplus/evalplus/master/README.md",
        "accessed": "2026-11-09"
      },
      {
        "title": "EvalPlus LICENSE (raw)",
        "url": "https://raw.githubusercontent.com/evalplus/evalplus/master/LICENSE",
        "accessed": "2026-11-09"
      },
      {
        "title": "evalplus 0.3.1 on PyPI",
        "url": "https://pypi.org/project/evalplus/0.3.1/",
        "accessed": "2026-11-09"
      },
      {
        "title": "HumanEval+ dataset card",
        "url": "https://huggingface.co/datasets/evalplus/humanevalplus",
        "accessed": "2026-11-09"
      },
      {
        "title": "BigCodeBench README (raw)",
        "url": "https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md",
        "accessed": "2026-11-09"
      },
      {
        "title": "BigCodeBench LICENSE (raw)",
        "url": "https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/LICENSE",
        "accessed": "2026-11-09"
      },
      {
        "title": "bigcodebench 0.2.5 on PyPI",
        "url": "https://pypi.org/project/bigcodebench/0.2.5/",
        "accessed": "2026-11-09"
      },
      {
        "title": "bigcode/bigcodebench on Hugging Face",
        "url": "https://huggingface.co/datasets/bigcode/bigcodebench",
        "accessed": "2026-11-09"
      },
      {
        "title": "bigcode-project/bigcodebench GitHub repository metadata (archived)",
        "url": "https://api.github.com/repos/bigcode-project/bigcodebench",
        "accessed": "2026-11-09"
      },
      {
        "title": "LiveCodeBench README (raw)",
        "url": "https://raw.githubusercontent.com/LiveCodeBench/LiveCodeBench/main/README.md",
        "accessed": "2026-11-09"
      },
      {
        "title": "LiveCodeBench LICENSE (raw)",
        "url": "https://raw.githubusercontent.com/LiveCodeBench/LiveCodeBench/main/LICENSE",
        "accessed": "2026-11-09"
      },
      {
        "title": "Agentless README (raw)",
        "url": "https://raw.githubusercontent.com/OpenAutoCoder/Agentless/main/README.md",
        "accessed": "2026-11-09"
      },
      {
        "title": "Agentless LICENSE (raw)",
        "url": "https://raw.githubusercontent.com/OpenAutoCoder/Agentless/main/LICENSE",
        "accessed": "2026-11-09"
      }
    ],
    "last_verified": "2026-11-09",
    "published": "2026-11-09",
    "updated": "2026-11-09"
  },
  "markdown": "\n## Three function-level suites, not one more SWE-bench clone\n\n[Agent benchmarks](https://indexagentica.com/compare/agent-benchmarks/) already covers repository-scale issue repair ([SWE-bench](https://indexagentica.com/entries/swe-bench/)), terminals and tool-using agents. [Newer agent eval sets](https://indexagentica.com/compare/newer-agent-eval-sets/) covers data-science, graded terminal corpora and multi-agent conflict. This page is about **code-generation benchmarks**: give a model a prompt, get a program, run tests.\n\nThe three subjects—[EvalPlus](https://indexagentica.com/entries/evalplus/), [BigCodeBench](https://indexagentica.com/entries/bigcodebench/) and [LiveCodeBench](https://indexagentica.com/entries/livecodebench/)—overlap in “Python coding” but differ in what fails a solution. The recommendations below are selection advice based on those documented differences, not a claim that one suite is universally harder.\n\n**Agentless is not in the table.** [Agentless](https://indexagentica.com/entries/agentless/) is a three-phase **repair baseline** for SWE-bench-style issues (localize, sample patches, validate with tests). It is a method for repository bugs, not a HumanEval-style generation suite. Confusing the two leads people to report Agentless “scores” as if they were another fixed prompt set.\n\n## EvalPlus: more tests on familiar tasks\n\nEvalPlus keeps the HumanEval/MBPP problem shape and attacks **underspecified tests**. Its README states HumanEval+ has **80×** more tests than HumanEval and MBPP+ has **35×** more than MBPP. The HumanEval+ Hugging Face card lists **164** examples. MBPP+ was cleaned from 399 to **378** tasks in the documented `v0.2.0` upgrade.\n\nThat makes EvalPlus the right first stop when a model already looks strong on classic pass@k numbers and you want to know whether the code is merely overfit to weak oracles. **EvalPerf** (COLM'24) is the efficiency companion: correctness is not enough if the solution is asymptotically or practically slow.\n\n**Operational notes.** PyPI **evalplus 0.3.1** requires Python **≥3.9**. The README documents a one-command evaluate path and Docker evaluation via `ganler/evalplus`. Generated samples are untrusted: prefer containerized execution. Offline, we installed 0.3.1 outside the site checkout, downloaded the HumanEval+ release used by the package, and confirmed `sanitize()` extracts a function body from a markdown-wrapped hand sample. We did **not** run model inference, Docker evaluation or untrusted code execution.\n\n## BigCodeBench: library calls and instructions\n\nBigCodeBench targets **software-engineering-oriented** Python tasks with diverse function/library calls. The project announces **1,140** tasks, with **Complete** (docstring/completion) and **Instruct** (natural-language instruction) splits, and a **Hard** subset of **148** tasks.\n\nUse it when HumanEval-style puzzles are too narrow: the failure mode you care about is composing APIs and following denser instructions, not only implementing a single algorithm.\n\n**Operational notes.** PyPI **bigcodebench 0.2.5** remains installable. The GitHub repository is **archived (read-only)** as of this check; treat the package and Hugging Face dataset as the living distribution channels, and expect slower community fixes. The README documents sanitization/calibration artifacts and isolation options (including Docker and slower remote backends). As with EvalPlus, do not evaluate raw model output on a privileged host. We did not install BigCodeBench or run its evaluator in this draft.\n\n## LiveCodeBench: time-windowed contest problems\n\nLiveCodeBench continuously collects problems from **LeetCode, AtCoder and Codeforces** and versions the dataset by date window. Documented releases include **release_v1** (400 problems, May 2023–Mar 2024) through **release_v6** (1,055 problems, May 2023–Apr 2025). The runner defaults toward the latest release; pin `--release_version` for reproducibility.\n\nBeyond code generation, the README lists scenarios for **code execution**, **test output prediction** and **self-repair**. Scoring uses **pass@1** and **pass@5** with a modified APPS-style checker; time limits can shift scores by less than half a point.\n\nUse LiveCodeBench when contamination is the primary concern: filter by problem release date relative to a model's training cutoff. That is a different guarantee from “more unit tests on old tasks.”\n\n**Operational notes.** Install from the GitHub repository with **uv** as documented. License is **MIT**. We did not clone the full runner or download every release shard for this page.\n\n## Where Agentless fits\n\nAgentless ([OpenAutoCoder/Agentless](https://github.com/OpenAutoCoder/Agentless), MIT) solves each issue with:\n\n1. **Localization** — files → classes/functions → edit sites  \n2. **Repair** — sample multiple candidate patches  \n3. **Patch validation** — regression tests plus generated reproduction tests to rank patches  \n\nIt is evaluated on **SWE-bench** settings (Lite/Verified artifacts are linked from the README), not on HumanEval+. If your question is “can this system fix a real repository issue?”, start from SWE-bench and related agent baselines. If your question is “does this model emit a correct function for a prompt?”, use the three suites above.\n\n## A safe selection sequence\n\n1. **Name the claim.** Correctness under weak vs strong tests (EvalPlus), library-heavy tasks (BigCodeBench), or post-cutoff contest problems (LiveCodeBench).\n2. **Pin versions.** Record package version, dataset release (especially LiveCodeBench), split (Complete/Instruct or Full/Hard) and decoding settings.\n3. **Isolate execution.** Assume every completion is hostile. Prefer the projects' Docker paths or your own sandbox; see [Run untrusted code in a sandbox](https://indexagentica.com/guides/run-untrusted-code-in-a-sandbox/).\n4. **Keep generation separate from scoring.** Sanitize/extract code, then evaluate. A chatty markdown wrapper is not a failed algorithm until after extraction.\n5. **Do not mix leaderboards.** Pass rates across these suites are not commensurate with SWE-bench resolve rates or with each other under different prompts and sandboxes.\n\nFor designing an evaluation harness more generally, see [Evaluate an AI agent](https://indexagentica.com/guides/evaluate-an-ai-agent/).\n\n## Verification limits\n\nChecked on **2026-11-09** against the project READMEs and licenses, PyPI metadata for evalplus **0.3.1** and bigcodebench **0.2.5**, the HumanEval+ dataset card (**164** examples, Apache-2.0), and GitHub metadata showing BigCodeBench **archived**. Offline smoke test: imported evalplus 0.3.1 and ran `sanitize` on a hand-written wrapped sample after loading HumanEval+.\n\n**Not run:** model inference, full EvalPlus/BigCodeBench/LiveCodeBench evaluation jobs, Docker evaluators, Agentless SWE-bench reproduction, or any paid API. No cross-suite accuracy ranking is claimed.\n",
  "raw": "---\nid: code-generation-eval-benchmarks\ntype: comparison\ntitle: Code-generation evals compared — EvalPlus, BigCodeBench and LiveCodeBench\nsummary: \"Pick a code-generation evaluation suite by what it stress-tests: expanded unit tests, library-heavy software tasks, or continuously dated contest problems. Agentless is a SWE-bench repair baseline, not a fourth benchmark.\"\ndescription: \"A source-checked comparison of EvalPlus (HumanEval+/MBPP+/EvalPerf), BigCodeBench and LiveCodeBench for LLM code generation. Covers task design, scoring, contamination handling, isolation guidance and licenses. Positions Agentless as a three-phase SWE-bench repair pipeline rather than a code-generation benchmark. No model leaderboard ranking; sanitizer tested offline on EvalPlus 0.3.1 without running untrusted generated code.\"\nauthor: Agentica Author\ntags: [benchmarks, code-generation, evaluation, python, coding-agents]\nentries: [evalplus, bigcodebench, livecodebench, agentless, swe-bench]\nrelated: [agent-benchmarks, newer-agent-eval-sets, evaluate-an-ai-agent, run-untrusted-code-in-a-sandbox]\nsubjects: [evalplus, bigcodebench, livecodebench]\ncriteria:\n  - What it evaluates\n  - Task set\n  - Scoring\n  - Freshness and contamination\n  - Running and isolation\n  - License and status\nrows:\n  evalplus:\n    What it evaluates: \"Correctness of function-level Python solutions under many more tests than the original HumanEval/MBPP sets, plus optional code-efficiency evaluation (EvalPerf)\"\n    Task set: \"HumanEval+ (164 tasks on the Hugging Face card) and MBPP+ (378 tasks after the v0.2.0 cleanup); EvalPerf is a separate efficiency suite\"\n    Scoring: \"Execution against base and plus tests; the README highlights large score drops when plus tests are added. EvalPerf measures efficiency on performance-exercising tasks\"\n    Freshness and contamination: \"Fixed curated suites derived from HumanEval/MBPP; not a continuously dated live contest stream\"\n    Running and isolation: \"PyPI evalplus 0.3.1 (Python >=3.9); generation/evaluation CLIs and a documented Docker path (ganler/evalplus). Treat generated code as untrusted\"\n    License and status: \"Apache-2.0 (package and HumanEval+ dataset card). Active PyPI release 0.3.1\"\n  bigcodebench:\n    What it evaluates: \"Practical Python programming with diverse library calls and complex instructions, closer to tool-using software tasks than toy algorithms\"\n    Task set: \"1,140 tasks (Complete and Instruct splits); BigCodeBench-Hard is a 148-task harder subset\"\n    Scoring: \"Execution-based evaluation after sanitization/calibration; packages report pass@k-style result files\"\n    Freshness and contamination: \"Fixed 2024 benchmark (arxiv 2406.15877); not continuously extended by contest date\"\n    Running and isolation: \"PyPI bigcodebench 0.2.5; README documents local backends and slower remote/e2b options, plus a Docker evaluate image. Isolate execution\"\n    License and status: \"Apache-2.0. GitHub repository is archived (read-only); PyPI and Hugging Face dataset remain available\"\n  livecodebench:\n    What it evaluates: \"Holistic coding ability across scenarios such as code generation, code execution, test-output prediction and self-repair\"\n    Task set: \"Continuously collected contest problems from LeetCode, AtCoder and Codeforces; documented releases grow from 400 problems (release_v1) to 1,055 (release_v6)\"\n    Scoring: \"pass@1 and pass@5 via a modified APPS-style checker; time limits can move scores by under 0.5 points\"\n    Freshness and contamination: \"Problems are time-stamped; filter by release version or start/end dates so evaluation can target problems released after a model's cutoff\"\n    Running and isolation: \"Clone the repo and install with uv; runner scenarios are selected by flag. Still execute model code only in an isolated environment\"\n    License and status: \"MIT (repository LICENSE). Actively versioned releases; pin release_version for reproducibility\"\nverdict: \"Use EvalPlus when you care whether a solution survives many more unit tests than HumanEval/MBPP. Use BigCodeBench when you need library-heavy, instruction-rich Python tasks. Use LiveCodeBench when you need time-windowed contest problems to reduce contamination. For repository issue repair, look at SWE-bench and baselines such as Agentless—not these three suites.\"\nsources:\n  - title: EvalPlus README (raw)\n    url: https://raw.githubusercontent.com/evalplus/evalplus/master/README.md\n    accessed: 2026-11-09\n  - title: EvalPlus LICENSE (raw)\n    url: https://raw.githubusercontent.com/evalplus/evalplus/master/LICENSE\n    accessed: 2026-11-09\n  - title: evalplus 0.3.1 on PyPI\n    url: https://pypi.org/project/evalplus/0.3.1/\n    accessed: 2026-11-09\n  - title: HumanEval+ dataset card\n    url: https://huggingface.co/datasets/evalplus/humanevalplus\n    accessed: 2026-11-09\n  - title: BigCodeBench README (raw)\n    url: https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/README.md\n    accessed: 2026-11-09\n  - title: BigCodeBench LICENSE (raw)\n    url: https://raw.githubusercontent.com/bigcode-project/bigcodebench/main/LICENSE\n    accessed: 2026-11-09\n  - title: bigcodebench 0.2.5 on PyPI\n    url: https://pypi.org/project/bigcodebench/0.2.5/\n    accessed: 2026-11-09\n  - title: bigcode/bigcodebench on Hugging Face\n    url: https://huggingface.co/datasets/bigcode/bigcodebench\n    accessed: 2026-11-09\n  - title: bigcode-project/bigcodebench GitHub repository metadata (archived)\n    url: https://api.github.com/repos/bigcode-project/bigcodebench\n    accessed: 2026-11-09\n  - title: LiveCodeBench README (raw)\n    url: https://raw.githubusercontent.com/LiveCodeBench/LiveCodeBench/main/README.md\n    accessed: 2026-11-09\n  - title: LiveCodeBench LICENSE (raw)\n    url: https://raw.githubusercontent.com/LiveCodeBench/LiveCodeBench/main/LICENSE\n    accessed: 2026-11-09\n  - title: Agentless README (raw)\n    url: https://raw.githubusercontent.com/OpenAutoCoder/Agentless/main/README.md\n    accessed: 2026-11-09\n  - title: Agentless LICENSE (raw)\n    url: https://raw.githubusercontent.com/OpenAutoCoder/Agentless/main/LICENSE\n    accessed: 2026-11-09\nlast_verified: 2026-11-09\npublished: 2026-11-09\nupdated: 2026-11-09\n---\n\n## Three function-level suites, not one more SWE-bench clone\n\n[Agent benchmarks](https://indexagentica.com/compare/agent-benchmarks/) already covers repository-scale issue repair ([SWE-bench](https://indexagentica.com/entries/swe-bench/)), terminals and tool-using agents. [Newer agent eval sets](https://indexagentica.com/compare/newer-agent-eval-sets/) covers data-science, graded terminal corpora and multi-agent conflict. This page is about **code-generation benchmarks**: give a model a prompt, get a program, run tests.\n\nThe three subjects—[EvalPlus](https://indexagentica.com/entries/evalplus/), [BigCodeBench](https://indexagentica.com/entries/bigcodebench/) and [LiveCodeBench](https://indexagentica.com/entries/livecodebench/)—overlap in “Python coding” but differ in what fails a solution. The recommendations below are selection advice based on those documented differences, not a claim that one suite is universally harder.\n\n**Agentless is not in the table.** [Agentless](https://indexagentica.com/entries/agentless/) is a three-phase **repair baseline** for SWE-bench-style issues (localize, sample patches, validate with tests). It is a method for repository bugs, not a HumanEval-style generation suite. Confusing the two leads people to report Agentless “scores” as if they were another fixed prompt set.\n\n## EvalPlus: more tests on familiar tasks\n\nEvalPlus keeps the HumanEval/MBPP problem shape and attacks **underspecified tests**. Its README states HumanEval+ has **80×** more tests than HumanEval and MBPP+ has **35×** more than MBPP. The HumanEval+ Hugging Face card lists **164** examples. MBPP+ was cleaned from 399 to **378** tasks in the documented `v0.2.0` upgrade.\n\nThat makes EvalPlus the right first stop when a model already looks strong on classic pass@k numbers and you want to know whether the code is merely overfit to weak oracles. **EvalPerf** (COLM'24) is the efficiency companion: correctness is not enough if the solution is asymptotically or practically slow.\n\n**Operational notes.** PyPI **evalplus 0.3.1** requires Python **≥3.9**. The README documents a one-command evaluate path and Docker evaluation via `ganler/evalplus`. Generated samples are untrusted: prefer containerized execution. Offline, we installed 0.3.1 outside the site checkout, downloaded the HumanEval+ release used by the package, and confirmed `sanitize()` extracts a function body from a markdown-wrapped hand sample. We did **not** run model inference, Docker evaluation or untrusted code execution.\n\n## BigCodeBench: library calls and instructions\n\nBigCodeBench targets **software-engineering-oriented** Python tasks with diverse function/library calls. The project announces **1,140** tasks, with **Complete** (docstring/completion) and **Instruct** (natural-language instruction) splits, and a **Hard** subset of **148** tasks.\n\nUse it when HumanEval-style puzzles are too narrow: the failure mode you care about is composing APIs and following denser instructions, not only implementing a single algorithm.\n\n**Operational notes.** PyPI **bigcodebench 0.2.5** remains installable. The GitHub repository is **archived (read-only)** as of this check; treat the package and Hugging Face dataset as the living distribution channels, and expect slower community fixes. The README documents sanitization/calibration artifacts and isolation options (including Docker and slower remote backends). As with EvalPlus, do not evaluate raw model output on a privileged host. We did not install BigCodeBench or run its evaluator in this draft.\n\n## LiveCodeBench: time-windowed contest problems\n\nLiveCodeBench continuously collects problems from **LeetCode, AtCoder and Codeforces** and versions the dataset by date window. Documented releases include **release_v1** (400 problems, May 2023–Mar 2024) through **release_v6** (1,055 problems, May 2023–Apr 2025). The runner defaults toward the latest release; pin `--release_version` for reproducibility.\n\nBeyond code generation, the README lists scenarios for **code execution**, **test output prediction** and **self-repair**. Scoring uses **pass@1** and **pass@5** with a modified APPS-style checker; time limits can shift scores by less than half a point.\n\nUse LiveCodeBench when contamination is the primary concern: filter by problem release date relative to a model's training cutoff. That is a different guarantee from “more unit tests on old tasks.”\n\n**Operational notes.** Install from the GitHub repository with **uv** as documented. License is **MIT**. We did not clone the full runner or download every release shard for this page.\n\n## Where Agentless fits\n\nAgentless ([OpenAutoCoder/Agentless](https://github.com/OpenAutoCoder/Agentless), MIT) solves each issue with:\n\n1. **Localization** — files → classes/functions → edit sites  \n2. **Repair** — sample multiple candidate patches  \n3. **Patch validation** — regression tests plus generated reproduction tests to rank patches  \n\nIt is evaluated on **SWE-bench** settings (Lite/Verified artifacts are linked from the README), not on HumanEval+. If your question is “can this system fix a real repository issue?”, start from SWE-bench and related agent baselines. If your question is “does this model emit a correct function for a prompt?”, use the three suites above.\n\n## A safe selection sequence\n\n1. **Name the claim.** Correctness under weak vs strong tests (EvalPlus), library-heavy tasks (BigCodeBench), or post-cutoff contest problems (LiveCodeBench).\n2. **Pin versions.** Record package version, dataset release (especially LiveCodeBench), split (Complete/Instruct or Full/Hard) and decoding settings.\n3. **Isolate execution.** Assume every completion is hostile. Prefer the projects' Docker paths or your own sandbox; see [Run untrusted code in a sandbox](https://indexagentica.com/guides/run-untrusted-code-in-a-sandbox/).\n4. **Keep generation separate from scoring.** Sanitize/extract code, then evaluate. A chatty markdown wrapper is not a failed algorithm until after extraction.\n5. **Do not mix leaderboards.** Pass rates across these suites are not commensurate with SWE-bench resolve rates or with each other under different prompts and sandboxes.\n\nFor designing an evaluation harness more generally, see [Evaluate an AI agent](https://indexagentica.com/guides/evaluate-an-ai-agent/).\n\n## Verification limits\n\nChecked on **2026-11-09** against the project READMEs and licenses, PyPI metadata for evalplus **0.3.1** and bigcodebench **0.2.5**, the HumanEval+ dataset card (**164** examples, Apache-2.0), and GitHub metadata showing BigCodeBench **archived**. Offline smoke test: imported evalplus 0.3.1 and ran `sanitize` on a hand-written wrapped sample after loading HumanEval+.\n\n**Not run:** model inference, full EvalPlus/BigCodeBench/LiveCodeBench evaluation jobs, Docker evaluators, Agentless SWE-bench reproduction, or any paid API. No cross-suite accuracy ranking is claimed.\n",
  "html": "<h2>Three function-level suites, not one more SWE-bench clone</h2>\n<p><a href=\"/compare/agent-benchmarks/\">Agent benchmarks</a> already covers repository-scale issue repair (<a href=\"/entries/swe-bench/\">SWE-bench</a>), terminals and tool-using agents. <a href=\"/compare/newer-agent-eval-sets/\">Newer agent eval sets</a> covers data-science, graded terminal corpora and multi-agent conflict. This page is about <strong>code-generation benchmarks</strong>: give a model a prompt, get a program, run tests.</p>\n<p>The three subjects—<a href=\"/entries/evalplus/\">EvalPlus</a>, <a href=\"/entries/bigcodebench/\">BigCodeBench</a> and <a href=\"/entries/livecodebench/\">LiveCodeBench</a>—overlap in “Python coding” but differ in what fails a solution. The recommendations below are selection advice based on those documented differences, not a claim that one suite is universally harder.</p>\n<p><strong>Agentless is not in the table.</strong> <a href=\"/entries/agentless/\">Agentless</a> is a three-phase <strong>repair baseline</strong> for SWE-bench-style issues (localize, sample patches, validate with tests). It is a method for repository bugs, not a HumanEval-style generation suite. Confusing the two leads people to report Agentless “scores” as if they were another fixed prompt set.</p>\n<h2>EvalPlus: more tests on familiar tasks</h2>\n<p>EvalPlus keeps the HumanEval/MBPP problem shape and attacks <strong>underspecified tests</strong>. Its README states HumanEval+ has <strong>80×</strong> more tests than HumanEval and MBPP+ has <strong>35×</strong> more than MBPP. The HumanEval+ Hugging Face card lists <strong>164</strong> examples. MBPP+ was cleaned from 399 to <strong>378</strong> tasks in the documented <code>v0.2.0</code> upgrade.</p>\n<p>That makes EvalPlus the right first stop when a model already looks strong on classic pass@k numbers and you want to know whether the code is merely overfit to weak oracles. <strong>EvalPerf</strong> (COLM&#39;24) is the efficiency companion: correctness is not enough if the solution is asymptotically or practically slow.</p>\n<p><strong>Operational notes.</strong> PyPI <strong>evalplus 0.3.1</strong> requires Python <strong>≥3.9</strong>. The README documents a one-command evaluate path and Docker evaluation via <code>ganler/evalplus</code>. Generated samples are untrusted: prefer containerized execution. Offline, we installed 0.3.1 outside the site checkout, downloaded the HumanEval+ release used by the package, and confirmed <code>sanitize()</code> extracts a function body from a markdown-wrapped hand sample. We did <strong>not</strong> run model inference, Docker evaluation or untrusted code execution.</p>\n<h2>BigCodeBench: library calls and instructions</h2>\n<p>BigCodeBench targets <strong>software-engineering-oriented</strong> Python tasks with diverse function/library calls. The project announces <strong>1,140</strong> tasks, with <strong>Complete</strong> (docstring/completion) and <strong>Instruct</strong> (natural-language instruction) splits, and a <strong>Hard</strong> subset of <strong>148</strong> tasks.</p>\n<p>Use it when HumanEval-style puzzles are too narrow: the failure mode you care about is composing APIs and following denser instructions, not only implementing a single algorithm.</p>\n<p><strong>Operational notes.</strong> PyPI <strong>bigcodebench 0.2.5</strong> remains installable. The GitHub repository is <strong>archived (read-only)</strong> as of this check; treat the package and Hugging Face dataset as the living distribution channels, and expect slower community fixes. The README documents sanitization/calibration artifacts and isolation options (including Docker and slower remote backends). As with EvalPlus, do not evaluate raw model output on a privileged host. We did not install BigCodeBench or run its evaluator in this draft.</p>\n<h2>LiveCodeBench: time-windowed contest problems</h2>\n<p>LiveCodeBench continuously collects problems from <strong>LeetCode, AtCoder and Codeforces</strong> and versions the dataset by date window. Documented releases include <strong>release_v1</strong> (400 problems, May 2023–Mar 2024) through <strong>release_v6</strong> (1,055 problems, May 2023–Apr 2025). The runner defaults toward the latest release; pin <code>--release_version</code> for reproducibility.</p>\n<p>Beyond code generation, the README lists scenarios for <strong>code execution</strong>, <strong>test output prediction</strong> and <strong>self-repair</strong>. Scoring uses <strong>pass@1</strong> and <strong>pass@5</strong> with a modified APPS-style checker; time limits can shift scores by less than half a point.</p>\n<p>Use LiveCodeBench when contamination is the primary concern: filter by problem release date relative to a model&#39;s training cutoff. That is a different guarantee from “more unit tests on old tasks.”</p>\n<p><strong>Operational notes.</strong> Install from the GitHub repository with <strong>uv</strong> as documented. License is <strong>MIT</strong>. We did not clone the full runner or download every release shard for this page.</p>\n<h2>Where Agentless fits</h2>\n<p>Agentless (<a href=\"https://github.com/OpenAutoCoder/Agentless\">OpenAutoCoder/Agentless</a>, MIT) solves each issue with:</p>\n<ol><li><strong>Localization</strong> — files → classes/functions → edit sites  </li><li><strong>Repair</strong> — sample multiple candidate patches  </li><li><strong>Patch validation</strong> — regression tests plus generated reproduction tests to rank patches  </li></ol>\n<p>It is evaluated on <strong>SWE-bench</strong> settings (Lite/Verified artifacts are linked from the README), not on HumanEval+. If your question is “can this system fix a real repository issue?”, start from SWE-bench and related agent baselines. If your question is “does this model emit a correct function for a prompt?”, use the three suites above.</p>\n<h2>A safe selection sequence</h2>\n<ol><li><strong>Name the claim.</strong> Correctness under weak vs strong tests (EvalPlus), library-heavy tasks (BigCodeBench), or post-cutoff contest problems (LiveCodeBench).</li><li><strong>Pin versions.</strong> Record package version, dataset release (especially LiveCodeBench), split (Complete/Instruct or Full/Hard) and decoding settings.</li><li><strong>Isolate execution.</strong> Assume every completion is hostile. Prefer the projects&#39; Docker paths or your own sandbox; see <a href=\"/guides/run-untrusted-code-in-a-sandbox/\">Run untrusted code in a sandbox</a>.</li><li><strong>Keep generation separate from scoring.</strong> Sanitize/extract code, then evaluate. A chatty markdown wrapper is not a failed algorithm until after extraction.</li><li><strong>Do not mix leaderboards.</strong> Pass rates across these suites are not commensurate with SWE-bench resolve rates or with each other under different prompts and sandboxes.</li></ol>\n<p>For designing an evaluation harness more generally, see <a href=\"/guides/evaluate-an-ai-agent/\">Evaluate an AI agent</a>.</p>\n<h2>Verification limits</h2>\n<p>Checked on <strong>2026-11-09</strong> against the project READMEs and licenses, PyPI metadata for evalplus <strong>0.3.1</strong> and bigcodebench <strong>0.2.5</strong>, the HumanEval+ dataset card (<strong>164</strong> examples, Apache-2.0), and GitHub metadata showing BigCodeBench <strong>archived</strong>. Offline smoke test: imported evalplus 0.3.1 and ran <code>sanitize</code> on a hand-written wrapped sample after loading HumanEval+.</p>\n<p><strong>Not run:</strong> model inference, full EvalPlus/BigCodeBench/LiveCodeBench evaluation jobs, Docker evaluators, Agentless SWE-bench reproduction, or any paid API. No cross-suite accuracy ranking is claimed.</p>"
}
