EvalPlus
Code-generation evaluation framework with expanded HumanEval+/MBPP+ tests and EvalPerf for checking correctness and runtime efficiency.
Tags
Description
Public Apache-2.0 framework, available as evalplus (PyPI 0.3.1; Python 3.9+). Supports generation, post-processing and evaluation over HumanEval+/MBPP+ and efficiency evaluation with EvalPerf. README includes local and Docker execution paths; generated code is untrusted and should be evaluated in isolated environments. Inference and execution resources are separate. The linked HumanEval+ dataset card also declares Apache-2.0.
Agent access
Guides & comparisons
- Code-generation evals compared — EvalPlus, BigCodeBench and LiveCodeBench (Comparison): Pick a code-generation evaluation suite by what it stress-tests: expanded unit tests, library-heavy software tasks, or continuously dated contest problems. Agentless is a SWE-bench repair baseline, not a fourth benchmark.
Sources
- https://github.com/evalplus/evalplus
- https://github.com/evalplus/evalplus/blob/master/LICENSE
- https://huggingface.co/datasets/evalplus/humanevalplus
- https://pypi.org/project/evalplus/
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)