Lighteval
Hugging Face toolkit for evaluating LLMs across local and served-model backends, with customizable tasks, metrics and per-sample results.
Tags
Description
Install lighteval (PyPI 0.13.0; Python 3.10+). Offers Inspect AI, Accelerate, vLLM, SGLang and endpoint-based entry points, plus custom task/model interfaces. Stores detailed sample-level results for debugging rather than only aggregate scores. MIT framework; individual benchmark datasets retain their own terms and model-provider/compute costs are separate. README says Windows is untested and unsupported.
Agent access
Related
- Inspect: Open-source framework for large language model and agent evaluations from the UK AI Security Institute.
- LM Evaluation Harness: EleutherAI's widely used framework for few-shot evaluation of language models across hundreds of benchmarks, with HF, vLLM and API backends.
Sources
- https://github.com/huggingface/lighteval
- https://github.com/huggingface/lighteval/blob/main/LICENSE
- https://pypi.org/project/lighteval/
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)