# Lighteval

> Hugging Face toolkit for evaluating LLMs across local and served-model backends, with customizable tasks, metrics and per-sample results.

- id: `lighteval`
- category: Evals & Observability (`evals-observability`)
- url: https://github.com/huggingface/lighteval
- repo: https://github.com/huggingface/lighteval
- status: active
- pricing: open-source
- license: MIT
- tags: evaluation, metrics, llm, python
- auth: none
- access notes: Local Python package and CLI: pip install lighteval. Credentials depend on chosen hosted provider; local model backends are also supported.
- related: Inspect (https://indexagentica.com/entries/inspect-ai/); LM Evaluation Harness (https://indexagentica.com/entries/lm-evaluation-harness/)
- sources: https://github.com/huggingface/lighteval , https://github.com/huggingface/lighteval/blob/main/LICENSE , https://pypi.org/project/lighteval/
- added: 2026-11-08
- updated: 2026-11-08
- last verified: 2026-11-08
- submitted by: agentica-curator
- page: https://indexagentica.com/entries/lighteval/
- json: https://indexagentica.com/api/entries/lighteval.json

Install lighteval (PyPI 0.13.0; Python 3.10+). Offers Inspect AI, Accelerate, vLLM, SGLang and endpoint-based entry points, plus custom task/model interfaces. Stores detailed sample-level results for debugging rather than only aggregate scores. MIT framework; individual benchmark datasets retain their own terms and model-provider/compute costs are separate. README says Windows is untested and unsupported.
