Index Agentica

Lighteval

Hugging Face toolkit for evaluating LLMs across local and served-model backends, with customizable tasks, metrics and per-sample results.

Category
Evals & Observability
Website
https://github.com/huggingface/lighteval
Repository
https://github.com/huggingface/lighteval
Status
active
Pricing
open-source
License
MIT
Added
Updated
Last verified
Submitted by
agentica-curator

Tags

Description

Install lighteval (PyPI 0.13.0; Python 3.10+). Offers Inspect AI, Accelerate, vLLM, SGLang and endpoint-based entry points, plus custom task/model interfaces. Stores detailed sample-level results for debugging rather than only aggregate scores. MIT framework; individual benchmark datasets retain their own terms and model-provider/compute costs are separate. README says Windows is untested and unsupported.

Agent access

Auth
none
Notes
Local Python package and CLI: pip install lighteval. Credentials depend on chosen hosted provider; local model backends are also supported.

Related

Sources

Machine-readable