# LM Evaluation Harness

> EleutherAI's widely used framework for few-shot evaluation of language models across hundreds of benchmarks, with HF, vLLM and API backends.

- id: `lm-evaluation-harness`
- category: Evals & Observability (`evals-observability`)
- url: https://github.com/EleutherAI/lm-evaluation-harness
- repo: https://github.com/EleutherAI/lm-evaluation-harness
- docs: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs
- status: active
- pricing: open-source
- license: MIT
- tags: evals, benchmarks, llm, python
- related: Inspect (https://indexagentica.com/entries/inspect-ai/); DeepEval (https://indexagentica.com/entries/deepeval/)
- sources: https://github.com/EleutherAI/lm-evaluation-harness , https://pypi.org/project/lm-eval/
- added: 2026-10-25
- updated: 2026-10-25
- last verified: 2026-10-25
- maintainer: EleutherAI
- submitted by: agentica-curator
- page: https://indexagentica.com/entries/lm-evaluation-harness/
- json: https://indexagentica.com/api/entries/lm-evaluation-harness.json

Install with `pip install lm-eval`; run tasks with `lm_eval --model hf --model_args pretrained=... --tasks ...`. PyPI `lm-eval` 0.4.13 (2026-08-31), Python >=3.10.