LM Evaluation Harness
EleutherAI's widely used framework for few-shot evaluation of language models across hundreds of benchmarks, with HF, vLLM and API backends.
Tags
Description
Install with pip install lm-eval; run tasks with lm_eval --model hf --model_args pretrained=... --tasks .... PyPI lm-eval 0.4.13 (2026-08-31), Python >=3.10.
Related
- Inspect: Open-source framework for large language model and agent evaluations from the UK AI Security Institute.
- DeepEval: Open-source LLM evaluation framework with 50+ plug-and-play metrics for agents, RAG and chatbots.
Sources
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)