Ragas
Open-source evaluation toolkit for LLM apps and agents: LLM-based and traditional metrics for RAG and agent workflows, plus automatic test-set generation.
Tags
Description
Score retrieval-augmented generation and agent runs with ready-made metrics (LLM-judged and non-LLM), generate production-aligned test datasets when you do not have one, and plug into LangChain and common observability tools. Install with pip install ragas. The project now lives at vibrantlabsai/ragas (Vibrant Labs).
Agent access
Related
- DeepEval: Open-source LLM evaluation framework with 50+ plug-and-play metrics for agents, RAG and chatbots.
- TruLens: Open-source, OpenTelemetry-native tracing and evaluation for LLM apps and agents, scoring each step with LLM judges and comparing app versions.
- Giskard: Open-source Python library for testing agentic systems: scenario-based checks and LLM-as-judge evals (giskard-checks) plus a vulnerability scanner for red teaming (giskard-scan).
- Arize Phoenix: Open-source AI observability and evaluation platform from Arize.
- Langfuse: Open-source agent evals and observability platform: trace, evaluate and improve LLM applications and agents.
Sources
- https://github.com/vibrantlabsai/ragas
- https://docs.ragas.io/en/stable/
- https://pypi.org/project/ragas/
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)