Evidently
Open-source Python framework to evaluate, test and monitor ML and LLM systems, with 100+ built-in metrics, LLM judges, test suites for CI and a monitoring UI.
Tags
Description
Reports compute data, ML and LLM quality evals (classification to RAG, text descriptors such as sentiment, length or denial detection, and LLM-as-a-judge); any Report becomes a Test Suite by adding pass/fail conditions, which suits regression testing and CI/CD checks. The monitoring UI can be self-hosted or used through Evidently Cloud, which Evidently says has a free tier plus extra features such as dataset and user management, alerting and no-code evals. Install with pip install evidently.
Agent access
Related
- DeepEval: Open-source LLM evaluation framework with 50+ plug-and-play metrics for agents, RAG and chatbots.
- Promptfoo: Open-source tool to test and red-team prompts, agents and RAG: automated evals, vulnerability scanning and model comparison.
- Arize Phoenix: Open-source AI observability and evaluation platform from Arize.
- Langfuse: Open-source agent evals and observability platform: trace, evaluate and improve LLM applications and agents.
- TruLens: Open-source, OpenTelemetry-native tracing and evaluation for LLM apps and agents, scoring each step with LLM judges and comparing app versions.
Guides & comparisons
- Data analysis agent stack (Stack): An agent that answers questions from your data: it queries DuckDB files, a ClickHouse warehouse or Postgres through read-only MCP servers, runs heavier analysis in a sandbox, reads spreadsheets and PDFs, and is tested and traced so you can trust the numbers.
Sources
- https://github.com/evidentlyai/evidently
- https://www.evidentlyai.com/
- https://pypi.org/project/evidently/
- https://docs.evidentlyai.com/llms.txt
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)