DeepEval
Open-source LLM evaluation framework with 50+ plug-and-play metrics for agents, RAG and chatbots.
Tags
Guides & comparisons
- Evaluate an AI agent with open-source eval and observability tools (Guide): A hands-on loop for agent quality: instrument with OpenTelemetry, collect traces, turn real traces into a dataset, write code, LLM-as-judge and human evaluators, run offline experiments, then monitor production and gate pull requests in CI.
- Observability and eval platforms for AI agents (Comparison): Langfuse, LangSmith, Arize Phoenix, Braintrust, Opik and Laminar compared on license, self-hosting, OpenTelemetry ingestion, MCP access to your traces, free tiers, paid entry prices and built-in evaluation.
- Open-source red-teaming tools for LLM apps and agents (Comparison): Promptfoo, DeepTeam and Giskard compared as red-teaming tools for LLM apps and agents. Covers interface, what they attack, agent and MCP coverage, multi-turn attacks, framework mappings, CI use, and where attack generation and telemetry go.
- Agent benchmarks compared (Comparison): Ten widely cited agent benchmarks (SWE-bench, Terminal-Bench, tau-bench, BFCL, MCPMark, GAIA, BrowseComp, WebArena, Mind2Web and OSWorld) compared on what they measure, how many tasks they have, where the agent runs, how runs are scored, where results are published and how they are licensed.
Sources
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)