{ "type": "guide", "id": "evaluate-an-ai-agent", "title": "Evaluate an AI agent with open-source eval and observability tools", "summary": "A hands-on loop for agent quality: instrument with OpenTelemetry, collect traces, turn real traces into a dataset, write code, LLM-as-judge and human evaluators, run offline experiments, then monitor production and gate pull requests in CI.", "author": "Agentica Author", "tags": [ "evals", "observability", "tracing", "opentelemetry", "ci", "llm-as-judge" ], "published": "2026-10-14", "last_verified": "2026-10-14", "difficulty": "intermediate", "time_estimate": "45 min", "entries": [ "langfuse", "opik", "laminar", "braintrust", "otel-genai-semconv", "openllmetry", "langfuse-mcp", "opik-mcp", "laminar-mcp", "braintrust-mcp", "arize-phoenix", "promptfoo", "deepeval", "inspect-ai" ], "links": { "html": "https://indexagentica.com/guides/evaluate-an-ai-agent/", "markdown": "https://indexagentica.com/guides/evaluate-an-ai-agent.md", "json": "https://indexagentica.com/api/longform/guides/evaluate-an-ai-agent.json", "source": "https://github.com/Drudley/indexagentica/blob/main/content-long/guides/evaluate-an-ai-agent.md" }, "status": "published", "prerequisites": [ "An agent you can call as a function (any framework)", "Python 3.10+ (examples are Python; every tool also has a TypeScript SDK)", "An account or self-hosted instance of the platform you pick, and its API keys" ], "entries_detail": [ { "id": "langfuse", "name": "Langfuse", "summary": "Open-source agent evals and observability platform: trace, evaluate and improve LLM applications and agents.", "url": "https://indexagentica.com/entries/langfuse/", "json": "https://indexagentica.com/api/entries/langfuse.json" }, { "id": "opik", "name": "Opik", "summary": "Comet's open-source platform to debug, evaluate and monitor LLM apps, RAG systems and agentic workflows.", "url": "https://indexagentica.com/entries/opik/", "json": "https://indexagentica.com/api/entries/opik.json" }, { "id": "laminar", "name": "Laminar", "summary": "Open-source observability platform purpose-built for AI agents: trace, evaluate and debug agent failures.", "url": "https://indexagentica.com/entries/laminar/", "json": "https://indexagentica.com/api/entries/laminar.json" }, { "id": "braintrust", "name": "Braintrust", "summary": "AI observability platform for agents: trace production, run evals and catch regressions before users see them.", "url": "https://indexagentica.com/entries/braintrust/", "json": "https://indexagentica.com/api/entries/braintrust.json" }, { "id": "otel-genai-semconv", "name": "OpenTelemetry GenAI Semantic Conventions", "summary": "OpenTelemetry conventions for spans, metrics and events emitted by GenAI clients, MCP and model providers, for interoperable agent observability.", "url": "https://indexagentica.com/entries/otel-genai-semconv/", "json": "https://indexagentica.com/api/entries/otel-genai-semconv.json" }, { "id": "openllmetry", "name": "OpenLLMetry", "summary": "Traceloop's open-source observability for GenAI/LLM applications, built on OpenTelemetry.", "url": "https://indexagentica.com/entries/openllmetry/", "json": "https://indexagentica.com/api/entries/openllmetry.json" }, { "id": "langfuse-mcp", "name": "Langfuse MCP Server", "summary": "Native, authenticated MCP server built into Langfuse so agents can read and write traces, prompts and other project data.", "url": "https://indexagentica.com/entries/langfuse-mcp/", "json": "https://indexagentica.com/api/entries/langfuse-mcp.json" }, { "id": "opik-mcp", "name": "Opik MCP Server", "summary": "MCP server for Comet's Opik so coding assistants can read traces, log scores and manage prompts; hosted with OAuth or local over stdio.", "url": "https://indexagentica.com/entries/opik-mcp/", "json": "https://indexagentica.com/api/entries/opik-mcp.json" }, { "id": "laminar-mcp", "name": "Laminar MCP Server", "summary": "Laminar's MCP server lets assistants query agent trace data: ask_agent, read-only ClickHouse SQL and LLM-optimized trace summaries.", "url": "https://indexagentica.com/entries/laminar-mcp/", "json": "https://indexagentica.com/api/entries/laminar-mcp.json" }, { "id": "braintrust-mcp", "name": "Braintrust MCP", "summary": "Remote Braintrust MCP server for Claude Code, Cursor, Codex and other clients to query logs and work with Braintrust data.", "url": "https://indexagentica.com/entries/braintrust-mcp/", "json": "https://indexagentica.com/api/entries/braintrust-mcp.json" }, { "id": "arize-phoenix", "name": "Arize Phoenix", "summary": "Open-source AI observability and evaluation platform from Arize.", "url": "https://indexagentica.com/entries/arize-phoenix/", "json": "https://indexagentica.com/api/entries/arize-phoenix.json" }, { "id": "promptfoo", "name": "Promptfoo", "summary": "Open-source tool to test and red-team prompts, agents and RAG: automated evals, vulnerability scanning and model comparison.", "url": "https://indexagentica.com/entries/promptfoo/", "json": "https://indexagentica.com/api/entries/promptfoo.json" }, { "id": "deepeval", "name": "DeepEval", "summary": "Open-source LLM evaluation framework with 50+ plug-and-play metrics for agents, RAG and chatbots.", "url": "https://indexagentica.com/entries/deepeval/", "json": "https://indexagentica.com/api/entries/deepeval.json" }, { "id": "inspect-ai", "name": "Inspect", "summary": "Open-source framework for large language model and agent evaluations from the UK AI Security Institute.", "url": "https://indexagentica.com/entries/inspect-ai/", "json": "https://indexagentica.com/api/entries/inspect-ai.json" } ], "related": [ { "type": "comparison", "id": "llm-red-teaming-tools", "title": "Open-source red-teaming tools for LLM apps and agents", "url": "https://indexagentica.com/compare/llm-red-teaming-tools/", "json": "https://indexagentica.com/api/longform/compare/llm-red-teaming-tools.json" }, { "type": "comparison", "id": "agent-observability-platforms", "title": "Observability and eval platforms for AI agents", "url": "https://indexagentica.com/compare/agent-observability-platforms/", "json": "https://indexagentica.com/api/longform/compare/agent-observability-platforms.json" }, { "type": "comparison", "id": "coding-agent-harnesses", "title": "Terminal coding agents compared", "url": "https://indexagentica.com/compare/coding-agent-harnesses/", "json": "https://indexagentica.com/api/longform/compare/coding-agent-harnesses.json" }, { "type": "stack", "id": "research-agent", "title": "Research agent stack", "url": "https://indexagentica.com/stacks/research-agent/", "json": "https://indexagentica.com/api/longform/stacks/research-agent.json" }, { "type": "comparison", "id": "agent-benchmarks", "title": "Agent benchmarks compared", "url": "https://indexagentica.com/compare/agent-benchmarks/", "json": "https://indexagentica.com/api/longform/compare/agent-benchmarks.json" }, { "type": "stack", "id": "data-analysis-agent", "title": "Data analysis agent stack", "url": "https://indexagentica.com/stacks/data-analysis-agent/", "json": "https://indexagentica.com/api/longform/stacks/data-analysis-agent.json" }, { "type": "guide", "id": "test-your-mcp-server", "title": "Test your MCP server with in-process tests and Inspector CI checks", "url": "https://indexagentica.com/guides/test-your-mcp-server/", "json": "https://indexagentica.com/api/longform/guides/test-your-mcp-server.json" }, { "type": "guide", "id": "test-an-agent-for-prompt-injection", "title": "Test a tool-using agent for prompt injection with AgentDojo", "url": "https://indexagentica.com/guides/test-an-agent-for-prompt-injection/", "json": "https://indexagentica.com/api/longform/guides/test-an-agent-for-prompt-injection.json" } ], "sources": [ { "title": "Langfuse observability get started", "url": "https://langfuse.com/docs/observability/get-started", "accessed": "2026-10-14" }, { "title": "Langfuse OpenTelemetry integration", "url": "https://langfuse.com/integrations/native/opentelemetry", "accessed": "2026-10-14" }, { "title": "Langfuse datasets", "url": "https://langfuse.com/docs/evaluation/experiments/datasets", "accessed": "2026-10-14" }, { "title": "Langfuse evaluation overview", "url": "https://langfuse.com/docs/evaluation/overview", "accessed": "2026-10-14" }, { "title": "Langfuse experiments in CI/CD", "url": "https://langfuse.com/docs/evaluation/experiments/experiments-ci-cd", "accessed": "2026-10-14" }, { "title": "Opik evaluation getting started", "url": "https://www.comet.com/docs/opik/evaluation/getting-started", "accessed": "2026-10-14" }, { "title": "Opik custom metric", "url": "https://www.comet.com/docs/opik/evaluation/metrics/custom_metric", "accessed": "2026-10-14" }, { "title": "Opik G-Eval", "url": "https://www.comet.com/docs/opik/evaluation/metrics/g_eval", "accessed": "2026-10-14" }, { "title": "Opik evaluate agents", "url": "https://www.comet.com/docs/opik/evaluation/evaluate_agents", "accessed": "2026-10-14" }, { "title": "Opik annotation queues", "url": "https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues", "accessed": "2026-10-14" }, { "title": "Opik test suites", "url": "https://www.comet.com/docs/opik/evaluation/advanced/building-test-suites", "accessed": "2026-10-14" }, { "title": "Opik online evaluation rules", "url": "https://www.comet.com/docs/opik/production/online-evaluation/rules", "accessed": "2026-10-14" }, { "title": "Opik OpenTelemetry integration", "url": "https://www.comet.com/docs/opik/integrations/opentelemetry", "accessed": "2026-10-14" }, { "title": "Laminar OpenAI integration", "url": "https://laminar.sh/docs/integrations/openai", "accessed": "2026-10-14" }, { "title": "Laminar observe decorator", "url": "https://laminar.sh/docs/tracing/structure/observe-decorator", "accessed": "2026-10-14" }, { "title": "Laminar OpenTelemetry", "url": "https://laminar.sh/docs/tracing/otel", "accessed": "2026-10-14" }, { "title": "Laminar adding data to datasets", "url": "https://laminar.sh/docs/datasets/adding-data", "accessed": "2026-10-14" }, { "title": "Laminar labeling queues", "url": "https://laminar.sh/docs/queues/quickstart", "accessed": "2026-10-14" }, { "title": "Laminar evaluations quickstart", "url": "https://laminar.sh/docs/evaluations/quickstart", "accessed": "2026-10-14" }, { "title": "Braintrust OpenTelemetry", "url": "https://www.braintrust.dev/docs/integrations/sdk-integrations/opentelemetry/send-traces-and-logs", "accessed": "2026-10-14" } ], "front_matter": { "id": "evaluate-an-ai-agent", "type": "guide", "title": "Evaluate an AI agent with open-source eval and observability tools", "summary": "A hands-on loop for agent quality: instrument with OpenTelemetry, collect traces, turn real traces into a dataset, write code, LLM-as-judge and human evaluators, run offline experiments, then monitor production and gate pull requests in CI.", "description": "A practical walkthrough of the full evaluation loop for an AI agent, using code from the official docs of Langfuse, Opik and Laminar, with Braintrust where it fits. It covers tracing over OpenTelemetry, building a dataset from production traces, three kinds of evaluators, offline experiments, online scoring and a regression gate in GitHub Actions. Snippets were checked against vendor docs on 2026-10-14. To choose a platform, see the comparison of observability and eval platforms.", "author": "Agentica Author", "difficulty": "intermediate", "time_estimate": "45 min", "prerequisites": [ "An agent you can call as a function (any framework)", "Python 3.10+ (examples are Python; every tool also has a TypeScript SDK)", "An account or self-hosted instance of the platform you pick, and its API keys" ], "tags": [ "evals", "observability", "tracing", "opentelemetry", "ci", "llm-as-judge" ], "entries": [ "langfuse", "opik", "laminar", "braintrust", "otel-genai-semconv", "openllmetry", "langfuse-mcp", "opik-mcp", "laminar-mcp", "braintrust-mcp", "arize-phoenix", "promptfoo", "deepeval", "inspect-ai" ], "sources": [ { "title": "Langfuse observability get started", "url": "https://langfuse.com/docs/observability/get-started", "accessed": "2026-10-14" }, { "title": "Langfuse OpenTelemetry integration", "url": "https://langfuse.com/integrations/native/opentelemetry", "accessed": "2026-10-14" }, { "title": "Langfuse datasets", "url": "https://langfuse.com/docs/evaluation/experiments/datasets", "accessed": "2026-10-14" }, { "title": "Langfuse evaluation overview", "url": "https://langfuse.com/docs/evaluation/overview", "accessed": "2026-10-14" }, { "title": "Langfuse experiments in CI/CD", "url": "https://langfuse.com/docs/evaluation/experiments/experiments-ci-cd", "accessed": "2026-10-14" }, { "title": "Opik evaluation getting started", "url": "https://www.comet.com/docs/opik/evaluation/getting-started", "accessed": "2026-10-14" }, { "title": "Opik custom metric", "url": "https://www.comet.com/docs/opik/evaluation/metrics/custom_metric", "accessed": "2026-10-14" }, { "title": "Opik G-Eval", "url": "https://www.comet.com/docs/opik/evaluation/metrics/g_eval", "accessed": "2026-10-14" }, { "title": "Opik evaluate agents", "url": "https://www.comet.com/docs/opik/evaluation/evaluate_agents", "accessed": "2026-10-14" }, { "title": "Opik annotation queues", "url": "https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues", "accessed": "2026-10-14" }, { "title": "Opik test suites", "url": "https://www.comet.com/docs/opik/evaluation/advanced/building-test-suites", "accessed": "2026-10-14" }, { "title": "Opik online evaluation rules", "url": "https://www.comet.com/docs/opik/production/online-evaluation/rules", "accessed": "2026-10-14" }, { "title": "Opik OpenTelemetry integration", "url": "https://www.comet.com/docs/opik/integrations/opentelemetry", "accessed": "2026-10-14" }, { "title": "Laminar OpenAI integration", "url": "https://laminar.sh/docs/integrations/openai", "accessed": "2026-10-14" }, { "title": "Laminar observe decorator", "url": "https://laminar.sh/docs/tracing/structure/observe-decorator", "accessed": "2026-10-14" }, { "title": "Laminar OpenTelemetry", "url": "https://laminar.sh/docs/tracing/otel", "accessed": "2026-10-14" }, { "title": "Laminar adding data to datasets", "url": "https://laminar.sh/docs/datasets/adding-data", "accessed": "2026-10-14" }, { "title": "Laminar labeling queues", "url": "https://laminar.sh/docs/queues/quickstart", "accessed": "2026-10-14" }, { "title": "Laminar evaluations quickstart", "url": "https://laminar.sh/docs/evaluations/quickstart", "accessed": "2026-10-14" }, { "title": "Braintrust OpenTelemetry", "url": "https://www.braintrust.dev/docs/integrations/sdk-integrations/opentelemetry/send-traces-and-logs", "accessed": "2026-10-14" } ], "related": [ "llm-red-teaming-tools", "agent-observability-platforms", "coding-agent-harnesses", "research-agent", "agent-benchmarks", "data-analysis-agent", "test-your-mcp-server", "test-an-agent-for-prompt-injection" ], "last_verified": "2026-10-14", "published": "2026-10-14" }, "markdown": "\n## The loop\n\nEvaluating an agent is a cycle, not a one-off benchmark:\n\n1. **Trace** every run, so you can see each model call, tool call and step.\n2. **Collect** real traces from development and production.\n3. **Curate** the interesting traces (failures, edge cases, typical requests) into a dataset with expected outputs.\n4. **Score** with evaluators: deterministic code checks, LLM-as-judge and human review.\n5. **Experiment** offline: run a candidate version against the dataset and compare scores.\n6. **Monitor** production with online evaluators, and **gate** pull requests in CI.\n\nThe examples use [Langfuse](https://indexagentica.com/entries/langfuse/), [Opik](https://indexagentica.com/entries/opik/) and [Laminar](https://indexagentica.com/entries/laminar/), all open source and self-hostable, plus [Braintrust](https://indexagentica.com/entries/braintrust/) for OpenTelemetry. Pick one platform and follow its column through every step. Mixing platforms only makes sense at the tracing layer. To choose between them, see [Observability and eval platforms for AI agents](https://indexagentica.com/compare/agent-observability-platforms/).\n\n## 1. Instrument with OpenTelemetry\n\nAll four platforms accept OTLP. Instrumenting with OpenTelemetry (with the [GenAI semantic conventions](https://indexagentica.com/entries/otel-genai-semconv/) or [OpenLLMetry](https://indexagentica.com/entries/openllmetry/)) keeps you portable. Each vendor also ships a thin SDK that sets OTel up for you.\n\n**Langfuse, plain OTLP.** Point any OTel exporter at the Langfuse endpoint. `AUTH_STRING` is base64 of `public_key:secret_key`.\n\n```bash\nOTEL_EXPORTER_OTLP_ENDPOINT=\"https://cloud.langfuse.com/api/public/otel\" # EU region\n# Self-hosted (>= v3.22.0): http://localhost:3000/api/public/otel\nOTEL_EXPORTER_OTLP_HEADERS=\"Authorization=Basic ${AUTH_STRING},x-langfuse-ingestion-version=4\"\n```\n\n**Langfuse SDK.** The SDK is built on OTel. Nest spans and generations around your agent's steps:\n\n```python\nfrom langfuse import get_client\n\nlangfuse = get_client()\n\nwith langfuse.start_as_current_observation(as_type=\"span\", name=\"process-request\") as span:\n span.update(output=\"Processing complete\")\n with langfuse.start_as_current_observation(as_type=\"generation\", name=\"llm-response\", model=\"gpt-3.5-turbo\") as generation:\n generation.update(output=\"Generated response\")\n\nlangfuse.flush() # in short-lived processes\n```\n\n**Laminar.** One call instruments supported LLM clients. `@observe()` turns your own functions into spans:\n\n```python\nfrom lmnr import Laminar, observe\nLaminar.initialize() # reads LMNR_PROJECT_API_KEY\n\n@observe()\ndef answer_question(question: str) -> str:\n response = openai.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=[{\"role\": \"user\", \"content\": question}],\n )\n return response.choices[0].message.content\n```\n\nFor raw OTel, Laminar's docs use the gRPC exporter at `https://api.lmnr.ai:8443/v1/traces` with a lowercase `authorization: Bearer ` header.\n\n**Braintrust.** Add its span processor to your existing tracer provider (`pip install \"braintrust[otel]\"`, then set `BRAINTRUST_API_KEY` and `BRAINTRUST_PARENT=project_name:my-otel-project`):\n\n```python\nfrom braintrust.otel import BraintrustSpanProcessor\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\n\nprovider = TracerProvider()\ntrace.set_tracer_provider(provider)\nprovider.add_span_processor(BraintrustSpanProcessor())\n```\n\n**Opik.** Wrap the client, for example `track_openai(OpenAI())` from `opik.integrations.openai`. Opik also accepts OpenTelemetry.\n\nMake sure a trace captures the agent's input, final output, every tool call with its arguments, and the user or session id. Those fields are what you will filter and score on later.\n\n## 2. Build a dataset from real traces\n\nSynthetic prompts miss the failures your users actually hit. Start from production traces: filter for low user feedback, errors or long runs, then save the trace input with a corrected expected output.\n\n**Langfuse** links each dataset item back to its source trace:\n\n```python\nlangfuse.create_dataset(name=\"support-agent-regression-set\")\n\nlangfuse.create_dataset_item(\n dataset_name=\"support-agent-regression-set\",\n input={\"text\": \"hello world\"},\n expected_output={\"text\": \"hello world\"},\n source_trace_id=\"\",\n source_observation_id=\"\", # optional\n)\n```\n\n**Opik** can search traces with a filter and push them into an annotation queue. Reviewers can then label them before you insert items into a dataset:\n\n```python\nimport opik\nclient = opik.Opik()\n\nqueue = client.get_traces_annotation_queue(\"queue-id\")\ntraces = client.search_traces(\n project_name=\"my-project\",\n filter_string='feedback_scores.user_frustration > 0.5',\n)\nqueue.add_traces(traces)\n\ndataset = client.get_or_create_dataset(name=\"my-eval-dataset\")\ndataset.insert([{\"input\": \"What is 2+2?\", \"expected_output\": \"4\"}])\n```\n\n**Laminar** works mostly in the UI. Select a span and choose *Add to dataset*: the span input becomes the datapoint's `data` and the output becomes its `target`. You can also import CSV, JSON or JSONL files. A labeling queue lets humans edit and approve targets before pushing them to a dataset.\n\nKeep the dataset small and sharp at first, a few dozen cases that each test one behaviour. Give every case a stable id in its metadata, so a CI gate can tell which case regressed.\n\n## 3. Write evaluators\n\nUse the cheapest evaluator that can catch the failure.\n\n**Code checks** cover exact matches, schema validity, the right tool being called, or length limits. They are deterministic and free. In Opik, a custom metric that checks the agent's first tool call against the expected one:\n\n```python\nfrom opik.evaluation.metrics import base_metric, score_result\n\nclass ToolSelectionQuality(base_metric.BaseMetric):\n def __init__(self, name: str = \"tool_selection_quality\"):\n self.name = name\n\n def score(self, tool_calls, expected_tool_calls, **kwargs):\n actual = tool_calls[0][\"function_name\"]\n expected = expected_tool_calls[0][\"function_name\"]\n return score_result.ScoreResult(\n name=self.name,\n value=1 if actual == expected else 0,\n reason=f\"Expected {expected}, got {actual}\",\n )\n```\n\nIn Laminar, an evaluator is just a function that returns a number:\n\n```python\ndef accuracy(output: str, target: str) -> int:\n return 1 if target.lower() in output.lower() else 0\n```\n\n**LLM-as-judge** grades what code can't, such as faithfulness, tone or whether the question was actually answered. Opik's G-Eval takes a task introduction and criteria:\n\n```python\nfrom opik.evaluation.metrics import GEval\n\nmetric = GEval(\n task_introduction=\"You are an expert judge tasked with evaluating the faithfulness of an AI-generated answer to the given context.\",\n evaluation_criteria=\"In the provided text the OUTPUT must not introduce new information beyond what's provided in the CONTEXT.\",\n)\n```\n\nOpik test suites can also take plain-language assertions, with a pass threshold over repeated runs, which smooths out judge noise:\n\n```python\nsuite = opik_client.get_or_create_test_suite(\n name=\"my-agent-tests\",\n project_name=\"my-agent\",\n global_assertions=[\"The response directly addresses the user's question\"],\n global_execution_policy={\"runs_per_item\": 2, \"pass_threshold\": 2},\n)\nresult = opik.run_tests(test_suite=suite, task=task)\n```\n\nLangfuse also offers managed LLM-as-a-judge evaluators that you configure in the UI.\n\n**Human review** calibrates the judges. Langfuse and Opik have annotation queues, and Laminar has labeling queues. Periodically send a sample of judged traces to humans and compare their verdicts with the judge's. Langfuse's score analytics reports human-versus-AI agreement directly.\n\n## 4. Run offline experiments\n\nAn experiment runs one version of your agent over the whole dataset and records a score per item, so you can compare versions side by side.\n\n**Opik:**\n\n```python\nfrom opik.evaluation import evaluate\nfrom opik.evaluation.metrics import Hallucination\n\ndef task(item):\n return {\"output\": call_agent(item[\"input\"])}\n\nevaluate(\n dataset=dataset,\n task=task,\n scoring_metrics=[Hallucination()],\n experiment_name=\"my-experiment-v1\",\n)\n```\n\n**Laminar:** `evaluate()` initializes tracing itself, so each datapoint gets a full trace next to its scores:\n\n```python\nfrom lmnr import evaluate\n\nevaluate(\n data=[{\"data\": {\"country\": \"France\"}, \"target\": \"Paris\"}],\n executor=capital_of_country,\n evaluators={\"accuracy\": accuracy},\n name=\"Capitals v1\",\n group_name=\"capitals\",\n)\n```\n\nPut the files under `evals/` and run them with `lmnr eval`, which picks up every `eval_*.py` or `*_eval.py` file.\n\nChange one thing per experiment (prompt, model, tool description) and name runs after that change. Then the comparison view tells you what caused the difference.\n\n## 5. Monitor online and gate in CI\n\n**Online scoring.** Run LLM-as-judge on live traffic at a sampling rate you can afford. In Opik, online evaluation rules are set up per project in the UI or the REST API, with a sampling percentage. Langfuse runs the same managed evaluators on incoming traces. Alert on falling scores and feed bad traces back into step 2.\n\n**Regression gate.** Langfuse publishes a GitHub Action that runs an experiment script on every pull request, comments the scores on the PR, and fails the job when the script raises `RegressionError`:\n\n```yaml\n# .github/workflows/eval.yml\non: pull_request\npermissions: {contents: read, pull-requests: write}\njobs:\n experiment:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n - uses: actions/setup-python@v6\n with: {python-version: \"3.14\"}\n - uses: langfuse/experiment-action@\n with:\n langfuse_public_key: ${{ secrets.LANGFUSE_PUBLIC_KEY }}\n langfuse_secret_key: ${{ secrets.LANGFUSE_SECRET_KEY }}\n langfuse_base_url: https://cloud.langfuse.com\n experiment_path: experiments/support-agent-gate\n dataset_name: support-agent-regression-set\n github_token: ${{ github.token }}\n```\n\n```python\n# experiments/support-agent-gate.py\nfrom langfuse import Evaluation, RegressionError, RunnerContext\n\nTHRESHOLD = 0.95\n\ndef experiment(context: RunnerContext):\n result = context.run_experiment(\n name=\"PR gate: support agent\",\n task=answer_support_question,\n evaluators=[exact_match],\n run_evaluators=[avg_accuracy],\n )\n accuracy = next((e.value for e in result.run_evaluations if e.name == \"avg_accuracy\"), None)\n if not isinstance(accuracy, (int, float)) or accuracy < THRESHOLD:\n raise RegressionError(result=result, metric=\"avg_accuracy\",\n value=float(accuracy or 0.0), threshold=THRESHOLD)\n return result\n\ndef answer_support_question(item, **kwargs):\n return my_agent(item.input[\"question\"]) # your agent\n\ndef exact_match(*, output, expected_output, **kwargs):\n passed = output.strip() == (expected_output or \"\").strip()\n return Evaluation(name=\"exact_match\", value=1.0 if passed else 0.0)\n\ndef avg_accuracy(*, item_results, **kwargs):\n scores = [e.value for i in item_results for e in i.evaluations if e.name == \"exact_match\"]\n return Evaluation(name=\"avg_accuracy\", value=sum(scores) / len(scores) if scores else 0.0)\n```\n\nLangfuse's docs recommend two gate policies: required cases must always pass, and cases that passed in an approved baseline must not start failing. They also say missing cases, task errors and invalid evaluator results should fail the gate rather than be skipped. Without the action, the same check works as a plain pytest or Vitest test around `langfuse.run_experiment(...)`, or around Opik's `run_tests` with `pass_rate`.\n\n## Pitfalls\n\n- **Judge drift.** Pin the judge model and prompt version, and store the evaluator version in experiment metadata.\n- **Leaky datasets.** Don't tune prompts on the same cases that gate CI. Keep a held-out set.\n- **Non-determinism.** Run flaky cases several times and use a pass threshold, not a single sample.\n- **Cost.** Online judges on 100% of traffic get expensive. Sample, and score cheaply with code checks first.\n\n## Going further\n\nEach platform also exposes its traces to coding agents over MCP: [Langfuse MCP](https://indexagentica.com/entries/langfuse-mcp/), [Opik MCP](https://indexagentica.com/entries/opik-mcp/), [Laminar MCP](https://indexagentica.com/entries/laminar-mcp/) and [Braintrust MCP](https://indexagentica.com/entries/braintrust-mcp/). An agent can then pull failing traces and propose dataset items. For local, framework-style test runners without a platform, look at [promptfoo](https://indexagentica.com/entries/promptfoo/), [DeepEval](https://indexagentica.com/entries/deepeval/) or [Inspect](https://indexagentica.com/entries/inspect-ai/).\n", "raw": "---\nid: evaluate-an-ai-agent\ntype: guide\ntitle: Evaluate an AI agent with open-source eval and observability tools\nsummary: \"A hands-on loop for agent quality: instrument with OpenTelemetry, collect traces, turn real traces into a dataset, write code, LLM-as-judge and human evaluators, run offline experiments, then monitor production and gate pull requests in CI.\"\ndescription: \"A practical walkthrough of the full evaluation loop for an AI agent, using code from the official docs of Langfuse, Opik and Laminar, with Braintrust where it fits. It covers tracing over OpenTelemetry, building a dataset from production traces, three kinds of evaluators, offline experiments, online scoring and a regression gate in GitHub Actions. Snippets were checked against vendor docs on 2026-10-14. To choose a platform, see the comparison of observability and eval platforms.\"\nauthor: Agentica Author\ndifficulty: intermediate\ntime_estimate: 45 min\nprerequisites:\n - An agent you can call as a function (any framework)\n - Python 3.10+ (examples are Python; every tool also has a TypeScript SDK)\n - An account or self-hosted instance of the platform you pick, and its API keys\ntags: [evals, observability, tracing, opentelemetry, ci, llm-as-judge]\nentries: [langfuse, opik, laminar, braintrust, otel-genai-semconv, openllmetry, langfuse-mcp, opik-mcp, laminar-mcp, braintrust-mcp, arize-phoenix, promptfoo, deepeval, inspect-ai]\nsources:\n - {title: Langfuse observability get started, url: \"https://langfuse.com/docs/observability/get-started\", accessed: 2026-10-14}\n - {title: Langfuse OpenTelemetry integration, url: \"https://langfuse.com/integrations/native/opentelemetry\", accessed: 2026-10-14}\n - {title: Langfuse datasets, url: \"https://langfuse.com/docs/evaluation/experiments/datasets\", accessed: 2026-10-14}\n - {title: Langfuse evaluation overview, url: \"https://langfuse.com/docs/evaluation/overview\", accessed: 2026-10-14}\n - {title: Langfuse experiments in CI/CD, url: \"https://langfuse.com/docs/evaluation/experiments/experiments-ci-cd\", accessed: 2026-10-14}\n - {title: Opik evaluation getting started, url: \"https://www.comet.com/docs/opik/evaluation/getting-started\", accessed: 2026-10-14}\n - {title: Opik custom metric, url: \"https://www.comet.com/docs/opik/evaluation/metrics/custom_metric\", accessed: 2026-10-14}\n - {title: Opik G-Eval, url: \"https://www.comet.com/docs/opik/evaluation/metrics/g_eval\", accessed: 2026-10-14}\n - {title: Opik evaluate agents, url: \"https://www.comet.com/docs/opik/evaluation/evaluate_agents\", accessed: 2026-10-14}\n - {title: Opik annotation queues, url: \"https://www.comet.com/docs/opik/evaluation/advanced/annotation_queues\", accessed: 2026-10-14}\n - {title: Opik test suites, url: \"https://www.comet.com/docs/opik/evaluation/advanced/building-test-suites\", accessed: 2026-10-14}\n - {title: Opik online evaluation rules, url: \"https://www.comet.com/docs/opik/production/online-evaluation/rules\", accessed: 2026-10-14}\n - {title: Opik OpenTelemetry integration, url: \"https://www.comet.com/docs/opik/integrations/opentelemetry\", accessed: 2026-10-14}\n - {title: Laminar OpenAI integration, url: \"https://laminar.sh/docs/integrations/openai\", accessed: 2026-10-14}\n - {title: Laminar observe decorator, url: \"https://laminar.sh/docs/tracing/structure/observe-decorator\", accessed: 2026-10-14}\n - {title: Laminar OpenTelemetry, url: \"https://laminar.sh/docs/tracing/otel\", accessed: 2026-10-14}\n - {title: Laminar adding data to datasets, url: \"https://laminar.sh/docs/datasets/adding-data\", accessed: 2026-10-14}\n - {title: Laminar labeling queues, url: \"https://laminar.sh/docs/queues/quickstart\", accessed: 2026-10-14}\n - {title: Laminar evaluations quickstart, url: \"https://laminar.sh/docs/evaluations/quickstart\", accessed: 2026-10-14}\n - {title: Braintrust OpenTelemetry, url: \"https://www.braintrust.dev/docs/integrations/sdk-integrations/opentelemetry/send-traces-and-logs\", accessed: 2026-10-14}\nrelated: [llm-red-teaming-tools, agent-observability-platforms, coding-agent-harnesses, research-agent, agent-benchmarks, data-analysis-agent, test-your-mcp-server, test-an-agent-for-prompt-injection]\nlast_verified: 2026-10-14\npublished: 2026-10-14\n---\n\n## The loop\n\nEvaluating an agent is a cycle, not a one-off benchmark:\n\n1. **Trace** every run, so you can see each model call, tool call and step.\n2. **Collect** real traces from development and production.\n3. **Curate** the interesting traces (failures, edge cases, typical requests) into a dataset with expected outputs.\n4. **Score** with evaluators: deterministic code checks, LLM-as-judge and human review.\n5. **Experiment** offline: run a candidate version against the dataset and compare scores.\n6. **Monitor** production with online evaluators, and **gate** pull requests in CI.\n\nThe examples use [Langfuse](https://indexagentica.com/entries/langfuse/), [Opik](https://indexagentica.com/entries/opik/) and [Laminar](https://indexagentica.com/entries/laminar/), all open source and self-hostable, plus [Braintrust](https://indexagentica.com/entries/braintrust/) for OpenTelemetry. Pick one platform and follow its column through every step. Mixing platforms only makes sense at the tracing layer. To choose between them, see [Observability and eval platforms for AI agents](https://indexagentica.com/compare/agent-observability-platforms/).\n\n## 1. Instrument with OpenTelemetry\n\nAll four platforms accept OTLP. Instrumenting with OpenTelemetry (with the [GenAI semantic conventions](https://indexagentica.com/entries/otel-genai-semconv/) or [OpenLLMetry](https://indexagentica.com/entries/openllmetry/)) keeps you portable. Each vendor also ships a thin SDK that sets OTel up for you.\n\n**Langfuse, plain OTLP.** Point any OTel exporter at the Langfuse endpoint. `AUTH_STRING` is base64 of `public_key:secret_key`.\n\n```bash\nOTEL_EXPORTER_OTLP_ENDPOINT=\"https://cloud.langfuse.com/api/public/otel\" # EU region\n# Self-hosted (>= v3.22.0): http://localhost:3000/api/public/otel\nOTEL_EXPORTER_OTLP_HEADERS=\"Authorization=Basic ${AUTH_STRING},x-langfuse-ingestion-version=4\"\n```\n\n**Langfuse SDK.** The SDK is built on OTel. Nest spans and generations around your agent's steps:\n\n```python\nfrom langfuse import get_client\n\nlangfuse = get_client()\n\nwith langfuse.start_as_current_observation(as_type=\"span\", name=\"process-request\") as span:\n span.update(output=\"Processing complete\")\n with langfuse.start_as_current_observation(as_type=\"generation\", name=\"llm-response\", model=\"gpt-3.5-turbo\") as generation:\n generation.update(output=\"Generated response\")\n\nlangfuse.flush() # in short-lived processes\n```\n\n**Laminar.** One call instruments supported LLM clients. `@observe()` turns your own functions into spans:\n\n```python\nfrom lmnr import Laminar, observe\nLaminar.initialize() # reads LMNR_PROJECT_API_KEY\n\n@observe()\ndef answer_question(question: str) -> str:\n response = openai.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=[{\"role\": \"user\", \"content\": question}],\n )\n return response.choices[0].message.content\n```\n\nFor raw OTel, Laminar's docs use the gRPC exporter at `https://api.lmnr.ai:8443/v1/traces` with a lowercase `authorization: Bearer ` header.\n\n**Braintrust.** Add its span processor to your existing tracer provider (`pip install \"braintrust[otel]\"`, then set `BRAINTRUST_API_KEY` and `BRAINTRUST_PARENT=project_name:my-otel-project`):\n\n```python\nfrom braintrust.otel import BraintrustSpanProcessor\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\n\nprovider = TracerProvider()\ntrace.set_tracer_provider(provider)\nprovider.add_span_processor(BraintrustSpanProcessor())\n```\n\n**Opik.** Wrap the client, for example `track_openai(OpenAI())` from `opik.integrations.openai`. Opik also accepts OpenTelemetry.\n\nMake sure a trace captures the agent's input, final output, every tool call with its arguments, and the user or session id. Those fields are what you will filter and score on later.\n\n## 2. Build a dataset from real traces\n\nSynthetic prompts miss the failures your users actually hit. Start from production traces: filter for low user feedback, errors or long runs, then save the trace input with a corrected expected output.\n\n**Langfuse** links each dataset item back to its source trace:\n\n```python\nlangfuse.create_dataset(name=\"support-agent-regression-set\")\n\nlangfuse.create_dataset_item(\n dataset_name=\"support-agent-regression-set\",\n input={\"text\": \"hello world\"},\n expected_output={\"text\": \"hello world\"},\n source_trace_id=\"\",\n source_observation_id=\"\", # optional\n)\n```\n\n**Opik** can search traces with a filter and push them into an annotation queue. Reviewers can then label them before you insert items into a dataset:\n\n```python\nimport opik\nclient = opik.Opik()\n\nqueue = client.get_traces_annotation_queue(\"queue-id\")\ntraces = client.search_traces(\n project_name=\"my-project\",\n filter_string='feedback_scores.user_frustration > 0.5',\n)\nqueue.add_traces(traces)\n\ndataset = client.get_or_create_dataset(name=\"my-eval-dataset\")\ndataset.insert([{\"input\": \"What is 2+2?\", \"expected_output\": \"4\"}])\n```\n\n**Laminar** works mostly in the UI. Select a span and choose *Add to dataset*: the span input becomes the datapoint's `data` and the output becomes its `target`. You can also import CSV, JSON or JSONL files. A labeling queue lets humans edit and approve targets before pushing them to a dataset.\n\nKeep the dataset small and sharp at first, a few dozen cases that each test one behaviour. Give every case a stable id in its metadata, so a CI gate can tell which case regressed.\n\n## 3. Write evaluators\n\nUse the cheapest evaluator that can catch the failure.\n\n**Code checks** cover exact matches, schema validity, the right tool being called, or length limits. They are deterministic and free. In Opik, a custom metric that checks the agent's first tool call against the expected one:\n\n```python\nfrom opik.evaluation.metrics import base_metric, score_result\n\nclass ToolSelectionQuality(base_metric.BaseMetric):\n def __init__(self, name: str = \"tool_selection_quality\"):\n self.name = name\n\n def score(self, tool_calls, expected_tool_calls, **kwargs):\n actual = tool_calls[0][\"function_name\"]\n expected = expected_tool_calls[0][\"function_name\"]\n return score_result.ScoreResult(\n name=self.name,\n value=1 if actual == expected else 0,\n reason=f\"Expected {expected}, got {actual}\",\n )\n```\n\nIn Laminar, an evaluator is just a function that returns a number:\n\n```python\ndef accuracy(output: str, target: str) -> int:\n return 1 if target.lower() in output.lower() else 0\n```\n\n**LLM-as-judge** grades what code can't, such as faithfulness, tone or whether the question was actually answered. Opik's G-Eval takes a task introduction and criteria:\n\n```python\nfrom opik.evaluation.metrics import GEval\n\nmetric = GEval(\n task_introduction=\"You are an expert judge tasked with evaluating the faithfulness of an AI-generated answer to the given context.\",\n evaluation_criteria=\"In the provided text the OUTPUT must not introduce new information beyond what's provided in the CONTEXT.\",\n)\n```\n\nOpik test suites can also take plain-language assertions, with a pass threshold over repeated runs, which smooths out judge noise:\n\n```python\nsuite = opik_client.get_or_create_test_suite(\n name=\"my-agent-tests\",\n project_name=\"my-agent\",\n global_assertions=[\"The response directly addresses the user's question\"],\n global_execution_policy={\"runs_per_item\": 2, \"pass_threshold\": 2},\n)\nresult = opik.run_tests(test_suite=suite, task=task)\n```\n\nLangfuse also offers managed LLM-as-a-judge evaluators that you configure in the UI.\n\n**Human review** calibrates the judges. Langfuse and Opik have annotation queues, and Laminar has labeling queues. Periodically send a sample of judged traces to humans and compare their verdicts with the judge's. Langfuse's score analytics reports human-versus-AI agreement directly.\n\n## 4. Run offline experiments\n\nAn experiment runs one version of your agent over the whole dataset and records a score per item, so you can compare versions side by side.\n\n**Opik:**\n\n```python\nfrom opik.evaluation import evaluate\nfrom opik.evaluation.metrics import Hallucination\n\ndef task(item):\n return {\"output\": call_agent(item[\"input\"])}\n\nevaluate(\n dataset=dataset,\n task=task,\n scoring_metrics=[Hallucination()],\n experiment_name=\"my-experiment-v1\",\n)\n```\n\n**Laminar:** `evaluate()` initializes tracing itself, so each datapoint gets a full trace next to its scores:\n\n```python\nfrom lmnr import evaluate\n\nevaluate(\n data=[{\"data\": {\"country\": \"France\"}, \"target\": \"Paris\"}],\n executor=capital_of_country,\n evaluators={\"accuracy\": accuracy},\n name=\"Capitals v1\",\n group_name=\"capitals\",\n)\n```\n\nPut the files under `evals/` and run them with `lmnr eval`, which picks up every `eval_*.py` or `*_eval.py` file.\n\nChange one thing per experiment (prompt, model, tool description) and name runs after that change. Then the comparison view tells you what caused the difference.\n\n## 5. Monitor online and gate in CI\n\n**Online scoring.** Run LLM-as-judge on live traffic at a sampling rate you can afford. In Opik, online evaluation rules are set up per project in the UI or the REST API, with a sampling percentage. Langfuse runs the same managed evaluators on incoming traces. Alert on falling scores and feed bad traces back into step 2.\n\n**Regression gate.** Langfuse publishes a GitHub Action that runs an experiment script on every pull request, comments the scores on the PR, and fails the job when the script raises `RegressionError`:\n\n```yaml\n# .github/workflows/eval.yml\non: pull_request\npermissions: {contents: read, pull-requests: write}\njobs:\n experiment:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v6\n - uses: actions/setup-python@v6\n with: {python-version: \"3.14\"}\n - uses: langfuse/experiment-action@\n with:\n langfuse_public_key: ${{ secrets.LANGFUSE_PUBLIC_KEY }}\n langfuse_secret_key: ${{ secrets.LANGFUSE_SECRET_KEY }}\n langfuse_base_url: https://cloud.langfuse.com\n experiment_path: experiments/support-agent-gate\n dataset_name: support-agent-regression-set\n github_token: ${{ github.token }}\n```\n\n```python\n# experiments/support-agent-gate.py\nfrom langfuse import Evaluation, RegressionError, RunnerContext\n\nTHRESHOLD = 0.95\n\ndef experiment(context: RunnerContext):\n result = context.run_experiment(\n name=\"PR gate: support agent\",\n task=answer_support_question,\n evaluators=[exact_match],\n run_evaluators=[avg_accuracy],\n )\n accuracy = next((e.value for e in result.run_evaluations if e.name == \"avg_accuracy\"), None)\n if not isinstance(accuracy, (int, float)) or accuracy < THRESHOLD:\n raise RegressionError(result=result, metric=\"avg_accuracy\",\n value=float(accuracy or 0.0), threshold=THRESHOLD)\n return result\n\ndef answer_support_question(item, **kwargs):\n return my_agent(item.input[\"question\"]) # your agent\n\ndef exact_match(*, output, expected_output, **kwargs):\n passed = output.strip() == (expected_output or \"\").strip()\n return Evaluation(name=\"exact_match\", value=1.0 if passed else 0.0)\n\ndef avg_accuracy(*, item_results, **kwargs):\n scores = [e.value for i in item_results for e in i.evaluations if e.name == \"exact_match\"]\n return Evaluation(name=\"avg_accuracy\", value=sum(scores) / len(scores) if scores else 0.0)\n```\n\nLangfuse's docs recommend two gate policies: required cases must always pass, and cases that passed in an approved baseline must not start failing. They also say missing cases, task errors and invalid evaluator results should fail the gate rather than be skipped. Without the action, the same check works as a plain pytest or Vitest test around `langfuse.run_experiment(...)`, or around Opik's `run_tests` with `pass_rate`.\n\n## Pitfalls\n\n- **Judge drift.** Pin the judge model and prompt version, and store the evaluator version in experiment metadata.\n- **Leaky datasets.** Don't tune prompts on the same cases that gate CI. Keep a held-out set.\n- **Non-determinism.** Run flaky cases several times and use a pass threshold, not a single sample.\n- **Cost.** Online judges on 100% of traffic get expensive. Sample, and score cheaply with code checks first.\n\n## Going further\n\nEach platform also exposes its traces to coding agents over MCP: [Langfuse MCP](https://indexagentica.com/entries/langfuse-mcp/), [Opik MCP](https://indexagentica.com/entries/opik-mcp/), [Laminar MCP](https://indexagentica.com/entries/laminar-mcp/) and [Braintrust MCP](https://indexagentica.com/entries/braintrust-mcp/). An agent can then pull failing traces and propose dataset items. For local, framework-style test runners without a platform, look at [promptfoo](https://indexagentica.com/entries/promptfoo/), [DeepEval](https://indexagentica.com/entries/deepeval/) or [Inspect](https://indexagentica.com/entries/inspect-ai/).\n", "html": "

The loop

\n

Evaluating an agent is a cycle, not a one-off benchmark:

\n
  1. Trace every run, so you can see each model call, tool call and step.
  2. Collect real traces from development and production.
  3. Curate the interesting traces (failures, edge cases, typical requests) into a dataset with expected outputs.
  4. Score with evaluators: deterministic code checks, LLM-as-judge and human review.
  5. Experiment offline: run a candidate version against the dataset and compare scores.
  6. Monitor production with online evaluators, and gate pull requests in CI.
\n

The examples use Langfuse, Opik and Laminar, all open source and self-hostable, plus Braintrust for OpenTelemetry. Pick one platform and follow its column through every step. Mixing platforms only makes sense at the tracing layer. To choose between them, see Observability and eval platforms for AI agents.

\n

1. Instrument with OpenTelemetry

\n

All four platforms accept OTLP. Instrumenting with OpenTelemetry (with the GenAI semantic conventions or OpenLLMetry) keeps you portable. Each vendor also ships a thin SDK that sets OTel up for you.

\n

Langfuse, plain OTLP. Point any OTel exporter at the Langfuse endpoint. AUTH_STRING is base64 of public_key:secret_key.

\n
OTEL_EXPORTER_OTLP_ENDPOINT="https://cloud.langfuse.com/api/public/otel" # EU region\n# Self-hosted (>= v3.22.0): http://localhost:3000/api/public/otel\nOTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ${AUTH_STRING},x-langfuse-ingestion-version=4"
\n

Langfuse SDK. The SDK is built on OTel. Nest spans and generations around your agent's steps:

\n
from langfuse import get_client\n\nlangfuse = get_client()\n\nwith langfuse.start_as_current_observation(as_type="span", name="process-request") as span:\n    span.update(output="Processing complete")\n    with langfuse.start_as_current_observation(as_type="generation", name="llm-response", model="gpt-3.5-turbo") as generation:\n        generation.update(output="Generated response")\n\nlangfuse.flush()  # in short-lived processes
\n

Laminar. One call instruments supported LLM clients. @observe() turns your own functions into spans:

\n
from lmnr import Laminar, observe\nLaminar.initialize()  # reads LMNR_PROJECT_API_KEY\n\n@observe()\ndef answer_question(question: str) -> str:\n    response = openai.chat.completions.create(\n        model="gpt-4o-mini",\n        messages=[{"role": "user", "content": question}],\n    )\n    return response.choices[0].message.content
\n

For raw OTel, Laminar's docs use the gRPC exporter at https://api.lmnr.ai:8443/v1/traces with a lowercase authorization: Bearer <project key> header.

\n

Braintrust. Add its span processor to your existing tracer provider (pip install "braintrust[otel]", then set BRAINTRUST_API_KEY and BRAINTRUST_PARENT=project_name:my-otel-project):

\n
from braintrust.otel import BraintrustSpanProcessor\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\n\nprovider = TracerProvider()\ntrace.set_tracer_provider(provider)\nprovider.add_span_processor(BraintrustSpanProcessor())
\n

Opik. Wrap the client, for example track_openai(OpenAI()) from opik.integrations.openai. Opik also accepts OpenTelemetry.

\n

Make sure a trace captures the agent's input, final output, every tool call with its arguments, and the user or session id. Those fields are what you will filter and score on later.

\n

2. Build a dataset from real traces

\n

Synthetic prompts miss the failures your users actually hit. Start from production traces: filter for low user feedback, errors or long runs, then save the trace input with a corrected expected output.

\n

Langfuse links each dataset item back to its source trace:

\n
langfuse.create_dataset(name="support-agent-regression-set")\n\nlangfuse.create_dataset_item(\n    dataset_name="support-agent-regression-set",\n    input={"text": "hello world"},\n    expected_output={"text": "hello world"},\n    source_trace_id="<trace_id>",\n    source_observation_id="<observation_id>",  # optional\n)
\n

Opik can search traces with a filter and push them into an annotation queue. Reviewers can then label them before you insert items into a dataset:

\n
import opik\nclient = opik.Opik()\n\nqueue = client.get_traces_annotation_queue("queue-id")\ntraces = client.search_traces(\n    project_name="my-project",\n    filter_string='feedback_scores.user_frustration > 0.5',\n)\nqueue.add_traces(traces)\n\ndataset = client.get_or_create_dataset(name="my-eval-dataset")\ndataset.insert([{"input": "What is 2+2?", "expected_output": "4"}])
\n

Laminar works mostly in the UI. Select a span and choose Add to dataset: the span input becomes the datapoint's data and the output becomes its target. You can also import CSV, JSON or JSONL files. A labeling queue lets humans edit and approve targets before pushing them to a dataset.

\n

Keep the dataset small and sharp at first, a few dozen cases that each test one behaviour. Give every case a stable id in its metadata, so a CI gate can tell which case regressed.

\n

3. Write evaluators

\n

Use the cheapest evaluator that can catch the failure.

\n

Code checks cover exact matches, schema validity, the right tool being called, or length limits. They are deterministic and free. In Opik, a custom metric that checks the agent's first tool call against the expected one:

\n
from opik.evaluation.metrics import base_metric, score_result\n\nclass ToolSelectionQuality(base_metric.BaseMetric):\n    def __init__(self, name: str = "tool_selection_quality"):\n        self.name = name\n\n    def score(self, tool_calls, expected_tool_calls, **kwargs):\n        actual = tool_calls[0]["function_name"]\n        expected = expected_tool_calls[0]["function_name"]\n        return score_result.ScoreResult(\n            name=self.name,\n            value=1 if actual == expected else 0,\n            reason=f"Expected {expected}, got {actual}",\n        )
\n

In Laminar, an evaluator is just a function that returns a number:

\n
def accuracy(output: str, target: str) -> int:\n    return 1 if target.lower() in output.lower() else 0
\n

LLM-as-judge grades what code can't, such as faithfulness, tone or whether the question was actually answered. Opik's G-Eval takes a task introduction and criteria:

\n
from opik.evaluation.metrics import GEval\n\nmetric = GEval(\n    task_introduction="You are an expert judge tasked with evaluating the faithfulness of an AI-generated answer to the given context.",\n    evaluation_criteria="In the provided text the OUTPUT must not introduce new information beyond what's provided in the CONTEXT.",\n)
\n

Opik test suites can also take plain-language assertions, with a pass threshold over repeated runs, which smooths out judge noise:

\n
suite = opik_client.get_or_create_test_suite(\n    name="my-agent-tests",\n    project_name="my-agent",\n    global_assertions=["The response directly addresses the user's question"],\n    global_execution_policy={"runs_per_item": 2, "pass_threshold": 2},\n)\nresult = opik.run_tests(test_suite=suite, task=task)
\n

Langfuse also offers managed LLM-as-a-judge evaluators that you configure in the UI.

\n

Human review calibrates the judges. Langfuse and Opik have annotation queues, and Laminar has labeling queues. Periodically send a sample of judged traces to humans and compare their verdicts with the judge's. Langfuse's score analytics reports human-versus-AI agreement directly.

\n

4. Run offline experiments

\n

An experiment runs one version of your agent over the whole dataset and records a score per item, so you can compare versions side by side.

\n

Opik:

\n
from opik.evaluation import evaluate\nfrom opik.evaluation.metrics import Hallucination\n\ndef task(item):\n    return {"output": call_agent(item["input"])}\n\nevaluate(\n    dataset=dataset,\n    task=task,\n    scoring_metrics=[Hallucination()],\n    experiment_name="my-experiment-v1",\n)
\n

Laminar: evaluate() initializes tracing itself, so each datapoint gets a full trace next to its scores:

\n
from lmnr import evaluate\n\nevaluate(\n    data=[{"data": {"country": "France"}, "target": "Paris"}],\n    executor=capital_of_country,\n    evaluators={"accuracy": accuracy},\n    name="Capitals v1",\n    group_name="capitals",\n)
\n

Put the files under evals/ and run them with lmnr eval, which picks up every eval_*.py or *_eval.py file.

\n

Change one thing per experiment (prompt, model, tool description) and name runs after that change. Then the comparison view tells you what caused the difference.

\n

5. Monitor online and gate in CI

\n

Online scoring. Run LLM-as-judge on live traffic at a sampling rate you can afford. In Opik, online evaluation rules are set up per project in the UI or the REST API, with a sampling percentage. Langfuse runs the same managed evaluators on incoming traces. Alert on falling scores and feed bad traces back into step 2.

\n

Regression gate. Langfuse publishes a GitHub Action that runs an experiment script on every pull request, comments the scores on the PR, and fails the job when the script raises RegressionError:

\n
# .github/workflows/eval.yml\non: pull_request\npermissions: {contents: read, pull-requests: write}\njobs:\n  experiment:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v6\n      - uses: actions/setup-python@v6\n        with: {python-version: "3.14"}\n      - uses: langfuse/experiment-action@<release tag>\n        with:\n          langfuse_public_key: ${{ secrets.LANGFUSE_PUBLIC_KEY }}\n          langfuse_secret_key: ${{ secrets.LANGFUSE_SECRET_KEY }}\n          langfuse_base_url: https://cloud.langfuse.com\n          experiment_path: experiments/support-agent-gate\n          dataset_name: support-agent-regression-set\n          github_token: ${{ github.token }}
\n
# experiments/support-agent-gate.py\nfrom langfuse import Evaluation, RegressionError, RunnerContext\n\nTHRESHOLD = 0.95\n\ndef experiment(context: RunnerContext):\n    result = context.run_experiment(\n        name="PR gate: support agent",\n        task=answer_support_question,\n        evaluators=[exact_match],\n        run_evaluators=[avg_accuracy],\n    )\n    accuracy = next((e.value for e in result.run_evaluations if e.name == "avg_accuracy"), None)\n    if not isinstance(accuracy, (int, float)) or accuracy < THRESHOLD:\n        raise RegressionError(result=result, metric="avg_accuracy",\n                              value=float(accuracy or 0.0), threshold=THRESHOLD)\n    return result\n\ndef answer_support_question(item, **kwargs):\n    return my_agent(item.input["question"])  # your agent\n\ndef exact_match(*, output, expected_output, **kwargs):\n    passed = output.strip() == (expected_output or "").strip()\n    return Evaluation(name="exact_match", value=1.0 if passed else 0.0)\n\ndef avg_accuracy(*, item_results, **kwargs):\n    scores = [e.value for i in item_results for e in i.evaluations if e.name == "exact_match"]\n    return Evaluation(name="avg_accuracy", value=sum(scores) / len(scores) if scores else 0.0)
\n

Langfuse's docs recommend two gate policies: required cases must always pass, and cases that passed in an approved baseline must not start failing. They also say missing cases, task errors and invalid evaluator results should fail the gate rather than be skipped. Without the action, the same check works as a plain pytest or Vitest test around langfuse.run_experiment(...), or around Opik's run_tests with pass_rate.

\n

Pitfalls

\n
  • Judge drift. Pin the judge model and prompt version, and store the evaluator version in experiment metadata.
  • Leaky datasets. Don't tune prompts on the same cases that gate CI. Keep a held-out set.
  • Non-determinism. Run flaky cases several times and use a pass threshold, not a single sample.
  • Cost. Online judges on 100% of traffic get expensive. Sample, and score cheaply with code checks first.
\n

Going further

\n

Each platform also exposes its traces to coding agents over MCP: Langfuse MCP, Opik MCP, Laminar MCP and Braintrust MCP. An agent can then pull failing traces and propose dataset items. For local, framework-style test runners without a platform, look at promptfoo, DeepEval or Inspect.

" }