{
  "type": "comparison",
  "id": "newer-agent-eval-sets",
  "title": "Newer agent eval sets for data, terminals and multi-agent conflict",
  "summary": "AgenticDataBench, TermGrade and CLASHarness compared: what each measures, task counts, how scoring works, how to run them, and licenses. Cross-links the broader agent-benchmarks comparison.",
  "author": "Agentica Author",
  "tags": [
    "benchmarks",
    "evaluation",
    "data-agents",
    "terminal-agents",
    "multi-agent",
    "computer-use"
  ],
  "published": "2026-11-06",
  "updated": "2026-11-06",
  "last_verified": "2026-11-06",
  "entries": [
    "agenticdatabench",
    "termgrade",
    "clasharness"
  ],
  "links": {
    "html": "https://indexagentica.com/compare/newer-agent-eval-sets/",
    "markdown": "https://indexagentica.com/compare/newer-agent-eval-sets.md",
    "json": "https://indexagentica.com/api/longform/compare/newer-agent-eval-sets.json",
    "source": "https://github.com/Drudley/indexagentica/blob/main/content-long/compare/newer-agent-eval-sets.md"
  },
  "status": "published",
  "comparison": {
    "subjects": [
      {
        "id": "agenticdatabench",
        "name": "AgenticDataBench",
        "summary": "Benchmark for data-science agents: 344 tasks over 97 real datasets in 15 domains, with 433 fine-grained skill labels for skill-level scoring.",
        "url": "https://indexagentica.com/entries/agenticdatabench/",
        "json": "https://indexagentica.com/api/entries/agenticdatabench.json"
      },
      {
        "id": "termgrade",
        "name": "TermGrade",
        "summary": "1,004 executable Linux terminal tasks with pytest verifiers and per-model pass rates, plus 36,144 graded agent trajectories, for training and evaluating agents.",
        "url": "https://indexagentica.com/entries/termgrade/",
        "json": "https://indexagentica.com/api/entries/termgrade.json"
      },
      {
        "id": "clasharness",
        "name": "CLASHarness",
        "summary": "Google Research benchmark and generator of 600 verifiable multi-agent conflict tasks in computer use: stale sub-agent reports, contradictory instructions, shared locks.",
        "url": "https://indexagentica.com/entries/clasharness/",
        "json": "https://indexagentica.com/api/entries/clasharness.json"
      }
    ],
    "criteria": [
      "Measures",
      "Tasks",
      "Verifier",
      "How to run",
      "License",
      "Extra releases"
    ],
    "rows": {
      "agenticdatabench": {
        "Measures": "LLM data agents on realistic data-science workflows across 15 domains (including B2B fintech), with skill-level scores",
        "Tasks": "344 tasks over 97 real datasets covering 433 skills; 98 tasks withheld as a private test set (246 public in the open testbed)",
        "Verifier": "Per-task evaluation function against a gold file (for example compare_csv on named columns); leaderboard submissions by email",
        "How to run": "Clone AgenticDataBench/AgenticDataBench; datasets on Hugging Face shawnzzzh/AgenticDataBench; open testbed under testbed/; baselines use harnesses such as smolagents, DA-Agent, Claude Code and Codex",
        "License": "Code Apache-2.0; Hugging Face dataset card apache-2.0",
        "Extra releases": "Skill-cluster tooling and an LLM task generator in-repo; paper arXiv 2607.01647"
      },
      "termgrade": {
        "Measures": "Executable Linux terminal agents; each task carries a measured pass rate per model rather than a hand-chosen difficulty label",
        "Tasks": "1,004 environments (container + instruction + pytest verifier), kept only if the reference solution passes",
        "Verifier": "pytest via tests/test.sh inside the container; the image ships without tests so the agent cannot read them; suite is copied in at scoring time",
        "How to run": "Hugging Face ai-and/termgrade-environments; build base image dsbase-venv312:local, then each task Dockerfile; grade with Terminus-2 via Harbor (as used for the release)",
        "License": "Apache-2.0 (environments and trajectories cards)",
        "Extra releases": "36,144 graded trajectories (21,910 passes as SFT, 14,234 failures kept); gemma-4-31B-it LoRA/merged checkpoint; +3.1 on Terminal-Bench 2.1 in their RL report"
      },
      "clasharness": {
        "Measures": "Multi-agent conflict resolution in computer-use settings: stale sub-agent reports, contradictory instructions, shared locks, cross-modal mismatches",
        "Tasks": "600 evaluation episodes (10 rounds × 60 packages from 20 conflict seeds); difficulty bands 236 / 224 / 140",
        "Verifier": "Verifiable by construction: ground truth pre-committed, reference trajectory executed, hybrid code and semantic graders compiled from observable side effects",
        "How to run": "Python 3.11+; pip install -e '.[dev]' from google-research/CLASHarness; download tasks-v8 / splits-v8 from Hugging Face yuxianglai117/CLASHarness; evaluate with --split total",
        "License": "Code Apache-2.0; dataset CC-BY-4.0",
        "Extra releases": "Generation harness and conflict plugins; recorded trajectories for 15 frontier models; paper arXiv 2610.19128"
      }
    },
    "table": [
      {
        "entry": "agenticdatabench",
        "name": "AgenticDataBench",
        "url": "https://indexagentica.com/entries/agenticdatabench/",
        "values": {
          "Measures": "LLM data agents on realistic data-science workflows across 15 domains (including B2B fintech), with skill-level scores",
          "Tasks": "344 tasks over 97 real datasets covering 433 skills; 98 tasks withheld as a private test set (246 public in the open testbed)",
          "Verifier": "Per-task evaluation function against a gold file (for example compare_csv on named columns); leaderboard submissions by email",
          "How to run": "Clone AgenticDataBench/AgenticDataBench; datasets on Hugging Face shawnzzzh/AgenticDataBench; open testbed under testbed/; baselines use harnesses such as smolagents, DA-Agent, Claude Code and Codex",
          "License": "Code Apache-2.0; Hugging Face dataset card apache-2.0",
          "Extra releases": "Skill-cluster tooling and an LLM task generator in-repo; paper arXiv 2607.01647"
        }
      },
      {
        "entry": "termgrade",
        "name": "TermGrade",
        "url": "https://indexagentica.com/entries/termgrade/",
        "values": {
          "Measures": "Executable Linux terminal agents; each task carries a measured pass rate per model rather than a hand-chosen difficulty label",
          "Tasks": "1,004 environments (container + instruction + pytest verifier), kept only if the reference solution passes",
          "Verifier": "pytest via tests/test.sh inside the container; the image ships without tests so the agent cannot read them; suite is copied in at scoring time",
          "How to run": "Hugging Face ai-and/termgrade-environments; build base image dsbase-venv312:local, then each task Dockerfile; grade with Terminus-2 via Harbor (as used for the release)",
          "License": "Apache-2.0 (environments and trajectories cards)",
          "Extra releases": "36,144 graded trajectories (21,910 passes as SFT, 14,234 failures kept); gemma-4-31B-it LoRA/merged checkpoint; +3.1 on Terminal-Bench 2.1 in their RL report"
        }
      },
      {
        "entry": "clasharness",
        "name": "CLASHarness",
        "url": "https://indexagentica.com/entries/clasharness/",
        "values": {
          "Measures": "Multi-agent conflict resolution in computer-use settings: stale sub-agent reports, contradictory instructions, shared locks, cross-modal mismatches",
          "Tasks": "600 evaluation episodes (10 rounds × 60 packages from 20 conflict seeds); difficulty bands 236 / 224 / 140",
          "Verifier": "Verifiable by construction: ground truth pre-committed, reference trajectory executed, hybrid code and semantic graders compiled from observable side effects",
          "How to run": "Python 3.11+; pip install -e '.[dev]' from google-research/CLASHarness; download tasks-v8 / splits-v8 from Hugging Face yuxianglai117/CLASHarness; evaluate with --split total",
          "License": "Code Apache-2.0; dataset CC-BY-4.0",
          "Extra releases": "Generation harness and conflict plugins; recorded trajectories for 15 frontier models; paper arXiv 2610.19128"
        }
      }
    ],
    "verdict": "Use AgenticDataBench when you need skill-level scores on data-science agents, TermGrade when you need pytest-verified terminal tasks with full pass and fail trajectories, and CLASHarness when the failure mode you care about is multi-agent conflict under computer use. For SWE-bench, Terminal-Bench, GAIA and similar older suites, start from the agent-benchmarks comparison."
  },
  "entries_detail": [
    {
      "id": "agenticdatabench",
      "name": "AgenticDataBench",
      "summary": "Benchmark for data-science agents: 344 tasks over 97 real datasets in 15 domains, with 433 fine-grained skill labels for skill-level scoring.",
      "url": "https://indexagentica.com/entries/agenticdatabench/",
      "json": "https://indexagentica.com/api/entries/agenticdatabench.json"
    },
    {
      "id": "termgrade",
      "name": "TermGrade",
      "summary": "1,004 executable Linux terminal tasks with pytest verifiers and per-model pass rates, plus 36,144 graded agent trajectories, for training and evaluating agents.",
      "url": "https://indexagentica.com/entries/termgrade/",
      "json": "https://indexagentica.com/api/entries/termgrade.json"
    },
    {
      "id": "clasharness",
      "name": "CLASHarness",
      "summary": "Google Research benchmark and generator of 600 verifiable multi-agent conflict tasks in computer use: stale sub-agent reports, contradictory instructions, shared locks.",
      "url": "https://indexagentica.com/entries/clasharness/",
      "json": "https://indexagentica.com/api/entries/clasharness.json"
    }
  ],
  "related": [
    {
      "type": "comparison",
      "id": "agent-benchmarks",
      "title": "Agent benchmarks compared",
      "url": "https://indexagentica.com/compare/agent-benchmarks/",
      "json": "https://indexagentica.com/api/longform/compare/agent-benchmarks.json"
    },
    {
      "type": "guide",
      "id": "evaluate-an-ai-agent",
      "title": "Evaluate an AI agent with open-source eval and observability tools",
      "url": "https://indexagentica.com/guides/evaluate-an-ai-agent/",
      "json": "https://indexagentica.com/api/longform/guides/evaluate-an-ai-agent.json"
    }
  ],
  "sources": [
    {
      "title": "AgenticDataBench site",
      "url": "https://agenticdatabench.github.io/",
      "accessed": "2026-11-06"
    },
    {
      "title": "AgenticDataBench GitHub README",
      "url": "https://github.com/AgenticDataBench/AgenticDataBench",
      "accessed": "2026-11-06"
    },
    {
      "title": "AgenticDataBench paper (arXiv 2607.01647)",
      "url": "https://arxiv.org/abs/2607.01647",
      "accessed": "2026-11-06"
    },
    {
      "title": "AgenticDataBench on Hugging Face",
      "url": "https://huggingface.co/datasets/shawnzzzh/AgenticDataBench",
      "accessed": "2026-11-06"
    },
    {
      "title": "TermGrade announcement (ai&)",
      "url": "https://www.aiand.com/newsroom/termgrade",
      "accessed": "2026-11-06"
    },
    {
      "title": "termgrade-environments on Hugging Face",
      "url": "https://huggingface.co/datasets/ai-and/termgrade-environments",
      "accessed": "2026-11-06"
    },
    {
      "title": "termgrade-trajectories on Hugging Face",
      "url": "https://huggingface.co/datasets/ai-and/termgrade-trajectories",
      "accessed": "2026-11-06"
    },
    {
      "title": "CLASHarness GitHub README",
      "url": "https://github.com/google-research/CLASHarness",
      "accessed": "2026-11-06"
    },
    {
      "title": "CLASHarness paper (arXiv 2610.19128)",
      "url": "https://arxiv.org/abs/2610.19128",
      "accessed": "2026-11-06"
    },
    {
      "title": "CLASHarness on Hugging Face",
      "url": "https://huggingface.co/datasets/yuxianglai117/CLASHarness",
      "accessed": "2026-11-06"
    }
  ],
  "front_matter": {
    "id": "newer-agent-eval-sets",
    "type": "comparison",
    "title": "Newer agent eval sets for data, terminals and multi-agent conflict",
    "summary": "AgenticDataBench, TermGrade and CLASHarness compared: what each measures, task counts, how scoring works, how to run them, and licenses. Cross-links the broader agent-benchmarks comparison.",
    "description": "A focused comparison of three late-2026 evaluation resources that are not yet rows in agent-benchmarks: AgenticDataBench (344 data-science tasks over 97 datasets with 433 skill labels), TermGrade (1,004 pytest-verified Linux terminal environments plus 36,144 graded trajectories), and CLASHarness (600 verifiable multi-agent computer-use conflict episodes). Numbers and licenses are taken from each project's paper, site, GitHub README and Hugging Face cards, checked on 2026-11-06.",
    "author": "Agentica Author",
    "tags": [
      "benchmarks",
      "evaluation",
      "data-agents",
      "terminal-agents",
      "multi-agent",
      "computer-use"
    ],
    "entries": [
      "agenticdatabench",
      "termgrade",
      "clasharness"
    ],
    "related": [
      "agent-benchmarks",
      "evaluate-an-ai-agent"
    ],
    "subjects": [
      "agenticdatabench",
      "termgrade",
      "clasharness"
    ],
    "criteria": [
      "Measures",
      "Tasks",
      "Verifier",
      "How to run",
      "License",
      "Extra releases"
    ],
    "rows": {
      "agenticdatabench": {
        "Measures": "LLM data agents on realistic data-science workflows across 15 domains (including B2B fintech), with skill-level scores",
        "Tasks": "344 tasks over 97 real datasets covering 433 skills; 98 tasks withheld as a private test set (246 public in the open testbed)",
        "Verifier": "Per-task evaluation function against a gold file (for example compare_csv on named columns); leaderboard submissions by email",
        "How to run": "Clone AgenticDataBench/AgenticDataBench; datasets on Hugging Face shawnzzzh/AgenticDataBench; open testbed under testbed/; baselines use harnesses such as smolagents, DA-Agent, Claude Code and Codex",
        "License": "Code Apache-2.0; Hugging Face dataset card apache-2.0",
        "Extra releases": "Skill-cluster tooling and an LLM task generator in-repo; paper arXiv 2607.01647"
      },
      "termgrade": {
        "Measures": "Executable Linux terminal agents; each task carries a measured pass rate per model rather than a hand-chosen difficulty label",
        "Tasks": "1,004 environments (container + instruction + pytest verifier), kept only if the reference solution passes",
        "Verifier": "pytest via tests/test.sh inside the container; the image ships without tests so the agent cannot read them; suite is copied in at scoring time",
        "How to run": "Hugging Face ai-and/termgrade-environments; build base image dsbase-venv312:local, then each task Dockerfile; grade with Terminus-2 via Harbor (as used for the release)",
        "License": "Apache-2.0 (environments and trajectories cards)",
        "Extra releases": "36,144 graded trajectories (21,910 passes as SFT, 14,234 failures kept); gemma-4-31B-it LoRA/merged checkpoint; +3.1 on Terminal-Bench 2.1 in their RL report"
      },
      "clasharness": {
        "Measures": "Multi-agent conflict resolution in computer-use settings: stale sub-agent reports, contradictory instructions, shared locks, cross-modal mismatches",
        "Tasks": "600 evaluation episodes (10 rounds × 60 packages from 20 conflict seeds); difficulty bands 236 / 224 / 140",
        "Verifier": "Verifiable by construction: ground truth pre-committed, reference trajectory executed, hybrid code and semantic graders compiled from observable side effects",
        "How to run": "Python 3.11+; pip install -e '.[dev]' from google-research/CLASHarness; download tasks-v8 / splits-v8 from Hugging Face yuxianglai117/CLASHarness; evaluate with --split total",
        "License": "Code Apache-2.0; dataset CC-BY-4.0",
        "Extra releases": "Generation harness and conflict plugins; recorded trajectories for 15 frontier models; paper arXiv 2610.19128"
      }
    },
    "verdict": "Use AgenticDataBench when you need skill-level scores on data-science agents, TermGrade when you need pytest-verified terminal tasks with full pass and fail trajectories, and CLASHarness when the failure mode you care about is multi-agent conflict under computer use. For SWE-bench, Terminal-Bench, GAIA and similar older suites, start from the agent-benchmarks comparison.",
    "sources": [
      {
        "title": "AgenticDataBench site",
        "url": "https://agenticdatabench.github.io/",
        "accessed": "2026-11-06"
      },
      {
        "title": "AgenticDataBench GitHub README",
        "url": "https://github.com/AgenticDataBench/AgenticDataBench",
        "accessed": "2026-11-06"
      },
      {
        "title": "AgenticDataBench paper (arXiv 2607.01647)",
        "url": "https://arxiv.org/abs/2607.01647",
        "accessed": "2026-11-06"
      },
      {
        "title": "AgenticDataBench on Hugging Face",
        "url": "https://huggingface.co/datasets/shawnzzzh/AgenticDataBench",
        "accessed": "2026-11-06"
      },
      {
        "title": "TermGrade announcement (ai&)",
        "url": "https://www.aiand.com/newsroom/termgrade",
        "accessed": "2026-11-06"
      },
      {
        "title": "termgrade-environments on Hugging Face",
        "url": "https://huggingface.co/datasets/ai-and/termgrade-environments",
        "accessed": "2026-11-06"
      },
      {
        "title": "termgrade-trajectories on Hugging Face",
        "url": "https://huggingface.co/datasets/ai-and/termgrade-trajectories",
        "accessed": "2026-11-06"
      },
      {
        "title": "CLASHarness GitHub README",
        "url": "https://github.com/google-research/CLASHarness",
        "accessed": "2026-11-06"
      },
      {
        "title": "CLASHarness paper (arXiv 2610.19128)",
        "url": "https://arxiv.org/abs/2610.19128",
        "accessed": "2026-11-06"
      },
      {
        "title": "CLASHarness on Hugging Face",
        "url": "https://huggingface.co/datasets/yuxianglai117/CLASHarness",
        "accessed": "2026-11-06"
      }
    ],
    "last_verified": "2026-11-06",
    "published": "2026-11-06",
    "updated": "2026-11-06"
  },
  "markdown": "\n## Why these three\n\nThe [agent benchmarks comparison](https://indexagentica.com/compare/agent-benchmarks/) covers established suites such as SWE-bench and Terminal-Bench. These three arrived later and fill gaps that comparison does not yet list:\n\n- **Data-science agents** that have to explore messy tables, not only write code against a fixed repo\n- **Terminal agents** graded with real pytest verifiers and released with every failing trajectory\n- **Multi-agent computer-use conflict**, where the hard part is fixing shared state, not only spotting a contradiction\n\nThey are different jobs. The table above is for picking which corpus to run, not for ranking models across them.\n\n## AgenticDataBench\n\nPaper arXiv 2607.01647 (Tsinghua University and Ant Digital Technologies). The abstract and body state **97 real datasets** across **15 domains**, **344** tasks covering **433** skills. The public GitHub README withholds **98** tasks for leaderboard integrity; the open `testbed/tasks/dev.jsonl` we fetched held **246** lines (344 − 98). Site metadata also says \"344 tasks across 15 domains\".\n\nEach public task carries a question, data sources, skill labels, a gold file and an `eval_func` (for example `compare_csv(...)`). Scoring is therefore programmatic against gold, not an LLM judge. Leaderboard results are emailed to the address on the project site. Code is Apache-2.0; the Hugging Face dataset card is `apache-2.0`.\n\nWe did not re-run agents. The paper reports baselines on smolagents, DA-Agent, Claude Code and Codex with several frontier models.\n\n## TermGrade\n\nReleased by ai& on 2026-10-08. **1,004** environments, each a container, an instruction and a **pytest** verifier, kept only when the reference solution passes. Six models were graded with the Terminus-2 scaffold via Harbor, producing **36,144** trajectories: **21,910** passes (usable as SFT) and **14,234** failures kept in full with per-test verdicts. Both Hugging Face cards are Apache-2.0.\n\nTheir RL write-up reports **+3.1** on Terminal-Bench 2.1 for a gemma-4-31B-it run trained on mid-pass-rate tasks. That figure is theirs; we did not reproduce it. The environments README shows an offline loop: build the base image, build a task image, run `solution/solve.sh`, copy tests in, run `tests/test.sh`, read `/logs/verifier/reward.txt`.\n\n## CLASHarness\n\nGoogle Research, paper arXiv 2610.19128. **600** multi-round episodes in stateful computer-use environments with mock sub-agents, approvals, locks and background daemons. Tasks are **verifiable by construction**: ground truth is pre-committed, a reference trajectory is executed, and hybrid code/semantic graders are compiled from side effects. Difficulty bands across the 600 episodes are **236 / 224 / 140**. Code is Apache-2.0; the Hugging Face dataset is CC-BY-4.0.\n\nThe README's reported panel of 15 frontier models finds that fixing state is harder than spotting conflicts, with high failure rates on closing actions such as releasing locks (they cite **58%–72%** failure on one closing-handover category). Treat those model scores as the authors' panel, not a re-run.\n\nQuick start: Python 3.11+, `pip install -e \".[dev]\"`, download `tasks-v8` / `splits-v8` from Hugging Face, evaluate with `--split total`.\n\n## How this fits the rest of the site\n\n- Broader suite landscape: [Agent benchmarks](https://indexagentica.com/compare/agent-benchmarks/)\n- Building an eval harness in general: [Evaluate an AI agent](https://indexagentica.com/guides/evaluate-an-ai-agent/)\n\n## What we verified\n\nChecked on 2026-11-06 against each project's primary sources listed above, including the AgenticDataBench PDF, the public `dev.jsonl` task count, TermGrade and CLASHarness READMEs and Hugging Face cards. We did not download full datasets end-to-end or run any agent. An \"AgenticDataBench v1.1\" label appears in some secondary write-ups; we did not find a clear v1.1 tag in the paper or README, so this page does not claim a version string.\n",
  "raw": "---\nid: newer-agent-eval-sets\ntype: comparison\ntitle: Newer agent eval sets for data, terminals and multi-agent conflict\nsummary: \"AgenticDataBench, TermGrade and CLASHarness compared: what each measures, task counts, how scoring works, how to run them, and licenses. Cross-links the broader agent-benchmarks comparison.\"\ndescription: \"A focused comparison of three late-2026 evaluation resources that are not yet rows in agent-benchmarks: AgenticDataBench (344 data-science tasks over 97 datasets with 433 skill labels), TermGrade (1,004 pytest-verified Linux terminal environments plus 36,144 graded trajectories), and CLASHarness (600 verifiable multi-agent computer-use conflict episodes). Numbers and licenses are taken from each project's paper, site, GitHub README and Hugging Face cards, checked on 2026-11-06.\"\nauthor: Agentica Author\ntags: [benchmarks, evaluation, data-agents, terminal-agents, multi-agent, computer-use]\nentries: [agenticdatabench, termgrade, clasharness]\nrelated: [agent-benchmarks, evaluate-an-ai-agent]\nsubjects: [agenticdatabench, termgrade, clasharness]\ncriteria:\n  - Measures\n  - Tasks\n  - Verifier\n  - How to run\n  - License\n  - Extra releases\nrows:\n  agenticdatabench:\n    Measures: \"LLM data agents on realistic data-science workflows across 15 domains (including B2B fintech), with skill-level scores\"\n    Tasks: \"344 tasks over 97 real datasets covering 433 skills; 98 tasks withheld as a private test set (246 public in the open testbed)\"\n    Verifier: \"Per-task evaluation function against a gold file (for example compare_csv on named columns); leaderboard submissions by email\"\n    How to run: \"Clone AgenticDataBench/AgenticDataBench; datasets on Hugging Face shawnzzzh/AgenticDataBench; open testbed under testbed/; baselines use harnesses such as smolagents, DA-Agent, Claude Code and Codex\"\n    License: \"Code Apache-2.0; Hugging Face dataset card apache-2.0\"\n    Extra releases: \"Skill-cluster tooling and an LLM task generator in-repo; paper arXiv 2607.01647\"\n  termgrade:\n    Measures: \"Executable Linux terminal agents; each task carries a measured pass rate per model rather than a hand-chosen difficulty label\"\n    Tasks: \"1,004 environments (container + instruction + pytest verifier), kept only if the reference solution passes\"\n    Verifier: \"pytest via tests/test.sh inside the container; the image ships without tests so the agent cannot read them; suite is copied in at scoring time\"\n    How to run: \"Hugging Face ai-and/termgrade-environments; build base image dsbase-venv312:local, then each task Dockerfile; grade with Terminus-2 via Harbor (as used for the release)\"\n    License: \"Apache-2.0 (environments and trajectories cards)\"\n    Extra releases: \"36,144 graded trajectories (21,910 passes as SFT, 14,234 failures kept); gemma-4-31B-it LoRA/merged checkpoint; +3.1 on Terminal-Bench 2.1 in their RL report\"\n  clasharness:\n    Measures: \"Multi-agent conflict resolution in computer-use settings: stale sub-agent reports, contradictory instructions, shared locks, cross-modal mismatches\"\n    Tasks: \"600 evaluation episodes (10 rounds × 60 packages from 20 conflict seeds); difficulty bands 236 / 224 / 140\"\n    Verifier: \"Verifiable by construction: ground truth pre-committed, reference trajectory executed, hybrid code and semantic graders compiled from observable side effects\"\n    How to run: \"Python 3.11+; pip install -e '.[dev]' from google-research/CLASHarness; download tasks-v8 / splits-v8 from Hugging Face yuxianglai117/CLASHarness; evaluate with --split total\"\n    License: \"Code Apache-2.0; dataset CC-BY-4.0\"\n    Extra releases: \"Generation harness and conflict plugins; recorded trajectories for 15 frontier models; paper arXiv 2610.19128\"\nverdict: \"Use AgenticDataBench when you need skill-level scores on data-science agents, TermGrade when you need pytest-verified terminal tasks with full pass and fail trajectories, and CLASHarness when the failure mode you care about is multi-agent conflict under computer use. For SWE-bench, Terminal-Bench, GAIA and similar older suites, start from the agent-benchmarks comparison.\"\nsources:\n  - title: AgenticDataBench site\n    url: https://agenticdatabench.github.io/\n    accessed: 2026-11-06\n  - title: AgenticDataBench GitHub README\n    url: https://github.com/AgenticDataBench/AgenticDataBench\n    accessed: 2026-11-06\n  - title: \"AgenticDataBench paper (arXiv 2607.01647)\"\n    url: https://arxiv.org/abs/2607.01647\n    accessed: 2026-11-06\n  - title: AgenticDataBench on Hugging Face\n    url: https://huggingface.co/datasets/shawnzzzh/AgenticDataBench\n    accessed: 2026-11-06\n  - title: \"TermGrade announcement (ai&)\"\n    url: https://www.aiand.com/newsroom/termgrade\n    accessed: 2026-11-06\n  - title: termgrade-environments on Hugging Face\n    url: https://huggingface.co/datasets/ai-and/termgrade-environments\n    accessed: 2026-11-06\n  - title: termgrade-trajectories on Hugging Face\n    url: https://huggingface.co/datasets/ai-and/termgrade-trajectories\n    accessed: 2026-11-06\n  - title: CLASHarness GitHub README\n    url: https://github.com/google-research/CLASHarness\n    accessed: 2026-11-06\n  - title: \"CLASHarness paper (arXiv 2610.19128)\"\n    url: https://arxiv.org/abs/2610.19128\n    accessed: 2026-11-06\n  - title: CLASHarness on Hugging Face\n    url: https://huggingface.co/datasets/yuxianglai117/CLASHarness\n    accessed: 2026-11-06\nlast_verified: 2026-11-06\npublished: 2026-11-06\nupdated: 2026-11-06\n---\n\n## Why these three\n\nThe [agent benchmarks comparison](https://indexagentica.com/compare/agent-benchmarks/) covers established suites such as SWE-bench and Terminal-Bench. These three arrived later and fill gaps that comparison does not yet list:\n\n- **Data-science agents** that have to explore messy tables, not only write code against a fixed repo\n- **Terminal agents** graded with real pytest verifiers and released with every failing trajectory\n- **Multi-agent computer-use conflict**, where the hard part is fixing shared state, not only spotting a contradiction\n\nThey are different jobs. The table above is for picking which corpus to run, not for ranking models across them.\n\n## AgenticDataBench\n\nPaper arXiv 2607.01647 (Tsinghua University and Ant Digital Technologies). The abstract and body state **97 real datasets** across **15 domains**, **344** tasks covering **433** skills. The public GitHub README withholds **98** tasks for leaderboard integrity; the open `testbed/tasks/dev.jsonl` we fetched held **246** lines (344 − 98). Site metadata also says \"344 tasks across 15 domains\".\n\nEach public task carries a question, data sources, skill labels, a gold file and an `eval_func` (for example `compare_csv(...)`). Scoring is therefore programmatic against gold, not an LLM judge. Leaderboard results are emailed to the address on the project site. Code is Apache-2.0; the Hugging Face dataset card is `apache-2.0`.\n\nWe did not re-run agents. The paper reports baselines on smolagents, DA-Agent, Claude Code and Codex with several frontier models.\n\n## TermGrade\n\nReleased by ai& on 2026-10-08. **1,004** environments, each a container, an instruction and a **pytest** verifier, kept only when the reference solution passes. Six models were graded with the Terminus-2 scaffold via Harbor, producing **36,144** trajectories: **21,910** passes (usable as SFT) and **14,234** failures kept in full with per-test verdicts. Both Hugging Face cards are Apache-2.0.\n\nTheir RL write-up reports **+3.1** on Terminal-Bench 2.1 for a gemma-4-31B-it run trained on mid-pass-rate tasks. That figure is theirs; we did not reproduce it. The environments README shows an offline loop: build the base image, build a task image, run `solution/solve.sh`, copy tests in, run `tests/test.sh`, read `/logs/verifier/reward.txt`.\n\n## CLASHarness\n\nGoogle Research, paper arXiv 2610.19128. **600** multi-round episodes in stateful computer-use environments with mock sub-agents, approvals, locks and background daemons. Tasks are **verifiable by construction**: ground truth is pre-committed, a reference trajectory is executed, and hybrid code/semantic graders are compiled from side effects. Difficulty bands across the 600 episodes are **236 / 224 / 140**. Code is Apache-2.0; the Hugging Face dataset is CC-BY-4.0.\n\nThe README's reported panel of 15 frontier models finds that fixing state is harder than spotting conflicts, with high failure rates on closing actions such as releasing locks (they cite **58%–72%** failure on one closing-handover category). Treat those model scores as the authors' panel, not a re-run.\n\nQuick start: Python 3.11+, `pip install -e \".[dev]\"`, download `tasks-v8` / `splits-v8` from Hugging Face, evaluate with `--split total`.\n\n## How this fits the rest of the site\n\n- Broader suite landscape: [Agent benchmarks](https://indexagentica.com/compare/agent-benchmarks/)\n- Building an eval harness in general: [Evaluate an AI agent](https://indexagentica.com/guides/evaluate-an-ai-agent/)\n\n## What we verified\n\nChecked on 2026-11-06 against each project's primary sources listed above, including the AgenticDataBench PDF, the public `dev.jsonl` task count, TermGrade and CLASHarness READMEs and Hugging Face cards. We did not download full datasets end-to-end or run any agent. An \"AgenticDataBench v1.1\" label appears in some secondary write-ups; we did not find a clear v1.1 tag in the paper or README, so this page does not claim a version string.\n",
  "html": "<h2>Why these three</h2>\n<p>The <a href=\"/compare/agent-benchmarks/\">agent benchmarks comparison</a> covers established suites such as SWE-bench and Terminal-Bench. These three arrived later and fill gaps that comparison does not yet list:</p>\n<ul><li><strong>Data-science agents</strong> that have to explore messy tables, not only write code against a fixed repo</li><li><strong>Terminal agents</strong> graded with real pytest verifiers and released with every failing trajectory</li><li><strong>Multi-agent computer-use conflict</strong>, where the hard part is fixing shared state, not only spotting a contradiction</li></ul>\n<p>They are different jobs. The table above is for picking which corpus to run, not for ranking models across them.</p>\n<h2>AgenticDataBench</h2>\n<p>Paper arXiv 2607.01647 (Tsinghua University and Ant Digital Technologies). The abstract and body state <strong>97 real datasets</strong> across <strong>15 domains</strong>, <strong>344</strong> tasks covering <strong>433</strong> skills. The public GitHub README withholds <strong>98</strong> tasks for leaderboard integrity; the open <code>testbed/tasks/dev.jsonl</code> we fetched held <strong>246</strong> lines (344 − 98). Site metadata also says &quot;344 tasks across 15 domains&quot;.</p>\n<p>Each public task carries a question, data sources, skill labels, a gold file and an <code>eval_func</code> (for example <code>compare_csv(...)</code>). Scoring is therefore programmatic against gold, not an LLM judge. Leaderboard results are emailed to the address on the project site. Code is Apache-2.0; the Hugging Face dataset card is <code>apache-2.0</code>.</p>\n<p>We did not re-run agents. The paper reports baselines on smolagents, DA-Agent, Claude Code and Codex with several frontier models.</p>\n<h2>TermGrade</h2>\n<p>Released by ai&amp; on 2026-10-08. <strong>1,004</strong> environments, each a container, an instruction and a <strong>pytest</strong> verifier, kept only when the reference solution passes. Six models were graded with the Terminus-2 scaffold via Harbor, producing <strong>36,144</strong> trajectories: <strong>21,910</strong> passes (usable as SFT) and <strong>14,234</strong> failures kept in full with per-test verdicts. Both Hugging Face cards are Apache-2.0.</p>\n<p>Their RL write-up reports <strong>+3.1</strong> on Terminal-Bench 2.1 for a gemma-4-31B-it run trained on mid-pass-rate tasks. That figure is theirs; we did not reproduce it. The environments README shows an offline loop: build the base image, build a task image, run <code>solution/solve.sh</code>, copy tests in, run <code>tests/test.sh</code>, read <code>/logs/verifier/reward.txt</code>.</p>\n<h2>CLASHarness</h2>\n<p>Google Research, paper arXiv 2610.19128. <strong>600</strong> multi-round episodes in stateful computer-use environments with mock sub-agents, approvals, locks and background daemons. Tasks are <strong>verifiable by construction</strong>: ground truth is pre-committed, a reference trajectory is executed, and hybrid code/semantic graders are compiled from side effects. Difficulty bands across the 600 episodes are <strong>236 / 224 / 140</strong>. Code is Apache-2.0; the Hugging Face dataset is CC-BY-4.0.</p>\n<p>The README&#39;s reported panel of 15 frontier models finds that fixing state is harder than spotting conflicts, with high failure rates on closing actions such as releasing locks (they cite <strong>58%–72%</strong> failure on one closing-handover category). Treat those model scores as the authors&#39; panel, not a re-run.</p>\n<p>Quick start: Python 3.11+, <code>pip install -e &quot;.[dev]&quot;</code>, download <code>tasks-v8</code> / <code>splits-v8</code> from Hugging Face, evaluate with <code>--split total</code>.</p>\n<h2>How this fits the rest of the site</h2>\n<ul><li>Broader suite landscape: <a href=\"/compare/agent-benchmarks/\">Agent benchmarks</a></li><li>Building an eval harness in general: <a href=\"/guides/evaluate-an-ai-agent/\">Evaluate an AI agent</a></li></ul>\n<h2>What we verified</h2>\n<p>Checked on 2026-11-06 against each project&#39;s primary sources listed above, including the AgenticDataBench PDF, the public <code>dev.jsonl</code> task count, TermGrade and CLASHarness READMEs and Hugging Face cards. We did not download full datasets end-to-end or run any agent. An &quot;AgenticDataBench v1.1&quot; label appears in some secondary write-ups; we did not find a clear v1.1 tag in the paper or README, so this page does not claim a version string.</p>"
}
