Newer agent eval sets for data, terminals and multi-agent conflict
AgenticDataBench, TermGrade and CLASHarness compared: what each measures, task counts, how scoring works, how to run them, and licenses. Cross-links the broader agent-benchmarks comparison.
Comparison
| Entry | Measures | Tasks | Verifier | How to run | License | Extra releases |
|---|---|---|---|---|---|---|
| AgenticDataBench | LLM data agents on realistic data-science workflows across 15 domains (including B2B fintech), with skill-level scores | 344 tasks over 97 real datasets covering 433 skills; 98 tasks withheld as a private test set (246 public in the open testbed) | Per-task evaluation function against a gold file (for example compare_csv on named columns); leaderboard submissions by email | Clone AgenticDataBench/AgenticDataBench; datasets on Hugging Face shawnzzzh/AgenticDataBench; open testbed under testbed/; baselines use harnesses such as smolagents, DA-Agent, Claude Code and Codex | Code Apache-2.0; Hugging Face dataset card apache-2.0 | Skill-cluster tooling and an LLM task generator in-repo; paper arXiv 2607.01647 |
| TermGrade | Executable Linux terminal agents; each task carries a measured pass rate per model rather than a hand-chosen difficulty label | 1,004 environments (container + instruction + pytest verifier), kept only if the reference solution passes | pytest via tests/test.sh inside the container; the image ships without tests so the agent cannot read them; suite is copied in at scoring time | Hugging Face ai-and/termgrade-environments; build base image dsbase-venv312:local, then each task Dockerfile; grade with Terminus-2 via Harbor (as used for the release) | Apache-2.0 (environments and trajectories cards) | 36,144 graded trajectories (21,910 passes as SFT, 14,234 failures kept); gemma-4-31B-it LoRA/merged checkpoint; +3.1 on Terminal-Bench 2.1 in their RL report |
| CLASHarness | Multi-agent conflict resolution in computer-use settings: stale sub-agent reports, contradictory instructions, shared locks, cross-modal mismatches | 600 evaluation episodes (10 rounds × 60 packages from 20 conflict seeds); difficulty bands 236 / 224 / 140 | Verifiable by construction: ground truth pre-committed, reference trajectory executed, hybrid code and semantic graders compiled from observable side effects | Python 3.11+; pip install -e '.[dev]' from google-research/CLASHarness; download tasks-v8 / splits-v8 from Hugging Face yuxianglai117/CLASHarness; evaluate with --split total | Code Apache-2.0; dataset CC-BY-4.0 | Generation harness and conflict plugins; recorded trajectories for 15 frontier models; paper arXiv 2610.19128 |
Verdict: Use AgenticDataBench when you need skill-level scores on data-science agents, TermGrade when you need pytest-verified terminal tasks with full pass and fail trajectories, and CLASHarness when the failure mode you care about is multi-agent conflict under computer use. For SWE-bench, Terminal-Bench, GAIA and similar older suites, start from the agent-benchmarks comparison.
Why these three
The agent benchmarks comparison covers established suites such as SWE-bench and Terminal-Bench. These three arrived later and fill gaps that comparison does not yet list:
- Data-science agents that have to explore messy tables, not only write code against a fixed repo
- Terminal agents graded with real pytest verifiers and released with every failing trajectory
- Multi-agent computer-use conflict, where the hard part is fixing shared state, not only spotting a contradiction
They are different jobs. The table above is for picking which corpus to run, not for ranking models across them.
AgenticDataBench
Paper arXiv 2607.01647 (Tsinghua University and Ant Digital Technologies). The abstract and body state 97 real datasets across 15 domains, 344 tasks covering 433 skills. The public GitHub README withholds 98 tasks for leaderboard integrity; the open testbed/tasks/dev.jsonl we fetched held 246 lines (344 − 98). Site metadata also says "344 tasks across 15 domains".
Each public task carries a question, data sources, skill labels, a gold file and an eval_func (for example compare_csv(...)). Scoring is therefore programmatic against gold, not an LLM judge. Leaderboard results are emailed to the address on the project site. Code is Apache-2.0; the Hugging Face dataset card is apache-2.0.
We did not re-run agents. The paper reports baselines on smolagents, DA-Agent, Claude Code and Codex with several frontier models.
TermGrade
Released by ai& on 2026-10-08. 1,004 environments, each a container, an instruction and a pytest verifier, kept only when the reference solution passes. Six models were graded with the Terminus-2 scaffold via Harbor, producing 36,144 trajectories: 21,910 passes (usable as SFT) and 14,234 failures kept in full with per-test verdicts. Both Hugging Face cards are Apache-2.0.
Their RL write-up reports +3.1 on Terminal-Bench 2.1 for a gemma-4-31B-it run trained on mid-pass-rate tasks. That figure is theirs; we did not reproduce it. The environments README shows an offline loop: build the base image, build a task image, run solution/solve.sh, copy tests in, run tests/test.sh, read /logs/verifier/reward.txt.
CLASHarness
Google Research, paper arXiv 2610.19128. 600 multi-round episodes in stateful computer-use environments with mock sub-agents, approvals, locks and background daemons. Tasks are verifiable by construction: ground truth is pre-committed, a reference trajectory is executed, and hybrid code/semantic graders are compiled from side effects. Difficulty bands across the 600 episodes are 236 / 224 / 140. Code is Apache-2.0; the Hugging Face dataset is CC-BY-4.0.
The README's reported panel of 15 frontier models finds that fixing state is harder than spotting conflicts, with high failure rates on closing actions such as releasing locks (they cite 58%–72% failure on one closing-handover category). Treat those model scores as the authors' panel, not a re-run.
Quick start: Python 3.11+, pip install -e ".[dev]", download tasks-v8 / splits-v8 from Hugging Face, evaluate with --split total.
How this fits the rest of the site
- Broader suite landscape: Agent benchmarks
- Building an eval harness in general: Evaluate an AI agent
What we verified
Checked on 2026-11-06 against each project's primary sources listed above, including the AgenticDataBench PDF, the public dev.jsonl task count, TermGrade and CLASHarness READMEs and Hugging Face cards. We did not download full datasets end-to-end or run any agent. An "AgenticDataBench v1.1" label appears in some secondary write-ups; we did not find a clear v1.1 tag in the paper or README, so this page does not claim a version string.
Directory entries in this comparison
- AgenticDataBench: Benchmark for data-science agents: 344 tasks over 97 real datasets in 15 domains, with 433 fine-grained skill labels for skill-level scoring.
- TermGrade: 1,004 executable Linux terminal tasks with pytest verifiers and per-model pass rates, plus 36,144 graded agent trajectories, for training and evaluating agents.
- CLASHarness: Google Research benchmark and generator of 600 verifiable multi-agent conflict tasks in computer use: stale sub-agent reports, contradictory instructions, shared locks.
Related
- Agent benchmarks compared (Comparison)
- Evaluate an AI agent with open-source eval and observability tools (Guide)
Sources
- AgenticDataBench site, accessed
- AgenticDataBench GitHub README, accessed
- AgenticDataBench paper (arXiv 2607.01647), accessed
- AgenticDataBench on Hugging Face, accessed
- TermGrade announcement (ai&), accessed
- termgrade-environments on Hugging Face, accessed
- termgrade-trajectories on Hugging Face, accessed
- CLASHarness GitHub README, accessed
- CLASHarness paper (arXiv 2610.19128), accessed
- CLASHarness on Hugging Face, accessed