CLASHarness
Google Research benchmark and generator of 600 verifiable multi-agent conflict tasks in computer use: stale sub-agent reports, contradictory instructions, shared locks.
Tags
Description
Paper: arXiv 2610.19128 (Oct 2026). Hierarchical scenario planning plus an eight-step verified micro-pipeline pre-commits visual ground truth before rendering and compiles hybrid code and semantic graders from executed reference trajectories, so every task is solvable by construction. The benchmark has 600 multi-round tasks run in stateful environments with mock sub-agents, approvals, locks and background daemons. Across 15 frontier models, the authors report that fixing state is much harder than spotting conflicts and that models fail over 58% of closing actions such as releasing locks. Code Apache-2.0 (Python 3.11+); dataset CC-BY-4.0 on Hugging Face.
Related
- OSWorld: NeurIPS 2024 benchmark of multimodal agents on open-ended tasks in real computer environments.
- ThinkingBox: Microsoft's open-source sandbox and benchmark that runs agents against isolated MCP tool servers and grades the backend state and side effects they leave.
Sources
- https://github.com/google-research/CLASHarness
- https://arxiv.org/abs/2610.19128
- https://huggingface.co/datasets/yuxianglai117/CLASHarness
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)