MCPMark
Stress-testing benchmark for agentic models using real MCP environments (Notion, GitHub, Filesystem, Postgres, Playwright), with strict automated verifiers and sandboxed tasks.
Tags
Description
One-command tasks, isolated sandboxes, auto-resume and aggregated reports. Since June 2026 the default task set is MCPMark Verified (version-pinned environments, stabilized verifiers); earlier results are not directly comparable. Trajectory logs are published on Hugging Face. Paper: arXiv 2509.24002.
Related
- MCP-Universe: Salesforce AI Research benchmark and framework that evaluates LLMs and agents on real-world tasks against live MCP servers, with a public leaderboard.
- Berkeley Function Calling Leaderboard (BFCL): UC Berkeley leaderboard evaluating how accurately LLMs call functions and tools; part of the Gorilla project.
- Terminal-Bench: Continuously maintained benchmark and leaderboard (Stanford, Laude Institute) measuring how well AI agents complete hard, end-to-end tasks in real terminal environments.
- WebArena: A realistic web environment for building autonomous agents, with benchmark tasks used to evaluate web agents.
Guides & comparisons
- Agent benchmarks compared (Comparison): Ten widely cited agent benchmarks (SWE-bench, Terminal-Bench, tau-bench, BFCL, MCPMark, GAIA, BrowseComp, WebArena, Mind2Web and OSWorld) compared on what they measure, how many tasks they have, where the agent runs, how runs are scored, where results are published and how they are licensed.
Sources
- https://github.com/eval-sys/mcpmark
- https://mcpmark.ai
- https://arxiv.org/abs/2509.24002
- https://huggingface.co/datasets/Jakumetsu/mcpmark-trajectory-log
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)