MCP-Universe
Salesforce AI Research benchmark and framework that evaluates LLMs and agents on real-world tasks against live MCP servers, with a public leaderboard.
Tags
Description
Domains include web search, location navigation, browser automation, financial analysis, repository management and 3D design, stressing long-horizon reasoning, large unfamiliar tool spaces and time-sensitive ground truth. The framework also runs the MCPMark tasks and ships agent tooling (MCP+ context wrapper, a deep-research agent). Paper: arXiv 2508.14704.
Related
- MCPMark: Stress-testing benchmark for agentic models using real MCP environments (Notion, GitHub, Filesystem, Postgres, Playwright), with strict automated verifiers and sandboxed tasks.
- Berkeley Function Calling Leaderboard (BFCL): UC Berkeley leaderboard evaluating how accurately LLMs call functions and tools; part of the Gorilla project.
- τ-bench (tau2-bench): Sierra's benchmark for tool-agent-user interaction in real-world domains.
- GAIA: Benchmark for general AI assistants ("Benchmarking General AI Agents"), published as a Hugging Face dataset under the gaia-benchmark org.
Guides & comparisons
- Agent benchmarks compared (Comparison): Ten widely cited agent benchmarks (SWE-bench, Terminal-Bench, tau-bench, BFCL, MCPMark, GAIA, BrowseComp, WebArena, Mind2Web and OSWorld) compared on what they measure, how many tasks they have, where the agent runs, how runs are scored, where results are published and how they are licensed.
Sources
- https://github.com/SalesforceAIResearch/MCP-Universe
- https://mcp-universe.github.io/
- https://arxiv.org/abs/2508.14704
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)