Information (60)
Knowledge sources, documentation hubs, datasets, benchmarks and research.
Machine-readable: /api/information.json.
- AgentBench
Comprehensive benchmark evaluating LLMs as agents (ICLR'24); the current repo is a function-calling version (AgentBench FC) with containerized tasks.
- AgentDojo
ETH Zurich benchmark environment for prompt-injection attacks and defenses on tool-using LLM agents, with suites like workspace, banking, travel and slack.
- AgenticDataBench
Benchmark for data-science agents: 344 tasks over 97 real datasets in 15 domains, with 433 fine-grained skill labels for skill-level scoring.
- Aider LLM Leaderboards
Quantitative benchmarks of LLM code-editing skill maintained by the Aider project.
- AndroidWorld
Google Research environment and benchmark for autonomous Android control agents: 116 hand-crafted tasks across 20 apps on a live emulator.
- APIGen Function-Calling Dataset (xlam-function-calling-60k)
Salesforce's 60,000-example function-calling dataset generated by APIGen, each example checked for format, by real execution and semantically.
- APIGen-MT-5k
Salesforce dataset of 5,000 verified multi-turn agent trajectories (retail and airline, tau-bench style) for training and evaluating tool-calling agents.
- AppWorld
Benchmark and sandbox of 9 simulated day-to-day apps operable via 457 APIs and populated by ~100 simulated people, for evaluating function-calling and interactive coding agents.
- Arena (formerly LMArena)
Public, crowd-voted leaderboard comparing AI models on text, image and code through real-world head-to-head evaluation.
- Artificial Analysis
Independent benchmarks comparing AI models and API providers on quality, price, output speed and latency.
- arXiv cs.AI (Artificial Intelligence)
arXiv's listing of the latest preprints in Artificial Intelligence, a primary source for new agent research.
- Berkeley Function Calling Leaderboard (BFCL)
UC Berkeley leaderboard evaluating how accurately LLMs call functions and tools; part of the Gorilla project.
- BigCodeBench (deprecated)
Code-generation benchmark with 1,140 software-engineering-oriented Python tasks that exercise diverse library calls, plus a harder task subset.
- BrowseComp
OpenAI's benchmark of 1,266 hard-to-find, short-answer questions that test how persistently and creatively browsing agents search the web.
- BrowserGym
ServiceNow's Gym environment for web agents that bundles MiniWoB, WebArena, VisualWebArena, WorkArena, AssistantBench and more behind one API.
- BrowseSafe++
Perplexity benchmark and SDK for detecting indirect prompt injection in web pages: 38,675 attack/control page pairs in 80 languages, with a leaderboard and adaptive attacks.
- Building Effective AI Agents (Anthropic)
Anthropic engineering essay (Dec 19, 2024) on agent design patterns: workflows vs. agents, and when to use simple composable patterns.
- CLASHarness
Google Research benchmark and generator of 600 verifiable multi-agent conflict tasks in computer use: stale sub-agent reports, contradictory instructions, shared locks.
- Common Crawl
Open repository of web crawl data that anyone can access and analyze.
- DeepResearch Bench II
Apache-2.0 evaluation pipeline and 132-task deep-research benchmark with 9,430 expert-derived rubrics and mixed data licenses.
- Epoch AI Benchmarking Hub
Epoch AI's hub of benchmark results for leading AI models, with trends over time by benchmark and by model.
- GAIA
Benchmark for general AI assistants ("Benchmarking General AI Agents"), published as a Hugging Face dataset under the gaia-benchmark org.
- Gaia2
CC-BY-4.0 follow-up to GAIA: 800 dynamic, time-aware agent scenarios released on Hugging Face and run with Meta’s ARE framework.
- GDPval
OpenAI benchmark of real-world, economically valuable knowledge-work tasks across 44 occupations; a 220-task gold subset with files and rubrics is open on Hugging Face.
- GPQA
Graduate-level 'Google-proof' multiple-choice Q&A benchmark; code is MIT, while the Hugging Face dataset is CC-BY-4.0 and gated.
- Hugging Face Daily Papers
Daily curated feed of new AI research papers on Hugging Face.
- Hugging Face Datasets Hub
Hugging Face hub of ready-to-use datasets for AI models, plus the open-source `datasets` library for loading and manipulating them.
- Humanity's Last Exam
Multi-modal benchmark of 2,500 expert-written questions at the frontier of human knowledge, from the Center for AI Safety and Scale AI.
- InjecAgent
Benchmark of 1,054 test cases measuring how tool-using LLM agents fall for indirect prompt injections in tool outputs.
- LiveCodeBench
Holistic, contamination-free evaluation of LLMs for code, with a public leaderboard.
- MCP-Bench
Accenture's benchmark for tool-using agents on complex real-world tasks across 28 live MCP servers and 250 tools (finance, travel, science, academic search).
- MCP-Universe
Salesforce AI Research benchmark and framework that evaluates LLMs and agents on real-world tasks against live MCP servers, with a public leaderboard.
- MCPMark
Stress-testing benchmark for agentic models using real MCP environments (Notion, GitHub, Filesystem, Postgres, Playwright), with strict automated verifiers and sandboxed tasks.
- METR Task-Completion Time Horizons
METR's up-to-date measurements of how long tasks frontier AI models can complete autonomously (time horizons).
- Mind2Web
Dataset and benchmark for generalist web agents: 2,000+ open-ended tasks on 137 real websites across 31 domains, with crowdsourced action sequences (NeurIPS 2023).
- MLE-bench
OpenAI benchmark measuring how well AI agents perform machine learning engineering tasks.
- Models.dev
Open-source database of AI model specifications, pricing and features, also served as JSON.
- Multi-SWE-bench
ByteDance Seed's multilingual issue-resolving benchmark: 1,632 verified tasks across Java, TS, JS, Go, Rust, C and C++.
- Open-SWE-Traces
NVIDIA dataset of software-engineering agent trajectories (OpenHands, SWE-agent, mini-swe-agent, Claude Code, OpenCode) for distilling SWE agents; CC BY 4.0.
- OpenRouter LLM Rankings
LLM leaderboard by real-world usage, ranked by tokens processed through the OpenRouter API.
- OSWorld
NeurIPS 2024 benchmark of multimodal agents on open-ended tasks in real computer environments.
- SimpleQA
OpenAI factuality short-form QA benchmark, with a reference implementation in the MIT simple-evals repository.
- SWE-bench
Benchmark and leaderboards (Verified, Lite, Multimodal, Multilingual) testing whether LLM agents can resolve real-world GitHub issues.
- SWE-Bench Pro
Scale AI benchmark of long-horizon software-engineering tasks for AI agents, with an open dataset on Hugging Face and a public leaderboard.
- SWE-Gym
Environment and dataset for training software-engineering agents and verifiers on real GitHub tasks (ICML 2025).
- SWE-rebench
Nebius's continuously refreshed, decontaminated SWE agent benchmark and dataset: 21,000+ validated GitHub issue-PR tasks plus a time-windowed leaderboard.
- SWE-smith
Toolkit and datasets for scaling training data for software-engineering agents by synthesizing task instances from GitHub repos (NeurIPS 2025 D&B).
- TermGrade
1,004 executable Linux terminal tasks with pytest verifiers and per-model pass rates, plus 36,144 graded agent trajectories, for training and evaluating agents.
- Terminal-Bench
Continuously maintained benchmark and leaderboard (Stanford, Laude Institute) measuring how well AI agents complete hard, end-to-end tasks in real terminal environments.
- TheAgentCompany
Benchmark of 175 professional tasks in a simulated software company (self-hosted GitLab, Plane, ownCloud, RocketChat) for agents that browse, code, run programs and talk to coworkers.
- Toolathlon (Tool Decathlon)
Benchmark of 108 long-horizon agent tasks across 32 real applications and 604 tools (mostly MCP servers), each checked by a deterministic evaluation script.
- ToolBench (ToolLLM)
Large tool-use instruction dataset over 16,464 real RapidAPI REST APIs, with the ToolEval evaluator and ToolLLaMA models.
- VisualWebArena
MIT benchmark of 910 multimodal visual web tasks for agents, extending WebArena with visually grounded sites and evaluation scripts.
- WebArena
A realistic web environment for building autonomous agents, with benchmark tasks used to evaluate web agents.
- WebChoreArena
Apache-2.0 benchmark of 532 tedious web-chore tasks built on the WebArena sites, for evaluating browsing agents.
- WebShop
Princeton simulated e-commerce environment and benchmark for grounded language agents shopping from text instructions.
- WebVoyager
Multimodal end-to-end web agent research code and 643-task dataset across 15 real websites, with GPT-4V auto-eval protocol.
- Windows Agent Arena
Microsoft's scalable OS benchmark for multimodal agents that perform tasks inside a real Windows environment, parallelizable on Azure.
- WorkArena
ServiceNow browser-task benchmark for knowledge-work agents, evaluated through BrowserGym, with an Apache-2.0 Python package.
- τ-bench (tau2-bench)
Sierra's benchmark for tool-agent-user interaction (airline, retail, telecom), now in six languages and full-duplex voice with v2.0.