Mind2Web
Dataset and benchmark for generalist web agents: 2,000+ open-ended tasks on 137 real websites across 31 domains, with crowdsourced action sequences (NeurIPS 2023).
Tags
Description
Training data is on Hugging Face (osunlp/Mind2Web); test splits (cross-task, cross-website, cross-domain) are distributed as a zip to limit contamination and should not be redistributed unzipped. Follow-ups from the same group include Multimodal-Mind2Web (HTML paired with screenshots) and Online-Mind2Web: 300 tasks on 136 live websites, judged by WebJudge, with a public leaderboard. Dataset CC-BY-4.0; code MIT.
Related
- WebArena: A realistic web environment for building autonomous agents, with benchmark tasks used to evaluate web agents.
- BrowseComp: OpenAI's benchmark of 1,266 hard-to-find, short-answer questions that test how persistently and creatively browsing agents search the web.
- OSWorld: NeurIPS 2024 benchmark of multimodal agents on open-ended tasks in real computer environments.
- Hugging Face Datasets Hub: Hugging Face hub of ready-to-use datasets for AI models, plus the open-source `datasets` library for loading and manipulating them.
Guides & comparisons
- Agent benchmarks compared (Comparison): Ten widely cited agent benchmarks (SWE-bench, Terminal-Bench, tau-bench, BFCL, MCPMark, GAIA, BrowseComp, WebArena, Mind2Web and OSWorld) compared on what they measure, how many tasks they have, where the agent runs, how runs are scored, where results are published and how they are licensed.
Sources
- https://github.com/OSU-NLP-Group/Mind2Web
- https://arxiv.org/abs/2306.06070
- https://github.com/OSU-NLP-Group/Online-Mind2Web
- https://huggingface.co/datasets/osunlp/Mind2Web
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)