TheAgentCompany
Benchmark of 175 professional tasks in a simulated software company (self-hosted GitLab, Plane, ownCloud, RocketChat) for agents that browse, code, run programs and talk to coworkers.
Tags
Description
Self-hosted with Docker (30+ GB disk); each task is a Docker image with an encrypted evaluator. Scoring is result-based (primary) plus subcheckpoints (secondary), using deterministic and LLM-based evaluators; tasks include communicating with coworkers (an npc component in each task image). Task roles: software engineer, product manager, data scientist, human resources, financial staff and administrator. Paper: arXiv 2412.14161; maintainers' contacts are at CMU. Public leaderboard on the website. Release 1.0.0 (December 2024).
Related
- WebArena: A realistic web environment for building autonomous agents, with benchmark tasks used to evaluate web agents.
- AppWorld: Benchmark and sandbox of 9 simulated day-to-day apps operable via 457 APIs and populated by ~100 simulated people, for evaluating function-calling and interactive coding agents.
- OSWorld: NeurIPS 2024 benchmark of multimodal agents on open-ended tasks in real computer environments.
- OpenHands: Open-source, model-agnostic platform for cloud coding agents.
Sources
- https://github.com/TheAgentCompany/TheAgentCompany
- https://arxiv.org/abs/2412.14161
- https://the-agent-company.com
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)