ThinkingBox
Microsoft's open-source sandbox and benchmark that runs agents against isolated MCP tool servers and grades the backend state and side effects they leave.
Tags
Description
ThinkingBox defines tools as MCP servers, runs an LLM agent (with a simulated user) through stateful scenarios and test cases, and checks terminal database state, side effects and response requirements instead of the agent's own claims. ThinkingBox-Bench has 507 stateful business workflows, each run 20 times to measure consistency (paper: "One Success Isn't Reliability", arXiv 2608.19741). An OpenEnv adapter (openenv/OpenEnv envs/thinkingbox_env) exposes the pinned public benchmark through the standard reset/step interface for evaluation. Also usable for training-data generation and RL loops. MIT.
Related
- τ-bench (tau2-bench): Sierra's benchmark for tool-agent-user interaction (airline, retail, telecom), now in six languages and full-duplex voice with v2.0.
- AppWorld: Benchmark and sandbox of 9 simulated day-to-day apps operable via 457 APIs and populated by ~100 simulated people, for evaluating function-calling and interactive coding agents.
- Inspect: Open-source framework for large language model and agent evaluations from the UK AI Security Institute.
Sources
- https://github.com/microsoft/thinkingbox
- https://huggingface.co/blog/microsoft/thinkingbox
- https://arxiv.org/abs/2608.19741
- https://github.com/openenv/OpenEnv/tree/main/envs/thinkingbox_env
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)