# ThinkingBox

> Microsoft's open-source sandbox and benchmark that runs agents against isolated MCP tool servers and grades the backend state and side effects they leave.

- id: `thinkingbox`
- category: Evals & Observability (`evals-observability`)
- url: https://github.com/microsoft/thinkingbox
- repo: https://github.com/microsoft/thinkingbox
- docs: https://huggingface.co/blog/microsoft/thinkingbox
- status: active
- pricing: open-source
- license: MIT
- tags: benchmark, agent-evaluation, mcp, stateful, reliability, openenv
- related: τ-bench (tau2-bench) (https://indexagentica.com/entries/tau-bench/); AppWorld (https://indexagentica.com/entries/appworld/); Inspect (https://indexagentica.com/entries/inspect-ai/)
- sources: https://github.com/microsoft/thinkingbox , https://huggingface.co/blog/microsoft/thinkingbox , https://arxiv.org/abs/2608.19741 , https://github.com/openenv/OpenEnv/tree/main/envs/thinkingbox_env
- added: 2026-11-03
- updated: 2026-11-03
- last verified: 2026-11-03
- maintainer: Microsoft
- submitted by: agentica-curator
- page: https://indexagentica.com/entries/thinkingbox/
- json: https://indexagentica.com/api/entries/thinkingbox.json

ThinkingBox defines tools as MCP servers, runs an LLM agent (with a simulated user) through stateful scenarios and test cases, and checks terminal database state, side effects and response requirements instead of the agent's own claims. ThinkingBox-Bench has 507 stateful business workflows, each run 20 times to measure consistency (paper: "One Success Isn't Reliability", arXiv 2608.19741). An OpenEnv adapter (openenv/OpenEnv envs/thinkingbox_env) exposes the pinned public benchmark through the standard reset/step interface for evaluation. Also usable for training-data generation and RL loops. MIT.