Agent environment frameworks compared: Harbor vs AgentEnv Framework vs Seahaven
Three open-source Python frameworks for building the environments agent evals and RL runs happen in: Harbor (container tasks), Scale AI's AgentEnv Framework and Kiln AI's Seahaven (synthetic SQLite worlds). Covers tasks, grading, compute, agent interface, interop and license.
Comparison
| Entry | Unit of work | Grading | Where runs execute | Agent interface | Interop | Install | License |
|---|---|---|---|---|---|---|---|
| Harbor | A task folder: instruction.md, task.toml, an environment (container definition) and tests; tasks group into versioned datasets such as terminal-bench@2.0 | The task's tests check the container after the agent finishes | Local Docker by default; --env switches to cloud sandboxes (Daytona, Modal, LangSmith, Blaxel, Novita, Tensorlake, Runta, Sail) for thousands of parallel runs | Runs real agent harnesses (Claude Code, OpenHands, Codex CLI and more) inside the environment, plus a model flag | Official harness for Terminal-Bench 2.0; harbor datasets list includes third-party benchmarks such as SWE-Bench and Aider Polyglot; generates rollouts for RL | uv tool install harbor (PyPI 0.25.0, Python 3.12+) | Apache-2.0 |
| AgentEnv Framework | An environment (containerized server speaking agentenv-framework-protocol) plus tasks that run against it, built into versioned images and shared through a registry | Tasks score what the agent did in the environment | Local Docker by default; other sandbox providers (including Modal and E2B, per the docs) and custom ones via plugins; optional AWS and GCP store backends | Agents are deployed into sandboxes and described with A2A Agent Cards; they reach the environment through a gateway | Every task in a bundle can be served as an OpenEnv env (openenv extra); plugins add env types, providers, task steps and stores | uv tool install agentenv-framework, then agent-env run hello (PyPI 0.9.1364, Python 3.11.4+) | Apache-2.0 |
| Seahaven | A world: SQLite tables plus Python tool functions, with frozen fixtures as starting states; worlds compose (for example Kiln's Stripe World) | On final state, not transcripts: every row the agent changed is logged for the grader | In-process: hundreds of world instances per process, each with its own SQLite database; no containers needed | Tool calls against mocked APIs (tools, REST, sandboxed SQL, search); seahaven mcp serves one world to an MCP client | seahaven serve is an OpenEnv environment with a web console; worlds can be published to Hugging Face | uvx seahaven new <world> scaffolds a project (PyPI 0.6.0, Python 3.14+) | MIT |
Verdict: Evaluating real coding agents on terminal tasks, or running existing benchmarks such as Terminal-Bench 2.0 and SWE-Bench at scale? Use Harbor. Building a fleet of containerized environments for RL with a registry, plugins and A2A-described agents? AgentEnv Framework is Scale's production stack. Need a fast, reproducible fake of a SaaS app (CRM, billing) so tool-calling agents can be graded on what they changed? Seahaven worlds are just Python and SQLite and need no containers.
A benchmark is a fixed set of tasks and a score. An environment framework is what you use to build and run those tasks: it starts a clean world for each attempt, connects an agent, and checks the result. The same framework serves evals (score a model or harness) and reinforcement learning (generate thousands of graded rollouts). For the benchmarks themselves, see Agent benchmarks compared.
The three frameworks here model "the world" in different ways, and that choice decides most of the rest.
Three ways to model the world
A container per task (Harbor). Each task is a folder with an instruction, a container environment and tests. The agent, a real harness such as Claude Code or Codex CLI, works inside the container, and the tests check what's left behind. This is how Terminal-Bench works. Harbor is its official harness, so it fits anything a shell can reach: repos, builds, files and services. It is also the heaviest option, because every attempt is a container.
A containerized environment server (AgentEnv Framework). Scale AI's framework splits the environment (a long-running server that speaks an open protocol) from the tasks that run against it. Environments are built into versioned images and deployed behind a gateway, and agents are deployed into sandboxes and described with A2A Agent Cards. That's more structure than Harbor. It pays off when many teams share environments through a registry, and when one environment backs many tasks.
A synthetic world in a database (Seahaven). Kiln AI's framework doesn't run containers. A world is a set of SQLite tables plus Python functions that act as the agent's tools, for example a fake CRM or a fake Stripe. Each run copies a frozen fixture, so it starts from a known state with a fixed clock and seed. Seahaven logs every row the agent changes, so you grade the final state instead of reading transcripts. Hundreds of instances fit in one process. This suits tool-calling agents in the style of τ-bench or AppWorld, where the work is API calls rather than shell commands.
Where the compute goes
Harbor and AgentEnv start on local Docker and scale out through sandbox providers. Harbor lists Daytona, Modal, LangSmith, Blaxel and others behind one --env flag, and AgentEnv adds providers through plugins. If you're picking a provider, Code sandboxes compared covers Daytona, Modal and the rest, and Self-hosted agent sandboxes covers running your own. Seahaven skips this layer entirely, which makes it cheap to run but limits it to what you can mock in Python.
Interop: OpenEnv and MCP
AgentEnv and Seahaven can both serve environments through OpenEnv, an open interface for RL environments, so an OpenEnv-based trainer can drive either. Seahaven can also expose a world over MCP (seahaven mcp), which is useful when you want to try a world by hand from an editor or chat client before you automate it. Harbor's interop is with benchmarks: harbor datasets list shows third-party datasets such as SWE-Bench and Aider Polyglot that run under the same CLI.
Getting started cheaply
- Harbor: run one Terminal-Bench 2.0 task locally with
--n-concurrent 1before you pay for cloud sandboxes. - AgentEnv:
uvx --from agentenv-framework agent-env run helloruns a built-in task with no Docker, model or config. - Seahaven:
uvx seahaven new crm_worldscaffolds a world with tests, a fixture generator and anAGENTS.md, so a coding agent can build the rest.seahaven checkreports fixes for authoring mistakes.
For how evals fit with tracing and observability, see Evaluate an AI agent. To train or fine-tune on agent trajectories instead of generating them, NVIDIA's Open-SWE-Traces is an open dataset of software-engineering agent trajectories (CC BY 4.0).
Directory entries in this comparison
- Harbor: Framework from the Terminal-Bench creators for evaluating and optimizing agents and models in sandboxed environments, run in parallel across cloud sandbox providers.
- AgentEnv Framework: Scale AI's open-source Python SDK and CLI for building, deploying and scoring containerized agent environments and tasks for RL and evals.
- Seahaven: Kiln AI's open-source Python framework for synthetic, resettable 'worlds' (SQLite state plus mock tools) for agent evals and RL, served via OpenEnv or MCP.
- Terminal-Bench: Continuously maintained benchmark and leaderboard (Stanford, Laude Institute) measuring how well AI agents complete hard, end-to-end tasks in real terminal environments.
- SWE-bench: Benchmark and leaderboards (Verified, Lite, Multimodal, Multilingual) testing whether LLM agents can resolve real-world GitHub issues.
- τ-bench (tau2-bench): Sierra's benchmark for tool-agent-user interaction in real-world domains.
- AppWorld: Benchmark and sandbox of 9 simulated day-to-day apps operable via 457 APIs and populated by ~100 simulated people, for evaluating function-calling and interactive coding agents.
- Daytona: Secure and elastic infrastructure (sandboxes) for running AI-generated code.
- Modal: Cloud platform for production AI: inference, training, sandboxes and serverless functions on one platform.
- Claude Code: Anthropic's agentic coding tool that reads your codebase, edits files, runs commands and integrates with dev tools; available in terminal, IDE, desktop and browser.
- OpenAI Codex CLI: OpenAI's lightweight open-source coding agent that runs in your terminal.
- Open-SWE-Traces: NVIDIA dataset of software-engineering agent trajectories (OpenHands, SWE-agent, mini-swe-agent, Claude Code, OpenCode) for distilling SWE agents; CC BY 4.0.
Related
- Agent benchmarks compared (Comparison)
- Evaluate an AI agent with open-source eval and observability tools (Guide)
- Code sandboxes for AI agents (Comparison)
- Self-hosted sandboxes for agents compared (Comparison)
- Run agent-generated code safely in a sandbox (Guide)
- Test a tool-using agent for prompt injection with AgentDojo (Guide)
Sources
- Harbor README, accessed
- Harbor example task (examples/tasks), accessed
- PyPI: harbor, accessed
- AgentEnv Framework README, accessed
- AgentEnv Framework docs, accessed
- PyPI: agentenv-framework, accessed
- Seahaven README, accessed
- PyPI: seahaven, accessed