Most sandbox products give an agent a disposable machine: a filesystem it can keep writing to, a shell, packages it can install, sometimes a browser. Run agent-generated code safely in a sandbox and the comparisons of hosted code sandboxes and self-hosted agent sandboxes cover that kind.
\nPiston and Judge0 are a different, smaller thing. They take source code plus stdin, run it once under fixed limits, and return stdout, stderr, the exit code and timing. Nothing survives between calls. They came out of chat bots and online judges, so they support many languages and are built to run code from strangers.
\nThat shape fits an agent tool when you want:
\nIt is the wrong shape when the agent needs to install a package, keep files between steps, clone a repository or run a long job. Use a machine-style sandbox for that.
\nBoth projects put the Isolate sandbox (the one written for programming olympiads) inside a privileged Docker container. Isolate uses Linux namespaces, a chroot, a separate unprivileged uid per run and cgroups for limits. That is a container-class boundary on a kernel shared with the host, not a VM. Run them on a host that does nothing else important.
\nOn 2026-10-30 we ran Piston (image ghcr.io/engineer-man/piston, built 2025-02-08) under Docker on a Linux 6.12 host with cgroup v2, installed Python 3.12.0, GCC 10.2.0, Bash 5.2.0 and Lua 5.4.4, and ran every probe and the wrapper below against it. The results quoted are from those runs.
We also brought up Judge0 CE v1.13.1 with its release Compose file on the same host. The API came up, but every submission failed with an internal error, because that release needs cgroup v1 (details below). So the Judge0 sections describe the API from its docs and its own /config_info output; we did not see a Judge0 submission succeed.
The deciding fact is often your kernel's cgroup setup, not features.
\n| Piston | Judge0 CE v1.13.1 | |
|---|---|---|
| License | MIT | GPL-3.0 |
| Host requirement | cgroup v2 enabled and cgroup v1 disabled | cgroup v1: the release notes tell you to add systemd.unified_cgroup_hierarchy=0 to GRUB on Ubuntu 22.04 and reboot |
| Services | one container | four: API server, workers, PostgreSQL, Redis |
| Request model | synchronous POST /api/v2/execute; WebSocket for interactive runs | queued submissions; poll by token, use wait=true, or a callback URL |
| Languages | installed one by one as packages | fixed set in the image (47 in our CE v1.13.1 instance; Extra CE is a separate branch) |
| Auth | none | optional token header, off by default |
| Telemetry | none documented | on by default |
| Last release | rolling master image | v1.13.1, 2024-04-18 |
Most current distributions (Ubuntu 22.04 and later, Debian 12, Fedora) boot with cgroup v2 only. Check yours:
\nstat -fc %T /sys/fs/cgroup # cgroup2fs means cgroup v2 onlyOn a cgroup v2 host, Piston runs as documented and Judge0 v1.13.1 needs a kernel command-line change and a reboot. On a dedicated cgroup v1 host, either works.
\nStart the API, bound to localhost only. The README's command publishes port 2000 on every interface; do not do that on a machine with a public address, because the API has no authentication.
\nmkdir -p piston-data\ndocker run --privileged -v "$PWD/piston-data:/piston" -dit \\\n -p 127.0.0.1:2000:2000 \\\n -e PISTON_RUN_MEMORY_LIMIT=268435456 \\\n --name piston_api ghcr.io/engineer-man/piston--privileged is required: Isolate needs it to create namespaces and cgroups inside the container. That is one more reason to give Piston its own host or VM.
The image has no languages. List what can be installed, install what you need, and check what is ready:
\ncurl -s localhost:2000/api/v2/packages # every package, with "installed": true/false\ncurl -s -X POST localhost:2000/api/v2/packages \\\n -H 'content-type: application/json' -d '{"language":"python","version":"3.12.0"}'\ncurl -s localhost:2000/api/v2/runtimes # what /execute can use nowInstalling Lua 5.4.4 took about 3 seconds here. The package endpoint is as unauthenticated as /execute, so anyone who can reach the API can install runtimes. Expose only /api/v2/execute to the agent (see the hardening list).
Run code:
\ncurl -s -X POST localhost:2000/api/v2/execute -H 'content-type: application/json' -d '{\n "language": "python", "version": "3.12",\n "files": [{"name": "main.py", "content": "import helper; print(helper.X)"},\n {"name": "helper.py", "content": "X = 99"}],\n "stdin": ""\n}'{"run":{"signal":null,"stdout":"99\\n","stderr":"","code":0,"output":"99\\n","memory":3856000,"message":null,"status":null,"cpu_time":11,"wall_time":51},"language":"python","version":"3.12.0"}version is required and is a semver selector, so "3.12" matched 3.12.0. Leaving it out returns 400 {"message":"version is required as a string"}. The first file is the one that runs. Compiled languages add a compile object with the same fields as run. status is null on success, or one of RE (runtime error), SG (signal), TO (timeout), OL (stdout too long), EL (stderr too long) or XX (internal error).
We sent these snippets with default limits, except where noted:
\n| Probe | Result |
|---|---|
while True: pass | status: TO, SIGKILL, "Time limit exceeded (wall clock)" at about 3.1 s |
time.sleep(10) | TO at 3.1 s, so wall time is limited, not just CPU |
TCP connect to 1.1.1.1:80 | OSError: [Errno 101] Network is unreachable |
DNS lookup of example.com | gaierror: [Errno -3] Temporary failure in name resolution |
| Fork loop | stopped after 62 forks with [Errno 11] Resource temporarily unavailable |
Allocate 512 MB with run_memory_limit 256 MiB | killed, exit code 137, memory 268432000 |
| Print 5,000 characters | status: OL, "stdout length exceeded", SIGKILL |
Write to /tmp | allowed, as an unprivileged uid; the file was gone on the next run |
List /piston | saw ['packages']; writing to /piston/packages raised PermissionError |
| C program with a compile error | compile.status: RE, exit code 1; the run stage never started |
"language": "cobol" | 400, "cobol-* runtime is unknown" |
The defaults behind those numbers come from api/src/config.js and the configuration docs: networking disabled, 3 s run timeout and 3 s run CPU time, 10 s for compiling, 1,024 characters of output per stream, 64 processes, 2,048 open files, 10 MB maximum file size, and 64 concurrent jobs. The fork result (62 children plus the parent) matches the 64-process default. The README's security section still says processes are capped at 256 by default; the code and the configuration page say 64.
The memory default is "no limit". compile_memory_limit and run_memory_limit default to -1. Set them with PISTON_RUN_MEMORY_LIMIT and PISTON_COMPILE_MEMORY_LIMIT (bytes). With no server limit, a request could ask for run_memory_limit: 999999999999 and Piston ran it. With PISTON_RUN_MEMORY_LIMIT=268435456 set, the same request got 400 {"message":"run_memory_limit cannot exceed the configured limit of 268435456"}, and a 512 MB allocation was killed even when the request did not mention memory.
Per-request limits can only go down. Asking for run_timeout: 60000 against the default returned 400 {"message":"run_timeout cannot exceed the configured limit of 3000"}. To allow more, raise the server value (for example PISTON_RUN_TIMEOUT=10000; a 5 s sleep then completed), or use PISTON_LIMIT_OVERRIDES to raise limits for one language only.
The 1,024-character output cap is small for an agent that wants to read a traceback. PISTON_OUTPUT_MAX_SIZE=65536 let a 5,000-character print through. Truncate again in your wrapper so a chatty program does not fill the model's context.
The probe script, so you can rerun this against your own instance:
\n#!/usr/bin/env bash\n# piston-probe.sh: run hostile and normal snippets against a Piston API and print one line each.\nP=${PISTON_URL:-http://127.0.0.1:2000}\nrun() { # $1 label, $2 JSON body\n printf '%-11s ' "$1"\n curl -s -X POST "$P/api/v2/execute" -H 'content-type: application/json' -d "$2" |\n python3 -c 'import sys,json; d=json.load(sys.stdin); r=d.get("run",d); c=d.get("compile") or {}\nprint(json.dumps({"compile":c.get("status"),"status":r.get("status"),"code":r.get("code"),"signal":r.get("signal"),"stdout":(r.get("stdout") or "")[:70],"stderr":(r.get("stderr") or "")[-90:],"message":r.get("message"),"wall_ms":r.get("wall_time"),"mem":r.get("memory")}))'\n}\npy() { python3 -c 'import json,sys; print(json.dumps({"language":"python","version":"3.12","files":[{"content":sys.argv[1]}],**json.loads(sys.argv[2] if len(sys.argv)>2 else "{}")}))' "$@"; }\n\nrun hello "$(py 'print("hello from piston")')"\nrun timeout "$(py 'while True: pass')"\nrun sleep "$(py 'import time; time.sleep(10)')"\nrun network "$(py 'import socket; socket.create_connection(("1.1.1.1",80),timeout=3); print("NET OK")')"\nrun dns "$(py 'import socket; print(socket.gethostbyname("example.com"))')"\nrun forkbomb "$(py 'import os\nn=0\ntry:\n while True:\n if os.fork()==0:\n import time; time.sleep(5); os._exit(0)\n n+=1\nexcept OSError as e: print("forks before error:",n,e)')"\nrun memory "$(py 'b=bytearray(512*1024*1024); print("allocated", len(b))' '{"run_memory_limit":268435456}')"\nrun bigout "$(py 'print("x"*5000)')"\nrun write-tmp "$(py 'open("/tmp/x","w").write("1"); import os; print(os.getuid(), os.listdir("/tmp"))')"\nrun read-host "$(py 'import os; print(os.listdir("/piston"))')"\nrun unknown '{"language":"cobol","version":"*","files":[{"content":"x"}]}'The model should see one small tool with a fixed language list. The wrapper re-checks limits so the model cannot ask for more, and returns a compact result. This is standard-library Python and was run against the instance above; the schema is in the shape most function-calling APIs accept (rename input_schema to parameters where needed).
import json, os, urllib.request, urllib.error\n\nPISTON_URL = os.environ.get("PISTON_URL", "http://127.0.0.1:2000")\nALLOWED = {"python": "3.12", "bash": "5", "c": "10"} # language -> version selector\nMAX_CODE_BYTES = 64_000\nMAX_RETURN_CHARS = 4_000\n\nTOOL_SCHEMA = {\n "name": "run_code",\n "description": "Run a short, self-contained program and return its output. "\n "No network, no persistent files, about 3 s of run time.",\n "input_schema": {\n "type": "object",\n "properties": {\n "language": {"type": "string", "enum": sorted(ALLOWED)},\n "code": {"type": "string"},\n "stdin": {"type": "string"},\n },\n "required": ["language", "code"],\n },\n}\n\ndef _post(path, body, timeout=30):\n req = urllib.request.Request(PISTON_URL + path, data=json.dumps(body).encode(),\n headers={"content-type": "application/json"})\n try:\n with urllib.request.urlopen(req, timeout=timeout) as r:\n return json.load(r)\n except urllib.error.HTTPError as e:\n return {"message": json.load(e).get("message", str(e))}\n\ndef run_code(language, code, stdin=""):\n if language not in ALLOWED:\n return {"ok": False, "error": f"language must be one of {sorted(ALLOWED)}"}\n if len(code.encode()) > MAX_CODE_BYTES:\n return {"ok": False, "error": "code too large"}\n res = _post("/api/v2/execute", {\n "language": language, "version": ALLOWED[language],\n "files": [{"content": code}], "stdin": stdin or "",\n "run_timeout": 3000, "run_memory_limit": 256 * 1024 * 1024,\n })\n if "run" not in res: # 400: bad request, unknown runtime, limit above the server's\n return {"ok": False, "error": res.get("message", "execution failed")}\n stage = res.get("compile") if res.get("compile", {}).get("code") else res["run"]\n return {\n "ok": stage["code"] == 0 and stage["status"] is None,\n "stage": "compile" if stage is not res["run"] else "run",\n "exit_code": stage["code"], "signal": stage["signal"],\n "status": stage["status"], "message": stage["message"],\n "stdout": stage["stdout"][:MAX_RETURN_CHARS],\n "stderr": stage["stderr"][:MAX_RETURN_CHARS],\n }What it returned for six test calls:
\n| Call | Result |
|---|---|
Python print(sum(range(10))) | ok: true, stdout 45 |
Python reading stdin hi | ok: true, stdout HI |
| C with an undeclared variable | ok: false, stage: compile, status: RE, compiler error in stderr |
Python while True: pass | ok: false, status: TO, "Time limit exceeded (wall clock)" |
Python urlopen("http://1.1.1.1") | ok: false, status: RE, "Network is unreachable" in stderr |
language: "ruby" | rejected by the wrapper before any request |
Two habits matter more than the code. Treat stdout and stderr as data the model reads, never as instructions to follow. And because Piston blocks the network, tell the model so in the tool description; otherwise it will try pip install and waste turns.
If you would rather call a library than run a service, LLM Sandbox runs code through Docker, Kubernetes, Podman, Bubblewrap or NsJail backends from Python and ships an MCP server.
\nJudge0's release deployment, from the v1.13.1 release notes:
\nwget https://github.com/judge0/judge0/releases/download/v1.13.1/judge0-v1.13.1.zip\nunzip judge0-v1.13.1.zip && cd judge0-v1.13.1\n# set REDIS_PASSWORD and POSTGRES_PASSWORD in judge0.conf to long random values\ndocker-compose up -d db redis\nsleep 10s\ndocker-compose up -dThe Compose file runs four services. server and workers both use judge0/judge0:1.13.1 with privileged: true, and the server publishes 2358:2358 on all interfaces. Change that to 127.0.0.1:2358:2358 unless a gateway sits in front.
Submit code and wait for the result in one call (language_id 71 is Python 3.8.1 in CE):
curl -s -X POST 'localhost:2358/submissions?base64_encoded=false&wait=true' \\\n -H 'content-type: application/json' \\\n -d '{"source_code":"print(\\"hello\\")","language_id":71,"stdin":""}'Without wait=true you get a token back and poll GET /submissions/{token}. POST /submissions/batch takes up to 20 at once by default, and a callback_url field makes Judge0 call you when a run finishes. GET /languages lists ids, and GET /statuses returns the result codes, from 1 "In Queue" and 3 "Accepted" to 5 "Time Limit Exceeded", 6 "Compilation Error", 7 to 12 for runtime errors and 13 "Internal Error". GET /config_info shows the limits in force. The interactive docs are served at /docs on your instance and at ce.judge0.com.
The API came up and answered /about, /languages (47 languages), /statuses and /config_info. Every submission came back as status 13, Internal Error, with the message No such file or directory @ rb_sysopen - /box/script.py. Running Isolate's setup by hand inside the workers container shows why:
$ isolate --version\nThe process isolator 1.8.1\n$ isolate --cg -b 99 --init\nFailed to create control group /sys/fs/cgroup/memory/box-99/: No such file or directoryIsolate 1.8.1, bundled in the v1.13.1 image, expects the cgroup v1 memory controller at /sys/fs/cgroup/memory. On a cgroup v2 host that path does not exist, so the sandbox is never created. This is what the release notes' GRUB step is for. If you cannot change the host's boot parameters (most managed Kubernetes nodes and many cloud images), use Piston or a different sandbox, or give Judge0 a VM you control.
(Unrelated to Judge0: on our host the Compose network's DNS and container-to-container traffic did not work, so to get this far we put all four services in one network namespace. Do not copy that workaround. It puts PostgreSQL and Redis on the same localhost as the workers, which is exactly what a sandboxed program with networking enabled must not reach.)
\nThe live /config_info of the default install showed these, and judge0.conf documents each:
AUTHN_TOKEN is empty by default. Set it, and clients send it in the X-Auth-Token header (AUTHN_HEADER renames it). AUTHZ_TOKEN (header X-Auth-User) is a separate token for protected API calls, which are disabled while it is empty.enable_network defaults to false, but allow_enable_network defaults to true, so any client can send "enable_network": true with a submission. Set ALLOW_ENABLE_NETWORK=false.JUDGE0_TELEMETRY_ENABLE=false if you don't want that.wait=true allowed. Clients can raise limits up to the MAX_* values, so set the MAX_* values to what you would accept.v1.13.1 fixed three critical advisories reported by Daniel Cooper, all affecting versions up to 1.13.0:
\nchown following a symlink.enable_network, a submission could reach the PostgreSQL database on the internal Docker network and turn that into unsandboxed code execution as root.The lesson goes beyond Judge0. In both projects the API and the code it runs sit in the same privileged container, so the boundary is Isolate plus your settings. Run v1.13.1 or later, keep networking off, and put the whole stack on a host with nothing worth stealing.
\nPOST /api/v2/execute, or POST /submissions) and checks a token. Piston has no auth at all, and its package endpoint installs runtimes for anyone.PISTON_DISABLE_NETWORKING=true). In Judge0, set ALLOW_ENABLE_NETWORK=false so callers cannot turn it on.PISTON_RUN_MEMORY_LIMIT and PISTON_COMPILE_MEMORY_LIMIT, which default to unlimited. Judge0: set the MAX_* values. Then set tighter per-call values in the wrapper.PISTON_MAX_CONCURRENT_JOBS); Judge0 rejects submissions once 100 are queued. An agent stuck in a loop can fill either.When the agent outgrows one-shot runs and needs a persistent workspace, move to a machine-style sandbox: E2B or the others in hosted code sandboxes, or microsandbox and the others in self-hosted agent sandboxes.
" }