TermGrade
1,004 executable Linux terminal tasks with pytest verifiers and per-model pass rates, plus 36,144 graded agent trajectories, for training and evaluating agents.
Tags
Description
Released by ai& on 2026-10-08. termgrade-environments: each task is a container, an instruction and a pytest verifier, kept only if its reference solution passes, with measured pass rates for six models (Terminus-2 scaffold via Harbor). termgrade-trajectories: all 36,144 graded attempts with commands, raw terminal frames and per-test verdicts; the 21,910 passes form an SFT set, and the 14,234 failures are included. ai& also released the training selection and a gemma-4-31B-it checkpoint (LoRA and merged) that scores 46.1 on Terminal-Bench 2.1, +3.1 over the base model. Both datasets are Apache-2.0 on Hugging Face.
Related
- Terminal-Bench: Continuously maintained benchmark and leaderboard (Stanford, Laude Institute) measuring how well AI agents complete hard, end-to-end tasks in real terminal environments.
- Harbor: Framework from the Terminal-Bench creators for evaluating and optimizing agents and models in sandboxed environments, run in parallel across cloud sandbox providers.
Sources
- https://www.aiand.com/newsroom/termgrade
- https://huggingface.co/datasets/ai-and/termgrade-environments
- https://huggingface.co/datasets/ai-and/termgrade-trajectories
- https://huggingface.co/ai-and/termgrade-gemma4-31b-lora
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)