SWE-rebench
Nebius's continuously refreshed, decontaminated SWE agent benchmark and dataset: 21,000+ validated GitHub issue-PR tasks plus a time-windowed leaderboard.
Tags
Description
The Hugging Face dataset nebius/SWE-rebench (CC-BY-4.0; each task also carries its source repo's license) is built by an automated pipeline that extracts and validates interactive tasks from 3,400+ Python repositories, with prebuilt Docker images for thousands of tasks. The leaderboard at swe-rebench.com evaluates models and agents on fresh tasks within a selectable time window to limit contamination and reports resolved rate, pass@5, cost and tokens per problem. A SWE-rebench v2 (paper arXiv 2602.23866) is also published. Run evaluations with the MIT-licensed SWE-bench fork.
Related
- SWE-bench: Benchmark and leaderboards (Verified, Lite, Multimodal, Multilingual) testing whether LLM agents can resolve real-world GitHub issues.
- SWE-Gym: Environment and dataset for training software-engineering agents and verifiers on real GitHub tasks (ICML 2025).
- SWE-smith: Toolkit and datasets for scaling training data for software-engineering agents by synthesizing task instances from GitHub repos (NeurIPS 2025 D&B).
Sources
- https://swe-rebench.com
- https://huggingface.co/datasets/nebius/SWE-rebench
- https://github.com/SWE-rebench/SWE-bench-fork
- https://github.com/SWE-rebench/SWE-rebench-V2
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)