Multi-SWE-bench
ByteDance Seed's multilingual issue-resolving benchmark: 1,632 verified tasks across Java, TS, JS, Go, Rust, C and C++.
Tags
Description
Multi-SWE-bench extends SWE-bench-style evaluation beyond Python with 1,632 instances in 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, C++), curated from 2,456 candidates by 68 expert annotators. Lighter variants: mini (400 instances, 8 languages) and flash (300 instances). The companion Multi-SWE-RL dataset started with 4,723 instances for RL training. Evaluation is Docker-based and all data, code and container images are public. Accepted to the NeurIPS 2025 Datasets and Benchmarks track.
Agent access
Related
- SWE-bench: Benchmark and leaderboards (Verified, Lite, Multimodal, Multilingual) testing whether LLM agents can resolve real-world GitHub issues.
- SWE-Bench Pro: Scale AI benchmark of long-horizon software-engineering tasks for AI agents, with an open dataset on Hugging Face and a public leaderboard.
- SWE-rebench: Nebius's continuously refreshed, decontaminated SWE agent benchmark and dataset: 21,000+ validated GitHub issue-PR tasks plus a time-windowed leaderboard.
- SWE-Gym: Environment and dataset for training software-engineering agents and verifiers on real GitHub tasks (ICML 2025).
Sources
- https://github.com/multi-swe-bench/multi-swe-bench
- https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench
Machine-readable
- JSON
- Markdown
- Source file on GitHub (edit via pull request)