SWE-bench Verified
- Kind
- benchmark
- Maintainer
- OpenAI / Princeton
- Feeds axes
- coding
- Measures
- Whether a model can fix real reported bugs in established software projects, working the way a human programmer would — read the bug report, find the problem in the code, write a fix that passes the project's tests.
- Does not measure
- Building something new from scratch, visual/front-end work, languages other than Python, how fast or cheap the model is.
- Known issues
- Vendors test with different amounts of supporting machinery around the model, so their numbers aren't directly comparable; older test items may have leaked into training data, letting models "remember" answers.
- Trust
- high
- Last reviewed
- 2026-08-01
The test takes 500 real bug reports filed against popular open-source software projects, each one checked by humans to make sure it's solvable. The model gets the bug report and the project's code, and has to produce a fix. The fix counts only if the project's own test suite passes — the same bar a human contributor's fix would have to clear. The score is simply the percentage of bugs fixed.
How to read a score: the supporting machinery matters almost as much as the model. A vendor's self-reported number using their own custom setup is not comparable to a stripped-down run, so prefer numbers the leaderboard maintainers verified themselves. Rule of thumb: a 5-point gap between comparable setups is a real difference in ability; a 2-point gap is coin-flip territory.