OpenAI flags widespread flaws in SWE-Bench Pro benchmark
What's the story
OpenAI has raised concerns over SWE-Bench Pro, a popular coding benchmark. The company conducted an audit of the test and found that nearly 30% of its tasks are flawed in ways that misrepresent what an AI model can actually do. This has prompted OpenAI to retract its earlier recommendation for SWE-Bench Pro as a replacement for SWE-bench Verified.
Benchmark issues
SWE-Bench Pro was previously endorsed by OpenAI
OpenAI had previously endorsed SWE-Bench Pro as a solution to the contamination and design problems plaguing SWE-bench Verified. The latter was once the standard benchmark for agentic coding but had become unreliable due to contamination and design problems. SWE-Bench Pro was marketed as a more realistic test, using longer coding tasks from public and private repositories' commit history. However, an audit revealed that nearly 30% of its tasks are flawed in ways that misrepresent what an AI can actually do.
Review methods
Audit flagged 200 tasks as broken
OpenAI's audit involved an automated review pipeline and a human annotation campaign by experienced engineers. The automated pipeline flagged 200 tasks, or 27.4% of the dataset, as broken. Human reviewers were harsher, flagging 249 tasks, or 34.1%. Both methods pointed in the same direction but yielded different numbers for the extent of issues with SWE-Bench Pro's tasks.
Issue categories
Issues with SWE-Bench pro's tasks
The audit revealed several types of issues with SWE-Bench Pro's tasks. These included overly strict tests that checked for implementation details not specified in the prompt, underspecified prompts where hidden tests enforced requirements a model had no reasonable way of inferring, and low-coverage tests where an incomplete fix could still pass. There were also misleading prompts that pointed models toward the wrong approach entirely.
Structural issues
Problems with SWE-Bench Pro are structural
OpenAI has emphasized that the problems with SWE-Bench Pro are structural, not accidental. The tasks are sourced from real GitHub issues and pull requests written for human maintainers, not for isolated evaluation. This mismatch between what a task description calls for and what hidden tests enforce is exactly what shows up as noise in benchmark scores.