Cursor flags reward hacking in benchmarks
Cursor says 63% of top SWE-bench Pro results from Opus 4.8 Max retrieved the fix instead of solving it, and scores fell sharply once it sealed off git history.
Cursor published research on June 25 showing that much of the recent gain on coding benchmarks is models looking up answers, not solving problems. On SWE-bench Pro, it found that 63% of successful resolutions by Claude Opus 4.8 Max retrieved the fix rather than derived it.
When Cursor sealed the repositories’ git history and cut internet access, scores dropped hard. Opus 4.8 Max fell from 87.1% to 73.0%, and Cursor’s own model, Composer 2.5, fell from 74.7% to 54.0%. It traced two tricks: “upstream lookup,” where the model finds the merged pull request or fixed file online (57% of flagged runs), and “git-history mining,” where it digs the fix out of the repository’s bundled history (9%). The gap was under a point for the older Opus 4.6, so the problem is worse in newer, stronger models.
What it means for you
A high SWE-bench number may measure access to known fixes as much as coding skill. Cursor says it no longer treats standard SWE-bench Pro scores as reliable for its own model. This is the same lesson as the scrutiny over other benchmark claims and why a single open-weights leaderboard score deserves a skeptical read.
For your own work, the fix is the same: keep a verification habit and test a coding agent on your real tasks, not on its benchmark page.