SWE-bench Pro: Coding Agent Scores

Last verified: 2026-08-10. Dataset composition, harness settings, and leaderboard figures below reflect Scale's published methodology page as of that date. Benchmarks get revised; re-check before citing.

SWE-bench Pro numbers are still appearing in evaluation memos as if nothing changed. Two things did.

OpenAI audited the public task set and retracted its own recommendation. And leaderboard scores roughly tripled — partly for reasons that have nothing to do with models getting better.

The benchmark's design is genuinely careful. But both of those change how a Pro figure should be read, and neither is visible in the number itself.

What SWE-bench Pro Measures

SWE-bench Pro, built by Scale AI, evaluates coding agents on long-horizon tasks — reference solutions average around 107 lines across roughly 4 files, so these are not single-function fixes.

Task Sourcing and Repo Scope

The dataset is 1,865 tasks across 41 repositories, split three ways:

The copyleft choice is the clever part. GPL licensing creates a legal deterrent against inclusion in proprietary training corpora — not just an access barrier. The private subset goes further: code acquired through partnership agreements that was never public to scrape.

Tasks are built through a four-stage pipeline Scale documents in full — repository selection, Docker environment construction, commit harvesting, then human augmentation. Three human-in-the-loop checkpoints are documented: environment construction, issue augmentation, and test verification for relevance and flakiness.

Resolve Rate: The Actual Scoring Mechanism

This is where summaries get sloppy most often. The primary metric is Resolve Rate, and a task counts as resolved only when both conditions hold:

Calling it "Pass@1" describes the attempt budget, not the grading rule. The second condition is the one that matters for reading scores honestly: a patch that fixes the issue but breaks unrelated tests scores zero. That is stricter than most internal review processes, and it is the strongest thing the benchmark has going for it.

The 2026 Audit and What It Changed

On July 8, 2026, OpenAI published a datapoint-level audit of the 731-task public split and concluded that roughly 30% of tasks are broken, formally retracting its earlier recommendation that the research community treat Pro as a leading coding eval.

The failure categories are specific: hidden requirements, contradictory instructions, overly strict tests, and incomplete grading criteria. One documented example — instructions specify one leading space when converting to Markdown, the hidden grading test requires two. An agent that follows the prompt correctly is marked wrong.

This matters more than a normal quality complaint, because it inverts an assumption the benchmark's own design rests on. Scale's methodology states that human experts refine under-specified issues to ensure solvability. If that holds, a failure is the agent's fault. If ~30% of tasks are broken, you can no longer attribute failure to the agent without checking the task.

This does not make the benchmark worthless. It means the error bar is now wider than the gaps between adjacent models on the leaderboard.

Reading a Leaderboard Without Overfitting

Four checks, and the current leaderboard demonstrates why each one exists.

Check the harness. Scale's own leaderboard footnotes that older entries ran with capped cost and a 50-turn limit, while current entries run uncapped with a 250-turn limit. Top scores moved from roughly 23% to roughly 61% over that period. Some of that is real model progress. Some of it is being allowed to try five times longer. A score without its harness settings is not a measurement.

Check the confidence intervals. Scale publishes them — figures carry roughly ±3 points. They also publish a rank that accounts for overlap, which is why multiple models legitimately share rank 5 and rank 10. A two-point lead is noise dressed as a ranking.

Check the subset. Public, held-out, and private results are not interchangeable. Models score meaningfully lower on the private set — which is the point of having it.

Check the date. Given the audit and the harness change, a Pro figure from six months ago describes a different measurement than one from today.

Benefits and Limits

Contamination and Harness Variance

Contamination handling is Pro's real contribution, and the copyleft plus proprietary-partnership structure is a genuine mitigation rather than a marketing claim.

Harness variance is unsolved, and the leaderboard's own history proves it. Same benchmark, same tasks, different turn limit — scores nearly triple. When a vendor cites a Pro number, the follow-up is which harness, which turn limit, which cost cap. Most decks don't say.

What the Score Cannot Tell You

Resolve Rate compresses each task to one bit. It is silent on:

• How many attempts were consumed, and what the agent did while failing
• Whether it modified files outside the task's scope
• Whether a reviewer could follow the change
• Token cost and wall-clock time
• Behavior on genuinely ambiguous tasks — which, per the audit, some of these are
• Whether it stopped safely when it couldn't finish

For screening, that compression is acceptable. For a purchase decision, it isn't.

Who Should Rely on It

Use it as a filter when you need to eliminate obviously weaker candidates or calibrate how hard real repository work currently is.

Don't use it as a verdict when two candidates are within a few points, when you're evaluating for a specific language stack — performance varies substantially by language — or when the number has to survive procurement review.

Ignore it if your bottleneck isn't agent capability. If changes take three days because ownership is unclear and coverage is thin, a better agent produces faster changes to something nobody can safely review.

FAQ

How many tasks does SWE-bench Pro contain, and is the data open?

1,865 tasks across 41 repositories: 731 public, 858 held-out, 276 private. Only the public set is openly available — the dataset is published on GitHub. Held-out results are never published; private-set results appear on a separate leaderboard.

How is Resolve Rate actually calculated?

Percentage of tasks where the submitted patch satisfies both conditions: new fail-to-pass tests now pass, and all pre-existing pass-to-pass tests still pass. Both required. Partial credit does not exist.

Does the OpenAI audit invalidate the leaderboard?

It doesn't zero it out, but it widens the error bar beyond the gaps between adjacent models. Treat rankings as tiers rather than ordered lists, and treat small differences as unresolved. Scale's methodology and the accompanying paper remain worth reading for how the tasks were built — the audit disputes execution, not the design goals.

Is SWE-bench Pro replacing SWE-bench Verified?

Layering, not replacing — and both now carry documented quality concerns. Verified has a longer track record and more comparable published results; Pro has harder tasks and better contamination handling. Teams tracking progress cite both and label which is which.

Conclusion

SWE-bench Pro was built to solve a real problem, and its two-condition scoring rule is stricter than most internal review gates. That part holds up.

What changed in 2026 is that the measurement acquired a documented error rate, and the harness under it shifted enough to triple headline scores. Neither makes the benchmark useless. Both make single-number comparisons unusable.

So use it the way it still works: to decide what to test, on your own repository, with your own harness. Then test it.

Related Reading

• What SWE-bench measures and how the variants differ
• Tracing what an agent actually did during a run
• Building a coding-agent loop you can evaluate