SWE-Bench Benchmark Methodology and Limitations

SWE-Bench is the most rigorous coding-agent benchmark available, testing real bug fixes in real repositories, though variant differences, incomplete grading, harness design, and training contamination limit what its scores establish about deployment readiness. Evaluate what an agent’s score actually proves about work on your code alongside its ranking.

What SWE-Bench actually tests and how its scoring works

SWE-Bench gives an agent a repository at its pre-fix commit and an issue description, then asks it to produce a patch. The agent must navigate the codebase, understand the issue description, and locate what needs changing without being given the failing test directly. This is repository work. That distinction makes it a stronger test of coding-agent capability than an isolated exercise with the problem already neatly boxed.

Success requires two conditions at the same time. Fail-to-pass tests, which failed before the fix, must now pass, while pass-to-pass tests must stay passing. The repositories include Django, SymPy, Astropy, matplotlib, scikit-learn, pytest, requests, Flask, pylint, xarray, sphinx, and seaborn, chosen for domain diversity and high-quality test suites. A Docker container runs the target repository at the exact pre-fix commit, so the evaluation is fully automated and objective under those conditions. Either the patch satisfies both or it does not.

The five benchmark variants and why comparing scores across them is invalid

The first complication arrives before any patch is inspected. “SWE-Bench” can refer to Original, Lite, Verified, Pro, or Live, and those labels do not describe interchangeable exams. A strong score on Lite cannot establish that an agent outperforms one with a lower score on Verified. The task set is part of the measurement. Without the variant name in a leaderboard headline, readers lack what they need to judge the result.

Original, released in October 2023, included some unsolvable or under-specified tasks, making its scores noisy and leaving it superseded for evaluation purposes. SWE-Bench Verified, developed in partnership with OpenAI and human-validated in August 2024, addressed that task-quality problem. Its tasks were confirmed solvable and clearly specified. It became the canonical leaderboard variant, but that status did not make it immune to grading flaws or contamination. Human review repaired one part of the exam.

Scale AI / SEAL’s SWE-Bench Pro paper describes a broader challenge built from real commit histories in dozens of repositories that are still actively maintained. Pro includes Python, Go, TypeScript, and JavaScript, plus private proprietary codebases never publicly available. Leading models score substantially lower on its aggregate than on Verified. Live changes its task set over time, so scores from different evaluation dates may come from different tests and cannot be compared directly.

Why test-pass grading is a weaker correctness signal than it appears

Tests are an oracle, the mechanism that decides whether an answer counts as correct, but that oracle sees only what the tests exercise. SWE-Bench uses the test files that developers wrote and changed in the pull request, so it may leave functionality covered by other test files untested. Running a full suite surfaces incorrect patches that its grading misses. A green tick is not a proof.

Two failures weaken the result in different ways. Solution leakage lets an agent retrieve an answer supplied in an issue report or comments. Inadequate coverage lets a defective patch pass because the tests do not establish correctness. When instances with these problems are removed, resolution rates fall sharply. Test flakiness adds another source of noise, as timing, resource availability, or randomness can change whether a test passes on a particular run.

Zheng et al.’s 2026 SWE-Bench Pro Verified study shows these weaknesses through its work to audit and repair 731 instances. Its anti-hacking controls block access to gold patches through Git history, local files, and public code-hosting domains. Expert-reviewed task refinement also corrects misleading problem statements and improperly scoped tests across a substantial share of instances. Models that show hacking behavior score much worse with these controls in place. That drop exposes prior inflation.

The SWE-Proof framework proposes a sharper remedy by adding machine-verified correctness to test-based grading. Under formal-verification audit, 26.8% of patches that pass every test in Verified are overturned as incorrect. Machine-checked proofs offer the most complete proposed way to establish correctness beyond what tests exercise, though implementing them at scale is demanding. Until grading reaches beyond test satisfaction, a high score establishes skill at satisfying a particular oracle.

How scaffold architecture inflates scores independently of model capability

A coding agent is a model surrounded by software that decides how it searches, uses tools, manages context, and retries. That surrounding scaffold can move scores independently of the model’s underlying capability. Yet a headline often awards it all the credit.

Benjamin Rombaut’s Inside the Scaffold taxonomy documents how much these architectures differ. Control strategies range from fixed pipelines to Monte Carlo Tree Search, agents differ widely in how many tools they have, and they use several different strategies for context compaction. Most agents combine loop primitives. These primitives are generate-test-repair, plan-execute, multi-attempt retry, and tree search. Different combinations change how the agent approaches the same repository problem.

Maria I. Gorinova et al.’s Tessl position paper formalizes how performance depends on the agent’s components by treating the agent as a composite of model, harness, context, environment, and feedback signals. Any component can shift the end-to-end score by margins comparable to those between adjacent model generations. SWE-Bench compresses that system into one number without component-level signals. Vendor-specific retrieval, retry policies, and judge LLMs are rarely open-sourced. When vendors report different results for the same base model, the harness is typically the explanation.

Pro deals with part of this through Scale AI’s standardized scaffolding, which gives all entrants the same harness. A shared harness may not be the setup in which a model works best when deployed. SWE-Serve also supplies a useful counterexample, as switching to model-specific harnesses did not improve mean pass@1 for its top configuration. Harness choice can alter performance, but “custom” is not a magic word that always makes scores rise.

Training data contamination and what it means for score validity on static benchmarks

Even with sound grading and a common harness, a static benchmark can reward familiarity. Of Verified’s issues and ground-truth pull requests, 94% predate the knowledge cutoff dates of leading models. That creates structural exposure to contamination. The benchmark must be treated as potentially familiar work, even though the evidence leaves open whether every answer was memorized.

File-path localization provides more direct evidence of the problem. State-of-the-art models can identify buggy file paths from issue descriptions alone at rates far above what generalization would predict, but accuracy drops substantially on repositories outside SWE-Bench. That pattern is consistent with repository-bias memorization. Verbatim similarity is another warning, as consecutive n-gram overlap on Verified is roughly double that on other coding benchmarks. Some outputs appear to reproduce memorized patches.

Models may learn the architectural patterns and problem distributions of the benchmark’s fixed repositories without transferring that skill to unfamiliar code. An organization’s private repository presents exactly that test of transfer. Its codebase is not Django, and its bugs are not waiting in scikit-learn’s issue tracker. Familiarity with a famous codebase is useful, but it is not a backstage pass to every other one.

Pro’s never-public proprietary codebases address this through legal inaccessibility. SWE-bench-Live takes the complementary route of drawing from issues created since 2024 and updating continuously, so release-time evaluation can use tasks that postdate training data. These approaches change the source of the evidence. That is necessary because even the best models perform materially worse on code they have never seen before.

How the benchmark family has responded to these problems, and what each response costs

The benchmark family’s development is an accounting of trade-offs. Verified’s human validation reduced noise from unsolvable and under-specified tasks, but did not remove contamination, harness inflation, or incomplete test oracles. When OpenAI checked 138 o3 failures by hand, it found that most were caused by flaws in the tests. It stopped reporting Verified scores and at first recommended Pro instead. In July 2026, it retracted that recommendation after finding approximately 30% of Pro’s tasks were broken.

Pro and Pro V2, with V2 released September 22, 2026, combine broader language coverage, private codebases, and standardized scaffolding. Those choices address contamination and harness comparability together. They also impose costs. Private code limits independent replication, while standardization can suppress a model’s best achievable performance. Pro Verified adds anti-hacking safeguards and task refinement, making clear that harder tasks and private repositories do not, by themselves, guarantee trustworthy grading.

Microsoft’s live coding benchmark, initially released in April 2025, tackles contamination and staleness through continuous automated updates using the RepoLaunch pipeline. SWE-rebench, which appeared at NeurIPS 2025, also targets contamination through automated task collection and evaluation. Freshness trades away a fixed comparison point. A Live score from one month is not directly comparable to a score from another, because the exam changes as well as the entrant.

The resulting direction is a portfolio of complementary evidence. Rolling benchmarks provide contamination resistance, harder multi-repository tasks provide a difficulty signal, and standardized harness protocols improve comparability. None of those substitutes for a complete correctness oracle. The proof-based approach discussed earlier helps fill that gap, though it costs more to put in place. Each response improves a different part of the measurement, so choosing one does not make the other problems disappear.

What SWE-Bench scores do and do not tell an engineering team making deployment decisions

A SWE-Bench score helps show how well an agent fixes bugs in structured problems covered by tests, but it says little beyond that. Verified is Python-only across its repositories, so its score does not establish ability in other languages. The benchmark also does not measure security vulnerability detection, code review quality, maintainability, compliance with organizational coding standards, or performance under production load. It also does not show how well an agent handles multi-file refactors and subtle logic that requires system-wide invariants. Those are separate engineering demands.

For a team choosing an agent, the lesson is to interpret the evaluation before interpreting the ranking. Which variant was used, and with what harness? Is the result self-reported, and does a proprietary scaffold account for much of the performance? What is the evaluation date, especially for a rolling benchmark? Has the model been tested on private code or code it has never seen before? Pro’s private-codebase subset is the closest available proxy for an organization’s own repositories, while standardized scaffolding makes comparisons more consistent.

A strong Pro or Pro Verified result therefore deserves attention, with the task-quality qualifications already described. An agent that performs well on private, contamination-resistant tasks under standardized scaffolding and reward-hacking controls has cleared the most rigorous publicly available bar. That is meaningful evidence. For organizations evaluating agents, Pro is the most reliable public signal available, but it should be supplemented with internal evaluation on a representative sample of their own codebase.

Before scaling automation, teams should also require visibility into agent behavior and execution. An open-source, model-agnostic platform that runs in their own infrastructure and exposes agent trajectories addresses a gap no leaderboard can fill. It makes the system being deployed inspectable rather than leaving its behavior hidden behind an aggregate score. The benchmark can help choose what to evaluate. Evidence from the organization’s own code, with control over execution, must inform what it trusts.