Evaluating Coding Agent Benchmarks

Benchmarks are a proxy. They are not the thing itself. Every benchmark score you read is actually the answer to a very specific question, and if that question doesn't match the one you're trying to answer, the score is noise at best and misleading at worst.

Here's the core argument: coding agent benchmarks have gone through three distinct generations. Function-level correctness came first. Then repository-level patch generation. Now we're watching end-to-end project development benchmarks emerge. Each generation was designed to fix a blind spot in the one before it. Understanding that lineage is the only way to interpret a score honestly.

The benchmark a team chooses also shapes what they believe agents can do. Choose the wrong one and you'll make real decisions based on imaginary capability. That's not a hypothetical risk. It's happening right now, in procurement decisions and build-vs-buy calls and hiring plans.

So let's go through each generation, what it gets right, what it gets wrong, and what you should actually do with any of it.

What Function-Level Benchmarks Measured and Why They Stopped Being Useful

Two benchmarks defined this era. HumanEval, from OpenAI in 2021, gave models 164 handwritten Python problems. Each one came with a function signature, a docstring, and a body. The model had to complete the function. Success was measured by running test cases, averaging about 7.7 per problem.

MBPP, from Google the same year, took a broader approach. Roughly 1,000 crowd-sourced entry-level Python problems. More coverage of the standard library. Similarly isolated in structure, with an average of about 3 test cases per problem.

These benchmarks were genuinely useful for their time. Reproducible. Controlled. You could run them across many models and get a clean comparison. That was the design intent, and it worked.

Then the scores caught up to the benchmark.

Top pass@1 rates on both now cluster in the 90–95% range. That saturation is the problem. When every frontier model is scoring within a few points of every other frontier model, the benchmark has stopped discriminating. It can't tell you which model to pick. That's not evidence that coding is solved. It's evidence that the test is too easy for the students taking it — like giving calculus students an arithmetic quiz and calling it a placement exam.

The structural problem runs deeper than saturation, though. Real engineering work is never a clean function signature handed to you in a vacuum. There's no repository context. No multi-file dependencies. No ambiguous requirements written by a product manager at 11pm. No iterative debugging when the first attempt fails.

High scores on HumanEval don't predict performance on repository-level tasks. The skills being tested are largely different. Knowing that your model scores well here tells you something narrow and increasingly irrelevant.

So why do these benchmarks still appear in product marketing? Three reasons: they're easy to run, easy to cite, and the scores are high. None of those reasons have anything to do with actual capability. That's worth remembering every time you see one of these numbers on a landing page.

What SWE-Bench Introduced and Why It Became the De Facto Standard

SWE-bench landed in November 2023, from researchers at Princeton, Stanford, and the University of Chicago. The premise was straightforward: stop giving models toy problems and start giving them real work.

The task structure was a genuine departure. The model receives a real GitHub issue description plus the full repository code. It must produce a diff that resolves the issue. Success is measured by whether the existing test suite passes after the patch is applied. Tests that were written independently of the agent, before the agent existed, by humans who had no idea a benchmark was coming.

The dataset: 2,294 real issues drawn from 12 popular Python repositories. Django, SymPy, matplotlib, scikit-learn, pytest, requests, Flask, and several others. These are production codebases, not tutorial projects.

What this required that function-level benchmarks didn't:

• Navigating a large, unfamiliar codebase to find the relevant code in the first place
• Interpreting ambiguous, human-written issue descriptions that may be incomplete, misleading, or assume context the model doesn't have
• Producing a minimal, targeted patch rather than rewriting from scratch
• Passing tests written by someone else, with no knowledge of how the model would approach the solution

That last point is important. When a model writes a function and the benchmark tests it with predetermined cases, there's a certain kind of cheating available. You can overfit to the test. SWE-bench's test cases were written before any agent ever saw the problem. That's a meaningfully harder constraint.

In August 2024, OpenAI released SWE-bench Verified. Expert software engineers reviewed 1,699 problems, each independently evaluated by three experts, and the result was a curated set of 500 problems with flawed tasks removed. The fact that curation was necessary at all was an early signal. The original set had quality problems. That signal would matter later.

SWE-bench became the de facto standard because it was the first benchmark that required agents to behave like junior engineers on real work, not students on homework. The field rallied around it fast.

That rapid adoption is also what broke it.

Why SWE-Bench Verified Was Retired and What That Reveals About Benchmark Design

On February 23, 2026, OpenAI's Frontier Evals team announced the retirement of SWE-bench Verified. Not a quiet deprecation. A public statement recommending that others stop reporting these scores too.

Two failures brought it down.

Failure one: flawed test design. OpenAI audited the tasks that models most often failed. They found that at least 59.4% of the audited problems have flawed test cases. These tests reject functionally correct solutions. So a meaningful share of recorded "failures" were never real failures. The benchmark was punishing correct work.

The 2024 Verified curation process was supposed to catch this. It didn't catch enough.

Failure two: training data contamination. SWE-bench draws from open-source repositories. Frontier models train on open-source repositories. The overlap was catastrophic. Models could reproduce the exact human-written gold-patch solution using only the task ID as a prompt. GPT-5.2's chain-of-thought revealed that it knew a specific argument name required by the test, an argument name that was never mentioned anywhere in the problem description. That's not reasoning. That's memorization.

Multiple frontier models demonstrated verbatim recall of reference solutions. The benchmark had become a test of what the model had seen during training, not what it could figure out.

The slowdown in score improvement was another tell. Scores moved from 74.9% to 80.9% in the six months before retirement. That's a stall. When a benchmark starts compressing scores near the top, it usually means one of two things: the models are genuinely near the ceiling of what's testable, or the benchmark is running out of signal. In this case, it was the latter.

The deeper lesson here isn't specific to SWE-bench. It's a structural risk for any static benchmark built from a public corpus. At some point, that corpus becomes training data. When it does, the benchmark stops measuring capability and starts measuring memory — like asking someone to recite a map rather than navigate unfamiliar terrain. The larger the model, the worse the contamination problem gets. This will happen again with other benchmarks if the field doesn't design around it.

The Scaffold Gap That Benchmark Scores Routinely Hide

Here's something that doesn't get talked about enough, partly because it's inconvenient for vendors and partly because it requires reading methodology sections.

There are two families of benchmark scores, and they are not comparable.

Vendor-scaffold scores: the model provider runs the benchmark using their own tuned agent harness, optimized for that benchmark's specific task format.

Standardized scores: an independent party runs every model through identical scaffolding, the same retrieval setup, the same tool-use framework, the same prompting structure.

The gap between those two families is 10 to 30 percentage points. Most of it comes from context retrieval and tool-use quality, not from anything inherent to the model itself.

A mid-tier model with a carefully engineered scaffold routinely outscores a frontier model run naively. Two runs of the exact same base model can differ by double digits depending on the harness. The number you see on a leaderboard reflects the model-plus-scaffold system. It does not reflect the model alone.

When a vendor publishes a benchmark score, the first question isn't "how good is the model?" The question is: who ran this, and what harness did they use? The methodology is the message.

This problem doesn't disappear with better benchmarks. It requires standardized evaluation infrastructure to control for. Until that's in place, comparing scores across vendors is like comparing race times run on entirely different tracks — the number means something, it just doesn't mean what most people assume.

What SWE-Bench Pro Changed and What It Still Doesn't Fix

Scale AI released SWE-bench Pro in September 2025. The design choices directly address the failures that retired SWE-bench Verified.

The dataset: 1,865 real-world software tasks across 41 professional repositories, scored Pass@1. The difficulty jump is immediate. The same frontier models clearing 80–95% on SWE-bench Verified solve roughly 59% under standardized scaffolding here. That's a meaningful spread. The benchmark is discriminating again.

The architecture is a three-way structure designed to prevent gaming:

• Public Set (731 tasks): openly released for evaluation and publication; uses GPL-licensed repositories to deter training data inclusion
• Held-Out Set (858 tasks): not publicly released; used to detect overfitting between public evaluation rounds
• Commercial Set (276 tasks): drawn from private startup codebases; tests generalization to genuinely unseen, proprietary code

As of June 2026, GPT-5.4 at its highest compute setting leads at 59.1% on Scale's public set under standardized scaffolding. The top vendor-aggregate score is Claude Opus 4.8 at 69.2%. The gap between those two figures is the scaffold gap in practice, visible in a single comparison.

What SWE-bench Pro gets right is real. Copyleft and private licensing block training data inclusion. Harder tasks, most requiring multi-file changes, resist easy memorization. The held-out set catches benchmark-specific overfitting before it compounds.

What it still doesn't address:

• Language diversity: still primarily single-language at its core; most enterprise codebases aren't
• Still patch-based: measures editing within a bounded repository; doesn't test building something new from a specification
• Code quality: a patch can pass every test and still be something no senior engineer would want to maintain
• Cost: running 1,865 tasks at full agent scaffolding is expensive enough to exclude smaller teams from running independent evaluations

Those aren't minor quibbles. They're structural gaps that the next generation of benchmarks will need to address.

The Dimensions No Current Benchmark Reliably Measures

Let's be direct about what's missing.

Language diversity. The original SWE-bench's 12 Python repositories are a narrow slice of how software actually gets built. Multi-SWE-bench, from ByteDance Seed in April 2025, makes a real effort here: 2,132 curated instances across 8 languages including Python, Java, TypeScript, JavaScript, Go, Rust, C, and C++, built by 68 expert annotators. It was accepted to NeurIPS 2025's Datasets and Benchmarks track. The open empirical question is whether strong Python performance generalizes to Go or Rust. Different paradigms, different idiomatic patterns, different runtime behaviors. That's a genuine research question, not an assumption.

Code quality. A model can pass every test case with code a human engineer would flatly refuse to review. No current widely-adopted benchmark penalizes for unmaintainable solutions, poor naming, or architectural choices that create future technical debt. We're measuring whether the tests pass, not whether the code is good. Those are different things.

Scale and context. Real enterprise codebases run to millions of lines. Finding the relevant file, tracing dependencies, assembling context before writing a single line of patch. That's a different kind of challenge. LongCodeBench pushes context well beyond the roughly 50,000-token limit of standard SWE-bench, with samples up to a million tokens. That's closer to the real problem.

Iterative development. Most benchmarks test one-pass generation. Real development involves debugging, retrying, and revising. An agent that scores well on single-shot tasks may behave very differently when it has a feedback loop and needs to respond to failure.

From-scratch construction. Patch benchmarks test editing. They don't test building. ProjDevBench, evaluating across coding agents in 2026, finds the best overall performer at 77.85%, with performance gaps widening specifically on from-scratch construction tasks. About 42% of failures traced to wrong answers and 14% to time limit violations. Specification alignment, edge case handling, and resource management emerge as distinct failure modes that patch benchmarks never surface, because patch benchmarks never ask the question.

How Emerging Benchmarks Are Trying to Close Specific Gaps

Each new benchmark is essentially an argument about what the previous ones got wrong. That's a healthy dynamic, even if no single benchmark has solved the whole problem.

Anti-memorization through freshness. Datacurve's DeepSWE uses 113 original tasks authored from scratch across 91 active open-source repositories in five languages. Because the tasks are new, nothing has been memorized. The effect is visible: where SWE-bench Verified compressed the field near saturation, DeepSWE still spreads scores from high percentages down to single digits. It's still discriminating at the frontier. That's exactly what a useful benchmark needs to do.

Automated pipeline construction. SWE-bench++ converts GitHub pull requests into reproducible tasks through programmatic sourcing, environment synthesis, test oracle extraction, and quality assurance. This makes large-scale generation feasible without requiring manual curation at every step. It also converts hard instances into training data through hint-guided trajectory synthesis, which is useful beyond evaluation alone.

Extended context. LongCodeBench introduces debugging tasks with context up to a million tokens. The question it asks is whether agents can reason over an entire large codebase, not just a retrieved snippet. That's a qualitatively different challenge.

What no current benchmark handles well:

• Code review quality as an output (some work is happening here, but nothing widely adopted)
• Long-horizon, multi-session tasks that mirror how engineers actually work across days and handoffs
• Security and correctness under adversarial inputs

The pattern is consistent across all of these efforts. Each closes one or two gaps and opens others. There is no single benchmark that covers the full surface of real software engineering work. That's not a failure of execution. It's a reflection of how broad and contextual real software engineering actually is.

A Practical Framework for Choosing the Right Benchmark for the Job

The first question isn't "which benchmark scores highest?" The first question is "what decision am I actually trying to make?"

That reframe changes everything.

If you're evaluating models for a specific codebase type, language diversity matters more than overall pass rate. A benchmark that doesn't include your language stack is telling you almost nothing. Multi-SWE-bench is more relevant than SWE-bench for polyglot shops. DeepSWE is more relevant if contamination is a concern.

If you're comparing vendors, the scaffold gap is your biggest enemy. Vendor-reported scores reflect their optimized harness, not a neutral environment. Push for standardized scores, or run your own evaluation on a held-out set of tasks from your actual codebase. Anything else is comparing apples and guided missiles.

If you're assessing patch-level capability, SWE-bench Pro under standardized scaffolding is currently the most honest signal available at scale. The held-out and commercial sets exist precisely to catch gaming. Use the Scale SEAL scores, not the vendor's self-reported numbers.

If you're building something new, not maintaining something existing, patch benchmarks are the wrong tool entirely. ProjDevBench and WebGen-Bench are more relevant. They test different capabilities. An agent that's great at editing existing code may be mediocre at generating new systems from specifications. That distinction matters enormously for from-scratch development work.

If code quality matters to your team, currently no benchmark will tell you what you need to know. You'll need human review, internal rubrics, or proxy metrics like test coverage and static analysis scores. Benchmark scores measure correctness. They don't measure craftsmanship.

A few practical rules of thumb:

• Always ask who ran the evaluation and with what harness. The methodology is the message.
• Treat any score above 85% on a benchmark as a saturation warning. It means the benchmark probably can't differentiate between frontier systems anymore.
• Look for benchmarks with held-out sets. Public-only benchmarks get gamed. It's not malicious; it's incentive physics.
• Weight freshness. A benchmark built from tasks that postdate a model's training cutoff is worth more than one built from data the model has probably seen.

The honest summary is this: no current benchmark reliably tells you whether an agent can do your specific job. What benchmarks can tell you is roughly how capable a system is at a well-defined class of tasks, in controlled conditions, with specific scaffolding. That's genuinely useful information. Just don't mistake it for more than it is.

The generation of benchmarks that's emerging right now is more rigorous, more contamination-resistant, and harder to game than what came before. That's real progress. But the gap between "passes the benchmark" and "works in your codebase" is still wide. Closing that gap is your job, not the benchmark's.