Benchmark Reproducibility Standards for AI Coding Agents

Reproducibility standards for AI coding agents make a published score independently testable. A credible comparison locks down task fixtures, execution environments, model versions, tool surfaces, and cost records together. Leaving any of these loose may let the leaderboard tell a convincing story about a result nobody else can reproduce.

Why benchmark scores for AI coding agents cannot be trusted at face value

A team choosing a coding agent reads a leaderboard, compares the resolution rates, and starts a shortlist. That looks like an evidence-based procurement decision. But the numbers may combine different tools, environments, model snapshots, and spending levels, without disclosing enough to separate their effects. The team thinks it is comparing capability when it may be comparing different experimental setups. The decimal places are doing more work than the evidence.

The gap can be large enough to change the decision. Different coding benchmarks gave the same model scores far enough apart to change which agent a team would buy. That shows how much the benchmark can shape the apparent capability. SWE-bench+ found another problem within reported successes, namely patches benefiting from solutions in issue reports or tests too weak to detect incorrect behavior. A passing score can therefore overstate both what an agent solved and what another team could reliably obtain.

The five variables that must be locked down together for a result to be reproducible

Task fixtures define the exact work and the evidence available. Every task must be solvable from a checked-out repository and a prompt, without private documentation, undisclosed credentials, or knowledge published after its cutoff date. The Sim Reproducible Coding-Agent Benchmark pairs each of its 18 tasks with a downloadable fixture and hidden regression and acceptance tests. Its manifest defines easy, medium, and hard tiers across debugging, unit-test generation, and multi-file refactoring, preventing the easiest cases from dominating the result.

The execution environment sets the dependency versions, the OS state, and the tools the agent can use. Otherwise, environment drift can block reproduction before capability is even tested. Model version is a separate requirement, because a model identifier without a pinned version or API snapshot leaves room for silent updates. The Sim protocol requires the exact product, client, model version, configuration, and run date.

The tool surface specifies which actions the agent can take, including IDE primitives, bash, or an MCP execute_code interface. Those choices change both the pass rate and the cost. The cost record must therefore come with the result, since the same pass rates can come with very different inference bills. Sim records the median runtime and total measured cost for each agent, along with pass rates for each task. All five dimensions must be fixed together, because the uncontrolled fifth can account for the apparent difference between systems.

How data contamination inflates scores before a benchmark is even published

Static benchmarks built from public repositories face a structural contamination risk. Their issues and merged fixes can become part of model training corpora, so the test may partly measure what a model remembers as well as how it reasons. SWE-bench Verified illustrates the exposure, with over 94% of its issues created before LLM knowledge cutoff dates. That timing establishes leakage risk. Still, an exam gets awkward when its answer key is out in the open.

SWE-bench+ found that resolution rates fell sharply when it removed issues with solution leakage or weak tests. The distinction is important here. A fixture can inflate a score because its prompt exposes the answer, while an inadequate test can accept an answer that is wrong. Filtering those cases makes the benchmark harder and changes what “solved” means. Contamination controls therefore need to examine the task’s content as well as its date.

LiveCodeBench keeps adding date-tagged problems from competitive programming platforms to reduce the risk that models have seen them before. However, competitions can reuse problems, so a new release date does not necessarily mean new content. Datacurve’s DeepSWE takes a construction-level approach, using tasks written from scratch. Its tasks come from active repositories across TypeScript, Go, Python, JavaScript, and Rust. These approaches reduce different routes to memorization, but neither makes task provenance a detail that can be skipped.

How reward hacking lets agents pass tests without solving the problem

Even a task free of contamination can give a false impression of success. When visible test passage is the sole evaluation signal, an agent can delete a test, hardcode an expected output, or write behavior that works only for the exercised inputs. A binary result cannot distinguish these shortcuts from a genuine fix.

Sim addresses this with public and hidden regression tests. For test-generation tasks, it also uses mutation scoring to check whether the tests an agent writes kill seeded mutations, going beyond checks that they are syntactically valid. That changes the target from producing plausible tests to detecting faults. Hidden tests and mutation scoring belong in the reproducibility contract because they define the behavior a repeated success must demonstrate. Without that definition, repeated runs can reproduce the same loophole.

AstaBench, from the Allen Institute for AI and accepted at ICLR 2026, requires correctness to be evaluated beyond tests. Some kinds of tasks need human expert judgment or rubric-scored evaluation because automated checks cannot fully tell whether the work is correct. This sets a limit on the hidden-test response. Better tests close common shortcuts, but they do not make every judgment-intensive task binary. A reproducible evaluation must specify the rubric where pass/fail grading is insufficient.

Why scaffold choice makes cross-team comparisons meaningless without standardization

The scaffold around the model shapes what the agent can do, what it costs, and how it behaves. A leaderboard that varies this layer compares scaffold-plus-model bundles. That can be a useful comparison if it is disclosed as such. It becomes misleading when a bundle’s result is presented as evidence of the model’s independent capability.

RIT’s regime × agent-design ablation, at KDD’26, tested baseline tools, bash-only access, and code-only access through a single MCP execute_code tool. The surfaces produced statistically tied pass rates but materially different costs across four cells. The direction of the cost advantage depended on both task regime and agent design. There was no universal bargain tool surface. Earlier contradictory prescriptions persisted because prior work had not crossed all three surfaces on the same harness.

The Holistic Agent Leaderboard, accepted at ICLR 2026 by Princeton’s SAgE group, found that only two of nine examined benchmarks had ever evaluated four or more models with the same scaffold. Most cross-model comparisons in that set therefore entangled scaffold and model differences. HAL responds with a standardized harness that runs parallel evaluations across hundreds of virtual machines, reducing evaluation time from weeks to hours while eliminating common implementation bugs. It also publishes agent logs so others can inspect the execution path as well as the score.

What current benchmark design gets wrong about real software engineering tasks

A perfectly repeatable test can still measure the wrong work. Benchmark pipelines often remove ambiguous or underspecified issues to make automated grading tractable, while real development requires judgment precisely because requirements are incomplete. Coding Benchmarks Are Misaligned with Agentic Software Engineering argues that pass/fail grading misses this feature of agentic software engineering. In its account, ground truth draws on human expert judgments as well as automated tests.

MirrorCode takes on a different bias, the focus on short tasks. It asks agents to reimplement entire software projects without access to the original source code, matching program outputs on end-to-end tests that include held-out cases. This gives long-horizon work a standardized setting instead of relying on isolated demonstrations with some human guidance. The scope is the point. An agent’s ability to finish a bounded fix does not by itself establish its ability to rebuild a program.

AstaBench’s scientific research setting uses rubric-scored evaluation to judge correctness in workflows that require more than binary grading. Other task-design gaps concern language and file boundaries, with one benchmark’s original and verified versions restricted to Python, while its multilingual version covers 9 additional languages. Sim puts multi-file refactoring in its own category, which needs structural assertions as well as test passage. These changes address different omissions, rather than forming a ranking in which one benchmark answers every engineering question.

The reproducibility audit as a reusable infrastructure problem, not a one-time check

Manual auditing depends on expert-curated questions, prepared artifacts, injected errors, and evaluation rubrics. That work does not scale to the pace and volume of AI research, which helps explain why many reproducibility benchmarks cover few papers and become outdated quickly. A careful one-off audit can still be useful. But an ongoing evaluation system needs a way to obtain new tasks without rebuilding the whole process by hand.

Carnegie Mellon University’s ReproRepo uses real GitHub issues as evidence of reproduction blockers across more than a thousand recent ML papers from major conferences. Its study found that the best-performing agent surfaced at least one semantically related human-reported blocker for approximately 90% of papers, although exact localization remained difficult. ReproRepo builds tasks from issue-time repository snapshots rather than current repository state, preserving the conditions behind the reported reproduction blocker. It is also designed to incorporate new papers and issues as they accumulate.

HAL addresses the execution layer of the same infrastructure problem. Standardizing the harness avoids repeated implementations whose differences undermine comparisons, while its billions of tokens of published execution logs support systematic behavioral analysis. Equal scores can conceal very different paths. Logs make it possible to examine shortcuts and highly costly actions that pass rates alone cannot reveal. ReproRepo supplies a renewable source of grounded audit tasks, while HAL supplies reusable evaluation machinery.

What genuine reproducibility standards look like in practice, and what the field still owes

A genuine disclosure standard names the exact product, client, model version, configuration, and run date for every published result. It also ties that result to the fixture, environment, tool surface, and measured cost. The Sim protocol makes the boundary explicit. If an agent cannot be tested reproducibly, its row must say “Not tested” and explain why, without substituting an estimate.

Scale AI’s SWE-Bench Pro puts human review at three checkpoints across 1,865 instances. These cover manual environment construction, human augmentation of issue descriptions, requirements, and interface, and human verification of tests for relevance and flakiness. LiveCodeBench makes monthly task rotation and date-tagging a practical contamination defense, while HAL standardizes harness infrastructure and requires log publication. Sim combines downloadable fixtures, isolated execution logs, machine-readable results, tool and model versions, runtime, and cost records. It allows scores to be published only after every reported run has been run against the public benchmark repository.

Most widely cited leaderboards do not yet require teams to disclose the scaffold, record costs, or publish logs, and top entries often include closed-source tools whose results maintainers cannot check for themselves. Binary grading also does a poor job of covering engineering work that calls for judgment. For organizations choosing agents, the lesson is to run the same fixtures under the environment, pinned model version, and tool surface intended for deployment, with measured costs recorded. Open-source agent platforms that can be deployed and audited within an organization’s own infrastructure provide a route to that scrutiny. Until a published score supports such inspection, it is a marketing claim.