Blog · Research & news · 22 October 2026

Two numbers, one model: what a benchmark score doesn’t tell you

By Dr. Iain Sanders · about a 5-minute read

Two scores for the same model on the same day: 62.7% under the standard harness and 99.9% under the provider adapter harness

In early September 2026, OpenAI announced that its new model, GPT-6 Astra, had scored 99.9% on ARC-AGI-3, a test built to measure reasoning on problems a model has not seen before. On the same day, the ARC Prize Foundation, which owns the test, published its own verified result for the same model: 62.7%.

Neither number is wrong. Both were produced on the same day, for the same model, on the same benchmark.

Same model, different set-up

The difference is the harness: the set-up a test is run in, including the instructions the model gets, the tools it may use and what it can keep between steps. Under the Foundation’s standard set-up, the model scored 62.7%. Under a provider-adapter set-up, which lets the model carry its internal working state between requests and condense long conversations, the reported results were between 98.6% and 99.9%, depending on settings. Independent reporting made the same point: the near-99% figure only reproduces under the provider’s own configuration, while the standard-harness result is about 62.7%.

Chapter 1 of How Far Can You Trust AI? works through the case. About 36 of the 37 points between the two headline numbers come from the harness, and about one from the setting that controls how long the model reasons. Within hours, comparison charts were circulating that placed the high figure beside other models’ standard-harness results as if they measured the same thing.

It is not just one test

A June 2026 analysis of SWE-bench, a coding benchmark, found that the agent “scaffolding” wrapped around a model can move scores by roughly 10 to 20 points without any change to the model. In its cleanest example, three agent systems running the same model on SWE-bench Pro scored between 50.2% and 55.4%. That analysis is a secondary source, so treat the exact figures with care, but the direction matches the ARC case: a benchmark score measures the model and its set-up together.

What the latest research adds

A July 2026 paper on arXiv by Mark Esposito and Liu Zhang, The Benchmark Ceiling, comes at the problem from the supply side. It argues that a benchmark score is a noisy stand-in for real quality, and that the signal weakens as models master easier items and as the hard items needed to tell frontier models apart become scarce and costly to write. It recommends procedural transparency and independent governance of benchmarks, rather than publishing live test items. The paper does not look at harness differences, so it complements the ARC story rather than repeating it.

Three questions to ask before you trust a score

What the book does with it

The book does not try to pick the “right” number. It treats the gap as the finding. Its divergence rule says that when two records of the same measurement differ by 20% or more of the larger value, the evidence grade of both drops one step until someone explains the gap. Here the gap was about 37%, well over the threshold. The vendor is not accused of anything; confidence in both numbers is simply lowered until the difference is explained, by a rule that gives the same answer however it is run.

Read Chapter 1 free

The free excerpt includes Chapter 1, “The Number That Was Two Numbers”.

Get the free excerpt

Sources and notes

Figures are as reported by these sources as at 22 October 2026, several of them through secondary reporting; check the originals before relying on them. This post is general information, not advice for your situation. It was drafted with the help of Claude, an AI system made by Anthropic, and reviewed by the author, who is responsible for the content.

Dr. Iain Sanders

Dr. Iain Sanders is the author of How Far Can You Trust AI?. About the author · Subscribe for news · More articles