In early September 2026, OpenAI announced that its new model, GPT-6 Astra, had scored 99.9% on ARC-AGI-3, a test built to measure reasoning on problems a model has not seen before. On the same day, the ARC Prize Foundation, which owns the test, published its own verified result for the same model: 62.7%.
Neither number is wrong. Both were produced on the same day, for the same model, on the same benchmark.
Same model, different set-up
The difference is the harness: the set-up a test is run in, including the instructions the model gets, the tools it may use and what it can keep between steps. Under the Foundation’s standard set-up, the model scored 62.7%. Under a provider-adapter set-up, which lets the model carry its internal working state between requests and condense long conversations, the reported results were between 98.6% and 99.9%, depending on settings. Independent reporting made the same point: the near-99% figure only reproduces under the provider’s own configuration, while the standard-harness result is about 62.7%.
Chapter 1 of How Far Can You Trust AI? works through the case. About 36 of the 37 points between the two headline numbers come from the harness, and about one from the setting that controls how long the model reasons. Within hours, comparison charts were circulating that placed the high figure beside other models’ standard-harness results as if they measured the same thing.
It is not just one test
A June 2026 analysis of SWE-bench, a coding benchmark, found that the agent “scaffolding” wrapped around a model can move scores by roughly 10 to 20 points without any change to the model. In its cleanest example, three agent systems running the same model on SWE-bench Pro scored between 50.2% and 55.4%. That analysis is a secondary source, so treat the exact figures with care, but the direction matches the ARC case: a benchmark score measures the model and its set-up together.
What the latest research adds
A July 2026 paper on arXiv by Mark Esposito and Liu Zhang, The Benchmark Ceiling, comes at the problem from the supply side. It argues that a benchmark score is a noisy stand-in for real quality, and that the signal weakens as models master easier items and as the hard items needed to tell frontier models apart become scarce and costly to write. It recommends procedural transparency and independent governance of benchmarks, rather than publishing live test items. The paper does not look at harness differences, so it complements the ARC story rather than repeating it.
Three questions to ask before you trust a score
- What set-up produced this number, and is it the same set-up used for the scores it is being compared with?
- What settings and what cost were involved, such as the reasoning effort and the total spend?
- Who ran it: the vendor, or someone independent who published their method?
What the book does with it
The book does not try to pick the “right” number. It treats the gap as the finding. Its divergence rule says that when two records of the same measurement differ by 20% or more of the larger value, the evidence grade of both drops one step until someone explains the gap. Here the gap was about 37%, well over the threshold. The vendor is not accused of anything; confidence in both numbers is simply lowered until the difference is explained, by a rule that gives the same answer however it is run.
Read Chapter 1 free
The free excerpt includes Chapter 1, “The Number That Was Two Numbers”.
Sources and notes
- How Far Can You Trust AI?, Edition 2.11, Chapter 1 (the book’s own reading of the ARC Prize Foundation’s results page for GPT-6 Astra, arcprize.org/blog/astra, read 19 September 2026).
- Independent benchmark roundup: Four Week MBA, GPT-6 Astra’s independent benchmarks.
- SWE-bench and scaffolding: Digital Applied, as at 16 June 2026.
- Esposito, M. and Zhang, L., The Benchmark Ceiling: Human Judgment, Evaluation Scarcity, and the Political Economy of AI Capability Measurement, arXiv 2607.01254.
Figures are as reported by these sources as at 22 October 2026, several of them through secondary reporting; check the originals before relying on them. This post is general information, not advice for your situation. It was drafted with the help of Claude, an AI system made by Anthropic, and reviewed by the author, who is responsible for the content.
