Blog · Method · 3 November 2026

Why the top scorer can be the wrong choice: gates before averages

By Dr. Iain Sanders · about a 5-minute read

Gates before averages: the highest score is blocked by a gate, and the same evidence gives three different winners under three weightings

Most AI leaderboards and procurement scorecards add up weighted scores and stop. Chapter 8 of How Far Can You Trust AI? explains why that is risky. A control system with no containment is not “slightly worse”. It is unusable. But a weighted sum will still rank it second.

Gates first, weights second

The book borrows the logic of a tender, the formal invitation to bid for a contract: pass or fail on the mandatory requirements first, weighted scoring second. A gate is a minimum level on one question that a system must reach before its score counts at all. It works like the height line at a fairground ride: below it you do not ride, however good the rest.

Weights come from the use, and are published first

A single score is legitimate only on one condition, in the book’s words: the weights come from the application, not from the assessor, and are published before the scores. So the score is never “this model”. It is “this model, for this use”. Each profile carries a date and a digest, a fingerprint of its name, weights, gates and date. Change a weight and the digest changes. Publish the digest before you score, and nobody, including the author, can tune the answer afterwards. Five core profiles were published on 7 September 2026, for uses such as controlling critical infrastructure, choosing an enterprise assistant, or putting a model in front of the public.

The case that settles it

Under the book’s research profile, written for an evaluator or auditor, the highest floor among the eight assessed models, meaning the score backed by evidence, belonged to a system that was blocked. The profile gates transparency at level 3. That system’s transparency evidence was largely its own documentation, and the book’s self-attestation rule caps such evidence at level 2. A plain weighted sum would have picked it. A researcher relying on that pick would have been relying on documentation that only its author can verify.

Same evidence, different winner

The book also scores the same two worked examples, one closed model and one open model, under three published weightings. Under an infrastructure weighting the closed example wins clearly. Under a balanced weighting it wins by one point. Under a sovereignty-first weighting the open example wins. The evidence did not change. The ranking flipped. The book’s conclusion: any single-number ranking is a statement about the ranker’s priorities, not about the models.

Decisive, or only provisional?

The book counts a lead as decisive only if the leader’s proven score beats every other eligible rival’s claimed ceiling. Otherwise it is provisional: a runner-up could overtake simply by proving what it already claims, so the right action is to ask for the missing evidence before choosing. As at 7 September 2026, no lead in any of the book’s sixteen profiles was decisive.

Limits the book states

Three questions to ask

Read more, free

This article draws on Chapter 8, “Gates, Not Averages”. The free excerpt includes the Preface, Part 0, Chapters 1, 5 and 6 and the seven decision cards.

Get the free excerpt

Sources and notes

The assessments grade the public evidence as at 7 September 2026, not the products, and were not re-verified for this post. This post is general information, not advice for your situation. It was drafted with the help of Claude, an AI system made by Anthropic, and reviewed by the author, who is responsible for the content.

Dr. Iain Sanders

Dr. Iain Sanders is the author of How Far Can You Trust AI?. About the author · Subscribe for news · More articles