Most AI leaderboards and procurement scorecards add up weighted scores and stop. Chapter 8 of How Far Can You Trust AI? explains why that is risky. A control system with no containment is not “slightly worse”. It is unusable. But a weighted sum will still rank it second.
Gates first, weights second
The book borrows the logic of a tender, the formal invitation to bid for a contract: pass or fail on the mandatory requirements first, weighted scoring second. A gate is a minimum level on one question that a system must reach before its score counts at all. It works like the height line at a fairground ride: below it you do not ride, however good the rest.
Weights come from the use, and are published first
A single score is legitimate only on one condition, in the book’s words: the weights come from the application, not from the assessor, and are published before the scores. So the score is never “this model”. It is “this model, for this use”. Each profile carries a date and a digest, a fingerprint of its name, weights, gates and date. Change a weight and the digest changes. Publish the digest before you score, and nobody, including the author, can tune the answer afterwards. Five core profiles were published on 7 September 2026, for uses such as controlling critical infrastructure, choosing an enterprise assistant, or putting a model in front of the public.
The case that settles it
Under the book’s research profile, written for an evaluator or auditor, the highest floor among the eight assessed models, meaning the score backed by evidence, belonged to a system that was blocked. The profile gates transparency at level 3. That system’s transparency evidence was largely its own documentation, and the book’s self-attestation rule caps such evidence at level 2. A plain weighted sum would have picked it. A researcher relying on that pick would have been relying on documentation that only its author can verify.
Same evidence, different winner
The book also scores the same two worked examples, one closed model and one open model, under three published weightings. Under an infrastructure weighting the closed example wins clearly. Under a balanced weighting it wins by one point. Under a sovereignty-first weighting the open example wins. The evidence did not change. The ranking flipped. The book’s conclusion: any single-number ranking is a statement about the ranker’s priorities, not about the models.
Decisive, or only provisional?
The book counts a lead as decisive only if the leader’s proven score beats every other eligible rival’s claimed ceiling. Otherwise it is provisional: a runner-up could overtake simply by proving what it already claims, so the right action is to ask for the missing evidence before choosing. As at 7 September 2026, no lead in any of the book’s sixteen profiles was decisive.
Limits the book states
- Profiles are a judgement. The author wrote the critical-infrastructure profile after about thirty years in energy infrastructure and may have weighted it accordingly.
- Gates at level 2 are low. They are set where the field is, not where it should be.
- A profile’s digest proves that a profile with those weights existed. It does not prove that it existed before the assessment.
Three questions to ask
- What is the use, and who set the weights?
- Were the weights published before the scores were known?
- Are there pass-or-fail minimums, or can strength in one area buy past a weakness in another?
Read more, free
This article draws on Chapter 8, “Gates, Not Averages”. The free excerpt includes the Preface, Part 0, Chapters 1, 5 and 6 and the seven decision cards.
Sources and notes
- How Far Can You Trust AI?, Edition 2.11, Chapter 8 “Gates, Not Averages”, including the research-profile table and the three-weighting comparison (public evidence as at 7 September 2026).
The assessments grade the public evidence as at 7 September 2026, not the products, and were not re-verified for this post. This post is general information, not advice for your situation. It was drafted with the help of Claude, an AI system made by Anthropic, and reviewed by the author, who is responsible for the content.
