Imagine you are choosing an AI system for your organisation. You send two vendors the same question: can a member of the public appeal a decision your system makes about them?
Vendor A answers: “Yes, we have a full complaints process,” and describes how it works. Vendor B does not answer at all.
How should a scorecard treat vendor B? Most scorecards, ratings and audits do the same thing: they mark the row “not assessed”, or drop it, and work out the overall result from what is left. That sounds fair. It is not.
Why dropping the blank rewards secrecy
If the result is worked out only from the rows that are present, a vendor that reports a poor answer does worse than a vendor that says nothing. The best move for anyone with something to hide is to leave that question out.
The book makes the same point with a neutral score. Suppose silence scored a neutral 2 on a scale of 0 to 4. A vendor whose record would show a measured 0 could simply stay silent and be given a 2 instead. Saying nothing would be the winning strategy, and it would win most often for the vendors with the most to hide. The scorecard would end up selecting for opacity: the more it mattered, the worse it would work.
The rule: silence scores like failure
Chapter 6 of How Far Can You Trust AI? is called “The Silence Rule”. A question with no evidence at all scores 0 on the assured scale, exactly as a failed answer would. A missing value is never guessed, averaged or filled in from other vendors.
In the example, vendor A’s claim rests only on its own description of its process. The book treats a company’s evidence about itself as weak evidence, so A scores at most 2 on that question. That is much less than the brochure suggests, but a long way above B’s 0.
Two zeros that are not the same
The rule does not treat every zero alike. The book scores two things for each question: how well the answer is backed by evidence, and how wide the gap is between what is claimed and what is assured.
- A measured zero has a width of 0. It means: we looked, and it is not there.
- Silence has a width of 4, the maximum. It means: nobody has looked.
Both sit at the floor of the score, so silence can never beat a bad answer. But the scorecard still shows which one you are looking at.
What it showed in the book’s own assessment
The book assesses eight AI models using public evidence as at 7 September 2026. On one question, whether a person can contest a decision, five of the eight had no route that the assessors could find: no appeal route, no human review and no regulator complaint path. Under a scorecard that drops blank rows, those five would have been invisible on that question.
Under the silence rule, each of the five scores 0. For a public-facing use, the book sets a minimum of 2 on this question, so all five fall below it. The book’s roadmap classes the gap as the cheapest kind to close: not “prove what you claim” and not “build what you lack”, but simply publish something. This is a statement about the public record, not proof that no process exists.
Two limits, stated plainly
- The rule cannot tell “nobody looked” from “somebody looked and the vendor refuses to say”. Both count as absences. The book records a refusal separately, in a log of sources that were looked at and not used.
- It makes the author’s own assessments look worse, because many questions have no public evidence. That is deliberate, but it has a cost: a wide, mostly blank assessment can be dismissed as “not really an assessment”. It is one, of the state of the public record.
Three questions to ask before you trust any rating
- Which questions did the vendor not answer, and does the scorecard show those gaps, or quietly drop them?
- Where a row says “not applicable”, who decided that, and on what grounds?
- Is the overall figure an average over only the answered questions? If so, a vendor can raise its score by saying less.
What the book does with it
In the book’s reference program, the silence rule is the first check the scoring function makes, before any other rule is applied, so nothing later can route around it. The aim is a method that gives the same answer however and by whoever it is run, and that cannot be gamed by staying quiet.
Read Chapter 6 free
The free excerpt includes Chapter 6, “The Silence Rule”, in full.
Sources and notes
- How Far Can You Trust AI?, Edition 2.11, Chapter 6 “The Silence Rule”, and the contestability column of the eight-model assessment in Chapter 4 (public evidence as at 7 September 2026).
- The vendors A and B above are an illustration, not real companies.
The figures about the eight models come from the book’s own assessment of the public record as at 7 September 2026; they grade the evidence, not the products. This post is general information, not advice for your situation. It was drafted with the help of Claude, an AI system made by Anthropic, and reviewed by the author, who is responsible for the content.
