One question dominates the public conversation about AI governance: is it AGI yet? AGI means artificial general intelligence, AI as capable as people across the board. Chapter 2 of How Far Can You Trust AI? argues that the question is a governance dead end, because any set of rules keyed to the arrival of AGI will fire too late, or never.
A system that crossed every line without the label
The book’s example is the system from Chapter 1, the one with two scores. Its own vendor’s published risk framework rates it at the “Critical” tier for cybersecurity capability, as recorded in the book’s evidence records. Its system card, the document a lab publishes with a model, reports days-long tasks run unsupervised, and the ability to start and direct further AI agents by itself. It also reports lower “chain-of-thought monitorability” than its predecessor, meaning people can see less of what the model is doing and why. The vendor found and disclosed that.
Yet the same system fails the best-validated definition of general intelligence: on long-term memory, on physical interaction and on adapting to unfamiliar situations. A launch briefing closed with “Welcome to the AGI era” (as reported by Axios on 3 September 2026), but the vendor’s own safety publications make no such claim, and the owner of the benchmark says it is “not claiming that it is AGI”. No developer or regulator has formally classified it as AGI.
Three definitions, three failures
- The economic definition (outperforming people at most valuable work) is tied, in reporting, to a contractual trigger. It fails the self-attestation test: the claim comes from a party with a financial interest in its timing, like a student marking their own exam when a prize depends on the date they pass.
- The psychometric definition measures cognitive versatility the way psychologists measure people. It is sound, but it gives you nothing to act on. A system could score below the bar everywhere and still be the most dangerous thing ever shipped.
- The operational tests, such as the “coffee test” (can a robot make coffee in an unfamiliar kitchen?), measure embodiment, not risk. A burglar who cannot cook is no less able to pick your lock.
The question that works
The book replaces the definition with an operative question: what can this system already do, and what can we already prove about controlling it? It then adds one rule: a profile’s gates should rise with the system’s demonstrated capability in the matching area. In plain words, the more a system has shown it can do in one area, the more proof of control it must show in that area.
What that looks like
The book’s assessment of the Chapter 1 system finds a flat governance profile: every one of the twelve governance questions sits at an assured level of 2 or 3. Its capability profile is jagged: superhuman in formal mathematics, “Critical” in cyber, partial or absent in memory, embodiment and adaptation. The risk is the product of the two. In the evidence assembled on 7 September 2026, its containment, the ability to stop and undo a wrong action at machine speed, was described in documentation, with no record of independent exercise. The book likens it to a fire plan that exists on paper but has never had a drill.
Under the proposed capability-indexed gate, a system one tier above baseline on cyber capability would have its containment gate raised from 2 to 3. Its proven containment is 2, so it would be ineligible for critical-infrastructure control and for agentic automation. The book is explicit that this rule is a documented proposal and is not yet in its reference program.
Limits the book states
Capability tiers are self-assessed by the vendor, so they enter at the weakest usable evidence grade. A vendor that declines to publish a tier could lower its own gates. The book’s suggested remedy, to treat an undisclosed tier as the highest by its silence rule, is a policy choice it has not tested. And a capability-indexed gate only means something if the assessment was recorded before the capability was demonstrated.
Three questions to ask
- What can the system already do, area by area, and who says so?
- What has been independently shown about controlling it in each of those areas?
- When its capability goes up, does the required proof of control go up with it?
Read more, free
This article draws on Chapter 2, “The Wrong Question”. The free excerpt includes the Preface, Part 0, Chapters 1, 5 and 6 and the seven decision cards.
Sources and notes
- How Far Can You Trust AI?, Edition 2.11, Chapter 2 “The Wrong Question”, with the Chapter 1 system’s evidence as at 7 September 2026.
- Launch-briefing quotation: as reported by Axios, 3 September 2026, and cited in the book. The vendor’s tier is the vendor’s own, self-assessed, and was not re-read for this post.
Figures and quotations come from the book’s own assessment of the public record as at 7 September 2026; they grade the evidence, not the products. This post is general information, not advice for your situation. It was drafted with the help of Claude, an AI system made by Anthropic, and reviewed by the author, who is responsible for the content.
