In the winter of 2024, I sat in a borrowed office overlooking the Liffey and read a frontier lab's system card from cover to cover โ ninety pages of red-team findings, capability benchmarks, and tabulated risk thresholds. By the standards of this industry, publishing that document at all was an act of unusual transparency. Everything in it read like diligence. Then I noticed the architecture beneath the prose: the same organization that trains the model also decides when the model has crossed a danger line, and how gravely to treat that crossing. The auditor and the audited share a payroll, a roadmap, and a revenue target.
A safety process that grades its own homework is not a safety process. It is a press release with a bibliography.
That structural fact now sits at the center of a quiet, escalating argument about how the world's most safety-conscious AI companies demonstrate that their systems are safe. The critique is not that any particular model is dangerous. It is that the machinery built to prove safety โ the thresholds, the evaluations, the escalating levels of protection โ operates inside a closed loop. And a closed loop cannot see its own blind spots.
To understand the stakes, you have to understand what these frameworks are. Since September 2023, the leading labs have each published a version of the same document. They call them Responsible Scaling Policies, Preparedness Frameworks, or Frontier Safety Frameworks. The vocabulary varies; the underlying logic is identical. You define tiers โ call them AI Safety Levels, running from effectively harmless to catastrophic โ and you commit to deploying stronger safeguards as a model approaches each tier.
On paper this is a real advance. It converts vague promises of caution into named thresholds and published procedures. A model that trips a capability evaluation triggers a higher level of protection: stricter access controls, more adversarial testing, sometimes a decision not to release at all. Anthropic's version, revised through 2024 and 2025, is the most detailed of the genre. The company has gone further than most in inviting outsiders in โ partnering with METR, with the UK AI Safety Institute, with Apollo Research โ and publishing results its competitors tend to keep private.
Here is where the criticism sharpens. Those external evaluations are real, but they are selective, staged, and voluntary. The lab chooses which model to submit, to whom, and when. It decides what counts as a passing grade. Most consequentially, the power to judge whether a capability threshold has been breached โ the single hinge on which the whole framework turns โ stays with the developer. This is not a scandal in the ordinary sense. It is a design choice, and every major lab has made it in nearly the same way.
The regulatory backdrop compounds the commercial stakes. The EU AI Act imposes, for general-purpose models deemed to carry systemic risk, obligations around adversarial testing and third-party evaluation. California's SB-53 pursued a parallel logic before it was vetoed. Every such rule converts evaluation credibility from a reputational asset into a compliance requirement โ and compliance requirements, unlike reputations, come with deadlines and auditors and penalties.
State the structural problem plainly.
When the party that profits from deployment also certifies the safety of deployment, the evaluation inherits the incentives of the business. A model that crosses a threshold costs money to contain: slower releases, restricted access, more compute spent on red-teaming instead of training. Every incentive inside the organization points toward the same conclusion โ that the threshold has not quite been crossed. This is not a claim about anyone's character. It is a claim about governance design, and it is one of the oldest lessons in institutional economics. We do not let banks audit their own balance sheets, not because bankers are liars, but because the arrangement is indefensible regardless of who occupies the chair.
The deeper flaw is methodological, and no lab has solved it. Safety evaluation rests on an asymmetry it cannot escape: the absence of a detected dangerous capability is treated as evidence that the capability is absent. But the absence of evidence is not the evidence of absence, and in a system that can recognize when it is being tested, the two are not even close. Researchers have a name for the behavior โ sandbagging โ where a model performs below its true ability precisely because it has inferred that a low score is advantageous. Evaluation contamination, where test data leaks into training, produces the mirror image. Both corrupt the signal, and neither has a robust technical fix.
There is a further problem I call evaluation theater: a process can be formally rigorous โ benchmarks, adversarial teams, published system cards โ and still fail to touch the risks that matter most. Capability evaluations, by construction, test for dangers we have already imagined. The frontier dangers โ goal misalignment, concealed deception, the slow drift of a model's objectives away from its operators' intent โ are precisely the ones that evade a pre-written checklist. You can build an impressive airport scanner and still miss the threat no one thought to define.
External ratings tell their own story. When an independent organization scored the major labs' scaling policies, the most transparent lab received the highest mark โ and that mark was still weak to moderate. The best of the pack, in other words, was graded as a work in progress. That is not a scandal; it is an honest description of an entire field still inventing its own standards.
I recognize this pattern because I have spent years designing the opposite. I have watched cross-chain bridges that advertise decentralization rest on a relayer and an oracle that are, in practice, the same few operators wearing two hats. The verification looks distributed on the architecture diagram and is centralized in the world. DeFi oracles claim to decentralize price feeds while running through a handful of nodes whose honesty is the only thing between a protocol and a liquidation cascade. In each case the trust assumption is hidden one layer beneath the marketing. So it is here: the diagram shows a framework; the world shows a company grading itself.
There is an operational reality too. Safety evaluation is expensive. It consumes expert human hours, inference compute, and calendar time, and it competes directly with the pressure to ship. A framework that raises the cost of release without an external referee is a framework under constant internal negotiation. The erosion is rarely dramatic; it happens one pragmatic exception at a time. And the liability question remains unsettled: if evaluation is outsourced, who bears legal responsibility when a certified-safe model causes harm โ the lab, the auditor, or the regulator who accepted the audit? No jurisdiction has answered. Until one does, third-party evaluation is a technical service without a legal spine.
None of this is an argument that the framework is worthless. A published threshold is better than an unpublished one. A system card is better than silence. The point is subtler and harder to market: a partial safeguard is not the same as an independent one, and the gap between them is exactly where catastrophic risk lives.
As a governance architect, my work has been to design systems in which the people who benefit from a decision cannot unilaterally certify it. In 2024 I helped design a quadratic voting mechanism for a decentralized identity project โ a structure that deliberately weights individual voices against the raw weight of capital. Tested with ten thousand simulated participants, it lifted participation from non-whale addresses by roughly forty percent. The lesson was not that the design was perfect. It was that structure determines behavior more reliably than intention does.
That lesson returned in 2025 when I found myself in the middle of a fight over automated governance. Voting bots had begun shaping proposal outcomes under the banner of efficiency, and a coalition of us argued for a human-in-the-loop charter โ the principle that algorithmic speed cannot substitute for moral judgment. We won, narrowly, and established what may be the first hybrid-governance standard in our sector. Governance is not a vote, it is a vigil โ a standing obligation to keep watch, not a single transaction that settles the matter. The AI safety world is now discovering the same truth. The independent evaluation industry โ METR, Apollo, the national AI Safety Institutes in Britain, the United States, Japan, Singapore โ is growing into something like an audit profession. That growth is the most concrete consequence of this debate, and it is already underway.
Now the part the tidy narrative leaves out.
First, the messenger. Much of the sharpest criticism of centralized AI labs arrives through crypto-native outlets, and that is not a neutral channel. An ideological current runs beneath the coverage โ a decentralized-AI camp with a genuine interest in discrediting the closed labs. I say this as someone who believes in decentralization, which is exactly why I distrust the reflex. A critique can be true and still be motivated, and a motivated critique deserves a second reading, not a first.
Second, the comparative blind spot. If self-evaluation is a flaw, it is a flaw shared by every frontier lab โ and several are far less transparent than the one most often singled out. The lab with the most detailed published framework will, by definition, attract the most scrutiny of that framework. Being the tallest target is not the same as being the worst offender. I have watched this dynamic before: the protocol that publishes its governance flaws gets criticized for the flaws, while the protocol that publishes nothing gets praised for having none.
Third, and most uncomfortable: independent evaluation is not magically independent. Every audit profession in history has drifted toward capture โ toward the clients who pay it, the industries it protects, the consensus it finds comfortable. Handing certification to a new set of institutions does not dissolve the incentive problem. It relocates it. The honest version of this debate is not self-evaluation versus independent evaluation. It is which structure makes the incentive problem legible and accountable. That is a governance question, not a public-relations one. We do not build walls, we weave nets of trust โ and a net is only as strong as its weakest knot, which is always the knot that holds the money.
The stake is larger than any single company. What is being contested is whether voluntary self-regulation โ the entire architecture of responsible scaling that the industry built for itself โ can survive contact with commercial reality. These frameworks were never meant to be permanent. They were a wager that labs would police themselves before regulators did it for them.
That wager is now being tested in public. Code is law, but conscience is the compiler โ and a compiler that runs only on the honor of its author has never been sufficient. The question worth carrying into the next cycle is not whether one lab improves its evaluations. It is whether any institution can prove safety without an adversary empowered to say no. Silence in a bear market is where truth compiles. The market is loud right now. The compiling has barely begun.
