Our Claims Can Break the Build

How a number normally goes stale

A site publishes a figure. The underlying facts move, or the original calculation turns out to have been wrong, and the text stays exactly where it is. Nothing breaks, because text does not break. It simply becomes untrue, quietly, and usually nobody notices until a reader does.

That is the ordinary failure mode of every published claim on the internet, and it is not caused by dishonesty. It is caused by the fact that prose has no mechanism for noticing that it has stopped being right.

What we do instead

Every engine-checkable figure we publish is registered rather than typed, and the registry entry names the measurement that produced it. A test re-runs that measurement on every build and compares the result to the published figure.

If they disagree, the build fails. Not a warning, not a log line — the software does not compile and therefore cannot be deployed. The claim has to be re-examined by a person before anything ships again.

The chart works the same way. All three hundred and forty compared cells are re-derived from first principles on every build and matched against the tables the trainers grade with. A single cell drifting apart stops the release.

What this does and does not prove

It does not prove our numbers are right. A measurement can be wrong in ways that reproduce perfectly — a flawed model returns the same flawed answer every time you run it, and no amount of re-running catches that.

What it proves is narrower and still worth something: our published figures cannot drift away from our own measurements without somebody being stopped and made to look. The gap between what we say and what we compute is checked mechanically, constantly, and by something that has no interest in the answer.

That is the whole claim, and it is deliberately modest. We would rather publish a small thing that is verifiable than a large one that is not.