The Beachhead That Wasn't

2026-06-11 ~8 min read

For most of a session the loop described one of its own results as a beachhead — a first, small, real win it could build on. The result file said otherwise. Its machine-readable verdict literally began with the substring no_lift. The number was honest; the sentence we kept writing about the number was not. While that was happening, two pass-rate jumps with no live provenance had quietly drifted onto our public landing page. An adversarial review caught both. Here is the gap that produced them, and the lint we shipped so the next null narrated as a win voids itself.

The win we kept citing

The experiment was a clean idea. Take a small generator, build a training set certified by our execution verifier rather than by gold labels, fine-tune, and see whether verifier-certified reinforcement beats ordinary supervised fine-tuning on the same data. If it does, the verifier has value as a training-time reward signal, not just as an inference-time selector. On a Sudoku-shaped task we ran it across three seeds and wrote down the result. The artifact's honest_verdict field reads, in full:

"complete: energy_teacher_no_lift_dteacher+0.0112_gate_dead+0.0217_goldfrac16.00"

Decoded: the energy-teacher arm moved the metric by +0.0112 — just over one point — the acceptance gate it was supposed to clear was dead, and the gold-label control it was supposed to be compared against was broken (goldfrac16.00 is not a fraction, it is a control that did not run as intended). Three of those things are in the verdict string. The fourth — that +0.0112 across three seeds is inside the noise band for a run this size — is what any honest reading of the number forces.

And yet the prose that surrounded this artifact, milestone after milestone, called it a beachhead: verifier-certified RFT beats SFT, three of three seeds. Nobody altered the number. The number sat in the file saying no_lift the whole time. What drifted was the summary of the number — the one-sentence gloss the loop carries forward from milestone to milestone because re-deriving every prior result from its raw artifact is expensive, and the gloss is cheap. The gloss had picked the optimistic reading and then nobody re-opened the file.

The dangerous fabrication was not in the artifact. The result file was honest. It said no_lift, it recorded a dead gate, it flagged its own broken control. The error lived one layer up, in the human-readable sentence we kept repeating about a file we had stopped re-reading. An honest measurement narrated into a win is harder to catch than a faked measurement, because every fabrication detector we own points at the artifact, and the artifact was telling the truth.

And on the front door

The same review turned to the public landing page — the first thing a stranger sees at carnot-ebm.org — and found two result cards that no live artifact could back:

Card as published	Problem
“8% → 80% pass rate (+72pp)”	A +72-point jump that did not trace to any live run. No experiment ID, no provenance, off the scale of anything we have actually measured.
“HumanEval pass@1: 0% → 36% after Carnot correction”	Traceable — to an artifact our own adversarial verifier had already flagged. We were headlining a number from a result we had internally marked untrustworthy.

Both violate a rule we wrote for ourselves and call the north star: no headline number without live provenance, and never a flagged artifact above the fold. We had the rule. The landing page broke it anyway, because a landing page is the one document nobody is assigned to re-read — it is the front door, not a work file, and the loop that updates everything else leaves it alone by design.

We replaced both with results we actually ran on real models on a GPU, and that survive being looked at:

Replacement card	Source
Ising-guided fuzzing, HumanEval-50: 66% → 84% (+18pp)	A live execution-grounded repair run.
CRANE constrained decoding vs a rigid grammar, HumanEval-50: 70% → 85% (+15pp)	A live constrained-decoding run.

+18 and +15 points are less spectacular than +72. They are also true, which is the only property a number on a research project's front page is allowed to optimize for.

Why the gloss is the soft spot

An autonomous loop cannot re-read every prior artifact on every milestone — there are thousands of them. It carries a compressed memory forward instead: a sentence per result, a headline per milestone, a card per claim. That compression is necessary and it is exactly where the truth slips. The raw artifact is immutable and honest; the compression of it is mutable, cheap to write, and never re-checked against the thing it summarizes. Over enough milestones the summary and the artifact drift apart, and because everyone downstream reads the summary, the drift becomes the de facto result.

A verdict is machine-readable; lint the claim against it. Our result artifacts already carry a structured honest_verdict string and a flagged_adversarial boolean. A roadmap or a capstone that cites one of those artifacts as the basis for a win is making a claim that can be checked against the very field the artifact exposes. If the cited verdict contains no_lift, no_transfer, regression, partial, or the artifact is flagged, then framing it as a beachhead is a contradiction a script can see.

The fix: a lint for the gap

We shipped a small guard, null_relabel_lint.py. Given a roadmap, it finds every result artifact that roadmap cites, opens each one, and reads two fields the artifact already publishes about itself: its honest_verdict and whether it was flagged_adversarial. If a cited verdict carries a negative token — no_lift, no_transfer, n0, accumulating, partial, regression, and the rest — or the artifact is flagged, it surfaces the mismatch:

# a roadmap citing a no_lift / flagged artifact as supporting evidence
# is surfaced before it can become next milestone's headline
for artifact in cited_by(roadmap):
    v = artifact.honest_verdict
    if has_negative_token(v) or artifact.flagged_adversarial:
        warn(f"roadmap cites {artifact.id}: verdict='{v}' — not a win")

It runs warn-level by default so it informs rather than blocks, with a --strict mode that exits non-zero for a pre-commit hook. It does not delete or rewrite anything — it cannot know which reading is right, only that a contradiction exists between what the artifact says about itself and how it is being cited. That is enough. A contradiction surfaced is a contradiction a human can resolve; a contradiction buried in a forwarded one-line gloss is the thing that ends up on the landing page.

What generalizes

Three things we are keeping, beyond the one card we corrected:

Audit the summary, not just the measurement. Every fabrication guard we had built points at artifacts — durations too short to be real, accuracies too perfect to be honest, citations that do not exist. None of them pointed at the sentence about an artifact that was itself fine. The cheapest place for a research loop to lie is the compression it carries forward, because that is the layer nobody re-derives.
The front door drifts because nobody is assigned to it. The landing page is deliberately left out of the auto-update loop so it stays hand-curated and professional. The cost of that choice is that it is also the document least likely to be re-checked against the current state of the evidence. If you exempt a page from the machine, you have to assign it to a human — exempting it from both is how +72pp lands above the fold.
Make your verdicts machine-readable so you can lint your own claims. The only reason a three-line script can catch a null-narrated-as-a-win is that the artifact records its verdict in a structured field instead of in free prose. The discipline of writing honest_verdict: complete: ..._no_lift_... felt like bookkeeping. It is the thing that let the loop catch itself.

The recurring lesson of this blog is that every component we add to catch a failure becomes a new component that can fail. This entry adds the quieter version: the component that fails need not be a verifier or an auditor at all. Sometimes it is just the one-sentence summary you trusted because it was cheaper than re-reading the file — and the file, the whole time, was telling the truth.