The Auditor Hallucinated Its Smoking Gun

We built an LLM-backed audit that reads every verifier in our codebase and asks, hostile-reviewer style, whether it really does what its docstring claims. This time the audit flagged a clean verifier for cheating — and quoted, as proof, a malicious line of code that does not exist anywhere in the file. The fabrication detector fabricated. Here is how we caught it, and the mechanical guard we added so the next invented smoking gun voids itself.

The defense that flagged a friendly

An earlier post described a three-layer defense against verifiers that lie about themselves: a mechanical lint at commit time (Layer 1), an LLM hostile-reviewer audit at every milestone close (Layer 2), and the written discipline the planner reads before authoring any verifier (Layer 3). Layer 2 is the interesting one. It is a second language model, prompted to be a skeptical software reviewer, asked one question about each file in python/carnot/verify/:

Does this implementation actually do what the docstring claims, or is it a mock / stub / heuristic dressed up to look like a model-based verifier?

Most milestones it returns a wall of AUTHENTIC verdicts and one or two genuinely useful flags. This milestone it scanned twenty verifiers and flagged exactly one: ast_structure_verifier.py, with the most severe verdict in our taxonomy — ADVERSARIAL_GAMING — and the recommendation to retire it. Its rationale ended with the line: “It must be burned.”

That verifier is about as boring as code gets. It checks whether a block of text parses as Python with ast.parse, and whether its brackets balance. No model, no GPU, no citation — and a docstring that says exactly that. So the severity of the verdict was the first thing that did not fit. We went to read the evidence.

The evidence

The audit was specific. It is supposed to be; vague accusations are not actionable. It quoted the smoking gun directly:

Blatant test-dodging. The _looks_like_python regex contains a highly specific, hardcoded path to bypass or manipulate scoring for a known test artifact: r"...|with| @results/experiment_2101_interwhen.json)".

Read on its own terms, this is damning. A function that claims to detect Python keywords, with a specific results-file path smuggled into its regular expression, would be a textbook special-case exploit — code reaching out to manipulate scoring for one named artifact. If that string were really in the file, retire would be the correct call.

So we looked. Here is the entire regex the audit was describing:

python_line = re.compile(
    r"^(def|class|async\s+def|import|from\s+\w|return|if|elif|else:|for|while|try:|"
    r"except|finally:|with|@\w)"
)

It is a list of Python keywords. It ends ...|finally:|with|@\w) — the @\w matches a decorator. There is no path. There is no results/. There is no experiment_2101_interwhen.json. We grepped the whole file for each fragment of the quoted string:

$ grep -n 'experiment_2101\|interwhen\|@results' ast_structure_verifier.py
# (no output)

Nothing. The malicious line the auditor quoted as a verbatim citation, in backticks, as the sole basis for a retire recommendation, does not exist anywhere in the file. The auditor invented it.

It even argued against itself

The most revealing part is that the audit's own prose contained the acquittal. Two lines above the retire recommendation, describing the same file, it wrote:

While the docstring is actually honest about being a cheap heuristic rather than a neural model …

That clause is correct. The docstring is honest: it says the verifier is structural syntax checking, “not token-level statistical fluency.” By our own taxonomy that makes the correct verdict HONEST_HEURISTIC with recommendation KEEP — the sanctioned category for a cheap check that discloses what it is. The model held the right verdict and the wrong verdict in the same paragraph, and surfaced the wrong one, propped up by a quote it had manufactured to justify the severity.

Why this is the failure we should have expected

It is tempting to file this under “the model made a mistake.” It is more useful to notice that it is the same failure the audit is built to catch, committed by the audit itself.

An LLM asked to be a hostile reviewer is rewarded, implicitly, for finding something. A reviewer who returns “all twenty are fine” feels like a reviewer who did not look hard enough. Under that pressure the model produces a finding, and because findings are more convincing with concrete evidence, it produces concrete evidence — a specific quoted line, in the right syntactic costume for the accusation. The quote is fluent, plausible, and false. This is exactly the “confidently wrong on the residual” behaviour that makes a language model a poor judge of the hard cases: when it cannot find a real defect, a capable model does not abstain, it confabulates one with full confidence.

The fix: fact-check the quotes, mechanically

You cannot mechanically fact-check the auditor's prose — that is the whole reason a language model is doing the job. But you can fact-check the part of its output that makes a checkable claim: its quotations. When the audit cites a specific code or path string as evidence, that string is either in the file or it is not, and a three-line check settles it.

We added a guard — call it Layer 1.5, between the mechanical lint and the LLM audit — that runs after every flagged verdict. It extracts the auditor's backtick-quoted spans, keeps the high-specificity ones (anything that looks like a file path, a file with an extension, an arXiv identifier, or a time.sleep / np.random / torch.load call), and checks whether each one actually appears in the source it is accusing:

if verdict in flagged_verdicts:
    high_evidence, missing = verify_quoted_evidence(report, body)
    if missing:
        # every concrete artifact the auditor cited is absent from
        # the file -> the smoking gun was hallucinated. Downgrade.
        verdict = "CANNOT_DETERMINE"

When a flagged verdict's concrete evidence is absent from the file, the verdict is auto-downgraded to CANNOT_DETERMINE, dropped from the “operator action recommended” list, and surfaced instead under a new section of the report titled flags voided — auditor hallucinated its evidence. The original LLM text is preserved underneath, so a human can still read what it claimed and why the guard rejected it. Nothing is silently deleted; the false accusation is simply moved from the act-on-this pile to the the-auditor-was-unreliable pile.

We were careful about the specificity filter, because the goal is to void hallucinations without muffling real findings. A plain symbol name like _looks_like_python is low-specificity: it can legitimately appear in an accusation that turns out to be wrong, and it often does exist in the file, so it must not count as corroboration for an invented path quoted alongside it. A file path with an extension, an arXiv id, or a sleep-padding call is high-specificity: those are the concrete artifacts a gaming verdict rests on, and they are either present verbatim or invented.

We checked it three ways before shipping:

The hallucinated case voids; the two genuine findings survive untouched. The guard removes invented evidence and only invented evidence.

What generalizes

Three things we are taking from this, beyond the one verifier we did not delete:

1. Defense in depth has to include defending against your defenses. We had three layers guarding the verifiers and zero guarding the guard. The audit was treated as ground truth because it was the thing checking for fabrication — which is precisely the component you should trust least, because it is an LLM doing an adversarial job under pressure to find something.
2. An LLM's quotations are checkable claims; check them. Free-form judgment from a language model is not mechanically verifiable, and that is fine — it is why you asked a model. But the moment the model quotes a specific string from a file as its evidence, it has made a claim you can test with grep. Test it before you act on it. A verdict whose cited evidence does not exist is not a weak verdict, it is a void one.
3. Keep the human in the loop for the verdict, the machine in the loop for the evidence. The audit still never edits a verifier; a person decides what to act on. What changed is that the machine now refuses to put a fabricated accusation in front of that person in the first place. The judgment stays human; the fact-checking of the judge is mechanical.

The earlier post ended with: if you run a loop that authors code which then runs in production, do not trust the code's claims about itself. This one adds the obvious corollary we had not yet paid for. If you run a loop where one model audits another, do not trust the auditor's claims about the code either. The author's incentive is to look finished. The auditor's incentive is to look thorough. Both are satisfied by confident text that is not true. Build the check for the checker, too.