In our last post, we showed three models agreeing on 9 of 10 extracted facts and presented it as evidence that our Code Prompt had become deterministic code instead of a fancier prompt. A reader looked at the actual table and asked the obvious question: if 7 of those 10 rows show source: model, what exactly did we prove? Three LLMs happening to agree on a guess is not the same claim as "this is a tested function." We agreed, and went and built the thing that actually answers that question: a unit test suite with zero model calls in it. It caught two real bugs. This post is the correction, in public, with the diff.
We try to run this project against our own standard, which means the Revision principle applies to us too: no conclusion is permanent, and when a reader shows you the evidence didn't say what you claimed, the right move is a dated correction, not a quiet edit to the old post. So the old post stays up as-is — it's an accurate record of what we believed at the time we published it — and this is what changed.
What the last post actually showed
The core table compared 10 extracted facts across three models on one test claim. Here it is again, with the column that matters highlighted:
| Fact | gpt-5-mini | claude-haiku-4.5 | gemini-2.5-flash | Source |
|---|---|---|---|---|
| has_citation | False | False | False | rule |
| states_failure_condition | True | False | False | model |
| has_defined_terms | False | False | False | model |
| states_scope_or_limits | False | False | False | model |
| documents_methods | True | True | True | model |
| has_measurable_terms | True | True | True | rule |
| makes_prediction | True | True | True | model |
| addresses_competing_explanations | True | True | True | rule |
| discloses_assumptions_or_conflicts | False | False | False | model |
| states_revision_criteria | False | False | False | model |
Only 3 rows say rule. Those are the only ones that never touched a model at all — a regex ran, returned a boolean, done. The other 7 say model, meaning on this claim, our regex for those facts found no match and fell through to an LLM call. The fact that 6 of those 7 happened to agree across three models isn't nothing, but it isn't a test of a function either. It's three separate guesses that landed in the same place once. Change the claim, and there's no code there that guarantees they'll land in the same place again — because for those 7 rows, there was no code deciding the answer, a model was.
We wrote the post around the framing "cross-model disagreement dropped from 10/10 to 1/10," which is true, and reads like a determinism win. But determinism was never actually demonstrated for 7 of those 10 facts — only that three particular models guessed alike, once, on one input.
The actual test
The fix isn't a better cross-model comparison. It's a test that doesn't involve a model at all: call each fact function directly, with no fallback available, on inputs specifically chosen to be unambiguous — one where the pattern should clearly be present, one where it clearly shouldn't — and assert the exact boolean that comes back. Then call it 100 times on the same input and assert every single call returns the same thing.
That's test_facts.py now: 43 assertions across the 10 fact functions, three per fact (positive case, negative case, 100x-repeat determinism check), with model_fn never passed in. If a test in this file fails, it means the regex is wrong. It says nothing about which LLM is or isn't involved, because none is.
What it caught
First run: 41 passed, 2 failed. Both were real bugs, not test-writing mistakes:
| Function | Input | Expected | Got | Root cause |
|---|---|---|---|---|
has_citation | "As shown by Wei et al., 2022, this effect is robust." | True | False | Regex only matched citations inside parentheses, e.g. "(Wei et al., 2022)" — missed the same citation written inline in prose with no parens. |
discloses_assumptions_or_conflicts | "We assume the evaluation dataset is representative of real-world usage." | True | False | Regex matched the noun "assumption" but not the verb "assume" / "assumes" / "assuming" — a narrower pattern than the fact it was meant to detect. |
Both are one-line regex fixes. Both are now covered by a test that will fail again immediately if either regression ever comes back. 43 of 43 pass now, with zero model calls anywhere in the run.
One more thing we should have been clearer about
The 70% fallback rate quoted in the last post came from a second, more elaborate claim we used specifically to exercise the fallback path — not from the main claim in the comparison table above, which actually falls back 0% of the time even before any of these fixes, because that claim simply doesn't contain citation-shaped or hedging language for the rules to find. Running the two different claims side by side without saying so made it easy to conflate "how often does this claim need a model" with "how good is our rule coverage in general." Those are different questions, and we should have kept them visibly separate.
Where this leaves the project
The unit test suite, not the cross-model table, is now the actual evidence for the "this is code, not a fancier prompt" claim. Cross-model comparisons still have a role — they're the honest way to measure how well the fallback path behaves when the rules run out, which will always be a real part of the system for phrasing the regex hasn't seen yet. But they don't get to stand in for a test of the deterministic core anymore. Going forward, any claim we make about determinism gets backed by an assertion with no model in the call path, and any claim about the fallback behavior gets labeled as exactly that.
Full test suite, the two regex fixes, and the rest of the Code Prompt source are part of our ongoing work and will be linked here as that project opens up further.