Our fact-extraction rules for the CRG-RIS Code Prompt passed 43 out of 43 tests: one positive case and one negative case per function, all green. That's a weaker guarantee than it looks like — none of those tests ever tried to break anything. We went back and deliberately probed each rule with adversarial phrasing designed to fool it. Nine confirmed bugs came back, five of them the exact same failure mode: a bare word-root regex matching an unrelated word that happens to contain it. All nine are fixed and now permanent regression tests. The bigger finding: in a rule-based system like this, a false match is more dangerous than a missed one, and our test suite had only ever been checking for misses.
Two posts ago we corrected ourselves in public: a cross-model agreement table wasn't proof of determinism, because most of the facts it displayed were still resolved by a model call. The real proof of "this is code, not a prompt" is a unit test that never touches a model — test_facts.py, 43 assertions, all passing, all with model_fn never even passed in.
That correction was accurate as far as it went. But 43 passing tests covering 10 functions is roughly 4 tests per function — one positive, one negative, one determinism check, occasionally a second positive variant. That's enough to prove each rule can work. It says almost nothing about whether it reliably only works when it should.
Going looking for the failure, on purpose
So we stopped writing tests that confirm the rules and started writing tests designed to fool them — the same adversarial instinct behind the CRG-RIS Refutability principle, aimed at our own code instead of an external claim. For each of the 10 fact functions, we asked: what's a sentence that should trigger this rule but is phrased unusually? And separately: what's a sentence that obviously shouldn't trigger it, but happens to contain a word that looks like what the regex is searching for?
Fourteen probes, run directly against the functions with no model involved. Nine came back wrong.
| Function | Failing input | Bug |
|---|---|---|
has_citation | "Smith and Jones 2020, this holds." | Missed — pattern required parentheses or a comma before the year |
has_defined_terms | "We define success as any accuracy above 90%." | Missed — pattern required "define" immediately followed by "as," no words allowed between |
has_defined_terms | "The refined process means better output." | False match — "process means better output" is causal, not definitional |
states_scope_or_limits | "This has assumed importance in the field." | False match — idiom, not a disclosed assumption |
documents_methods | "The dataset was proprietary and never released." | False match — the sentence's actual meaning is the opposite of documented |
has_measurable_terms | "Room 202% occupancy this year." | False match — any bare percentage matched, regardless of relevance |
addresses_competing_explanations | "This versus that is a common metaphor." | False match — generic "versus" usage, not a ruled-out alternative |
discloses_assumptions_or_conflicts | "The fundamentals of chemistry are well understood." | False match — "fund" matched inside "fundamentals" |
states_revision_criteria | "This is a revised edition of the textbook." | False match — a book printing, not a stated revision criterion |
The pattern behind the pattern
Five of the nine bugs — fund\w*, revis\w*, assum\w+, and two others — were the identical mistake wearing different clothes: a regex written to detect a concept ("funding," "revision," "assumption") implemented as a bare word-root match, which actually just detects a substring. Substrings show up in unrelated words constantly. "Fund" is the first four letters of "fundamentals." "Revis" is the first five of "revised," which shows up in "revised edition" as often as in "we will revise this claim." The regex doesn't know the difference; it was never asked to.
The fix, every time, was the same shape too: stop trusting the root alone, and require a second, co-occurring signal — a nearby word, a specific verb form, a grammatical pattern — before calling it a match. fund\w* became fund(ed|ing|s)?\b, which requires a full word boundary and so no longer bleeds into "fundamentals." revis\w* became "revis* near if/when/evidence," which requires the word to be doing the job of stating a criterion, not just appearing.
One bug was a different shape entirely, and harder: documents_methods matching "dataset was proprietary and never released" isn't a stemming problem, it's a negation problem — the sentence contains the right keyword and means the opposite of what the keyword implies. Regex is bad at negation in general. We didn't try to out-clever it with a smarter positive pattern; we added an explicit override list of non-availability phrases ("proprietary," "never released," "withheld") that short-circuits the function to False before the positive pattern even runs.
Why false positives, specifically, are the risk that matters here
A missed match — a rule that should fire and doesn't — falls through to the model fallback. That costs an API call and adds to the fallback_rate we've been tracking, but it's self-correcting: the model still gets asked, and the run's transcript honestly shows a model: source, not a rule source. Nothing is hidden.
A false positive is worse, structurally. It returns True tagged with source: "rule" — the label that's supposed to mean "trust this, it's deterministic and correct by construction." Nobody double-checks a rule result the way they'd sanity-check a model guess, because the entire point of the rule-first architecture is that rules don't need double-checking. A wrong answer wearing the "rule" label is the one failure mode this system is specifically built to make you not notice. Five of our nine bugs were exactly that.
Writing the test is not the same as fixing the bug
One more thing worth being honest about: our first attempt at fixing the "means" false positive wasn't actually correct. It passed in a scratch script, so we moved on. Then we converted the nine probes into permanent, named pytest cases in test_facts.py instead of leaving them as a throwaway script — and the very next run failed. The "fixed" pattern still matched "process means better output" because it only required any word before "means," and "process" satisfied that. The real fix required a quoted term before "means" specifically — a narrower, more honest rule that admits it can't catch every definitional use of "means" and lets the harder cases fall to the model instead of guessing.
If we'd trusted the scratch script's clean output, that regression would have shipped. The lesson generalizes past this one bug: a probe that finds a defect and a test that prevents its return are not the same artifact, and only one of them belongs in the repository.
Where things stand
test_facts.py is now 54 assertions — the original 43 plus 11 new permanent regression tests covering every adversarial case above, still with zero model calls anywhere in the file. The baseline claim from the earlier posts (score: 5/30, Speculative, 0% fallback) is unchanged by any of this — none of the fixes touched a pattern that claim happened to hit, which is itself a small useful data point: the bugs we found weren't visible in the one claim we'd been testing against all along. They only showed up once we went looking for them on purpose.
Full diff — the nine regex fixes and the eleven new tests — is part of the ongoing Code Prompt source and will be linked here as that project opens up further. Next up: expanding the rule library's coverage on more varied real-world claims, tracked the same way — adversarial probe first, confirmed bug only, permanent test always.