Our last round of testing found 9 bugs by adversarially inventing sentences designed to fool our own regex. This round we stopped inventing and pulled 8 real arXiv cs.CL abstracts instead, ran them through all 10 fact-extraction functions, and manually checked every result against a careful reading of the source text. Found 3 more real gaps — all misses this time, not false positives, which is evidence the earlier hardening held up under real text. Fixing one of them briefly reintroduced an old bug via a subtle regex gotcha: a character class written as [A-Z] inside a pattern compiled with re.IGNORECASE silently matches lowercase letters too. All fixed, all now permanent tests using the real abstract text. The suite is 60/60, still zero model calls, and we now have an actual coverage number instead of a guess: 0% fallback across 8 genuine academic abstracts.
Two posts ago we found 9 bugs by writing sentences specifically designed to break our own rules — "the fundamentals of chemistry," "a revised edition of the textbook," the kind of phrasing a regex author invents specifically because they know where their own pattern is weak. That's a real and necessary kind of testing. It's also a limited one: you can only invent the bugs you already suspect exist. It's very good at catching false positives, because you're deliberately constructing traps. It's much worse at catching false negatives, because you'd have to imagine every real phrasing pattern a rule might miss — and if you could imagine it, you probably would have written the rule to catch it already.
So this round we didn't invent anything. We fetched 8 abstracts straight from arXiv's cs.CL listing — real, current research prose, written by people with no idea our regex existed — and ran them through the fact extractor with no model fallback available, so every result you see below came from a rule or didn't come at all.
Three real gaps, all misses
| Function | Missed text | Why it missed |
|---|---|---|
has_measurable_terms | "the final Krippendorff's alpha reached 0.743, 0.723, and 0.722..." | Required a percentage adjacent to a comparison word like "improvement" — missed plainly-reported statistics with no such word nearby |
addresses_competing_explanations | "VEXMLM reaches 80.0% accuracy versus 77.0% (XLM-R)" | Our own earlier false-positive fix required "versus" to sit next to a word like "baseline" or "explanation" — real papers compare against a named model far more often than they use the word "baseline" itself |
documents_methods | "We use binary trees as an evaluation scaffold... apply this protocol across problem domains" | Recognized "method," "procedure," "dataset," "benchmark" — never "scaffold" or "protocol," which is how this abstract actually describes its methodology |
The middle one is worth sitting with. Two posts ago, we tightened addresses_competing_explanations specifically because "versus" was matching generic metaphors like "this versus that." The fix required an anchor word from a fixed list — prompting|explanation|hypothesis|baseline|condition|mechanism. It was a real fix for a real bug. It was also narrower than real usage: nobody writing a comparison table says "versus baseline," they say "versus XLM-R." Fixing a false positive introduced a false negative, and neither the original bug nor this one would have shown up without testing against text nobody wrote to prove a point.
A bug inside the fix for a bug
Repairing the "versus" gap needed a pattern that recognizes "versus" followed by a proper noun — a capitalized model or system name — without falling back to matching every generic "X versus Y." The natural regex is "versus, then a capitalized word": \b(versus|vs\.?)\s+[A-Z][a-zA-Z0-9\-]*\b.
Running the full test suite immediately after adding it failed a test that had been green for two posts: "This versus that is unrelated." came back True again — the exact false positive we'd already fixed once. The cause is a regex detail easy to forget: this whole pattern is compiled with re.IGNORECASE, and that flag doesn't just make literal text case-insensitive — it case-folds character classes too. [A-Z] under IGNORECASE doesn't mean "an uppercase letter." It means "a letter, case notwithstanding," so it happily matched the lowercase "t" in "versus that."
The fix is a scoped override: (?-i:[A-Z][a-zA-Z0-9\-]*) turns case-sensitivity back on for just that one alternative, so the proper-noun check only fires on text that's actually capitalized, while the rest of the pattern stays case-insensitive as intended. It's a two-line diff, but it's the kind of bug that's invisible until you have a regression test old enough to catch it firing again — which is the entire argument for keeping every adversarial case as a permanent, named test instead of a scratch script you run once and discard.
What this adds up to
test_facts.py is now 60 assertions: the original 43, the 11 adversarial false-positive cases from two posts ago, and 6 new ones from this round — the 3 real gaps, plus a fourth for the pipeline/scaffold language, plus a permanent regression guard for the IGNORECASE case-folding bug specifically, so it can never come back unnoticed. All 60 pass with zero model calls anywhere in the file.
We also now have something we didn't have before: an actual coverage number instead of an assumption. Across the 8 real abstracts times 10 facts — 80 checks total — 13 resolved True, and every single one of the 80 resolved via a rule. Zero model calls across all 8 genuine academic abstracts. That's not proof the rule library is complete; a wider or more adversarial sample would likely find more gaps, the same way this round found gaps the last round didn't. But it's a real, measured data point about how this thing behaves on text nobody wrote for our benefit, which is the only kind of evidence that actually earns confidence under our own standard.
The working method for this project now has two legs, and both matter: invented adversarial phrasing to hunt for rules that are too broad, and real external text to hunt for rules that are too narrow. Neither one substitutes for the other. Full diff and the arXiv-derived test cases are part of the ongoing Code Prompt source and will be linked here as that project opens up further.