Our first Code Prompt (v1.0) asked an LLM to holistically score a claim against the CRG-RIS matrix in one shot. Run against three models on an identical claim, it diverged on all 10 of 10 scoring criteria — the wrong result for something meant to be model-agnostic. We rebuilt it as v1.1: ten narrow, named fact-extraction functions, each resolved by a deterministic regex rule first, with the model invoked only as a scoped fallback answering one yes/no question — never the whole claim. All scoring math is now pure code with zero model calls. Same claim, same three models: total disagreement collapsed to a single named fact out of ten, and running with no model at all now produces byte-identical output, every time, on any machine.
Earlier today we published a falsification: a "Code Prompt" — our name for a deterministic, versioned instruction set meant to give two parties consistent output from any LLM — scored the identical one-sentence research claim as Speculative on one model and Supported on two others, an 8-point-plus swing across every single one of the CRG-RIS matrix's 10 criteria. We logged it instead of quietly patching it, per our own standard's Refutability principle: state what would prove you wrong, and report it when it does.
The feedback on the root cause was direct: that wasn't really a Code Prompt. It was a better-organized system prompt with a strict output format bolted on. The model was still the thing doing all the thinking — reading the claim, weighing all 10 criteria, deciding all 10 scores, in one entangled pass. "Code Prompt" should mean what the name says: variables, functions, loops. Code that runs the same way regardless of which model is anywhere near it.
The rebuild: two layers, one of them has no model in it at all
v1.1 splits the problem in half instead of handing it over whole.
Layer 1 — fact extraction. Instead of one big "score this claim" instruction, we ask 10 narrow, specific, yes/no questions: does the text contain a citation-shaped substring? Does it use falsification language like "would be disproven if"? Does it name a measurable quantity? Each question has a deterministic regex implementation that runs first — plain pattern matching, the same result on every machine, forever.
Layer 2 — scoring. Once the 10 facts are known, turning them into the 0–3 per-criterion scores, the 0–30 total, and the confidence label is pure arithmetic and lookup tables. This layer never calls a model. Not sometimes — never, in any run, by construction.
The fallback, scoped tight. Regex can't catch every phrasing. So each fact function has an escape hatch: if the rule can't decide, ask a model — but only that one narrow question, about that one fact, never "evaluate this whole claim." "Does this claim state what observation would prove it false? Answer only true or false." That's the entire scope of what the model ever sees per call.
Same claim, same three models, very different outcome
We reran the identical test claim from the v1.0 post — "Large language models exhibit better factual accuracy when prompted with chain-of-thought reasoning compared to direct-answer prompting" — through v1.1, using the same three models as fallback: openai/gpt-5-mini, anthropic/claude-haiku-4.5, google/gemini-2.5-flash.
| Version | Totals across 3 models | Confidence bands | Criteria disagreeing |
|---|---|---|---|
| v1.0 (holistic scoring) | 6 / 19 / 23 | Speculative / Supported / Supported | 10 of 10 |
| v1.1 (code-first, fact-fallback) | 15 / 12 / 12 | Hypothesis / Hypothesis / Hypothesis | 1 of 10 |
All three models now land in the same confidence band. The full 10-fact breakdown shows exactly where the one disagreement lives:
| Fact | gpt-5-mini | claude-haiku-4.5 | gemini-2.5-flash | Source |
|---|---|---|---|---|
| has_citation | False | False | False | rule |
| states_failure_condition | True | False | False | model (disagreement) |
| has_defined_terms | False | False | False | model |
| states_scope_or_limits | False | False | False | model |
| documents_methods | True | True | True | model |
| has_measurable_terms | True | True | True | rule |
| makes_prediction | True | True | True | model |
| addresses_competing_explanations | True | True | True | rule |
| discloses_assumptions_or_conflicts | False | False | False | model |
| states_revision_criteria | False | False | False | model |
3 of 10 facts resolved by rule alone — guaranteed identical regardless of which model backs the fallback, because no model was called for them at all. Of the 7 that did fall back to a model, 6 came back unanimous across all three models. Exactly one — states_failure_condition — split, with gpt-5-mini alone answering True. That's a 70% fallback rate on this particular claim, logged and visible in the run output, not hidden inside a black-box holistic judgment.
Why narrowing the question narrowed the disagreement
In v1.0, one ambiguous instruction — should the evaluator credit the claim with outside knowledge it recognizes, or judge the literal text alone? — had room to ripple through all 10 scores at once, because everything was one entangled inference. A model's stance on that single ambiguity determined its Evidence score, its Reproducibility score, its Traceability score, all downstream of the same interpretive choice, in one pass with no seams.
In v1.1 there are seams everywhere on purpose. Each fact is its own function, asked in isolation, with no visibility into the other nine. A model can't let its read of "does this state a failure condition" bleed into "does this document methods" — they're separate calls, separate scopes, separate opportunities to disagree that don't compound each other. And 30% of the facts on this claim never reached a model at all, so there was no surface for disagreement there in the first place.
The remaining disagreement isn't noise anymore, either — it's a specific, addressable target. We now know exactly which function, on exactly which kind of phrasing, needs either a better regex rule or a clearer fallback question. That's the real dividend of decomposition: not (only) less disagreement, but disagreement you can point at and go fix, instead of a holistic score that came out different for reasons buried somewhere in a reasoning trace you don't have access to.
What's still not code
We're not overstating this. 7 of the 10 fact functions on this claim still needed a model call — the regex rule library is young, and plenty of real phrasing won't match our current patterns. fallback_rate is now a first-class, printed, logged number precisely because it's the honest measure of how much of a given run is actually deterministic code versus how much is still delegated judgment. Right now, on this claim, that number is 70%. Progress looks like driving it toward 0% — not by making the model smarter, but by writing more code.
Full source for facts.py, scoring.py, and the runner, plus the raw run transcripts behind every number in this post, are part of our ongoing Code Prompt work and will be linked here as that project opens up further.