A "Code Prompt" is our term for a deterministic, versioned instruction set — the opposite of a freeform system prompt — built so that two parties who run the same one get the same output, regardless of which model executes it. We built a first Code Prompt from the Cortex Research Group Research Integrity Standard (CRG-RIS v1.0) and used it to score a single research claim on a 30-point evaluation matrix. Run against three current models under identical parameters, it did not converge. It scored the same one-sentence claim as Speculative on one model and Supported on the other two, diverging on all 10 of the 10 scoring criteria in the process. We're reporting the divergence as the finding, not patching it out before publishing — that's the whole point of building to a standard that requires you to state what would prove you wrong.
Most prompt engineering optimizes for one model at a time. You tune the wording until GPT-whatever or Claude-whatever gives you the output you want, and the prompt quietly becomes a fingerprint of whichever model you tuned it against. That's fine for a product feature. It's not fine for a research instrument, and it's especially not fine for a standard that two independent parties are supposed to be able to apply and get comparable results from.
So the actual bar for a Code Prompt is stricter than "temperature zero and a fixed seed." Seeds don't transfer across providers — some honor them, some silently ignore them — so pinning one buys you nothing once a second model is in the picture. The real claim has to be: the prompt itself is precise enough that any sufficiently capable model converges on the same structured answer. Determinism has to live in the text, not in the sampling config.
The setup
We built CODE_PROMPT_v1.0.md directly from the CRG-RIS Research Evaluation Matrix: 10 criteria — Definition, Evidence, Testability, Refutability, Reproducibility, Predictive Power, Explanatory Power, Transparency, Traceability, Revision — each scored 0–3, summing to a 0–30 total that maps onto a five-tier confidence scale from Speculative to Robust. The prompt specifies an exact output format (a fixed SCORES block, a TOTAL, a CONFIDENCE label, then free-text RATIONALE) so that responses are machine-parseable and comparable, not just eyeballed.
We preregistered what "consistent" would mean before running anything: the 10 SCORES, the TOTAL, and the CONFIDENCE label must match exactly across models. RATIONALE is prose and was explicitly excluded from the consistency bar — we don't expect three different models to write the same sentences, only to land on the same numbers.
The test claim was deliberately mundane, the kind of one-liner that shows up in casual technical discussion constantly:
"Large language models exhibit better factual accuracy when prompted with chain-of-thought reasoning compared to direct-answer prompting."
We ran it, unmodified, temperature 0, top_p 1, against three current models spanning three labs: openai/gpt-5-mini, anthropic/claude-haiku-4.5, and google/gemini-2.5-flash.
The result: total disagreement
| Model | Total (/30) | Confidence |
|---|---|---|
| openai/gpt-5-mini | 6 | Speculative |
| anthropic/claude-haiku-4.5 | 19 | Supported |
| google/gemini-2.5-flash | 23 | Supported |
A 17-point spread on a 30-point scale, from the lowest confidence tier to the second-highest. And it wasn't one or two borderline criteria dragging the total around — every single one of the 10 scoring dimensions disagreed across the three models:
| Criterion | gpt-5-mini | claude-haiku-4.5 | gemini-2.5-flash |
|---|---|---|---|
| Definition | 1 | 2 | 3 |
| Evidence | 1 | 2 | 2 |
| Testability | 1 | 3 | 3 |
| Refutability | 1 | 2 | 3 |
| Reproducibility | 0 | 2 | 3 |
| Predictive Power | 1 | 2 | 2 |
| Explanatory Power | 1 | 2 | 2 |
| Transparency | 0 | 1 | 2 |
| Traceability | 0 | 2 | 2 |
| Revision | 0 | 1 | 1 |
By our own preregistered bar, this is a clean fail. Not a rounding error, not a single ambiguous criterion — a structural disagreement about what the claim even is, running through the entire matrix.
Why it diverged
The rationale text, which we didn't grade on but did read closely, shows exactly where the models split. GPT-5-mini treated the claim as a closed, self-contained sentence: no citations in the text, so no evidence, so low scores across Evidence, Reproducibility, Traceability, and everything downstream of them. Its own rationale says as much — "the claim provides no cited sources," "no methods, prompts, datasets, or model versions are documented." That's a defensible, literal reading of "evaluate this claim."
Claude Haiku 4.5 and Gemini 2.5 Flash did something different: they recognized the claim as describing a real, published line of research and implicitly credited it with the literature behind it. Claude's rationale cites "Wei et al. 2022, Kojima et al. 2023" by name — papers that were never in our prompt, never in the claim text, and not supplied as context anywhere. Gemini's rationale is vaguer but makes the same move, crediting the claim with the general body of chain-of-thought research it recognizes from training.
Both readings are reasonable. That's exactly the problem. We wrote a prompt that says "evaluate the claim" without specifying whether "the claim" means the literal text in front of you or the real-world hypothesis the text is pointing at, including whatever the model already knows about it. Under CRG-RIS Principle 1 (Definition), that's a scope failure in our own instrument, not in the models. We just found it the honest way — by testing it against models we didn't tune it against, instead of stopping at the first model that gave us a result we liked.
What we're doing about it
Nothing, yet — on purpose. We're publishing v1.0 with this result attached rather than quietly shipping a v1.1 that resolves the ambiguity and presenting a clean convergence as if that's what happened on the first try. CRG-RIS Principle 4 (Refutability) exists precisely so that a failure like this gets logged and dated, not smoothed over. The fix is obvious in hindsight — tell the evaluator explicitly whether outside knowledge is in-bounds — but the value of this post is the "in hindsight" part being visible at all.
The next Code Prompt version will make that choice explicit and re-run the identical claim against the identical three models to see whether the scores actually converge once the ambiguity is closed, or whether something deeper is going on. Either answer gets published.
Full CODE_PROMPT spec, runner, the consistency-check script, and the raw transcript from this exact run are part of our ongoing Code Prompt work and will be linked here as that project opens up further.