Promptest scores prompts against a five-category rubric using an LLM judge, and the rubric works exactly as advertised: bad, weak, good, and excellent prompts land in tight, cleanly separated score bands. So a team at Cortex Research Group's CRG-RIS lab asked the obvious follow-up question — does a higher score actually predict a better real-world coding outcome? They preregistered a falsification condition, ran four prompts spanning all four tiers against an identical bug with independent coding agents, and reported what happened even though it wasn't the answer the rubric's existence implies. The rubric is reliable. It is also, on this task, not predictive. And the thing that actually mattered was buried in the token bill, not the score.
There's a quiet assumption baked into every "prompt quality" tool: that a higher score should translate into a better outcome when the prompt is actually used. It's a reasonable assumption. It's also an empirical claim, and empirical claims are supposed to be tested, not inferred from the fact that the scoring rubric feels rigorous.
The CRG-RIS evaluation of Promptest tested it anyway, in two layers, against a standard the study calls the Cortex Research Group Research Integrity Standard — a discipline built around preregistration, falsifiability, and reporting negative results as the headline rather than burying them in a limitations section.
Layer 1: the rubric is reliable
The first question is the boring one, and it's boring because it holds up. Eleven prompts, hand-labeled into four tiers (bad, weak, good, excellent), were scored five times each through live calls to Promptest's actual production judge pipeline — not a reimplementation, the real judgePrompt() function imported directly from source.
The tiers separated cleanly, with no overlap:
| Tier | Mean score (0–100) |
|---|---|
| Bad | 7.5 |
| Weak | 23.3 |
| Good | 83.5 |
| Excellent | 92.6 |
A 35-point gap separates "weak" from "good" with zero overlap, and within-prompt consistency was tight (standard deviation under 3 points) for nine of eleven prompts. The one outlier, a rate-limiter prompt sitting right on a rubric boundary, showed more judge disagreement — which reads less like unreliability and more like the judge correctly detecting genuine ambiguity in a prompt that omitted the file name, the exact contract, and any tests.
So: the rubric measures something real and does it consistently. That's Layer 1, and it's a clean pass.
Layer 2: does the score predict anything?
Layer 2 is where it gets interesting, because this is where the study stopped grading the rubric on its own terms and started asking whether it predicts something outside itself.
The setup: one fixed Node/TypeScript repository with a single, objectively verifiable bug — a login() function that threw an uncaught exception on a wrong password instead of returning a clean HTTP 401. A hidden acceptance test settled correctness with no room for judgment calls. Four prompts, one from each Layer 1 tier, all describing the identical bug in the identical file, were preregistered before execution and handed to four independent coding-agent runs, each working from its own isolated copy with zero knowledge of the study design.
The preregistered falsification condition was blunt: if bad or weak-tier prompts matched or beat good/excellent on turns, correctness, or scope discipline, the predictive-power claim is falsified for this task. No hedging room was built in after the fact.
All four prompts produced a correct fix. All four. A 7.5-scoring "fix my code" prompt fixed the bug exactly as successfully, on the hidden test, as a 92.6-scoring prompt with a named root cause and a verify command. Self-reported turn counts, if anything, ran the wrong direction — the lowest-tier prompts self-reported fewer turns than the good-tier prompt.
By the preregistered bar, the hypothesis failed. The study calls this the central negative finding and reports it as exactly that, rather than reframing "no difference in outcome" as some quieter kind of win.
The real signal was hiding in the token bill
Here's where the paper earns its keep. Self-reported turn counts turned out to be a bad proxy for what actually happened — the "good" prompt self-reported 6 turns but made 9 real API calls; the "weak" prompt self-reported 1 turn and also made 9. Once the study pulled actual usage from the session transcripts instead of trusting the agent's own turn count, a pattern that rubric score never touched fell out:
| Prompt tier | Rubric score | Actual API calls | Actual cost |
|---|---|---|---|
| Bad ("fix my code") | 7.6 | 8 | $0.1206 |
| Weak (names function only) | 14.4 | 9 | $0.1345 |
| Good (scoped + exact contract) | 88.0 | 9 | $0.1264 |
| Excellent (+ root cause + verify cmd) | 93.0 | 5 | $0.0796 |
Bad, weak, and good clustered tightly together at $0.12–$0.13 — rubric score explained almost none of the spread between them. Only the prompt that named the bug's exact root cause and exact location broke away, at 37–41% lower cost and roughly half the API calls.
Scope discipline told the same story. Three of the four variants — bad, weak, and good — independently converged on a broader change to a shared compareSecret() comparator function, stripping out its fail-closed throw behavior. That change was serious enough to trip an actual security-policy flag, describing it as weakening a credential-comparison guard with no authorization for that specific change. The one prompt that avoided it wasn't the highest-scoring one in some fine-grained sense — it was the one prompt that told the agent exactly where the bug lived and what "done" looked like. Rubric tier didn't predict which three prompts would wander into the shared security-critical function. Specificity about root cause and location did.
What this actually argues for
The honest reading isn't "rubrics are useless." Layer 1 shows the opposite — a well-built LLM-judge rubric can be a precise, repeatable instrument. The honest reading is narrower and more useful: a high overall quality score is not the same claim as "this prompt will produce a cheap, correctly-scoped fix," and treating them as interchangeable is exactly the kind of unearned inference the CRG-RIS discipline exists to catch.
What predicted cost and scope discipline in this study wasn't politeness, structure, or five-category polish — it was whether the prompt named the specific root cause and the specific location. That's a narrower, sharper, more testable claim than "write a good prompt," and it point at something practical: when you're briefing a coding agent, the highest-leverage sentence you can add isn't the one that makes the prompt read better. It's the one that tells the agent exactly where to look and exactly when to stop.
Two other details are worth noting, because they say something about how the study was run, not just what it found. A full round of Layer 2 was discarded and rerun after the team discovered their own test fixture contained a leftover // BUG: comment functioning as an answer key — contaminating results in a way that would have made the prompt text irrelevant. And the same live-call methodology that produced Layer 1's clean numbers also surfaced two real production bugs in Promptest itself: a markdown-fence regex that discarded legitimate JSON when a judge's example output happened to contain a code fence, and an overly conservative token cap that silently truncated the judge's response on exactly the longest, most detail-rich "excellent" tier prompts. Both are fixed and live in production now, as a direct byproduct of the study actually exercising real inputs instead of a mocked pipeline.
The full paper, preregistration, raw judge output, and the four Layer 2 agent runs are public: github.com/cortexresearch/promptest-crg-ris-study.