The 12-Prompt Eval I Run Before I Trust Any Model Upgrade

This post is about evaluating a new model against a frozen task set before you change production defaults. It is not about training an academic benchmark. By the end you will be able to run a half-day eval that answers one question: does this model earn the default slot on our work.

I do not upgrade on launch day. I upgrade when the scoreboard says so.

Why twelve, not two

Two prompts create anecdotes. Twelve create a pattern. I want failures to cluster. If the new model only fails JSON, I patch JSON. If it fails voice, length, and honesty together, I keep the old default.

The set below is the lab's current kit for marketing and ops clients. Swap domains if you write code or legal. Keep the shape: constraints, faithfulness, structure, refusal, repair.

The twelve prompts (compressed)

Run each against old default and candidate, same system scaffold, temperature locked where possible. Three trials each. Majority wins.

You can paste your real client inputs into each row. Toy inputs lie.

Rubric (binary, on purpose)

For each trial, pass only if all apply:

• Constraints: every hard rule in the prompt held.
• Faithfulness: no new facts, numbers, or quotes that were not in the input.
• Format: schema or structure match when required.
• Voice: would you send it to the client without a shame rewrite.

Score the model, not your affection for the vendor. A pretty paragraph that invents a "47% lift" is a fail on #6 and #2.

Promotion rule I use: candidate must win or tie on at least 9 of 12 jobs by majority of three runs, and must not fail #3, #6, or #11. Those three are the trust breaks. Everything else is quality of life.

The bad eval (do not do this)

Open ChatGPT. Ask "are you better at writing now." Paste one blog intro. Declare victory on Slack.

That is not an eval. That is a mood.

How I log it

A single markdown table in the client repo:

| job | old_pass | new_pass | notes | date | models |

Check it into git. When someone asks "why did we switch," you have a file, not a vibe. When the next wave hits, you re-run the same twelve instead of inventing a new ritual.

Interaction with the migration post

If you are mid-upgrade, pair this with Upgrade Your Prompt Stack for Sonnet 5 and GPT-5.6. Eval first tells you whether to move. Migration patches tell you what to change if the model is close but constraint-soft.

If the candidate fails the trust breaks, no amount of "be concise" poetry will save it. Keep the old default. Re-test next month.

Tiny experiment

This week, freeze your twelve inputs in a folder. Do not improve them mid-eval. Run old vs new once. Put the table in Slack without commentary. Ask the team which model they would ship. Compare their gut to the table. The gap is why the lab exists.

The scaffold survives the temperature change. The recipe assumes you already wrote the brief. The fail-state is upgrading on a demo video.

Never promote a model that fails honesty. Never promote a model that fails honesty.

Sources

• Upgrade Your Prompt Stack for Sonnet 5 and GPT-5.6
• The Constraint Goes First
• The De-Slop Prompt Stack
• The July Model Wave Is Not a Race You Need to Win