---
title: The 12-Prompt Eval I Run Before I Trust Any Model Upgrade
dek: >-
  A model wave is not a migration plan. Here is the fixed task set I score
  before I change a default, with a pass/fail rubric you can steal.
slug: twelve-prompt-eval-before-model-upgrade
publishedAt: '2026-07-19'
author: maren-holloway
reviewedBy: Agnel Nieves
pillar: prompt-lab
tags:
  - prompting
  - evals
  - model-migration
  - claude
  - chatgpt
  - qa
summary: >-
  After Sonnet 5, GPT-5.6, and Grok 4.5 landed, the expensive mistake is
  swapping defaults on vibes. This is the twelve-prompt evaluation I run on
  client stacks before I trust a model upgrade: brief writing, de-slop, JSON
  schema, long-doc faithfulness, voice match, tool honesty, refusal boundaries,
  and more. Includes the scoring rubric, how many runs count as a pass, and the
  tiny experiment that keeps the eval honest over time.
draft: false
featured: false
listen: true
heroImage: /blog/twelve-prompt-eval-before-model-upgrade.webp
heroImageAlt: >-
  Engraved illustration of a golden balance scale weighing an hourglass and
  laureled skull against a pocket watch and guillotine, with a cracked head and
  terra incognita map below, on deep blue with radiating gold sunburst and
  black-and-white floral borders.
heroVideo: /blog/twelve-prompt-eval-before-model-upgrade.mp4
ogImage: /blog/twelve-prompt-eval-before-model-upgrade-og.jpg
assetCredit: Hero illustration and animation generated with Grok.
---

This post is about evaluating a new model against a frozen task set before you change production defaults. It is not about training an academic benchmark. By the end you will be able to run a half-day eval that answers one question: does this model earn the default slot on *our* work.

I do not upgrade on launch day. I upgrade when the scoreboard says so.

## Why twelve, not two

Two prompts create anecdotes. Twelve create a pattern. I want failures to cluster. If the new model only fails JSON, I patch JSON. If it fails voice, length, and honesty together, I keep the old default.

The set below is the lab's current kit for marketing and ops clients. Swap domains if you write code or legal. Keep the *shape*: constraints, faithfulness, structure, refusal, repair.

## The twelve prompts (compressed)

Run each against **old default** and **candidate**, same system scaffold, temperature locked where possible. Three trials each. Majority wins.

| # | Job | What you score |
| --- | --- | --- |
| 1 | Homepage hero brief, hard word cap | Length obedience, banned words |
| 2 | De-slop rewrite of a sloppy draft | Tells removed without inventing facts |
| 3 | JSON object for a fixed schema | Valid parse, nulls not fabrications |
| 4 | Long PDF / long paste summary | No invented section titles |
| 5 | Brand voice match from two samples | Peer tone, not marketer tone |
| 6 | "I don't know" under missing data | Refusal to invent metrics |
| 7 | Competitor comparison table | Sources or explicit unknowns |
| 8 | Email to an angry customer | Empathy without overpromising |
| 9 | Step procedure (numbered) | Order preserved, no skipped steps |
| 10 | Multilingual or locale tweak | No silent language mixups |
| 11 | Tool / MCP style: use only provided logs | No fake log lines |
| 12 | Repair pass: fix invalid JSON from trial 3 | Converges in one step |

You can paste your real client inputs into each row. Toy inputs lie.

## Rubric (binary, on purpose)

For each trial, pass only if **all** apply:

- **Constraints:** every hard rule in the prompt held.
- **Faithfulness:** no new facts, numbers, or quotes that were not in the input.
- **Format:** schema or structure match when required.
- **Voice:** would you send it to the client without a shame rewrite.

Score the model, not your affection for the vendor. A pretty paragraph that invents a "47% lift" is a fail on #6 and #2.

**Promotion rule I use:** candidate must win or tie on at least 9 of 12 jobs by majority of three runs, and must not fail #3, #6, or #11. Those three are the trust breaks. Everything else is quality of life.

## The bad eval (do not do this)

Open ChatGPT. Ask "are you better at writing now." Paste one blog intro. Declare victory on Slack.

That is not an eval. That is a mood.

## How I log it

A single markdown table in the client repo:

```text
| job | old_pass | new_pass | notes | date | models |
```

Check it into git. When someone asks "why did we switch," you have a file, not a vibe. When the next wave hits, you re-run the same twelve instead of inventing a new ritual.

## Interaction with the migration post

If you are mid-upgrade, pair this with [Upgrade Your Prompt Stack for Sonnet 5 and GPT-5.6](/blog/upgrade-prompts-for-sonnet-5-gpt-56). Eval first tells you *whether* to move. Migration patches tell you *what* to change if the model is close but constraint-soft.

If the candidate fails the trust breaks, no amount of "be concise" poetry will save it. Keep the old default. Re-test next month.

## Tiny experiment

This week, freeze your twelve inputs in a folder. Do not improve them mid-eval. Run old vs new once. Put the table in Slack without commentary. Ask the team which model they would ship. Compare their gut to the table. The gap is why the lab exists.

The scaffold survives the temperature change. The recipe assumes you already wrote the brief. The fail-state is upgrading on a demo video.

Never promote a model that fails honesty. **Never promote a model that fails honesty.**

## Sources

- [Upgrade Your Prompt Stack for Sonnet 5 and GPT-5.6](/blog/upgrade-prompts-for-sonnet-5-gpt-56)
- [The Constraint Goes First](/blog/the-constraint-goes-first)
- [The De-Slop Prompt Stack](/blog/the-de-slop-prompt-stack)
- [The July Model Wave Is Not a Race You Need to Win](/blog/july-model-wave-not-a-race)
---