Shipping Model Changes: Shadow Traffic, Canaries, and a Rollback That Works.

Quality regressions do not page anyone

A bad code deploy throws exceptions, error rates spike, someone gets paged. A bad model deploy returns confident, well-formatted, plausible answers that are worse than yesterday's. Every dashboard stays green. You find out from a customer three weeks later, or from an internal user who quietly stopped using the feature.

That asymmetry is the entire reason model deployment needs its own discipline. The eval gate answers whether a candidate is good enough on a fixed set of examples. It cannot tell you how the candidate behaves on the live distribution, which is always broader and stranger than your golden set. Deployment strategy is how you find that out without betting production on it.

The prerequisite: swapping must be trivial

Everything below assumes you can change which model serves traffic without a code deploy, and change it back in seconds. If a rollback requires a pull request, a build, and a deploy pipeline, you will hesitate during the exact ten minutes when hesitation is expensive.

The primitive is an alias, resolved at request time:

# routing.yaml, hot-reloaded by the gateway. No rebuild, no restart.
routes:
  credit-memo:
    stable: { model: qwen3.5-32b-fp8@sha256:9c1f…, prompt: credit-memo/v7 }
    canary: { model: qwen3.5-32b-fp8@sha256:4ad2…, prompt: credit-memo/v8 }
    canary_percent: 5
    cohort: internal_users # who is eligible for canary
    shadow: { model: llama4-70b-fp8@sha256:77b0…, prompt: credit-memo/v8 }
    rollback_to: stable

Note that a route pins model artifact and prompt version together. That pairing matters: a prompt tuned for one model is not automatically correct for another, and treating them as independent knobs produces combinations nobody evaluated.

Shadow traffic: measure without risking anything

Shadow mode sends a copy of real production requests to the candidate, discards its responses, and records them for comparison. Users are never exposed. You get the candidate's behavior on the true input distribution, including the malformed, adversarial, and deeply weird inputs your golden set does not contain.

async def handle(req, route):
    primary = await invoke(route.stable, req)      # served to the user

    if route.shadow and sampled(req, rate=0.10):
        # Fire and forget. Never awaited on the user path, never surfaced.
        asyncio.create_task(shadow_compare(route.shadow, req, primary))

    return primary

async def shadow_compare(cfg, req, primary):
    try:
        async with timeout(30):
            candidate = await invoke(cfg, req)
    except Exception as e:
        return record_shadow_error(req.id, e)      # candidate failures are data
    record_shadow(
        request_id=req.id,
        agreement=semantic_agreement(primary.text, candidate.text),
        schema_ok=validates(candidate.text, req.schema),
        delta_tokens=candidate.tokens - primary.tokens,
        delta_latency_ms=candidate.latency_ms - primary.latency_ms,
    )

Three implementation rules. Shadow work never blocks or affects the user response, including on timeout or exception. Shadow requests carry a header marking them as such, so downstream tools can refuse to act on them, which matters enormously for agents: an unmarked shadow agent will happily execute real tool calls twice. And sample rather than mirroring everything, because shadowing at 100% doubles inference cost for no additional signal.

What you are looking for in the shadow data is not "is the candidate better," which shadow mode cannot tell you without labels. It is: where do the two disagree most, and what do those cases look like? Sort disagreements by magnitude, read fifty of them, and you will learn more about the candidate than any benchmark provides. The disagreements also make excellent new golden-set cases.

Canary: exposure with a tripwire

Once shadow looks clean, expose the candidate to a small, deliberately chosen slice of real traffic.

Choose the cohort by blast radius, not randomly. Internal users first, then low-stakes workflows, then a percentage of general traffic, and never the highest-stakes path until last. In a regulated setting, some cohorts should be permanently excluded from canaries; a customer-facing suitability determination is not where you learn about a new model.

The rollback triggers must be defined and automated before the canary starts, because rollback decisions made during an incident are made badly:

ROLLBACK_TRIGGERS = [
    ("schema_valid_rate",   lambda c, s: c < 0.995,                "structure broken"),
    ("error_rate",          lambda c, s: c > s * 2 + 0.005,        "errors doubled"),
    ("p95_latency_ms",      lambda c, s: c > s * 1.5,              "latency regression"),
    ("refusal_rate",        lambda c, s: c > s + 0.05,             "over-refusing"),
    ("human_edit_rate",     lambda c, s: c > s + 0.10,             "humans fixing output"),
    ("cost_per_request",    lambda c, s: c > s * 1.4,              "cost regression"),
    ("safety_probe_hits",   lambda c, s: c > 0,                    "safety failure"),
]

human_edit_rate is the most valuable signal on that list and the one most teams do not have. If your interface lets a person accept, edit, or reject model output, the edit rate is a continuous, free, human-graded quality metric on live traffic. A model that is subtly worse shows up here days before anyone files a complaint. If your product does not capture it today, adding it is the most valuable instrumentation you can build this quarter.

refusal_rate catches the opposite failure: a candidate that is "safer" because it declines more work. That is a regression, and it is invisible to accuracy metrics.

Let the canary run long enough to cover a full business cycle. An hour at 5% on a Tuesday morning tells you nothing about month-end close.

Rollback

Rollback is a config change, applied by an on-call engineer without a deploy, and it must be tested. Practice it on a normal Wednesday, time it, and write the number down. If it takes more than a minute, fix that before you need it.

Two things that quietly break rollbacks. Caches: a prompt cache or response cache keyed without the model hash will keep serving the bad model's output after you have rolled back, which produces a genuinely maddening incident. Include the model hash and prompt hash in every cache key. State written by the bad version: if the canary wrote records, opened tickets, or sent messages, rolling the model back does not roll those back. Know which artifacts came from the canary cohort, which is one more reason every decision record carries the model hash.

Prompt changes are deploys

The single most common way teams bypass all of this is the prompt edit. Prompts feel like copy, they live in a config UI or a database row, and someone changes one on a Friday afternoon to fix a phrasing complaint.

A prompt change can alter behavior more than a model change. Treat it identically: version controlled, reviewed, gated by evals, canaried, rollback-able. If your prompts live in a database that a non-engineer can edit through an admin panel, then your production model behavior is editable by a non-engineer without review, and that is worth knowing before an auditor points it out.

The same applies to tool schemas, retrieval configuration, chunking parameters, and the system's temperature setting. Everything that changes behavior goes through the pipeline.

After the rollout

Promotion is not the end. Keep the previous stable version resolvable for at least a full retention cycle so you can answer questions about decisions it made, and keep a scheduled sample of live traffic scored against the same rubric your gate uses, so that slow drift surfaces as a trend instead of a surprise.

The end state worth building toward is boring: a model change is a config commit, it shadows for a day, canaries to internal users for a day, expands over a week, and rolls back automatically if any tripwire fires. Nobody is heroic and nothing is dramatic, which is the correct emotional register for changing the thing that writes your credit memos.