Structured Output That Actually Holds: Constrained Decoding and the Retry Ladder.

Two percent is not a rounding error

A model that returns valid JSON 98% of the time feels fine in a notebook. Put it behind a document pipeline processing 50,000 items a night and it is 1,000 failures, most of them silent, some of them halfway through a multi-step agent run that already wrote to three systems.

Structured output is where LLM features stop being demos. This post covers the three layers that make it reliable: constraining generation so invalid output is impossible, designing schemas the model can actually satisfy, and a retry ladder that degrades predictably instead of looping.

Layer 1: make invalid output unrepresentable

There are three mechanisms, and they are not equivalent.

Prompting ("respond only with JSON matching this shape") is a request. The model usually complies and occasionally wraps the output in a markdown fence, adds a preamble, or trails off mid-object when it hits the token limit.

JSON mode guarantees syntactic validity but not conformance to your schema. You get parseable JSON with the wrong field names.

Constrained decoding is the one that actually solves it. The schema is compiled into a state machine, and at every decoding step the sampler masks out any token that would make the output invalid. Malformed output is not unlikely, it is unreachable.

Self-hosted, this is a first-class feature: vLLM exposes guided decoding backed by libraries like Outlines and XGrammar (the parameter is guided_json in older releases and folded into structured-output options in newer ones, so check your version), and it accepts JSON Schema, regex, a choice list, or a full context-free grammar.

from pydantic import BaseModel, Field
from typing import Literal

class ExtractedInvoice(BaseModel):
    vendor_name: str
    invoice_number: str
    total_cents: int = Field(ge=0)
    currency: Literal["USD", "EUR", "GBP"]
    due_date: str = Field(pattern=r"^\d{4}-\d{2}-\d{2}$")
    confidence: Literal["high", "medium", "low"]

resp = client.chat.completions.create(
    model="internal-general-v3",
    messages=[...],
    extra_body={"guided_json": ExtractedInvoice.model_json_schema()},
    temperature=0,
)

Through hosted APIs the equivalents are strict JSON-schema response formats and native tool calling. Tool calling is usually the better instrument when the model must also decide whether to produce output at all, since "call this tool or don't" is a cleaner contract than "return this object or a sentinel."

Two costs to know. Compiling a grammar takes real time on first use, so cache compiled schemas rather than rebuilding per request. And over-constraining can degrade quality: if the grammar forces a token the model finds deeply improbable, you get a valid object with worse contents. A model boxed into an enum it does not believe in will pick the least-bad member, which is why the confidence field above earns its place.

Layer 2: schemas designed for models

The schema that fits your database is usually not the schema the model handles best. Five rules that measurably improve conformance:

Flat beats nested. Every level of nesting is another chance to close a brace early. Two shallow calls usually beat one deeply nested one.

Enums beat free text wherever the value space is closed. Literal["approved","rejected","needs_review"] cannot drift into "Approved (pending)".

Avoid unions and anyOf. They compile into large state machines and confuse models. Prefer a discriminator field plus a flat set of optional fields.

Name fields semantically. total_cents outperforms amt, because the field name is part of the prompt. Include units in the name and you eliminate an entire class of error.

Give the model an escape hatch. This is the one teams skip and regret. If a field cannot be determined from the input, the model needs somewhere honest to put that, or it will invent a value. An explicit not_found enum member, a nullable field with an accompanying missing_fields: list[str], or a confidence: low signal all work. Without one, your extraction pipeline's failure mode is confident fabrication, which is far worse than a null.

Integers for money, ISO strings with a regex for dates, and no floats where exactness matters. A model that emits 1234.56 will eventually emit 1234.560000000001.

Layer 3: the retry ladder

Even with constrained decoding, calls fail: timeouts, truncation at max tokens, semantically valid output that violates a business rule. The response should be a ladder with a defined bottom, not a while True.

async def extract(doc: str, schema: type[BaseModel], budget: int = 3):
    attempts, last_error = [], None

    for step in range(budget):
        cfg = LADDER[min(step, len(LADDER) - 1)]
        raw = await invoke(cfg.model, render(cfg.prompt, doc, last_error),
                           guided_json=schema.model_json_schema(),
                           temperature=cfg.temperature)
        attempts.append(raw)

        try:
            obj = schema.model_validate_json(raw.text)     # structural
            violations = business_rules(obj, doc)          # semantic
            if violations:
                last_error = f"Rule violations: {violations}"
                continue
            return Ok(obj, attempts=step + 1)
        except ValidationError as e:
            last_error = summarize_errors(e)               # feed back exactly what broke

    # Bottom of the ladder: fail closed, hand to a human, keep the evidence.
    return NeedsReview(reason=last_error, attempts=attempts)

LADDER = [
    Rung(model="small-v2",  prompt="extract/v4", temperature=0.0),
    Rung(model="small-v2",  prompt="extract/v4-repair", temperature=0.2),
    Rung(model="general-v3", prompt="extract/v4-repair", temperature=0.0),
]

Four properties make this work. The ladder escalates (cheap model first, stronger model only when needed), which is also the cost pattern. It feeds the specific error back into the repair prompt, since "field due_date did not match pattern" is far more actionable to a model than "invalid output". It has a hard budget, because unbounded retries against a model that cannot do the task is how a nightly job becomes a five-figure invoice. And it fails closed into human review rather than returning a partial object, which matters enormously when the caller is an agent that will act on whatever it receives.

Semantic validation deserves emphasis. Schema conformance says the shape is right; business rules say the content is possible. Line items summing to the stated total, a due date after the invoice date, a counterparty that exists in your vendor master. These catch the failures that structural validation cannot see, and they are cheap deterministic code.

Retries and side effects

The moment retries touch tool calls, idempotency stops being optional. A retried extraction is harmless. A retried create_payment is a duplicate payment.

Every mutating tool needs an idempotency key derived from the logical operation, not from the attempt: hash the (session, step, normalized arguments) and let the downstream system deduplicate. This belongs in the tool gateway, next to the rest of the enforcement, so that no individual tool implementation has to remember it.

Streaming, honestly

Streaming and structured output pull against each other. You can stream constrained JSON and parse it incrementally with a partial parser, but you cannot validate business rules until it is complete, and you cannot un-show a field you already rendered.

The pattern that works: stream the human-facing prose, do not stream the machine-consumed object. If the UI needs progressive feedback for a long extraction, stream status events (a per-field "extracted" ping) rather than the partial object itself. Rendering a half-parsed object leads to values flickering on screen and correcting themselves, which reads as a broken product even when the final answer is right.

What to measure

Four metrics, on a dashboard, per prompt version:

• First-attempt validity. The real health signal. If it drops, something changed: model, prompt, or input distribution.
• Repair rate and ladder depth. How often you escalate, and to which rung. Rising depth is a quality regression that accuracy metrics miss.
• Per-field null and not_found rates. A field that suddenly goes 40% null means an upstream format changed. This is the earliest warning in the whole system.
• Human-review rate from the bottom of the ladder, which is both your cost and your accuracy backstop.

Wire first-attempt validity and schema conformance into the promotion gate as a near-binary floor, and into the canary rollback triggers. Structure is one of the few LLM properties you can assert deterministically, which makes it the most reliable early-warning signal you have.

The short version

Constrain generation so invalid output cannot be produced. Design schemas for the model: flat, enumerated, semantically named, with an honest escape hatch. Escalate through a bounded ladder that feeds errors back and fails closed into human review. Make mutating tools idempotent before you add retries. Do not stream the object. Then measure first-attempt validity like you measure uptime, because for a pipeline it is the same thing.