LLM Cost Engineering: Token Accounting, Cache Economics, and Cascade Routing.

The metric almost everyone optimizes is the wrong one

Teams negotiate token prices, switch to a cheaper model, and watch the bill go up. The reason is that the unit that matters is cost per successfully completed task, and the cheaper model needed three attempts, escalated 30% of the time, and produced output a human had to fix.

Everything below is about that number. It has four inputs: tokens consumed per attempt, attempts per task, the share of work that escalates, and the human time spent cleaning up. Optimizing any one in isolation reliably moves the others in the wrong direction.

Where the tokens actually go

Instrument before you optimize. In nearly every system we audit, the distribution surprises the team that built it.

The system prompt is a tax on every call. A 2,000-token system prompt in a feature handling a million calls a month is two billion tokens of pure overhead. It is also the easiest thing to cache, which we come to below.

Retrieved context usually dominates. A RAG call with k=10 chunks at 800 tokens each is 8,000 tokens of input against maybe 300 tokens of output. Input is cheaper per token, but at 25:1 ratios the input side is the whole bill. Halving k, or reranking 20 candidates down to 5 good ones, cuts more cost than any model swap. Measure the accuracy delta before you do it, then keep the smallest k that holds quality.

Agent loops are quadratic. This is the one that produces shocking invoices. Each turn typically re-sends the entire conversation, so an agent that takes n turns consumes roughly O(n²) input tokens. A 40-turn run is not four times a 10-turn run, it is closer to sixteen.

turn 1:  3k tokens
turn 10: 3k + 9 × (tool results + reasoning)   ≈ 25k
turn 40: ...                                    ≈ 180k

Total for a 40-turn run: several million input tokens.

The mitigations are structural: compact tool results before they enter history (a 50KB JSON response becomes a 400-token summary), summarize older turns past a threshold, and cap turns with a budget that fails into human handoff. Turn caps are also a safety control, since a looping agent and a hijacked agent look identical on a cost graph.

Reasoning tokens are invisible and expensive. Models that think before answering bill for those tokens. A task that looks like 500 output tokens can be 5,000. Track reasoning tokens as a separate line or your forecasts will be wrong by an order of magnitude.

Prompt caching, and the one rule that makes it work

Prompt caching stores the computed state for a prefix so repeated calls skip recomputation. Hosted APIs bill cached input at a steep discount; self-hosted, this is vLLM's prefix caching, where the win is throughput and latency rather than a line item.

The mechanism is a prefix match, and that single fact determines your whole prompt layout:

[ system prompt ][ tool schemas ][ few-shot examples ][ retrieved docs ][ user turn ]
└──────────── stable, cacheable ─────────────┘└──── volatile ────┘

Anything that changes invalidates everything after it. Put a timestamp, a request ID, or the user's name at the top of your system prompt and your cache hit rate is zero, permanently, for every call. We have found exactly this in production more than once: a well-meaning Current time: {now} line at the top of a 3,000-token system prompt, silently costing the full uncached rate on every request.

Order stable to volatile, keep the stable block genuinely stable (version it, do not template it), and put retrieved documents after the fixed material. Then measure the hit rate, because a cache you have not measured is a cache you do not have.

Cascade routing with a quality floor

Send everything to the strongest model and you overpay for the 80% of requests that are easy. Send everything to a small model and you pay in retries, escalations, and human corrections.

The cascade: try cheap, verify, escalate only on failure. It only saves money when the verifier is cheap and reliable, and it is worth writing down when that holds.

async def cascade(task, budget: CostBudget):
    small = await invoke("small-v2", task)          # ~1/15th the cost

    verdict = verify(small, task)                    # deterministic where possible
    if verdict.ok:
        record(task, tier="small", escalated=False)
        return small

    record(task, tier="small", escalated=True, reason=verdict.reason)
    return await invoke("general-v3", task)          # the strong model

verify is the whole design. Deterministic checks are ideal: schema validity, business rules, arithmetic that must reconcile, a citation that must exist in the retrieved set. A cheap classifier is acceptable. An LLM judge as expensive as the strong model defeats the purpose entirely.

The arithmetic that tells you whether to bother: with escalation rate e, cheap cost c, strong cost s, and verification cost v, the cascade costs c + v + e·s against a baseline of s. At c = s/15 and v ≈ 0, it wins while escalation stays under roughly 90%. That sounds generous until you notice the second-order effects: escalated requests pay double and take twice as long, so p95 latency degrades even when the average improves. Watch the tail, not the mean.

One trap: routing by a model's self-reported confidence. Models are poorly calibrated, and a small model is often most confident exactly where it is wrong. Route on verifiable properties.

Also boring and effective: batch APIs for anything not latency-sensitive (nightly extraction, backfills, evals), typically at half price, and turning off reasoning for tasks that do not need it.

Attribution, or you cannot manage any of it

You cannot control what you cannot attribute. Cost belongs on every request, tagged at the gateway:

@dataclass
class UsageRecord:
    request_id: str
    tenant_id: str          # who to bill or show
    feature: str            # "credit-memo", "support-triage"
    model_hash: str         # cost per model version, not per alias
    prompt_version: str     # prompt changes move cost
    input_tokens: int
    cached_input_tokens: int   # measure your cache hit rate
    reasoning_tokens: int
    output_tokens: int
    cost_micros: int
    attempt: int            # retries are cost
    escalated_from: str | None
    task_succeeded: bool    # THE denominator

task_succeeded is what turns this from a token report into cost per successful task. cached_input_tokens is what tells you whether your prompt layout is working. attempt is what reveals that your "cheap" model is not.

With this you can answer the questions that actually drive decisions: which feature is 60% of the bill, which tenant is unprofitable, whether last week's prompt change raised cost per task, and what your cache hit rate is by feature.

Guardrails so a bug cannot become an invoice

Runaway spend is nearly always a loop, not gradual growth. Enforce at the gateway, not in application code:

• Per-tenant and per-feature daily ceilings, with the request rejected rather than queued when hit.
• Per-session token budgets for agents, failing into human handoff.
• A circuit breaker on rate of change: if hourly spend for a feature exceeds several times its trailing average, page someone and throttle. This catches the retry storm at hour one instead of on the monthly invoice.
• Alerts on cost per successful task, not total spend. Total spend rising with usage is a business succeeding; cost per task rising is a regression.

The order of operations

In our experience the savings come in this order, and most teams try them in exactly the reverse: fix the cache-busting prefix (often 40% or more on chatty features, and it takes an afternoon), then cut retrieved context with reranking, then compact agent history and cap turns, then move batchable work to batch pricing, then cascade route, and only then negotiate rates or swap models.

The first four are engineering changes with no quality risk and no vendor conversation. The last two involve real trade-offs. Doing them in the wrong order is how teams end up with a slightly cheaper model, a worse product, and a bill that did not move.