Self-Hosting LLMs for Finance: The Stack That Passes a Bank Review.

The question changed this year

Two years ago, "should we self-host the model?" was a question you asked to look thorough before signing the API contract. The open-weight options were a step behind on quality, the serving stack was a research project, and the GPU math only worked if you ignored the engineers babysitting it.

That is no longer the situation. The current open-weight tier, DeepSeek V4 and V4 Flash from March and April, Qwen 3.5 under Apache 2.0 from February, Gemma 4 under Apache 2.0 from April, plus Llama 4 under Meta's community license, matches closed frontier models on most production workloads that finance teams actually run: document extraction, classification, summarization, structured drafting, and agent tool-calling against internal systems. The serving stack matured with them. vLLM is production-grade, SGLang wins on prefix-heavy RAG and multi-turn serving, and both speak an OpenAI-compatible API so your application code does not care what is behind it.

For a bank, an insurer, or a fund, that changes the default. When the model runs inside your boundary, an entire class of vendor questions disappears: no customer data leaves your network, no retention policy to negotiate, no third-party subprocessor to explain to your regulator, no surprise deprecation of the model version your validation work was done against.

The economics, honestly

Self-hosting is not automatically cheaper. It is cheaper at utilization. The break-even analyses published this year cluster around the same range: an owned or reserved GPU cluster beats API pricing once you sustain roughly 50 to 80 percent utilization, and a fully utilized cluster lands at a fraction of equivalent API cost over a multi-year horizon.

The corollary is the honest version of the pitch: if your workload is a few million tokens a day of bursty, latency-tolerant traffic, stay on an API behind a gateway and spend your effort on evals and monitoring. Self-hosting earns its keep when you have steady batch volume (nightly document processing, reconciliation, screening), strict data-boundary requirements, or both. Most regulated teams we see have both.

The reference architecture

The stack that holds up in review has four layers, and none of them is exotic.

Serving. vLLM for general and batch throughput, SGLang where the workload is dominated by shared prefixes, which is most RAG and most multi-turn assistants. Run it on your own metal or in your own cloud accounts. Pin the model by content hash, not by name.

Gateway. One internal endpoint, OpenAI-compatible, in front of every model you serve. This is where authentication, per-team quotas, request logging, and PII redaction live. It is also your swap point: when a better open-weight model ships, and this year one ships every few weeks, you re-qualify it behind the gateway and no application changes.

Isolation. The serving cluster gets no outbound internet. Weights come in through an internal registry, requests come in through the gateway, logs go out to your SIEM. If a model server can reach the internet, you have built an exfiltration path and called it infrastructure.

Evidence. Every request and response logged with model hash, prompt version, and caller identity. This is the difference between "we think the March model produced that memo" and an answer your auditor accepts.

apps ──▶ gateway (auth · quotas · logging · redaction)
              │
              ▼
      vLLM / SGLang cluster  (no egress · pinned weights)
              ▲
   internal model registry (hash-verified weights)
              │
   eval gate  (must pass before a model reaches the registry)

Yes, it is still a model under SR 11-7

Swapping an API for your own cluster does not move you out of model risk territory. If the system produces outputs that business decisions rely on, it is a model under SR 11-7, whether the weights came from a vendor contract or a download. What self-hosting changes is that you finally control the whole validation surface: the weights are frozen until you change them, the eval suite runs against the exact artifact in production, and no vendor can silently update the model out from under your validation work. Treat the eval gate as the promotion mechanism: no model hash reaches the serving registry without passing the suite your validators signed.

The U.S. Treasury's financial services AI guidance and the NIST AI Risk Management Framework both point the same direction: know what the model is, prove what it does, and log what it did. A self-hosted stack makes all three easier, not harder, because there is no vendor between you and the evidence.

Where teams get burned

Four failure modes come up in nearly every review we run.

1. Weights from anywhere. Engineers pull model files from public hubs straight onto production nodes, with remote code execution enabled in the loader. Your model pipeline is a software supply chain; treat downloads like unreviewed dependencies.
2. Quantization without re-validation. The INT4 version of a model is a different model. If you quantize to fit hardware, the eval suite runs again. "It is the same weights, just smaller" is not a sentence your validator should ever hear.
3. No egress control. The cluster that holds your most sensitive prompts can phone home. This is the first thing we look for and the most common finding.
4. One heroic engineer. The stack works because one person understands it. That is an operational risk finding, not an engineering achievement. Runbooks, on-call, and a second pair of hands are part of the build.

The order of operations

Audit what you have first: where prompts flow today, which vendors see what, what your utilization actually looks like. Then pick the one workload where the boundary requirement is hardest, usually the one nobody has been allowed to ship, and stand the stack up for that. Qualify one model, wire the eval gate, run it for a month with logging on. Expand from a working, evidenced system rather than a platform diagram.

Owning the stack is the point. When the review lands, the answer to "where does our data go?" is a network diagram you control, and the answer to "which model produced this?" is a hash in your own logs.