A vLLM Production Reference: GPU Sizing, KV Cache Math, and the Configs.

The question every self-hosting project stalls on

"How many GPUs do we need?" gets answered badly in most planning docs, usually by dividing parameter count by GPU memory and adding a fudge factor. That estimate is wrong in a specific and expensive way: it accounts for the weights, which are static, and ignores the KV cache, which is what actually determines how many concurrent users you can serve.

This post is the arithmetic, the configuration, and the benchmark methodology we use when standing up a self-hosted stack. Everything here is vendor-neutral and runs on any CUDA-capable fleet.

Part 1: the memory budget

Three things occupy GPU memory during inference. Size all three or you will discover the missing one in production.

Weights

Straightforward: parameter count times bytes per parameter.

A 70B model in BF16 does not fit on a single 80GB card once you account for anything else. That is the first fork in the road: quantize, or shard across GPUs with tensor parallelism.

KV cache: the part people forget

Every token in every active sequence stores a key and a value vector in each layer. The per-token cost:

bytes_per_token = 2 (K and V)
                × num_layers
                × num_kv_heads
                × head_dim
                × dtype_bytes

Note num_kv_heads, not num_attention_heads. Modern models use grouped-query attention (GQA), where many query heads share one KV head, and this is the single biggest reason KV cache got cheaper over the last two years. Read the values off the model's config.json; do not assume.

Worked example, a 32B-class model with 64 layers, 8 KV heads, head dim 128, cache in FP16:

2 × 64 × 8 × 128 × 2 bytes = 262,144 bytes ≈ 0.25 MB per token

32k-token context, one sequence:   ~8 GB
64 concurrent sequences @ 8k avg:  ~128 GB

That second line is the one that decides your cluster size. Weights for this model in FP8 are 32GB and static. Serving 64 users at 8k context needs four times that in cache alone. Concurrency is a memory purchase, not a software setting.

Two practical levers: quantize the cache to FP8 (halves it, with a quality check required), and cap --max-model-len to what the workload truly needs. A team that sets 128k context because the model supports it, when their documents are 6k tokens, has cut their concurrency by an order of magnitude for nothing.

Activations and overhead

Reserve headroom for activation memory, CUDA graphs, and fragmentation. In practice --gpu-memory-utilization 0.90 is a safe starting point; vLLM will use that fraction of the card and allocate the remainder after weights to KV blocks. Going to 0.95 buys concurrency and buys OOM crashes under burst.

Putting it together

usable_kv_bytes = (gpu_bytes × utilization) − weight_bytes − overhead
max_tokens_in_flight = usable_kv_bytes / bytes_per_token
max_concurrent_seqs  = max_tokens_in_flight / avg_sequence_length

Run this before you buy anything. It converts "we need some H100s" into a number you can defend in a budget review.

Part 2: the serving configuration

A production vLLM launch, with the reasoning attached.

vllm serve /models/qwen3.5-32b-fp8 \
  --served-model-name internal-general-v3 \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --disable-log-requests \
  --host 127.0.0.1 --port 8000

--served-model-name is an internal alias, not the upstream name. Applications reference internal-general-v3; swapping the underlying weights is then a deploy, not a code change across every caller. Version the alias so a rollback is unambiguous.

Load from a local path, never a hub URL. The path points at an artifact your registry already hash-verified. Combined with trust_remote_code left off, this closes the supply-chain path that has burned teams repeatedly.

--tensor-parallel-size shards the model across GPUs in one node. Use it when weights plus target cache exceed one card. It costs inter-GPU communication on every forward pass, so it is a fit decision, not a performance one; two cards do not give you double throughput on a model that fit in one.

--max-model-len is your concurrency dial, per the math above. Set it from measured p99 input length plus max generation, not from the model card.

--enable-prefix-caching is close to free money for RAG and multi-turn: shared prompt prefixes (system prompt, retrieved chunks, conversation history) are cached across requests instead of recomputed. On prefix-heavy workloads this is the single largest throughput win available, and it is why SGLang is worth benchmarking against vLLM for that shape of traffic.

--kv-cache-dtype fp8 roughly halves cache memory. Treat it as a model change and re-run evals; on most current models the quality delta is negligible, but "most" is not "yours."

--disable-log-requests matters for regulated deployments: the default request logging writes prompt content to stdout, which then lands in your log aggregator, which is very likely not cleared for client data. Log metadata at the gateway instead, where redaction lives.

Bind to localhost. vLLM's OpenAI-compatible server has no real authorization model. Anything on the network that reaches port 8000 can use your model and read the served model list. It sits behind the gateway, always.

Part 3: the deployment shape

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-general
spec:
  replicas: 2
  template:
    spec:
      # Weights arrive via a read-only volume from the internal registry.
      # No pod pulls from a public hub at runtime.
      containers:
        - name: vllm
          image: registry.internal/vllm:v0.9.2
          args: ["--model", "/models/qwen3.5-32b-fp8", "--port", "8000"]
          resources:
            limits:
              nvidia.com/gpu: 2
              memory: 64Gi
          volumeMounts:
            - { name: models, mountPath: /models, readOnly: true }
            - { name: shm, mountPath: /dev/shm }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 180 # weight loading is slow; be patient
          startupProbe:
            httpGet: { path: /health, port: 8000 }
            failureThreshold: 60
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: model-registry-ro }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 16Gi }

Three details that cause real outages if missed. Shared memory: tensor parallelism uses /dev/shm for inter-process communication, and the container default of 64MB will hang or crash multi-GPU serving. Probe timing: loading tens of gigabytes of weights takes minutes, and an impatient liveness probe will restart the pod forever in a loop that looks like a GPU fault. Read-only weights: the serving pod has no business writing to the model volume.

Around this, the network policy does the security work: no egress from the serving namespace, ingress only from the gateway.

Part 4: benchmark before you believe anything

Vendor throughput numbers are measured on workloads that are not yours. Measure with your own distribution of input and output lengths, at the concurrency you actually expect.

python -m vllm.entrypoints.cli.benchmark serve \
  --model internal-general-v3 --base-url http://127.0.0.1:8000 \
  --dataset-name random \
  --random-input-len 6000 --random-output-len 500 \
  --max-concurrency 32 --num-prompts 500

Four numbers matter, and they trade against each other:

• TTFT (time to first token): dominated by prompt processing. Prefix caching moves this most.
• TPOT (time per output token): the streaming feel. Memory bandwidth bound.
• Throughput (tokens/sec across all requests): what your batch jobs care about.
• p99 latency at target concurrency: what your users feel when the system is busy.

Sweep concurrency from 1 upward and plot throughput against p99. Throughput climbs, then flattens while p99 keeps rising: that knee is your operating limit, and it is the number that belongs in the capacity plan. Set --max-num-seqs near it so the scheduler queues instead of thrashing.

Re-run the sweep on every model change, every quantization change, and every vLLM upgrade. Ten minutes of benchmarking is cheaper than one incident review.

Part 5: what to monitor

Beyond the usual GPU utilization and memory, three vLLM-specific signals predict trouble:

KV cache utilization (vllm:gpu_cache_usage_perc). Sustained above ~90% means requests are about to queue or get preempted. This is your true capacity metric, more than GPU utilization.

Preemption count (vllm:num_preemptions_total). Non-zero means the scheduler is evicting sequences to make room and recomputing them later. Latency will be spiky and users will describe it as "randomly slow."

Queue time versus inference time. If queue time dominates, add replicas. If inference time dominates, you need faster hardware or a smaller model. These have completely different budget conversations attached, and conflating them wastes quarters.

The honest summary

Self-hosting is not hard because inference is hard; vLLM and SGLang solved the hard parts. It is hard because capacity planning is unforgiving and the failure modes are quiet. Do the KV cache math before the purchase order, cap context to the workload, keep the server behind a gateway with no egress, benchmark your own traffic shape, and alert on cache utilization rather than GPU utilization.

Get those five right and a self-hosted cluster is one of the more boring pieces of infrastructure you will run, which is precisely the goal when the alternative is explaining to an examiner where your customer data went.