LLM Inference Cost on Kubernetes: How to Cut Your Cost per Token

The Only Metric That Matters: Cost per Million Tokens

If you serve an open-source LLM on Kubernetes, your bill is not driven by requests, replicas, or even model size. It is driven by one ratio: GPU dollars per hour divided by tokens served per hour. A single A100 80GB runs roughly $3–5/hour on-demand at major clouds. Run it at 15% utilization and every million tokens costs you six times more than the exact same pod at 90% utilization. Most self-hosted inference setups I have audited sit at the bad end of that ratio — not because the GPU is slow, but because nobody is measuring the ratio at all.

This post is the cost-engineering companion to deploying and scaling LLMs on Kubernetes: how to compute cost per token from metrics you already have, and the four levers that actually move it — batching, model right-sizing, queue-based autoscaling, and cheaper GPU capacity.

Step 1: Measure Cost per Token with vLLM + Prometheus

vLLM exposes Prometheus metrics on its /metrics endpoint out of the box. The two counters you need:

vllm:prompt_tokens_total       # input tokens processed
vllm:generation_tokens_total   # output tokens produced

Add the DCGM exporter (ships with the NVIDIA GPU Operator) for hardware truth:

DCGM_FI_DEV_GPU_UTIL       # coarse GPU utilization %
DCGM_FI_PROF_SM_ACTIVE     # fraction of streaming multiprocessors doing work
DCGM_FI_DEV_FB_USED        # VRAM actually used (MiB)

Then wire cost per million tokens as a Prometheus recording rule. Hard-code the hourly node price per pool (or pull it from OpenCost — see the OpenCost vs Kubecost comparison for which tool fits your cluster):

groups:
  - name: llm-cost
    rules:
      - record: llm:tokens_per_second
        expr: |
          sum by (deployment) (
            rate(vllm:generation_tokens_total[10m])
            + rate(vllm:prompt_tokens_total[10m])
          )
      # a100-80gb on-demand, adjust to your negotiated rate
      - record: llm:gpu_cost_per_hour
        expr: |
          count by (deployment) (vllm:num_requests_running) * 3.67
      - record: llm:cost_per_million_tokens
        expr: |
          llm:gpu_cost_per_hour
          / (llm:tokens_per_second * 3600)
          * 1000000

Now you have a number to optimize instead of a vague feeling that GPUs are expensive. Graph it per deployment, per model, per team. If you already do showback, fold it into your per-namespace cost allocation so LLM workloads stop hiding inside a blended "ml" line item.

One warning about DCGM_FI_DEV_GPU_UTIL: it reports the fraction of time any kernel ran, so a GPU doing tiny batch-1 decodes can show 90% "utilization" while wasting most of its compute. DCGM_FI_PROF_SM_ACTIVE is the honest signal — if it sits below 0.3 while your queue is empty, you are paying for silicon that mostly waits.

Lever 1: Continuous Batching — Throughput Is Almost Free

The single biggest cost mistake is serving at low concurrency. LLM decoding is memory-bandwidth-bound, so a modern engine like vLLM can decode dozens of sequences concurrently for barely more wall-clock time than one. Going from batch size 1 to 32 typically multiplies tokens/second by 10–20x on the same GPU — which divides your cost per token by the same factor.

vLLM does this automatically (continuous batching), but its ceiling is configurable and the defaults are conservative for some workloads:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 64 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192 \
  --enable-prefix-caching

What each flag buys you:

• --max-num-seqs 64 raises the concurrent-sequence ceiling. Watch p95 time-to-first-token as you raise it; the tradeoff is real but usually favorable until the KV cache saturates.
• --gpu-memory-utilization 0.92 gives vLLM more VRAM for KV cache, which directly caps batch size. The default 0.90 leaves headroom you may not need on a dedicated node.
• --max-model-len 8192 — do not serve 128k context if your real p99 prompt is 6k tokens. KV cache is allocated per-token of possible context; oversizing it silently halves your effective batch.
• --enable-prefix-caching reuses KV cache across requests that share a prefix. If your traffic is agent-style — same long system prompt, different tails — this is a large win for both latency and throughput. The same prompt-structure discipline that makes agent token bills manageable makes prefix caching effective here.

Rule of thumb: before buying more GPUs, check vllm:num_requests_running. If it averages in the single digits on hardware that can batch 64, your next GPU dollar is wasted.

Lever 2: Right-Size the Model and Quantize

Cost per token scales with the hardware the model forces you onto. Three moves, in order of least to most effort:

Quantize. AWQ, GPTQ, or FP8 cuts weight memory roughly in half versus FP16 with minor quality loss for most serving workloads. That freed VRAM becomes KV cache (bigger batches) — or lets the model step down a hardware tier entirely:

--model TheBloke/Llama-3.1-8B-Instruct-AWQ --quantization awq

Step down the GPU. A quantized 8B model serves comfortably on an L4 (roughly $0.70–1.00/hour) instead of an A100 (roughly $3.70/hour). If tokens/second drops by 2x but the price drops by 4x, your cost per token just halved. Run the recording rule from Step 1 on both pools and let the number decide — never the spec sheet.

Route by task. Most production traffic mixes hard requests with trivial ones (classification, extraction, short summaries). A 8B model handles the trivial tier at a fraction of the 70B cost. Even a dumb router — regex on request type, or a header set by the caller — beats sending everything to the big model.

Lever 3: Autoscale on Queue Depth, Not GPU Utilization

HPA on GPU utilization fails for inference in both directions: a batch-1 workload shows high "utilization" while wasting the GPU (no scale-down where you want it), and a saturated engine shows ~100% flat (no gradient to scale up on). The signal that actually tracks user pain is vLLM's queue: vllm:num_requests_waiting.

KEDA's Prometheus scaler wires that up cleanly — the same pattern as any HPA/VPA/KEDA setup, just pointed at the right metric:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama31-8b
spec:
  scaleTargetRef:
    name: vllm-llama31-8b
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 600        # model load takes minutes; don't flap
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        query: |
          sum(vllm:num_requests_waiting{deployment="vllm-llama31-8b"})
        threshold: "16"       # queued requests per replica before scaling

Two inference-specific caveats:

• Cold starts are minutes, not seconds. Pulling a multi-GB image plus loading weights onto the GPU takes 2–5 minutes even with a warmed image cache. Set cooldownPeriod long, scale up early (low threshold), and pre-pull images with a DaemonSet on GPU nodes.
• Scale-to-zero is for internal tools only. minReplicaCount: 0 is a genuine saving for a dev assistant used a few hours a day, and a five-minute p100 latency for anything user-facing. Keep one warm replica for prod and let spot capacity absorb the peaks.

Lever 4: Spot GPUs and Fractional GPUs

Spot. Stateless inference is the ideal spot workload — an interrupted replica loses in-flight requests (clients retry) and nothing else. Spot GPU capacity is commonly 60–70% off on-demand. With Karpenter, let the baseline replica land on-demand and the scaled replicas chase spot:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-spot
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]   # spot preferred, falls back
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g6.xlarge", "g6.2xlarge"]
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized

Karpenter's consolidation also cleans up the other classic waste: GPU nodes idling after scale-down. If you are still on Cluster Autoscaler for GPU pools, the Karpenter vs Cluster Autoscaler cost comparison covers why that switch usually pays for itself.

Fractional GPUs. A whole A100 for a 3B embedding model is pure waste. On A100/H100, MIG partitions the card into isolated slices (nvidia.com/mig-1g.10gb: 1 instead of nvidia.com/gpu: 1) with real memory and fault isolation — fine for prod. Time-slicing (a GPU Operator config that oversubscribes nvidia.com/gpu) shares without isolation — one tenant can OOM another, so keep it for dev and batch.

A Worked Example

Take a Llama-3.1-8B service on one on-demand A100 at $3.67/hour, serving batch-heavy traffic at 2,500 tokens/second aggregate: that is 9M tokens/hour, or about $0.41 per million tokens. The same service left at the defaults — short max batch, 128k context reserved, single-digit concurrency — commonly does 300 tokens/second: $3.40 per million tokens. Same GPU, same model, 8x difference, all configuration. Quantize to AWQ, move to an L4 at $0.81/hour doing 900 tokens/second, and you are at $0.25 per million — below many hosted-API prices for comparable model quality, which is the actual bar self-hosting has to clear.

Those throughput numbers are illustrative arithmetic, not a benchmark of your stack — which is precisely why the recording rule comes first. Measure, pull one lever, watch llm:cost_per_million_tokens for a week, repeat. GPU bills do not shrink from architecture diagrams; they shrink from that loop.