Run a LiteLLM Proxy on Kubernetes as the LLM Gateway for Your DevOps AI Agents: Per-Agent Budgets, Rate Limits, Fallback Routing, and a Kill Switch

Every ops agent gets its own key, budget, and off switch

Put a LiteLLM proxy between your DevOps AI agents and the model providers, and three things you cannot do today become one config change: cap what each agent may spend per month, rate-limit it so a runaway loop cannot burn the whole org quota, and swap or fail over the model behind it without redeploying the agent. Agents stop holding provider API keys at all. They hold a virtual key issued by the gateway, scoped to one team, a model allowlist, and a budget that returns an error instead of a bill when it runs out.

This is the LLM-side twin of the MCP gateway pattern. That gateway controls what tools an agent may call against your infrastructure. This one controls what the agent may spend on thinking about it. You want both, because the two failure modes are independent: an agent with a perfect tool allowlist can still loop on a read-only query until the invoice arrives.

Here is the full setup on Kubernetes: the proxy deployment, the routing config, the per-agent keys, the one-line change in the agent, the Prometheus alerts, and the limits that nobody's quickstart mentions.

What the gateway owns

Keep responsibilities sharp or the proxy turns into another place where policy hides. The gateway owns:

• Credential custody. Provider keys live only in the proxy's Secret. Agents never see them, which closes the biggest hole described in secrets management for AI agents.
• Identity and budget. One LiteLLM team per agent, one virtual key per agent-environment pair. Budgets and rate limits attach to both layers.
• Model aliasing and failover. Agents ask for strong or cheap. The gateway decides which real model backs each alias today, and where to fail over when a provider returns 529s.
• Spend telemetry. One /metrics endpoint labelled by team, key alias, and model, instead of each agent instrumenting its own usage.

It does not own prompt design, tool-result truncation, or the turn budget inside the agent loop. Those stay in the harness, exactly as laid out in the token economics post. The gateway is the backstop for when the harness-side budget has a bug.

Deploy the proxy

Virtual keys, teams, and budgets need a database. Spend tracking across more than one replica needs Redis. Neither is optional for the setup below, so provision a small Postgres (a db.t4g.micro RDS or an in-cluster CloudNativePG instance both work) and a Redis before you start.

# litellm-config.yaml  (mounted as a ConfigMap)
# Model IDs below are illustrative; paste the exact IDs from your provider's model list.
model_list:
  - model_name: strong                      # alias agents ask for
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: strong                      # second deployment, same alias
    litellm_params:
      model: bedrock/anthropic.claude-sonnet-5
      aws_region_name: us-east-1
  - model_name: cheap
    litellm_params:
      model: anthropic/claude-haiku-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  redis_host: os.environ/REDIS_HOST          # shares rate-limit + budget counters across replicas
  redis_port: 6379
  allowed_fails: 3                           # failures per deployment before cooldown
  cooldown_time: 60                          # seconds a failing deployment is skipped

litellm_settings:
  num_retries: 2
  request_timeout: 90
  fallbacks:
    - cheap: ["strong"]                      # if every cheap deployment is cooling down
  context_window_fallbacks:
    - cheap: ["strong"]                      # oversized prompt on the small model: escalate
  drop_params: true                          # strip provider-specific params on failover
  callbacks: ["prometheus"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: false                   # config is the source of truth, not the UI

Two deployments share the alias strong. LiteLLM load-balances between them and, when the direct Anthropic endpoint trips allowed_fails, routes everything to Bedrock for cooldown_time seconds. The agent sees one slightly slower request, not a failed incident investigation.

Notice what is not in litellm_settings: response caching. LiteLLM can cache completions in Redis, and for a chatbot that is free money. For an ops agent it is a correctness bug, because "is the pod still crash-looping" must never return a ten-minute-old answer. Use Redis for the router's shared counters only.

# litellm-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
  namespace: agents
spec:
  replicas: 2
  selector:
    matchLabels: { app: litellm }
  template:
    metadata:
      labels: { app: litellm }
    spec:
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:main-stable
          args: ["--config", "/app/config.yaml", "--port", "4000"]
          ports: [{ containerPort: 4000 }]
          envFrom:
            - secretRef: { name: litellm-secrets }   # LITELLM_MASTER_KEY, DATABASE_URL, ANTHROPIC_API_KEY, REDIS_HOST
          volumeMounts:
            - { name: config, mountPath: /app/config.yaml, subPath: config.yaml }
          readinessProbe:
            httpGet: { path: /health/readiness, port: 4000 }
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /health/liveliness, port: 4000 }
            periodSeconds: 30
          resources:
            requests: { cpu: 250m, memory: 512Mi }
            limits: { memory: 1Gi }
      volumes:
        - name: config
          configMap: { name: litellm-config }
---
apiVersion: v1
kind: Service
metadata:
  name: litellm
  namespace: agents
spec:
  selector: { app: litellm }
  ports: [{ port: 4000, targetPort: 4000 }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: litellm
  namespace: agents
spec:
  minAvailable: 1
  selector:
    matchLabels: { app: litellm }

The readiness path is spelled /health/liveliness on the liveness side. That is LiteLLM's spelling, not a typo, and copying /health/liveness from habit gives you a 404 and a pod that restarts every 30 seconds.

Do not expose the Service outside the cluster. Agents reach it at http://litellm.agents.svc:4000, and the one NetworkPolicy you want allows ingress only from pods labelled as agents.

Create a team and a key per agent

The master_key is the admin credential. It creates teams and keys and must never reach an agent pod. Run this from a one-off Job or your laptop, not from the agent's manifest.

LITELLM=http://litellm.agents.svc:4000
MASTER="Authorization: Bearer $LITELLM_MASTER_KEY"

# One team per agent. The budget lives here so rotating a key never resets spend.
curl -s -X POST "$LITELLM/team/new" -H "$MASTER" -H 'Content-Type: application/json' -d '{
  "team_alias": "oncall-agent",
  "models": ["strong", "cheap"],
  "max_budget": 400,
  "budget_duration": "30d",
  "tpm_limit": 400000,
  "rpm_limit": 120,
  "metadata": {"owner": "sre-platform", "cost_center": "CC-4411"}
}' | tee /tmp/team.json
TEAM_ID=$(jq -r .team_id /tmp/team.json)

# One key per agent-environment. Tighter limits than the team; expires on its own.
curl -s -X POST "$LITELLM/key/generate" -H "$MASTER" -H 'Content-Type: application/json' -d "{
  \"team_id\": \"$TEAM_ID\",
  \"key_alias\": \"oncall-agent-prod\",
  \"models\": [\"strong\", \"cheap\"],
  \"max_budget\": 300,
  \"budget_duration\": \"30d\",
  \"rpm_limit\": 60,
  \"max_parallel_requests\": 8,
  \"duration\": \"90d\",
  \"metadata\": {\"env\": \"prod\"}
}" | jq -r .key

Three decisions are baked in here. The team budget exceeds the key budget, so a single environment cannot consume the whole team allowance and starve staging. Budgets have a budget_duration, so they reset monthly instead of silently becoming a lifetime cap that trips in month three. Keys expire, which forces a rotation path to exist before you need it.

The CI triage bot gets its own team with models: ["cheap"] only. It physically cannot request the strong model, no matter what its prompt says. That is the model-side equivalent of the per-agent tool allowlist in the MCP gateway.

Put the returned key in a Kubernetes Secret in the agent's namespace. The agent's manifest references that Secret and nothing else.

The agent change is one line

Every agent in this series calls the provider SDK directly. Switching to the gateway means pointing the client at the Service and passing the virtual key. LiteLLM exposes an Anthropic-format /v1/messages endpoint as well as the OpenAI-format /v1/chat/completions, so the Anthropic SDK works unchanged:

import os
import anthropic

client = anthropic.Anthropic(
    base_url=os.environ["LLM_GATEWAY_URL"],     # http://litellm.agents.svc:4000
    api_key=os.environ["LLM_GATEWAY_KEY"],      # the virtual key, not a provider key
)

resp = client.messages.create(
    model="strong",                              # the alias; the gateway picks the deployment
    max_tokens=1024,
    system=[{"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}}],
    tools=TOOLS,
    messages=messages,
    extra_headers={"x-litellm-tags": "run=" + RUN_ID},   # shows up in spend logs
)

The deterministic router from the token economics post now returns "cheap" or "strong" instead of a model ID. Upgrading every agent to a new model generation becomes a ConfigMap edit and a rolling restart of the proxy. No agent image changes.

Check that prompt caching survives the hop before you trust the cost model. Print resp.usage on a two-turn test and confirm cache_read_input_tokens is non-zero on the second call. If it is zero, the proxy is rewriting the system block, and your cache-hit-ratio alert will fire the moment you cut over.

What the agent sees when a limit trips

The harness has to handle three new responses, and it must not retry any of them:

Retrying a budget error is the worst thing you can do. The proxy denies it again, your retry loop spins, and the budget stays at zero while the agent produces nothing. Map the 400 to the same hard-stop path your circuit breaker uses, so a budget trip auto-demotes the agent and opens a ticket.

import anthropic

class LLMBudgetExhausted(RuntimeError):
    pass

def call_model(**kwargs):
    try:
        return client.messages.create(**kwargs)
    except anthropic.BadRequestError as e:
        msg = str(e)
        if "Budget has been exceeded" in msg or "ExceededBudget" in msg:
            raise LLMBudgetExhausted(msg) from e
        raise

Alert on budget, fallbacks, and cooldowns

With the prometheus callback enabled, scrape /metrics on port 4000. Check the callback's doc page for its current licensing note before you depend on it; the metric set has moved between the free and enterprise tiers more than once. The labels you care about are team_alias, api_key_alias, and model.

groups:
  - name: llm-gateway
    rules:
      - alert: AgentBudgetNearlyExhausted
        expr: |
          litellm_remaining_team_budget_metric
            / on (team, team_alias) litellm_team_max_budget_metric < 0.2
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.team_alias }} has under 20% of its monthly LLM budget left"

      - alert: AgentBudgetBurnTooFast
        # Spent >50% of a 30-day budget in the last 3 days = runaway or a new loop bug.
        expr: |
          sum by (team_alias) (increase(litellm_spend_metric[3d]))
            / on (team_alias) group_left litellm_team_max_budget_metric > 0.5
        labels: { severity: critical }

      - alert: LLMDeploymentCooledDown
        expr: increase(litellm_deployment_cooled_down[10m]) > 0
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.litellm_model_name }} at {{ $labels.api_base }} is in cooldown; traffic is on fallback"

      - alert: LLMFallbackFailing
        expr: increase(litellm_deployment_failed_fallbacks[10m]) > 0
        labels: { severity: critical }

The burn-rate alert is the one that matters. A remaining-budget alert tells you the month was expensive. A burn alert tells you this week an agent started behaving differently, which is almost always a prompt change or a tool that started returning five times more output. Pair it with the per-run spans from agent observability with OpenTelemetry: the x-litellm-tags header carries the run ID into the spend log, so one curl /spend/logs?request_id=... joins the gateway's cost to the trace of what the agent was doing.

The kill switch

Two commands, no redeploy:

# Block one agent-environment instantly. Existing in-flight requests finish; the next one gets 401.
curl -s -X POST "$LITELLM/key/block" -H "$MASTER" -H 'Content-Type: application/json' \
  -d '{"key": "sk-oncall-agent-prod-..."}'

# Put the whole team on a zero budget for the rest of the period.
curl -s -X POST "$LITELLM/team/update" -H "$MASTER" -H 'Content-Type: application/json' \
  -d "{\"team_id\": \"$TEAM_ID\", \"max_budget\": 0}"

Blocking the key is the right first move during an incident: it is reversible with /key/unblock, it leaves the spend history intact, and it works even if the agent is running somewhere you cannot reach. Deleting the key is for off-boarding.

Honest limits

The proxy is now on the critical path of every agent. Two replicas and a PodDisruptionBudget are the minimum. Size the Postgres connection pool for replicas × workers, because each LiteLLM worker opens its own pool and a cluster-wide node drain can briefly double connections.

Spend accuracy depends on LiteLLM's price table. It ships a model-to-price map and updates it regularly, but a brand new model ID, a Bedrock region with different pricing, or a provisioned-throughput commitment will be costed wrong until you set input_cost_per_token and output_cost_per_token on that deployment in model_list. Reconcile the gateway's monthly total against the provider invoice once; if they are within a few percent, trust the alerts.

Budgets are enforced at request start, not request end. A key with one dollar left can start a request that costs three. The overshoot is bounded by max_tokens and max_parallel_requests, which is why both are set on the key.

Fallback across providers changes behaviour. The same prompt on Bedrock and on the direct API should produce the same answer, but tool-call formatting edge cases differ by provider build. Run your agent eval suite against each deployment in the alias, not just the primary, or the first time you find out is during a real outage when the fallback is live.

This is not a tool gateway. A virtual key stops an agent from spending money. It does nothing to stop a hallucinated kubectl delete. That is the MCP gateway's job, and the two belong side by side in the agents namespace.

Takeaway

One LiteLLM proxy per cluster, one team per agent, one virtual key per agent-environment, budgets with a monthly reset, rate limits on both layers, two deployments behind each model alias, and the prometheus callback feeding a burn-rate alert. Agents switch over by changing a base URL and a key. From that point, "what does the on-call agent cost" is a PromQL query, "turn it off" is one HTTP call, and "move everything to the new model" is a ConfigMap edit. That is what it means to run LLM access as platform infrastructure instead of as a collection of API keys in a dozen Deployments.