What a DevOps AI Agent Costs to Run: Token Economics, Caching, and Model Routing

The cost of an ops agent is not the model you picked

The first surprise when you put an AI agent on-call is that the bill has almost nothing to do with how clever the model is. It comes from one structural fact: an agent re-sends its entire conversation on every turn. A six-step investigation with 3,000 tokens of genuinely unique content will bill you for 30,000 input tokens, because turn six carries turns one through five on its back.

That means the three levers that actually move your spend are: cache the part of the prompt that never changes, stop pouring raw tool output into context, and route cheap work to a cheap model. Everything else is rounding error. Here is the arithmetic, and the instrumentation to prove it on your own agent.

Where the tokens actually go

Take a realistic on-call agent: a policy system prompt plus six tool definitions with full JSON schemas (~2,400 tokens), an alert payload and cluster context (~800 tokens), tool results averaging ~600 tokens each because you summarize them, and ~150 output tokens of reasoning per turn.

Run the loop for six turns and the input side looks like this:

Roughly 90% of what you pay for is re-transmitted history. Output tokens — maybe 1,300 across the whole run — are a footnote even though they cost more per token. This is why "just use a smaller model" is usually the wrong first move: you can cut the same bill further by attacking the re-send, without giving up any reasoning quality.

Before you optimize anything, measure. Most providers expose a token-counting endpoint (Anthropic's is POST /v1/messages/count_tokens) that prices a request without running it. Wire it into a test and assert on the number:

def test_prefix_budget(client, system_prompt, tools):
    r = client.messages.count_tokens(
        model=MODEL, system=system_prompt, tools=tools,
        messages=[{"role": "user", "content": "placeholder"}],
    )
    # Tool schemas creep. Fail the build when the static prefix bloats.
    assert r.input_tokens < 3000, f"prefix grew to {r.input_tokens}"

That single test catches the most common regression in agent cost: someone adds a seventh tool with a chatty description, and every turn of every run gets 400 tokens heavier forever.

Lever 1: cache the prefix, and put volatile data after it

Your system prompt and tool definitions are byte-identical on every single turn of every single run. That is the ideal prompt-cache target. With caching, the first request writes the prefix to cache at a small premium and every subsequent request reads it back at a fraction of the base input rate.

resp = client.messages.create(
    model=MODEL,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"},   # breakpoint: everything above is cached
    }],
    tools=TOOLS,
    messages=messages,
)
u = resp.usage
print(u.input_tokens, u.cache_creation_input_tokens, u.cache_read_input_tokens)

Two details decide whether this works:

Order matters more than the flag. Caching matches on an exact prefix, so anything that changes per-run must sit after the breakpoint. Putting the current timestamp, cluster name, or alert ID in your system prompt invalidates the cache on every single invocation — you pay the write premium forever and never take a read. Keep the system prompt and tool schemas static; inject the volatile context as the first user message instead.

Cache the growing history too. Moving the breakpoint to the end of the message list each turn means each request pays full price only for its own delta. On the six-turn loop above, that drops full-rate input from 30,450 tokens to about 6,950, with the remaining ~23,500 billed as cache reads. Check your provider's current write and read multipliers on their pricing page before you model the savings — the multipliers change, the structure doesn't.

There's a scheduling consequence people miss. Cache entries have a TTL (five minutes by default on Anthropic, with a longer-TTL option at a higher write premium). An agent that fires once an hour never hits a warm cache and pays the write premium every time; an agent polling every two minutes keeps it warm and rides read rates all day. If you run a supervised agent loop on a timer, the timer interval is now a cost decision.

Lever 2: tool output is your biggest variable cost

Change one assumption in the table above — tool results at 4,000 tokens instead of 600, which is what you get from a raw kubectl logs --tail=200 or an unsummarized describe — and the same six-turn loop bills 81,450 input tokens instead of 30,450. Same model, same question, 2.7× the cost, and a worse answer because the signal is buried.

So summarize at the tool boundary, not in the prompt. The pattern from the read-only Prometheus MCP server — return first/last/min/max instead of 700 raw samples — generalizes to every tool you write:

MAX_TOOL_TOKENS = 800

def bounded(text: str, note: str = "") -> str:
    """Hard-cap any tool result before it enters context."""
    budget = MAX_TOOL_TOKENS * 4          # ~4 chars/token, good enough for a guard
    if len(text) <= budget:
        return text
    head, tail = text[: budget // 2], text[-budget // 2 :]
    return f"{head}\n...[truncated {len(text) - budget} chars]...\n{tail}\n{note}"

def get_pod_logs(namespace, pod, lines=200):
    raw = run(["kubectl", "logs", "-n", namespace, pod, f"--tail={lines}"])
    errors = [l for l in raw.splitlines() if "ERROR" in l or "FATAL" in l]
    if errors:
        # An error-only view is usually the whole answer at 5% of the tokens.
        return bounded("\n".join(errors[-40:]), note="filtered: ERROR/FATAL only")
    return bounded(raw)

Head-and-tail truncation beats a plain [:N] cut, because stack traces put the useful frames at the top and the exception at the bottom. And filtering to error lines before truncating usually answers the question outright — this is the same context engineering discipline that makes the agent more accurate, applied for its cost side effect.

Lever 3: route by task, not by taste

Most of what an ops agent does is not reasoning. It's classification: is this alert actionable? which runbook applies? is this a duplicate of an open incident? Those turns take ~2,500 input tokens, emit 50 output tokens, and do not need your most expensive model.

CHEAP, STRONG = "claude-haiku-4-5-20251001", "claude-sonnet-5"

def route(task: str, payload: dict) -> str:
    # Deterministic router. The model never picks its own tier.
    if task in ("classify_alert", "dedupe", "extract_fields", "summarize"):
        return CHEAP
    if task == "investigate" and payload.get("severity") in ("sev1", "sev2"):
        return STRONG
    return STRONG if payload.get("escalated") else CHEAP

Run the fleet math on 400 alerts/day. Send everything to the strong model at ~30k tokens each and you burn 12M tokens/day. Triage first — 340 alerts resolved by classification alone at ~2,500 tokens, 60 escalating to a full 30k-token investigation — and you're at 2.65M tokens/day, with only 1.8M of those on the expensive tier. That's an 85% cut in frontier-model tokens with no change to how deeply real incidents get investigated.

Two rules keep this honest. Make routing deterministic in code — an agent asked to choose its own model will escalate, because escalating always looks safer from inside the loop. And re-run your agent eval suite per tier: a downgrade that saves 60% and misclassifies one sev1 as noise is not a saving, it's an outage with a discount.

Lever 4: a hard budget, because loops don't stop themselves

A stuck agent re-reading the same logs is a runaway cost bug that no amount of prompt tuning prevents. Cap it in the harness:

MAX_TURNS, MAX_TOKENS, MAX_WALL_SECONDS = 12, 120_000, 180

class Budget:
    def __init__(self):
        self.turns = self.tokens = 0
        self.started = time.monotonic()

    def charge(self, usage) -> None:
        self.turns += 1
        self.tokens += (usage.input_tokens + usage.output_tokens
                        + usage.cache_creation_input_tokens
                        + usage.cache_read_input_tokens)

    def exceeded(self) -> str | None:
        if self.turns >= MAX_TURNS:
            return "turn_limit"
        if self.tokens >= MAX_TOKENS:
            return "token_limit"
        if time.monotonic() - self.started >= MAX_WALL_SECONDS:
            return "wall_clock"
        return None

When exceeded() trips, stop and hand off to a human with whatever the agent found — don't silently retry. Count cache reads in the total even though they're cheap; a loop burning 400k cached tokens is still a loop that isn't converging, and you want the abort signal either way.

Make cost a metric, not a monthly surprise

Emit usage as telemetry on every call, the same way you'd instrument any service. If you already trace your agents with OpenTelemetry, these are span attributes plus one counter:

TOKENS = Counter("agent_tokens_total", "Tokens billed",
                 ["agent", "model", "token_type"])

def record(agent, model, usage):
    for kind, n in (
        ("input", usage.input_tokens),
        ("output", usage.output_tokens),
        ("cache_write", usage.cache_creation_input_tokens),
        ("cache_read", usage.cache_read_input_tokens),
    ):
        TOKENS.labels(agent, model, kind).inc(n)

Then two recording rules turn that into the numbers a FinOps review can act on — cache effectiveness, and true unit economics:

groups:
  - name: agent-cost
    rules:
      - record: agent:cache_hit_ratio
        expr: |
          sum by (agent) (rate(agent_tokens_total{token_type="cache_read"}[1h]))
          /
          sum by (agent) (rate(agent_tokens_total{token_type=~"input|cache_read|cache_write"}[1h]))

      - record: agent:cost_per_incident_usd
        expr: |
          sum(increase(agent_cost_usd_total[7d]))
          / clamp_min(sum(increase(agent_incidents_resolved_total[7d])), 1)

Alert when agent:cache_hit_ratio drops below ~0.6 — that almost always means someone put something volatile above the cache breakpoint, and it will show up as a step change in spend days before the invoice does. cost_per_incident_usd is the number to defend the agent with: compare it to the loaded cost of the engineer-minutes it removed, exactly the unit-economics framing you'd apply to any other Kubernetes cost line item.

Honest limits

Retries are invisible cost. A 529 with automatic retry bills the full input again; count retried requests separately or your per-incident math quietly understates by 10–20%. Long context isn't free just because it's cached — cache reads still cost real money and still add latency, so a 100k-token context you cache is worse than a 10k-token context you curated. And these ratios move: verify current per-token prices and cache multipliers against your provider's pricing page before committing to a forecast.

Takeaway

An ops agent's bill is mostly re-transmitted history, so attack it in that order: cache the static prefix (and keep volatile data below the breakpoint), cap and summarize every tool result at the boundary, route classification work to a small model with a deterministic router, and put a hard turn/token/wall-clock budget around the loop. Then instrument tokens as a first-class metric and track cost per resolved incident. Done in that order, the same agent that looked expensive in month one runs at a fraction of the cost — and the FinOps agent you built to watch your cluster spend finally stops being the thing nobody measures.