Observability for DevOps AI Agents: Trace Tool Calls, Tokens, and Runaway Loops

An autonomous agent is a production service. Instrument it like one.

You gave your ops agent a scoped MCP server, wrapped it in a supervised loop, and evaluated it before prod. It's running. At 03:00 it takes an action, and on-call asks the one question every autonomous system eventually provokes: what did it do, and why? If your answer is "let me scroll the raw model logs," you're running a production service blind.

The fix is the same one you already apply to every other service: distributed tracing. Model each agent run as a trace, each turn and each tool call as a span, and attach the numbers that matter — tokens, cost, latency, and which guardrail fired. This guide instruments a DevOps agent with OpenTelemetry so that "what did the agent do" becomes a query, not an archaeology dig.

The span tree that mirrors how an agent actually runs

An agent run has a natural hierarchy, and it maps one-to-one onto OpenTelemetry spans:

agent.run  (root span — one incident triage)
├── agent.turn 1
│   ├── llm.chat        gen_ai.usage.* tokens, model, latency
│   └── tool.k_describe args, result size, exit
├── agent.turn 2
│   ├── llm.chat
│   └── tool.k_scale_preview
└── agent.turn 3
    └── llm.chat        (final answer, no tool call)

The root span is the unit you alert on: total tokens, total cost, wall-clock, and turn count for the whole run. Each agent.turn shows the reason→act cadence. Each tool.* span is the part with real blast radius — the kubectl or the Terraform preview — so that's where you spend your attributes.

OpenTelemetry has emerging GenAI semantic conventions for exactly this. They're still stabilizing, so pin your semconv version, but adopting the standard names (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens) means your traces render in any OTLP backend without custom dashboards.

Instrument the loop

Here's a minimal but honest instrumentation of a tool-calling loop. The tracing wraps the loop you already have — it doesn't change the agent's logic, it just makes every decision legible.

# tracing.py — OTel setup, exports OTLP to your collector
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))  # OTEL_EXPORTER_OTLP_ENDPOINT
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("dtc.ops-agent")
meter = metrics.get_meter("dtc.ops-agent")

# A few metrics you'll want as time series, not just traces
guardrail_hits = meter.create_counter("agent.guardrail.blocked")
tokens_total   = meter.create_counter("agent.tokens.total")

# agent.py — the loop, now traced
MAX_TURNS = 8

def run_agent(incident_state):
    with tracer.start_as_current_span("agent.run") as run_span:
        run_span.set_attribute("gen_ai.system", "anthropic")
        run_span.set_attribute("agent.incident_id", incident_state["id"])
        total_tokens = 0
        messages = build_initial_messages(incident_state)

        for turn in range(1, MAX_TURNS + 1):
            with tracer.start_as_current_span("agent.turn") as t:
                t.set_attribute("agent.turn", turn)

                with tracer.start_as_current_span("llm.chat") as llm:
                    resp = model.chat(messages)
                    it, ot = resp.usage.input_tokens, resp.usage.output_tokens
                    llm.set_attribute("gen_ai.request.model", resp.model)
                    llm.set_attribute("gen_ai.usage.input_tokens", it)
                    llm.set_attribute("gen_ai.usage.output_tokens", ot)
                    total_tokens += it + ot
                    tokens_total.add(it + ot, {"model": resp.model})

                if not resp.tool_calls:
                    run_span.set_attribute("agent.turns_used", turn)
                    run_span.set_attribute("agent.tokens_total", total_tokens)
                    return resp.text

                for call in resp.tool_calls:
                    messages.append(exec_tool_traced(call))

        run_span.set_status(trace.StatusCode.ERROR, "hit MAX_TURNS")
        run_span.set_attribute("agent.runaway", True)
        raise RuntimeError("agent exceeded turn budget")

Two details carry most of the value. The root span records turns_used and tokens_total for every run, and the MAX_TURNS exit is tagged agent.runaway=True — so a loop that spirals is a searchable event, not a silent budget leak. This is the runtime enforcement of the turn caps from loop engineering: the loop bounds the behavior, the trace proves it held.

The tool span is where the blast radius lives

The LLM span tells you what the model said. The tool span tells you what the agent did — and that's the one you'll pull up during a postmortem.

def exec_tool_traced(call):
    with tracer.start_as_current_span(f"tool.{call.name}") as span:
        span.set_attribute("tool.name", call.name)
        # Redact before recording — args can carry secrets or PII.
        span.set_attribute("tool.args", redact(call.args))
        span.set_attribute("tool.mutating", call.name in MUTATING_TOOLS)

        if call.name in MUTATING_TOOLS and not call.approved:
            guardrail_hits.add(1, {"tool": call.name, "reason": "unapproved_mutation"})
            span.set_attribute("tool.blocked", True)
            span.set_status(trace.StatusCode.ERROR, "guardrail: unapproved mutation")
            return tool_result(call, "BLOCKED: requires human approval")

        result = TOOLS[call.name](**call.args)
        span.set_attribute("tool.result_bytes", len(result))
        return tool_result(call, result)

Note what gets recorded: a tool.mutating flag so you can filter for the risky calls, a tool.blocked flag when a guardrail fires, and a counter metric for every block. That guardrail counter is your early-warning signal — the same idea the evals post calls online evaluation. A rising agent.guardrail.blocked rate means the model is repeatedly trying to do something it shouldn't, which is a prompt or model regression you want to see before it finds a gap. And always redact() before you record arguments — a trace backend is not where cluster secrets should land.

What to actually put on your dashboards

Traces are for the deep-dive on one run. Metrics are for the fleet-wide view. Three panels earn their place next to your Prometheus + Grafana setup:

• Cost per run (p50/p95). Derived from agent.tokens_total × model price. A creeping p95 is a chattier model or a context that's grown bloated — the context-engineering failure mode, made visible. This is the raw input to any real FinOps view of agent spend.
• Turns per run (histogram). Healthy triage clusters at 2–4 turns. A fat tail toward MAX_TURNS means agents are thrashing, not converging.
• Guardrail-block rate. Should be near-zero in steady state. Any sustained rise is an alert, not a chart you glance at.

# p95 tokens per agent run over 1h — the cost early-warning
histogram_quantile(0.95, sum by (le) (rate(agent_tokens_total_bucket[1h])))

# guardrail blocks per 5m by tool — should be flat at zero
sum by (tool) (rate(agent_guardrail_blocked_total[5m]))

Traces make the failure modes concrete

Every agent failure mode people wave at in the abstract becomes a specific trace query once you're instrumented:

• Runaway loop → agent.runaway=True, or trace with agent.turns_used == MAX_TURNS.
• Hallucinated action → an llm.chat span whose text claims a fix, with no tool.* span that actually mutated. The model said "I scaled it"; the trace shows it never called the tool.
• Silent cost blowout → agent.tokens_total far above the run's peers, usually one giant tool result stuffed back into context.
• Guardrail probing → repeated tool.blocked=True spans across runs from the same trigger.

This is the observability foundation that autonomous work actually depends on. The agent-harness-as-infrastructure post makes the point in one line — "unobservable automation is just an incident you haven't noticed yet." Tracing is how you notice. It's also the inverse of AI-powered observability: there you point AI at your telemetry; here you generate telemetry from the AI.

Honest limits

Tracing tells you what the agent did and what it cost. It does not tell you whether the action was correct — a perfectly-traced run can still have made the wrong call with full confidence. That judgment stays with your eval suite and the human approval gate. Redaction is also load-bearing and easy to get wrong: audit what lands in tool.args before you ship, because a trace backend is a long-lived, widely-read store. And the GenAI semantic conventions are still moving — pin the version and expect to adjust attribute names as they stabilize.

Start small: wrap the loop in a root span, add a span per tool call, record tokens and the runaway flag. That alone turns your 3 a.m. "what did it do?" from an unanswerable question into a single trace you can open — which is the whole difference between running an agent and operating one.