Shadow Mode for DevOps AI Agents: Earn Trust on Real Incidents Before Write Access

What shadow mode is, in one paragraph

Shadow mode means your AI agent runs on every real production alert with read-only tools, writes its proposed fix to a log — and nobody executes it. Meanwhile, on-call resolves the incident the normal way. Afterwards, a scorer compares what the agent would have done against what the human actually did, and the agreement rate per action class decides when the agent earns write access. It is the online counterpart to an offline eval suite, and the evidence base that justifies wiring up a human-in-the-loop approval gate later. This post builds the whole loop: the shadow runner, ground-truth capture, the agreement scorer, Prometheus metrics, and concrete promotion criteria.

The rollout ladder for an ops agent looks like this, and each rung has its own post on this site:

offline evals  ->  SHADOW MODE   ->  approval-gated writes  ->  narrow autonomy
(golden cases)     (this post)       (human clicks yes)         (earned, per action)

Most teams jump from rung one straight to rung three. Shadow mode is the rung that produces the numbers that make rung three defensible in a postmortem review.

Why offline evals aren't enough

Golden scenarios are frozen, cleaned-up incidents — you built them from postmortems, so they are by definition the failures you already understand. Production alerts are messier in three ways your eval suite can't simulate:

• Distribution shift. Real alert traffic is dominated by flapping, duplicates, and partial signals. An agent that aces 20 curated scenarios can still propose a rollout restart for every noisy CPU alert it sees at 2 a.m.
• Context rot. Live clusters carry weird half-states — a node cordoned last Tuesday, a Helm release stuck mid-upgrade — that no fixture captures. Shadow mode tests the agent against the cluster you actually have.
• Base rates. Offline you score pass/fail per scenario. Online you learn the number that matters operationally: out of 60 real alerts this month, how many times would the agent's action have been the right call? That is a precision estimate, and you can only get it from live traffic.

Shadow mode is cheap insurance: the agent has zero blast radius (read-only tools, no Slack pings to the incident channel), so the worst case is wasted tokens, a line item you should be tracking per run anyway.

The shadow runner

Tap your existing Alertmanager webhook — the same feed on-call gets — and fan it out to the agent asynchronously so shadow runs never delay real paging. The agent gets read-only tools only (the scoped describe/get/logs surface of a Kubernetes MCP server is exactly right), and its output is forced through the same structured propose_action schema you'll use later at the approval gate — so shadow data is directly comparable to gated data.

# shadow_runner.py — receives the Alertmanager webhook, records proposals, executes nothing
import json, hashlib
from flask import Flask, request

app = Flask(__name__)
LOG = "/var/log/agent/shadow.jsonl"   # append-only; this file IS the dataset

@app.post("/shadow/alert")
def on_alert():
    alert = request.json["alerts"][0]
    incident_id = hashlib.sha1(
        (alert["labels"]["alertname"] + alert["startsAt"]).encode()
    ).hexdigest()[:12]

    proposal = run_agent_readonly(alert)   # LLM + read-only MCP tools, no creds

    with open(LOG, "a") as f:
        f.write(json.dumps({
            "incident_id": incident_id,
            "alert": alert["labels"],
            "started_at": alert["startsAt"],
            "proposal": proposal,           # {"kind","namespace","workload","replicas","reason"} or null
            "tokens": proposal.pop("_tokens", None) if proposal else None,
        }) + "\n")
    return {"ok": True}

Two deliberate choices. First, proposal may be null — "no safe action" is a valid answer, and you want to measure how often the agent knows to say it. Second, the incident_id is derived from the alert, not generated, so the scorer can join proposals to outcomes later without shared state.

Everything the runner does should also be traced like any other service — spans per tool call, tokens per run — using the agent observability setup you'd want in production anyway. Shadow mode is a dress rehearsal for the telemetry too.

Ground truth: what did the human actually do?

This is the part most teams hand-wave, and it decides whether your agreement numbers mean anything. You need, per incident, the remediation the human actually performed. Three sources, in order of reliability:

1. Kubernetes events and audit log. Scale events, rollout restarts, and rollbacks all leave records with timestamps and actors. A watcher that captures ScalingReplicaSet, deployment.kubernetes.io/revision bumps, and kubectl-edit audit entries in the alert's namespace covers most remediations.
2. GitOps history. If your team ships fixes as PRs — the model argued for in GitOps for AI agents — then merged PRs touching the affected workload within the incident window are clean, reviewable ground truth.
3. Resolution notes. Weakest but necessary fallback: a one-line "what fixed it" field when on-call resolves the page. Humans skip it under pressure, so treat it as supplementary.

The join rule is simple and honest: a human action counts as the incident's resolution if it targets the same namespace/workload as the alert and lands between startsAt and resolution time. If nothing matches — the alert self-resolved — the ground truth is null, which is exactly what the agent should have proposed.

Scoring agreement

Now the scorer, run daily as a cron. The comparison must be structural, not textual — you are comparing action specs, never prose:

# shadow_score.py — join proposals to observed human actions, emit agreement stats
def match(proposal, human_action):
    """Returns one of: agree | partial | disagree | overeager | missed"""
    if proposal is None and human_action is None:
        return "agree"            # correctly proposed nothing for a self-resolving alert
    if proposal is None:
        return "missed"           # human acted, agent had no idea
    if human_action is None:
        return "overeager"        # agent proposed action; none was needed
    if proposal["kind"] != human_action["kind"]:
        return "disagree"
    same_target = (proposal["namespace"] == human_action["namespace"]
                   and proposal["workload"] == human_action["workload"])
    if not same_target:
        return "disagree"
    if proposal["kind"] == "scale":
        # exact replica match is too strict; within 1 of the human's choice counts
        close = abs(int(proposal["replicas"]) - int(human_action["replicas"])) <= 1
        return "agree" if close else "partial"
    return "agree"

The five buckets matter more than a single score. Disagree on target is the scary one — wrong workload means the agent misread the incident. Overeager is the AdSense-blog equivalent of alert fatigue: an agent that proposes restarts for self-resolving blips will train humans to rubber-stamp, which quietly destroys the value of the approval gate later. Missed is usually fine early on — it means the agent was conservative.

Push the counts to Prometheus so trust is a dashboard, not a feeling:

from prometheus_client import Counter

SHADOW_RESULT = Counter(
    "agent_shadow_result_total",
    "Shadow-mode agreement outcomes",
    ["action_kind", "result"],
)
# in the scorer loop:
SHADOW_RESULT.labels(action_kind=kind, result=verdict).inc()

A PromQL agreement rate per action class is then one expression: sum by (action_kind) (rate(agent_shadow_result_total{result="agree"}[30d])) / sum by (action_kind) (rate(agent_shadow_result_total[30d])) — plot it on the same board as your token spend.

Promotion criteria: make graduation boring

Decide the thresholds before you look at the data, per action class, and write them down where the team can veto them. Mine, for a mid-size cluster:

Three rules make this work in practice:

• Promote the narrowest class first. rollout_restart in one namespace, not "writes". Each class graduates separately, on its own evidence.
• Promotion changes the mode, not the leash. Graduating from shadow means the action now goes to the approval gate — a human still clicks yes. Autonomy, if ever, is a later, separate promotion with its own shadow-style evidence from approval outcomes.
• Demotion is automatic. One dangerous proposal — wrong namespace, out-of-bounds argument, a fabricated claim that it already acted — sends that action class back to shadow, regardless of its average. Averages don't page you; outliers do. Feed the offending case into your golden-scenario evals so it can never recur unnoticed, and into the agent's incident memory so the same misdiagnosis isn't repeated tomorrow.

Honest limits

Agreement with humans is a proxy, not truth. On-call under pressure sometimes picks a worse fix than the agent proposed — shadow scoring will punish the agent for being right differently. Skim the weekly disagree bucket by hand; occasionally you'll find the agent's proposal was better, which is signal about your runbooks, not the agent. Self-resolving alerts inflate the easy "agree on nothing" bucket, so track action-proposing incidents separately. And 20 samples is a floor, not statistics — at typical incident rates a quiet namespace can take two months to graduate a single action class. That slowness is the feature: an agent that needs two months of receipts to earn a restart button is exactly the deployment posture that lets you sleep after you grant it.

Start this week: point your Alertmanager webhook at a shadow runner, force structured proposals, and let the log accumulate. In a month you will know — with numbers — whether your agent deserves an approval button, and for exactly which actions.