The gap between reading and doing
Every safe DevOps agent I have shipped starts read-only: it queries metrics, reads logs, describes pods, and proposes a fix in English. That is genuinely useful, and it is where you should start. But the moment an on-call engineer reads the agent's suggestion, copies the command, and pastes it into a terminal at 3am, you have rebuilt the exact manual toil the agent was supposed to remove — and added a copy-paste transcription bug as a bonus.
The missing piece is a human-in-the-loop approval gate: the agent proposes a concrete write action, a human sees the exact change and approves it with one click, and then — and only then — a bounded executor applies it. The human keeps the judgment; the machine keeps the typing accurate. This post builds that gate end to end: a propose/approve/execute state machine, a Slack approval message with the real diff, a signed approval token that can't be forged or replayed, and an executor whose allowlist is the true blast-radius boundary. It is the natural next step after a read-only Kubernetes MCP server, and it leans on the same principle from the agent harness as infrastructure: the agent's power is defined by what the harness lets it do, not by what the prompt asks for.
Separate the three phases, always
The single most important design decision is that proposing, approving, and executing are three different processes with three different privilege levels. The agent that reasons never holds cloud or cluster write credentials. The executor that holds credentials never talks to the LLM. The approval sits between them as a signed handoff. Draw that boundary wrong and the approval becomes theater.
(LLM, no creds) (human, Slack) (executor, has creds)
propose() --> approve/deny --> execute()
| | |
action spec signed token allowlist check
Phase 1: the agent proposes a structured action
The agent's output must never be a shell string. Free-form text is unauditable and unexecutable-safely. Force the model to emit a structured action spec through a tool schema, exactly the constraint that makes any ops agent trustworthy. Using the Anthropic Python SDK with Claude Sonnet — the right latency and cost tier for an incident co-pilot:
import anthropic
PROPOSE_TOOL = {
"name": "propose_action",
"description": "Propose ONE remediation action for a human to approve.",
"input_schema": {
"type": "object",
"properties": {
"kind": {"enum": ["scale", "rollout_restart", "rollback"]},
"namespace": {"type": "string"},
"workload": {"type": "string"},
"replicas": {"type": "integer",
"description": "Target replicas; only for kind=scale."},
"reason": {"type": "string",
"description": "One sentence: why this fixes the symptom."},
},
"required": ["kind", "namespace", "workload", "reason"],
},
}
SYSTEM = (
"You are an on-call SRE co-pilot. Given an alert and cluster context, propose "
"exactly ONE reversible remediation via propose_action. Prefer the smallest "
"action that addresses the symptom. Never propose deletes or anything that "
"loses data. If no safe action fits, do not call the tool."
)
def propose(alert_context: str):
client = anthropic.Anthropic()
msg = client.messages.create(
model="claude-sonnet-5",
max_tokens=800,
system=SYSTEM,
tools=[PROPOSE_TOOL],
messages=[{"role": "user", "content": alert_context}],
)
for block in msg.content:
if block.type == "tool_use":
return block.input
return None
Notice the kind enum. The agent cannot invent an action type; it can only pick from three reversible verbs the executor knows how to run and, crucially, how to undo. What you feed into alert_context is its own discipline — the alert, recent deploys, and current state, and nothing sensitive the model shouldn't see — which I cover in context engineering for on-call AI agents.
Phase 2: show the real change and mint a signed token
Before a human approves anything, they must see the actual effect, not the agent's description of it. Compute a concrete before/after and render it. For a scale action that is trivial; for a rollback, show the image tag you are rolling back to. Then attach an approval token that is signed, scoped, and time-boxed so it cannot be forged, reused, or resurrected an hour later.
import hmac, hashlib, json, base64, time
SECRET = open("/etc/agent/approval.key", "rb").read()
def mint_token(action: dict, ttl_seconds: int = 300) -> str:
payload = {
"action": action,
"exp": int(time.time()) + ttl_seconds, # pass current time in from caller in prod
"nonce": action["workload"] + ":" + action["namespace"],
}
body = base64.urlsafe_b64encode(json.dumps(payload).encode()).decode()
sig = hmac.new(SECRET, body.encode(), hashlib.sha256).hexdigest()
return body + "." + sig
def verify_token(token: str, seen: set) -> dict:
body, sig = token.rsplit(".", 1)
expected = hmac.new(SECRET, body.encode(), hashlib.sha256).hexdigest()
if not hmac.compare_digest(sig, expected):
raise ValueError("bad signature")
payload = json.loads(base64.urlsafe_b64decode(body))
if payload["exp"] < int(time.time()):
raise ValueError("token expired")
if token in seen:
raise ValueError("token already used") # replay guard
seen.add(token)
return payload["action"]
Three properties matter here. The HMAC signature means only the proposer service can create a valid approval — a compromised Slack account can click, but cannot mint a token for a different, nastier action. The TTL means a stale approval from a past incident can't be replayed against a changed cluster. And the nonce plus a seen-set stops the same token from being redeemed twice. This is the same defensive posture as defending agents against prompt injection: assume every input to the executor is potentially hostile and verify it cryptographically.
The Slack approval message
Post the proposal with the diff and two buttons. The token rides inside the button's value, so the approval callback receives it directly:
def approval_blocks(action, token):
summary = (f"*{action['kind']}* `{action['workload']}` "
f"in `{action['namespace']}`\n"
f"Target replicas: {action.get('replicas', 'n/a')}\n"
f"Reason: {action['reason']}")
return [
{"type": "section",
"text": {"type": "mrkdwn", "text": f":robot_face: Proposed fix:\n{summary}"}},
{"type": "actions", "elements": [
{"type": "button", "style": "primary",
"text": {"type": "plain_text", "text": "Approve"},
"action_id": "approve", "value": token},
{"type": "button", "style": "danger",
"text": {"type": "plain_text", "text": "Deny"},
"action_id": "deny", "value": token},
]},
]
The approver sees the workload, the target state, and the agent's one-line justification — enough to make a call in seconds without leaving Slack. Because the TTL is five minutes, an approval that sits unanswered simply expires, and the incident falls back to a human doing it by hand. That fail-safe default — inaction on timeout — is deliberate.
Phase 3: the executor, where the blast radius really lives
The executor is the only component with cluster write access, and it trusts nothing from the LLM directly. It re-verifies the token, then re-validates the action against an allowlist and hard bounds before it runs a single command. The allowlist is your real security boundary; the prompt is not.
from kubernetes import client, config
ALLOWED_NS = {"payments", "web", "checkout"}
MAX_REPLICAS = 20
def execute(token: str, seen: set):
action = verify_token(token, seen) # signature + TTL + replay
ns, wl = action["namespace"], action["workload"]
if ns not in ALLOWED_NS:
raise PermissionError(f"namespace {ns} not allowlisted")
config.load_incluster_config()
apps = client.AppsV1Api()
if action["kind"] == "scale":
n = int(action["replicas"])
if not 1 <= n <= MAX_REPLICAS:
raise ValueError(f"replicas {n} out of bounds")
apps.patch_namespaced_deployment_scale(
wl, ns, {"spec": {"replicas": n}})
elif action["kind"] == "rollout_restart":
apps.patch_namespaced_deployment(wl, ns, {"spec": {"template":
{"metadata": {"annotations":
{"agent/restartedAt": str(action["nonce"])}}}}})
elif action["kind"] == "rollback":
apps.create_namespaced_deployment_rollback # or undo via revision history
else:
raise ValueError("unknown kind")
audit_log(action, actor="agent-approved")
The ALLOWED_NS set and MAX_REPLICAS bound are not decoration — they are what turns "the agent went haywire" from an outage into a rejected request. Even a perfectly forged approval for kubectl scale --replicas=9000 in kube-system dies at the namespace check. Give the executor's Kubernetes service account RBAC that matches: patch on deployments in three namespaces, nothing cluster-wide. If you need a refresher on scoping those permissions tightly, the RBAC and least-privilege patterns are the foundation the whole gate stands on.
Log every step for the postmortem
Every proposal, approval, denial, expiry, and execution goes to an append-only audit log with the actor, the token, and the full action spec. When you write the incident report, you want a clean record of the agent proposed X, Alex approved at 03:14, executed at 03:14:02, symptom cleared at 03:16 — which slots straight into an incident management runbook. Without that trail you cannot answer the only question that matters after an agent touches prod: who decided, and on what evidence.
Test the gate before it gates anything
An approval gate is production infrastructure, so it needs its own tests. Fixture a forged token and assert verify_token rejects it; fixture an expired token and a replayed one and assert the same; fixture an action targeting a non-allowlisted namespace and assert the executor refuses. Then run the propose step against saved alert contexts and measure whether the proposed action is the one a senior SRE would pick. This is the eval discipline from testing ops agents before production: you are measuring the gate's refusals as carefully as its successes, because a gate that approves the wrong thing is worse than no gate.
Where this fits — and where it doesn't
Start every workload in propose-only mode: the agent posts to Slack, the human copies the reasoning but executes by hand, and you watch for a few weeks whether its proposals are ones you'd actually approve. Only wire the executor once the proposals earn trust. Keep the action vocabulary small and reversible — scale, restart, roll back — and resist the urge to add delete or anything that loses state; those stay human-only. Fully autonomous remediation, the kind explored in autonomous incident response for SRE, is a real destination, but the approval gate is how you get there safely: it lets a human ratify hundreds of the agent's decisions until the precision is high enough to trust a narrow slice unattended. Until then, the gate gives you the best of both — the agent's speed at diagnosis and typing, the human's judgment on the trigger.