What a Circuit Breaker Does for an Agent
A circuit breaker for a DevOps AI agent is a layer of deterministic code between the model and your infrastructure that can stop the agent from acting, no matter what the model decides next. It has four parts: a kill switch that a human can flip in one command and that fails closed, per-run budgets on turns, mutations, and money, a breaker that trips when a tool keeps failing or the agent keeps retrying the same write, and auto-demotion that drops the agent to read-only when the service it just touched starts burning its SLO. Once tripped, only a human resets it.
None of this lives in the prompt. Telling the model "stop if things go wrong" is a suggestion. A breaker is enforcement, and it sits where the MCP gateway already intercepts every tool call.
Why MAX_TURNS Is Not Enough
Most agent loops already cap turns, and the observability setup tags runs that hit the cap. That protects you against one failure mode: the loop that never ends. It does nothing about the loop that ends on time having done damage on every turn.
The pattern that actually hurts looks like this. A remediation agent sees checkout pods in CrashLoopBackOff and restarts the deployment. The restart briefly drops capacity, latency climbs, a second alert fires, and the agent's next run sees new evidence of a broken service and restarts it again. Each run is under budget. Each action is individually reasonable. Together they are an outage with a bot at the keyboard. The fix is state that outlives a single run: a counter that notices the third rollout restart in ten minutes and refuses the fourth.
A breaker also covers the dull failure: the upstream API is down, every kubectl returns an error, and the agent burns tokens rephrasing the same failing call. That is money and noise, not risk, but the same mechanism handles both.
Layer 1: The Kill Switch
The switch is a ConfigMap with one field. Every mutating tool reads it before it acts. There are three modes: active, readonly, and paused.
apiVersion: v1
kind: ConfigMap
metadata:
name: agent-controls
namespace: agents
data:
mode: "active" # active | readonly | paused
reason: ""
changed_by: ""
The on-call engineer needs one memorised command, not a runbook:
kubectl -n agents patch configmap agent-controls --type merge \
-p '{"data":{"mode":"readonly","reason":"checkout incident INC-4821","changed_by":"riko"}}'
Two design decisions matter here. First, fail closed. If the agent cannot read the ConfigMap, because the API server is degraded, RBAC changed, or the namespace was deleted, it must behave as paused. An agent that keeps mutating things because its safety flag was unreachable has a safety flag in name only. Second, prefer readonly over paused. Scaling the gateway to zero, as the gateway post suggests for a security incident, stops diagnosis too. During an ordinary incident you want the agent to keep reading logs and metrics and to stop touching anything. Reserve paused for a suspected compromise or prompt injection.
# controls.py: fail-closed read of the kill switch
from kubernetes import client, config
config.load_incluster_config()
core = client.CoreV1Api()
def current_mode() -> str:
try:
cm = core.read_namespaced_config_map("agent-controls", "agents")
mode = cm.data.get("mode", "paused")
return mode if mode in ("active", "readonly", "paused") else "paused"
except Exception:
return "paused" # unreachable switch == closed switch
Cache this for a few seconds at most. A ten-minute cache turns the one-command kill switch into a ten-minute kill switch.
Layer 2: Per-Run Budgets
Turn caps count model calls. Budgets should count the things that cost money or change the world. Track them per run and refuse the tool call that would exceed any of them.
| Budget | Typical default | What exceeding it means |
|---|---|---|
| Mutating tool calls | 3 | The agent is doing more than the runbook step it was given |
| Distinct namespaces written | 1 | Scope drift: it started on checkout and is now editing payments |
| Wall clock | 10 min | Slow tools or a stuck approval, not a runaway loop |
| Model spend | $2 | Direct cap on the token bill for one run |
| Same mutation repeated | 1 | Retrying a write that already returned an error |
The namespace budget is the one teams skip and the one that catches real drift. A pod-restart agent has no business writing to a second namespace inside one run, and a model that has convinced itself it should is a model that has left its task.
# budget.py: enforced per run, in code, before the tool executes
from dataclasses import dataclass, field
@dataclass
class RunBudget:
max_mutations: int = 3
max_namespaces: int = 1
mutations: int = 0
namespaces: set = field(default_factory=set)
seen_writes: set = field(default_factory=set)
def admit(self, tool: str, args: dict, mutating: bool) -> None:
if not mutating:
return
key = (tool, tuple(sorted(args.items())))
if key in self.seen_writes:
raise PermissionError(f"budget: {tool} already attempted with identical args")
ns = args.get("namespace")
if ns and ns not in self.namespaces and len(self.namespaces) >= self.max_namespaces:
raise PermissionError(f"budget: run already writes to {sorted(self.namespaces)}, refusing {ns}")
if self.mutations >= self.max_mutations:
raise PermissionError("budget: mutation cap reached for this run")
self.mutations += 1
self.seen_writes.add(key)
if ns:
self.namespaces.add(ns)
The error messages are deliberate. They go back to the model as the tool result, so the model learns in-context that the run is over and writes its handoff instead of inventing a workaround.
Layer 3: The Breaker Itself
Budgets reset every run. The breaker does not. It is keyed on agent and tool, stored somewhere that survives the process (Redis, or a ConfigMap if you want zero dependencies), and it has the three classic states.
- Closed: calls flow. Every failure increments a counter in a sliding window.
- Open: calls to that tool are refused for a cooldown, and a mutating tool that opens flips the whole agent to
readonly. - Half-open: after the cooldown, one call is allowed through. If it is a mutation, it is still refused. Only read tools probe the breaker, because "let's see if the risky write works now" is not a probe, it is the incident again.
# breaker.py: per (agent, tool), persisted in Redis
import time, redis
r = redis.Redis(host="redis.agents.svc")
WINDOW, THRESHOLD, COOLDOWN = 600, 3, 1800 # 3 failures in 10 min -> open 30 min
def check(agent: str, tool: str, mutating: bool) -> None:
opened = r.get(f"brk:{agent}:{tool}:open")
if opened:
if time.time() - float(opened) < COOLDOWN:
raise PermissionError(f"breaker open for {tool}; retry after cooldown")
if mutating:
raise PermissionError(f"breaker half-open for {tool}; mutations need a human reset")
def record(agent: str, tool: str, ok: bool, mutating: bool) -> None:
k = f"brk:{agent}:{tool}:fail"
if ok:
r.delete(k)
return
r.zadd(k, {str(time.time()): time.time()})
r.zremrangebyscore(k, 0, time.time() - WINDOW)
if r.zcard(k) >= THRESHOLD:
r.set(f"brk:{agent}:{tool}:open", time.time())
if mutating:
set_mode("readonly", reason=f"breaker tripped on {tool}", changed_by=agent)
What counts as a failure needs one distinction. A 5xx from the Kubernetes API or a timeout means the infrastructure is unhappy, and tripping quickly is right. A 404 or 403 means the agent asked for something that does not exist or that it may not touch, which usually means a hallucinated resource name or an RBAC boundary doing its job. Count both, but log them with different reasons. Three hallucinated names in ten minutes is a model-quality problem for your evals; three timeouts is an infra problem for on-call.
Layer 4: The SLO Tripwire
The most important trip condition is not visible from inside the agent at all. It is the burn rate of the service the agent just changed. If the agent wrote to checkout and five minutes later checkout is burning its error budget at 14x, the agent is the prime suspect and it should lose write access until a human clears it.
Export what the middleware knows as Prometheus metrics: every tool call with agent, tool, outcome, mutating, and the target namespace. Then join it with the burn-rate alerts you already have from the error budget agent setup.
groups:
- name: agent-tripwire
rules:
- alert: AgentMutatedServiceNowBurning
expr: |
(
sum by (namespace) (increase(agent_tool_calls_total{mutating="true"}[15m])) > 0
)
and on (namespace)
(
slo:error_budget_burn_rate:1h > 14
)
for: 2m
labels:
severity: critical
action: agent_readonly
annotations:
summary: "Agent wrote to {{ $labels.namespace }} in the last 15m and it is now burning SLO"
An Alertmanager webhook receiver for action="agent_readonly" patches the ConfigMap. That is twenty lines of Python receiving the standard Alertmanager webhook payload. The alert may be a coincidence, and demoting the agent may cost you a genuinely helpful remediation. Accept that. A human can restore active in one command after looking; nobody can un-run a rollout restart.
Add a second, quieter rule for the money side:
- alert: AgentBreakerOpen
expr: max by (agent, tool) (agent_breaker_state) == 1
for: 0m
labels:
severity: warning
If that alert is firing three times a week for the same tool, the breaker is doing its job and your agent has a systematic problem worth a fix, not a reset.
Wiring the Layers in Order
In the gateway middleware the order is fixed and cheap: kill switch, budget, breaker, then the call, then record the outcome. Each check is a dictionary lookup or a Redis round trip, so the whole stack adds single-digit milliseconds to a tool call that the model will spend seconds thinking about.
async def on_call_tool(self, ctx, call_next):
tool, args = ctx.message.name, ctx.message.arguments or {}
mutating = tool in MUTATING_TOOLS # explicit allowlist, never inferred
mode = current_mode()
if mode == "paused" or (mode == "readonly" and mutating):
raise PermissionError(f"agent is {mode}: {tool} refused")
ctx.budget.admit(tool, args, mutating)
check(ctx.agent, tool, mutating)
try:
result = await call_next(ctx)
record(ctx.agent, tool, ok=True, mutating=mutating)
return result
except Exception:
record(ctx.agent, tool, ok=False, mutating=mutating)
raise
Note MUTATING_TOOLS is a hand-maintained list. Do not infer it from tool names. A tool called describe_and_fix_pod is mutating whatever its prefix says.
Reset Is a Human Act
Everything above ratchets one way. Nothing in the system flips readonly back to active automatically, not a timer, not a healthy-looking metric, and certainly not the agent. The reset is the same kubectl patch with a reason and a name, and it lands in the same audit trail as every action the agent took. When you review the week, the interesting question is never "how many times did it trip" but "what did the human see before choosing to reset".
If you find yourself resetting several times a day, do not raise the thresholds. Move the agent back to shadow mode for the tool that keeps tripping and find out why its proposals fail.
Test the Breaker Before It Matters
The breaker is code, so it gets tests, using the same pytest approach as testing MCP servers. Three cases cover most of the risk.
def test_kill_switch_fails_closed(monkeypatch):
monkeypatch.setattr(controls.core, "read_namespaced_config_map",
lambda *a, **k: (_ for _ in ()).throw(RuntimeError("api down")))
assert controls.current_mode() == "paused"
def test_repeated_write_refused():
b = RunBudget()
b.admit("k8s_rollout_restart", {"namespace": "checkout", "name": "api"}, True)
with pytest.raises(PermissionError, match="identical args"):
b.admit("k8s_rollout_restart", {"namespace": "checkout", "name": "api"}, True)
def test_three_failures_demote_agent(fake_redis, fake_controls):
for _ in range(3):
record("oncall", "k8s_scale", ok=False, mutating=True)
assert fake_controls.mode == "readonly"
Then do it live once a quarter. Point the agent at a staging cluster with a deliberately broken API endpoint and watch it trip. If the on-call rota cannot recite the kill-switch command from memory, the drill has done its job.
Honest Limits
A breaker catches repetition and collateral damage. It does not catch a single, well-formed, catastrophic action that succeeds on the first try. Deleting the right PVC in the wrong namespace returns 200. That class of risk belongs to approval gates and to least-privilege RBAC that makes the action impossible rather than merely unwise.
The SLO tripwire will also fire on coincidences, and every false trip removes an agent that might have been helping. That trade is correct for a team still building trust in its agents and may be wrong for a mature one. Tune the burn-rate threshold and the 15-minute window to your own incident history, not to a blog post.
Finally, the breaker is only as good as the identity behind it. If two agents share a token, one agent's trip demotes both, and one agent's clean record hides the other's failures. Per-agent identity at the gateway is the prerequisite for everything in this article.