What This Agent Does
A chaos engineering agent picks the next experiment worth running and writes the hypothesis — it does not get to run it. It reads each service's topology (replicas, PodDisruptionBudget, probes, upstream dependencies) and its SLO status through read-only tools, then proposes one Chaos Mesh experiment from a fixed catalogue: kill one pod of checkout-api, add 100 ms to calls from checkout-api to payments, burn CPU on one replica. The proposal has a falsifiable hypothesis and an abort condition. A human approves, a runner applies the CRD, and a deterministic watchdog kills the experiment the moment the SLO burn rate moves. The model never holds a kubeconfig with write access to anything.
Most teams that "do chaos engineering" ran GameDay once, killed a pod, and stopped. The reason isn't tooling — Chaos Mesh installs in five minutes — it's that designing a good experiment is tedious: you need to know the service's redundancy story, what SLO it carries, and what "steady state" even means for it, and then write all of that down. That is a reading-and-drafting job, which is what language models are good at. Running faults against production is not.
Chaos Mesh Setup: Namespace Filtering Is Not Optional
Install with namespace filtering enabled, so Chaos Mesh refuses to touch any namespace you haven't explicitly opted in:
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
--namespace chaos-mesh --create-namespace \
--set chaosDaemon.runtime=containerd \
--set chaosDaemon.socketPath=/run/containerd/containerd.sock \
--set controllerManager.enableFilterNamespace=true
# Opt in exactly the namespaces the agent may propose against
kubectl annotate namespace checkout chaos-mesh.org/inject=enabled
kubectl annotate namespace catalog chaos-mesh.org/inject=enabled
That annotation is the first blast-radius boundary and it lives outside the agent entirely: a PodChaos targeting the payments namespace is ignored by the controller no matter who created it. The agent's own allowlist (below) is a second, redundant check — defense in depth is the right posture for anything that injects faults on purpose.
The Catalogue Is Code; the Agent Fills in Blanks
The model never authors YAML. It selects a template and supplies a handful of bounded parameters; the wrapper renders the CRD. Three templates cover most first-year chaos programs:
# catalogue.py — every experiment the agent can ever propose
CATALOGUE = {
"pod_kill_one": {
"kind": "PodChaos",
"spec": lambda p: {
"action": "pod-kill",
"mode": "one", # never 'all', never a percentage
"gracePeriod": 0,
"selector": {"namespaces": [p["namespace"]],
"labelSelectors": {"app": p["app"]}},
},
"max_duration": "0s", # instantaneous
},
"upstream_delay": {
"kind": "NetworkChaos",
"spec": lambda p: {
"action": "delay",
"mode": "one",
"selector": {"namespaces": [p["namespace"]],
"labelSelectors": {"app": p["app"]}},
"direction": "to",
"target": {"mode": "all",
"selector": {"namespaces": [p["namespace"]],
"labelSelectors": {"app": p["target_app"]}}},
"delay": {"latency": f'{p["latency_ms"]}ms', "correlation": "25"},
"duration": f'{p["duration_s"]}s',
},
"max_duration": "120s",
},
"cpu_stress_one": {
"kind": "StressChaos",
"spec": lambda p: {
"mode": "one",
"selector": {"namespaces": [p["namespace"]],
"labelSelectors": {"app": p["app"]}},
"stressors": {"cpu": {"workers": 2, "load": p["cpu_load_pct"]}},
"duration": f'{p["duration_s"]}s',
},
"max_duration": "120s",
},
}
def render(template: str, p: dict, run_id: str) -> dict:
t = CATALOGUE[template]
return {
"apiVersion": "chaos-mesh.org/v1alpha1",
"kind": t["kind"],
"metadata": {"name": f"agent-{run_id}", "namespace": p["namespace"],
"labels": {"chaos-agent/run": run_id}},
"spec": t["spec"](p),
}
mode: one is hard-coded into every template on purpose. Chaos Mesh supports all, fixed-percent, and random-max-percent, and an agent that can choose among them will eventually reason its way to "the hypothesis is stronger if we kill 50%". The hypothesis is stronger. That is not the point.
Read Tools: Topology and SLO, Not Free-Form kubectl
The agent sees two tools. Both return computed facts, never raw objects, following the same discipline as the least-privilege kubectl agent — but this agent needs even less, because it only reads.
# chaos_tools.py
import os, httpx
from kubernetes import client, config
from fastmcp import FastMCP
config.load_incluster_config()
apps, core, policy = client.AppsV1Api(), client.CoreV1Api(), client.PolicyV1Api()
PROM = os.environ["PROM_URL"]
mcp = FastMCP("chaos-agent")
def _q(query: str) -> float:
r = httpx.get(f"{PROM}/api/v1/query", params={"query": query}, timeout=15).json()
res = r["data"]["result"]
return float(res[0]["value"][1]) if res else 0.0
@mcp.tool()
def service_profile(namespace: str, app: str) -> dict:
"""Redundancy facts for one Deployment: replicas, PDB, probes, upstreams."""
dep = apps.read_namespaced_deployment(app, namespace)
c = dep.spec.template.spec.containers[0]
pdbs = policy.list_namespaced_pod_disruption_budget(namespace).items
pdb = next((p for p in pdbs
if (p.spec.selector.match_labels or {}).get("app") == app), None)
# Upstreams from the Tempo/OTel service graph, last 1h
up = httpx.get(f"{PROM}/api/v1/query", params={"query":
f'sum by (server) (rate(traces_service_graph_request_total{{client="{app}"}}[1h])) > 0'},
timeout=15).json()["data"]["result"]
return {
"replicas_ready": dep.status.ready_replicas or 0,
"replicas_desired": dep.spec.replicas,
"pdb_min_available": pdb.spec.min_available if pdb else None,
"has_readiness_probe": c.readiness_probe is not None,
"has_startup_probe": c.startup_probe is not None,
"upstreams": sorted(s["metric"]["server"] for s in up),
"cpu_limit": (c.resources.limits or {}).get("cpu"),
}
@mcp.tool()
def slo_status(app: str) -> dict:
"""Burn rate now (5m) and 1h, plus 30d error budget remaining (0..1)."""
return {
"burn_rate_5m": round(_q(f'slo:burn_rate5m{{service="{app}"}}'), 2),
"burn_rate_1h": round(_q(f'slo:burn_rate1h{{service="{app}"}}'), 2),
"budget_remaining_30d": round(_q(f'slo:error_budget_remaining{{service="{app}"}}'), 3),
}
The slo: recording rules are the multi-window burn-rate set from the SLI/SLO implementation guide; if you already run the error budget agent, it reads the same series. The upstreams list comes from the Tempo service graph metrics — that one field is what lets the agent propose a specific dependency delay ("checkout-api → payments") instead of a generic one.
The Proposal Schema and the Prompt
Output is forced through a tool schema. Every parameter the runner uses is an enum or a bounded number; free text lives only in hypothesis and reasoning.
PROPOSE = {
"name": "propose_experiment",
"input_schema": {
"type": "object",
"properties": {
"template": {"enum": ["pod_kill_one", "upstream_delay", "cpu_stress_one"]},
"namespace": {"type": "string"},
"app": {"type": "string"},
"target_app": {"type": "string", "description": "upstream_delay only"},
"latency_ms": {"type": "integer", "minimum": 20, "maximum": 500},
"cpu_load_pct": {"type": "integer", "minimum": 20, "maximum": 80},
"duration_s": {"type": "integer", "minimum": 30, "maximum": 120},
"hypothesis": {"type": "string",
"description": "Falsifiable: '<fault> will not push <SLI> past <threshold>'"},
"abort_burn_rate_5m": {"type": "number", "minimum": 2, "maximum": 10},
"skip_reason": {"type": "string",
"description": "Set instead of a proposal when nothing is safe to run"},
"reasoning": {"type": "string"}
},
"required": ["reasoning"]
}
}
SYSTEM = (
"You design one chaos experiment per run for an SRE team. Call service_profile "
"and slo_status for the candidate service before proposing anything.\n"
"Rules you must respect (the runner enforces them too):\n"
"- Propose nothing if burn_rate_1h >= 1 or budget_remaining_30d < 0.5.\n"
"- pod_kill_one requires replicas_ready >= 2 and a PDB; otherwise the "
"hypothesis is already known to fail and the experiment teaches nothing.\n"
"- upstream_delay target_app must appear in the service's upstreams.\n"
"- Prefer the experiment whose hypothesis you are LEAST sure of. Re-running "
"a fault the service already survived last month has low value.\n"
"- The hypothesis names the SLI and a threshold, e.g. 'availability stays "
"above 99.9% over the run'. Vague hypotheses are rejected."
)
The "least sure" instruction is the useful part of having a model here. A catalogue plus a cron job can kill a pod every Tuesday; it cannot notice that checkout-api gained a new upstream (fraud-scoring) last week with no timeout configured, and that a 300 ms delay on that call is the experiment nobody has run.
Guardrails: Blast Radius the Model Can't Widen
Prompts are requests. The runner is policy, and it re-checks everything the prompt asked for using its own reads, not the model's claims:
ALLOWED_NS = {"checkout", "catalog"} # mirrors the chaos-mesh.org/inject set
WINDOW_UTC = range(9, 16) # weekday business hours only
def validate(p: dict, profile: dict, slo: dict, now) -> str | None:
if "skip_reason" in p: return None
if p["namespace"] not in ALLOWED_NS: return "namespace not allowlisted"
if now.weekday() > 4 or now.hour not in WINDOW_UTC: return "outside window"
if slo["burn_rate_1h"] >= 1 or slo["budget_remaining_30d"] < 0.5:
return "SLO not healthy enough to spend budget on chaos"
if profile["replicas_ready"] < 2 or profile["pdb_min_available"] is None:
return "no redundancy: experiment would be an outage, not a test"
if p["template"] == "upstream_delay" and p["target_app"] not in profile["upstreams"]:
return "target is not an observed upstream"
if owner_kind(p["namespace"], p["app"]) == "StatefulSet":
return "stateful workloads are out of scope for this agent"
if active_experiments() > 0: return "one experiment at a time"
return None
Two of these deserve a note. The redundancy check turns a class of "experiment" into a lint result: if a service has one replica and no PDB, you don't need chaos to learn it will go down — the agent should file that as a finding, not run it. And the StatefulSet exclusion is blunt on purpose. Killing a Postgres primary is a valid experiment, but it belongs to a human-designed GameDay, not to an automated proposer in its first quarter. The Human-in-the-Loop approval gate sits after validate: a Slack message carries the rendered YAML, the hypothesis, and the current burn rate, and only an approver's click creates the CRD. RBAC makes the split real — the agent's ServiceAccount has get/list on Deployments and PDBs and nothing on chaos-mesh.org; the runner's ServiceAccount has create/patch/delete on chaos CRDs, scoped by RoleBinding to the allowlisted namespaces, and no LLM access at all.
The Watchdog Aborts — the Model Is Not in the Loop
The abort path must not involve a language model, a queue, or a network call to a vendor. It's a loop polling one PromQL series:
# watchdog.py — runs alongside every experiment
import time, subprocess
def watch(run_id: str, app: str, abort_burn: float, duration_s: int):
deadline = time.time() + duration_s + 30
while time.time() < deadline:
burn = _q(f'slo:burn_rate5m{{service="{app}"}}')
if burn >= abort_burn:
# Pause is instant and reversible; delete finalizes cleanup.
subprocess.run(["kubectl", "annotate", "--overwrite", "-n", ns_of(run_id),
f"podchaos,networkchaos,stresschaos",
"-l", f"chaos-agent/run={run_id}",
"experiment.chaos-mesh.org/pause=true"], check=True)
subprocess.run(["kubectl", "delete", "podchaos,networkchaos,stresschaos",
"-n", ns_of(run_id), "-l", f"chaos-agent/run={run_id}"], check=True)
return {"outcome": "aborted", "burn_rate_at_abort": burn}
time.sleep(10)
return {"outcome": "completed"}
The pause annotation is Chaos Mesh's own kill switch — it recovers the injected fault (removes the tc qdisc, stops the stressor) without waiting for garbage collection — and the delete follows for hygiene. abort_burn_rate_5m comes from the proposal but is clamped to the 2–10 range by the schema; at a 99.9% target, burn rate 10 on a 5-minute window means about 1% of requests failing, which is the most damage a single sanctioned experiment should be allowed to do. If the watchdog itself can't reach Prometheus, it aborts. Losing observability during fault injection is not a condition under which the fault should continue.
The Report Closes the Loop
After the run, the agent gets one more turn with the outcome and the SLI time series (as numbers, via a tool), and writes the experiment record: hypothesis, what happened, verdict. The template is fixed and the record lands in the repo next to the catalogue, so the next run's prompt can include "experiments run in the last 90 days" and avoid repeats. A completed pod_kill_one that did dent availability is the good outcome — a readiness probe that passes before the app can serve, a PDB that exists but with minAvailable: 0, a client without retries — and that finding gets a ticket, in the same style as the postmortem agent's blameless draft.
Run the whole thing in shadow mode for a month first: proposals go to the channel with no approve button. If a proposal would have been dangerous and validate didn't catch it, that's a guardrail gap to fix before anything is ever applied — the shadow-mode playbook applies verbatim.
Honest Limits
The agent proposes from what it can read, and it can't read intent. A service with two replicas, a PDB, and a hidden shared dependency (one Redis, one NFS mount) looks redundant on paper and isn't; the topology tool won't see it unless the service graph does. The burn-rate abort is reactive with a 5-minute window — a fault that damages a low-traffic service will register slowly, which is why the duration cap is 120 seconds and not 20 minutes. Business-hours-only means you never learn how the system fails at 03:00 with the smaller on-call, which is where real incidents live; that gap is a deliberate trade for a first program, not a permanent stance. And none of this substitutes for the human-run GameDay that tests the things the catalogue excludes: stateful failover, a whole-AZ loss, the pager itself.
Start with the catalogue and the watchdog and no agent — a Tuesday cron that kills one pod of one service under a burn-rate abort is already more chaos engineering than most teams do. Add the proposer when the boring experiments are boring, and let it find the one you haven't thought of.