The safest write path already exists in your stack
If your cluster is managed by GitOps, your DevOps AI agent should never hold cluster credentials at all. Instead of teaching an LLM to run kubectl safely, point it at the thing your team already trusts for every human change: a pull request against the config repo. The agent proposes a change as a branch and PR, CI runs the same policy gates it runs on human PRs, a human (or CODEOWNERS rule) merges, and Argo CD applies it. The agent gets exactly zero new privileges — it inherits the review pipeline you already built.
This is the GitOps answer to the question I keep coming back to in this series: where does the blast radius live? With a direct-write agent, it lives in an executor you have to build and harden yourself, like the one in human-in-the-loop approval gates. With a PR-opening agent, it lives in branch protection, CI checks, and Argo CD's sync — infrastructure that already survived your last three years of human mistakes.
Why a PR beats a kubectl call
Four properties fall out of the Git write path for free, and every one of them is something you'd otherwise build by hand:
- Review is native. A PR diff is the exact change, rendered by GitHub, reviewed in a UI your team lives in. No custom Slack approval flow, no signed tokens.
- Rollback is
git revert. An agent change that goes bad is undone the same way any bad change is undone, and Argo CD reconciles the cluster back. Compare that to reconstructing what a directkubectl patchdid at 2am. - Audit is the commit log. Who proposed, who approved, what changed, when it synced — all answerable without building an audit system.
- The LLM never touches cluster credentials. The agent's token can write to one repo. It cannot write to the API server, because it has nothing that authenticates there. Argo CD is the only writer, exactly as in ArgoCD production best practices.
The trade-off is honesty about latency: a PR-plus-sync loop takes minutes, not seconds. This is the wrong pattern for urgent incident mitigation — keep a narrow approval-gated executor for that. It is the right pattern for the 95% of agent-proposed changes that are not urgent: resource rightsizing, image bumps, replica tuning, config rollouts.
(LLM, repo-scoped token) (CI + humans) (Argo CD, cluster creds)
propose_change tool ----> PR + policy gates ----> merge ----> sync to cluster
| | |
structured change kubeconform / kyverno the ONLY writer
Step 1: the agent proposes a structured change, never raw YAML
The most important rule: the model picks the change; deterministic code renders the YAML. If you let the LLM emit manifest text, you will eventually merge hallucinated fields that kubeconform happens not to catch. Constrain it to a tool schema with an enum of change kinds your renderer knows how to apply:
import anthropic
PROPOSE_TOOL = {
"name": "propose_git_change",
"description": "Propose ONE change to the GitOps config repo.",
"input_schema": {
"type": "object",
"properties": {
"kind": {"enum": ["set_image", "set_replicas", "set_resources"]},
"app": {"type": "string", "description": "App directory under clusters/prod/apps/"},
"container": {"type": "string"},
"value": {"type": "string",
"description": "New image tag, replica count, or 'cpu=500m,memory=512Mi'."},
"justification": {"type": "string",
"description": "Two sentences max, citing the evidence."},
},
"required": ["kind", "app", "value", "justification"],
},
}
SYSTEM = (
"You are a platform engineering assistant. Based on the utilization report, "
"propose at most ONE conservative change via propose_git_change. Never "
"propose a change that reduces capacity by more than 30% in one step. "
"If the evidence is ambiguous, propose nothing."
)
def propose(evidence: str):
client = anthropic.Anthropic()
msg = client.messages.create(
model="claude-sonnet-5",
max_tokens=600,
system=SYSTEM,
tools=[PROPOSE_TOOL],
messages=[{"role": "user", "content": evidence}],
)
for block in msg.content:
if block.type == "tool_use":
return block.input
return None
The renderer applies the change with yq against a checkout — boring, testable, and incapable of inventing fields:
import subprocess, pathlib
def render(change: dict, repo_dir: str) -> str:
path = f"{repo_dir}/clusters/prod/apps/{change['app']}/deployment.yaml"
if not pathlib.Path(path).is_file():
raise FileNotFoundError(f"unknown app {change['app']}") # no invented apps
if change["kind"] == "set_replicas":
subprocess.run(["yq", "-i",
f".spec.replicas = {int(change['value'])}", path], check=True)
elif change["kind"] == "set_image":
c = change["container"]
subprocess.run(["yq", "-i",
f'(.spec.template.spec.containers[] | select(.name == "{c}")).image'
f' = "{change["value"]}"', path], check=True)
return path
Note the existence check. The agent can only modify apps that already exist in the repo — the file-not-found error is your first hallucination guard, and it fires before anything reaches review. What evidence you feed the propose step (utilization queries, cost reports) is the same context discipline as a FinOps agent for Kubernetes cost — several of my agent PRs are exactly that agent's recommendations, routed through Git instead of an executor.
Step 2: branch, commit, open the PR with a scoped token
Give the agent a fine-grained PAT or GitHub App installation token scoped to the one config repo, with contents: write and pull_requests: write and nothing else. It can open PRs; it cannot merge them, because branch protection requires review it cannot give itself.
import requests
API = "https://api.github.com/repos/acme/k8s-config"
H = {"Authorization": "Bearer " + open("/etc/agent/gh-token").read().strip(),
"Accept": "application/vnd.github+json"}
def open_pr(change: dict, branch: str) -> str:
base = requests.get(f"{API}/git/ref/heads/main", headers=H).json()
requests.post(f"{API}/git/refs", headers=H, json={
"ref": f"refs/heads/{branch}", "sha": base["object"]["sha"]})
# ...commit the rendered file to the branch via the contents API...
pr = requests.post(f"{API}/pulls", headers=H, json={
"title": f"agent: {change['kind']} {change['app']} -> {change['value']}",
"head": branch, "base": "main",
"body": (f"**Proposed by ops-agent**\n\n{change['justification']}\n\n"
"Evidence and rendered diff below. Merge = approve."),
}).json()
requests.post(f"{API}/issues/{pr['number']}/labels",
headers=H, json={"labels": ["agent-proposed"]})
return pr["html_url"]
Two conventions matter here. The agent-proposed label lets you filter, meter, and — if things go wrong — bulk-close agent PRs. And the PR body carries the justification and evidence, because the reviewer's question is never "what changed" (the diff shows that) but "why should I believe this is safe."
Step 3: CI gates that treat the agent as an untrusted contributor
Agent PRs run the same gates as human PRs, plus stricter ones. Schema-validate every manifest, run policy-as-code, and hard-fail on paths the agent has no business touching:
name: agent-pr-gates
on:
pull_request:
paths: ["clusters/**"]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Block out-of-bounds paths for agent PRs
if: contains(github.event.pull_request.labels.*.name, 'agent-proposed')
run: |
git fetch origin main
CHANGED=$(git diff --name-only origin/main...HEAD)
echo "$CHANGED" | grep -vE '^clusters/prod/apps/[^/]+/deployment\.yaml$' \
&& { echo "agent touched a forbidden path"; exit 1; } || true
- name: Schema validation
run: |
kubeconform -strict -summary clusters/prod/apps/*/deployment.yaml
- name: Policy checks
run: |
kyverno apply policies/ --resource clusters/prod/apps/ --table
The path check is the piece teams skip and regret. Your prompt says "only touch deployments" — but the prompt is a request, not a boundary. The CI job is the boundary: even a fully compromised agent that commits a change to clusters/prod/rbac/ fails the gate and cannot merge. Layer Kyverno policies behind it so a technically-valid-but-noncompliant manifest (missing resource limits, latest tag) also dies in CI, deterministically, at zero token cost.
Then make review non-optional with branch protection plus CODEOWNERS:
# CODEOWNERS
clusters/prod/ @acme/platform-team
With "require review from code owners" enabled, the merge button is your human-in-the-loop approval gate — no custom tokens, no Slack callbacks, and the approval UX is one your team already uses daily.
Step 4: Argo CD is the only executor
After merge, Argo CD syncs the change. The agent never sees this step and needs nothing for it. Two settings worth checking for agent-originated changes: enable automated sync with self-heal off for the first weeks so you can watch what agent merges do before trusting auto-sync fully, and set sync windows so a Friday-evening merge doesn't roll out until Monday. If a synced change degrades a service, recovery is git revert plus sync — and if you want progressive protection on top, route the deployment through Argo Rollouts canary analysis so a bad agent change gets automatically rolled back by metrics before it takes the fleet down.
Rate limits and the runaway-PR failure mode
The GitOps path removes the scariest failure mode (direct cluster writes) but introduces a mundane one: a looping agent that opens forty PRs overnight. Bound it in the harness, not the prompt:
- One open agent PR per app. Before proposing, list open PRs with the
agent-proposedlabel for that app; if one exists, update it or stand down. - Daily budget. Cap total agent PRs per day (I use 5) and alert when the cap is hit — hitting it usually means bad evidence upstream, not forty good ideas.
- No self-re-trigger. Make sure the agent's own PRs and merges are excluded from whatever event stream triggers it, or you have built a loop.
These are the same bounded-autonomy rules as any agent harness: the model decides what to propose; code decides how often and where.
Where this pattern fits
Use the GitOps write path when the change is declarative, non-urgent, and lives in a repo Argo CD (or Flux) already watches: rightsizing, image bumps, replica and HPA tuning, config values. Keep a separate, narrowly-scoped approval-gated executor for the handful of imperative, time-critical actions — restarts and rollbacks mid-incident — where minutes matter. If you run both, you get a clean division: the executor handles the 3am page, the PR agent handles the daily grind of keeping the repo matched to reality. And because every agent decision lands as a commit, your postmortems and your capacity reviews read straight out of git log — which is exactly where an infrastructure decision trail belongs.