Build a Pod Security Standards Migration Agent: Dry-Run restricted Per Namespace, Classify Every Violation, and Ship securityContext PRs Before You Flip enforce

What This Agent Does

A Pod Security Standards migration agent answers the question every platform team asks and then postpones: if we set pod-security.kubernetes.io/enforce=restricted on this namespace right now, what breaks. It gets the answer from the API server itself with a server-side dry run of the label change, which returns one warning line per violating controller. It then enriches each violation with facts the admission controller does not have, such as whether the image already runs as a non-root user and whether the process binds a port below 1024. An LLM classifies each violation into one of four buckets, and only the bucket "fixable in the manifest" turns into a pull request with the exact securityContext block. The label change itself ships as a Git change too, in three steps: warn, then audit with the audit log watched for a week, then enforce pinned to a version.

The agent never labels a namespace outside a dry run and never patches a workload. The Kubernetes security best practices guide explains the three standards and the admission controller. This post is about getting a 60-namespace cluster from privileged to restricted without the Friday-afternoon outage.

Why Migrations Stall at warn

Pod Security Admission is built into the API server, so there is no webhook to fail and no webhook timeout to debug. The hard part is not enabling it. The hard part is the long tail of workloads that violate restricted for reasons nobody remembers.

Most teams set warn=restricted everywhere, see hundreds of warnings, and stop. Three things cause that.

• Warnings go to whoever runs kubectl. Argo CD runs kubectl. The developer who owns the Deployment never sees the warning, so nothing changes.
• A violation is not a fix. runAsNonRoot != true is one line in the warning. The fix is sometimes four lines of YAML and sometimes a new base image, and the warning cannot tell you which.
• Some violations are legitimate. A node exporter needs hostPID and a hostPath mount. The right answer is a namespace of its own at privileged, not a securityContext patch.

The agent exists to do the sorting, because sorting is the work.

Step 1: Ask the API Server What Would Break

The inventory step uses a trick that is documented but rarely used. When you add a PSA label to a namespace, the API server evaluates every existing pod against the new level and returns warnings for the ones that fail. With --dry-run=server, you get the evaluation without the label.

kubectl label --dry-run=server --overwrite ns payments \
  pod-security.kubernetes.io/enforce=restricted 2>&1 >/dev/null

Warning: existing pods in namespace "payments" violate the new PodSecurity enforce level "restricted:latest"
Warning: api-7c5ddbdf54-k2x9p (and 2 other pods): allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
Warning: ledger-sync-29184720-qx8vn: hostPath volumes, host namespaces

Warnings arrive on stderr, so the 2>&1 >/dev/null ordering keeps them and drops stdout. The API server already groups pods by controller ("and 2 other pods"), so one Deployment produces one line regardless of replica count. The inventory loop runs this for every namespace that is not already at the target level and for both baseline and restricted, because knowing a workload passes baseline but fails restricted changes the recommendation.

# inventory.py — read-only apart from server-side dry runs
import json, re, subprocess

LINE = re.compile(r"^Warning: (?P<pod>\S+)(?: \(and \d+ other pods\))?: (?P<checks>.+)$")

def dry_run(ns: str, level: str) -> list[dict]:
    out = subprocess.run(
        ["kubectl", "label", "--dry-run=server", "--overwrite", "ns", ns,
         f"pod-security.kubernetes.io/enforce={level}"],
        capture_output=True, text=True, timeout=30,
    ).stderr
    findings = []
    for line in out.splitlines():
        m = LINE.match(line)
        if m and "violate the new PodSecurity" not in line:
            findings.append({"namespace": ns, "pod": m["pod"], "level": level,
                             "checks": [c.strip() for c in m["checks"].split(",")]})
    return findings

def owner_of(ns: str, pod: str) -> dict:
    p = json.loads(subprocess.run(["kubectl", "get", "pod", pod, "-n", ns, "-o", "json"],
                                  capture_output=True, text=True).stdout)
    ref = (p["metadata"].get("ownerReferences") or [{}])[0]
    if ref.get("kind") == "ReplicaSet":          # walk up to the Deployment
        rs = json.loads(subprocess.run(["kubectl", "get", "rs", ref["name"], "-n", ns, "-o", "json"],
                                       capture_output=True, text=True).stdout)
        ref = (rs["metadata"].get("ownerReferences") or [ref])[0]
    return {"kind": ref.get("kind", "Pod"), "name": ref.get("name", pod)}

One honest caveat on permissions. Server-side dry run requires the same RBAC verb as the real request, so the agent's ServiceAccount needs patch on namespaces. That is more than the read-only posture used for the least-privilege agent RBAC on this site, so the agent gets a belt as well as braces: a ValidatingAdmissionPolicy that rejects any non-dry-run namespace write from that ServiceAccount.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: psa-agent-dry-run-only
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["UPDATE", "PATCH"]
        resources: ["namespaces"]
  validations:
    - expression: >-
        request.userInfo.username != 'system:serviceaccount:psa-agent:psa-agent'
        || (has(request.dryRun) && request.dryRun)
      message: "psa-agent may only dry-run namespace changes"

If the agent ever tries to apply a label for real, it is denied, logged, and you have a bug report instead of a changed namespace.

Step 2: Enrich With What Admission Cannot Know

The most common restricted fix is runAsNonRoot: true, and it is also the most common way to break a working pod. If the image's USER is root or unset and you add runAsNonRoot: true without a runAsUser, the kubelet refuses to start the container with the error covered in the CreateContainerConfigError guide. So the enricher checks the image before the model is allowed to suggest the line.

# USER from the image config; empty means root
crane config ghcr.io/acme/api@sha256:3f9c... | jq -r '.config.User // "root"'

# Ports the container declares; anything under 1024 needs NET_BIND_SERVICE or a port change
kubectl get deploy api -n payments -o jsonpath='{.spec.template.spec.containers[*].ports[*].containerPort}'

Each finding leaves the enricher with the controller, the violated checks, the image USER, the declared ports, the volume types, and whether the workload is a DaemonSet with any host* field set. The model sees that record and nothing else. It does not get kubeconfig, and it does not get the raw manifest, because it does not need either to classify.

Step 3: Classify, Then Fix Only One Bucket

The classifier runs with a fixed schema and a short system prompt. The job is categorisation, and the categories decide what the agent is allowed to do next.

You classify Kubernetes Pod Security Standards violations. For each finding, pick exactly one class:
- manifest_fix: the violation is fully resolved by adding or changing securityContext fields; the image USER is already non-root or a runAsUser can be set safely.
- image_fix: the container runs as root and binds a privileged port or writes to a root-owned path; the fix is a different image or Dockerfile change.
- needs_privilege: the workload legitimately requires host namespaces, hostPath, or added capabilities (CNI, node agents, storage drivers). Recommend isolation, not a patch.
- unknown: not enough evidence. Say what evidence would settle it.
Return JSON matching the schema. Never propose runAsNonRoot without a runAsUser when image_user is "root".

{
  "controller": "Deployment/api",
  "namespace": "payments",
  "class": "manifest_fix",
  "confidence": 0.93,
  "reason": "Image USER is 10001; port 8080; no host fields; fails only on missing securityContext fields.",
  "patch": {
    "allowPrivilegeEscalation": false,
    "capabilities": {"drop": ["ALL"]},
    "runAsNonRoot": true,
    "seccompProfile": {"type": "RuntimeDefault"}
  }
}

Only manifest_fix findings with confidence at or above 0.85 become pull requests. image_fix becomes an issue assigned to the owning team with the crane config output attached. needs_privilege becomes a recommendation to move the workload into a namespace labelled baseline or privileged, which is what node agents should have had from the start. unknown goes to a human. This is the same split the manifest review agent uses: deterministic tools find, the model explains and sorts, and nothing writes to the cluster.

On an example 58-namespace cluster, the first inventory run produced this distribution:

Seventy-one of ninety findings were four lines of YAML each. That is the ratio that makes automating the PRs worth it.

Step 4: Ship the Fix as a Pull Request

The agent follows the rule from GitOps for AI agents: change the source, not the cluster. For a Helm-managed service, that means the values file, not the rendered Deployment.

# charts/api/values-prod.yaml (added by the PR)
podSecurityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

The PR body carries the evidence: the dry-run warning line, the image USER, and the class with its reason. The CI check on the PR renders the chart and re-runs admission against the result, so the fix is verified by the same code path that would have rejected it.

helm template api charts/api -f charts/api/values-prod.yaml -n payments \
  | kubectl apply --dry-run=server --warnings-as-errors -n payments -f -

--warnings-as-errors turns PSA warnings into a non-zero exit, which is exactly what you want in CI once the target namespace carries warn=restricted. Before the PR, this command fails. After it, it passes. That is the merge gate.

For the image_fix bucket the PR is usually a one-line image swap, such as nginxinc/nginx-unprivileged on port 8080 with the Service targetPort updated to match, but the agent opens that as a draft because it changes a listening port and a human should confirm the ingress still routes.

Step 5: Ramp warn, audit, enforce

The namespace labels live in Git too, so the ramp is three small PRs per namespace, each gated on evidence from the previous stage.

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.31
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.31
    # enforce is added by the third PR, after seven clean audit days

Pin the version labels. restricted:latest means the set of checks changes on every cluster upgrade, and a workload that passed last month can start failing the day you move to a new minor. Pinning to the current minor makes an upgrade a deliberate second migration rather than a surprise.

audit mode is where the agent earns a week of trust. Every violating request is annotated in the API server audit log, which is the one place Argo CD syncs cannot hide from. On EKS the audit stream is the kube-apiserver-audit log stream in the cluster's CloudWatch log group, and the filter is one jq expression:

aws logs filter-log-events --log-group-name /aws/eks/prod/cluster \
  --log-stream-name-prefix kube-apiserver-audit \
  --filter-pattern '"pod-security.kubernetes.io/audit-violations"' \
  --start-time $(( $(date -d '7 days ago' +%s) * 1000 )) --query 'events[].message' --output text \
  | jq -r 'select(.annotations["pod-security.kubernetes.io/audit-violations"] != null)
           | [.objectRef.namespace, .objectRef.name,
              .annotations["pod-security.kubernetes.io/audit-violations"]] | @tsv' \
  | sort | uniq -c | sort -rn

The enforce PR for a namespace is opened automatically only when this query returns zero rows for seven consecutive days and the dry run from Step 1 returns no warnings. Both conditions are required, because the dry run only sees pods that exist right now, and a nightly CronJob that violates restricted exists for forty seconds a day.

Guardrails and Honest Limits

• The agent writes to Git only. Every cluster-facing call is a read or a server-side dry run, and the ValidatingAdmissionPolicy above enforces that even if the code is wrong.
• Keep a mutating backstop. PSA validates and never mutates. If you want defaults injected for teams that forget, the Kyverno policy-as-code guide shows a podSecurity rule set that can also mutate in the missing fields. Run both: PSA as the floor, Kyverno for the nicer error messages and defaults.
• Init containers count. The unknown bucket in the example was all init containers running chown on a volume. The usual fix is fsGroup in the pod securityContext plus fsGroupChangePolicy: OnRootMismatch, but the model is told to leave that to a human because getting it wrong corrupts permissions on persistent data.
• The LLM never sees secrets or manifests. It receives a short enrichment record and returns a class. If it hallucinates a field, the CI dry run in Step 4 fails the PR before anyone reviews it.
• Attribution. Each PR carries the run ID and the operator who approved the enforce stage, so a later incident can trace a securityContext change to the run that proposed it.

The migration that stalled at warn for a year finishes in about three weeks with this loop: one week of inventory and PR review, one week of audit, and one week of staggered enforce PRs, namespace by namespace, lowest tier first.