What This Agent Does
A Kubernetes RBAC right-sizing agent answers a question kubectl auth can-i cannot: not what a ServiceAccount may do, but what it has actually done in the last 90 days. It gets that answer from the API server audit log, where every request carries the subject, the verb, the resource, and an annotation naming the exact RoleBinding that allowed it. The agent joins that usage against the rules the subject is granted, computes the unused remainder, and opens a pull request with a narrower Role. It never edits a binding itself. After merge, it watches the same audit log for 403 responses from that subject and opens a revert PR if the shrink broke a workload.
It is the Kubernetes twin of the AWS IAM right-sizing agent. AWS gives you Access Analyzer and last-accessed data for free. Kubernetes gives you nothing unless you turn the audit log on and do the join yourself. That is the whole job.
Why RBAC Only Grows
Nobody writes verbs: ["*"] on purpose. It arrives in three ways.
- The midnight widening. A controller failed with
forbidden, somebody addedcluster-adminto get the deploy out, and the ticket to narrow it is still open. - Helm chart defaults. Many charts ship a ClusterRole with
listandwatchonsecretscluster-wide because one optional feature needs it. - Copied kubeconfigs. A CI job was given a human's
editbinding in 2024 and the human has since left.
The RBAC deep dive explains why these matter and how to enable audit logging. This post assumes the log exists and builds the thing that reads it.
Step 0: An Audit Policy That Captures Enough
The agent needs one event per request at Metadata level, for every verb, including reads. Most clusters log only writes, or only RequestReceived, and the join is useless. This policy is cheap and sufficient.
# audit-policy.yaml — Metadata for everything, no request bodies
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
- level: None
users: ["system:kube-proxy", "system:apiserver"]
verbs: ["watch"]
- level: None
nonResourceURLs: ["/healthz*", "/readyz*", "/livez*", "/metrics", "/version"]
- level: Metadata
On EKS, you do not control the policy, but enabling the audit control plane log type sends an equivalent stream to CloudWatch Logs under /aws/eks/<cluster>/cluster in streams named kube-apiserver-audit-*. The CloudWatch Logs cost agent shows why that group needs a retention policy. Audit logs at Metadata level run about 1 to 3 GB per day on a busy cluster, so set 90 days and move on.
One event looks like this, trimmed to the fields the agent uses.
{
"stage": "ResponseComplete",
"verb": "list",
"user": {"username": "system:serviceaccount:payments:ledger-sync"},
"userAgent": "ledger-sync/v2.3.0",
"objectRef": {"resource": "secrets", "namespace": "payments", "apiGroup": ""},
"responseStatus": {"code": 200},
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"ledger-sync\" of Role \"ledger-sync\" to ServiceAccount \"ledger-sync/payments\""
}
}
The authorization.k8s.io/reason annotation is the part most people do not know exists. It names the binding that granted the request, so the agent can attribute usage to a specific Role even when a subject is bound to several.
Step 1: Aggregate Usage Per Subject
On EKS the aggregation is one Logs Insights query. Field names containing dots or slashes need backticks.
fields @timestamp
| filter @logStream like /kube-apiserver-audit/
| filter user.username like /^system:serviceaccount:/
| filter responseStatus.code < 300
| stats count(*) as n, max(@timestamp) as last_seen
by user.username, verb, objectRef.apiGroup, objectRef.resource,
objectRef.subresource, objectRef.namespace,
`annotations.authorization.k8s.io/reason`
On a self-managed cluster, Loki does the same thing. The query below counts distinct tuples per subject from the audit stream, with the same window the Logs Insights query uses. If you have the Loki MCP server from this site, this is one more read-only query template.
sum by (username, verb, resource, subresource, apiGroup, namespace) (
count_over_time(
{job="kube-apiserver-audit"}
| json username="user.username", verb="verb",
resource="objectRef.resource", subresource="objectRef.subresource",
apiGroup="objectRef.apiGroup", namespace="objectRef.namespace",
code="responseStatus.code"
| username =~ "system:serviceaccount:.*"
| code < 300
[90d]
)
)
Both produce the same shape: a set of (subject, verb, apiGroup, resource, subresource, namespace) tuples with counts and last-seen timestamps. The agent writes them to a local SQLite table so the rest of the pipeline is testable without a cluster.
Step 2: What the Subject Is Granted
Granted permissions come from the API, not the log. rbac-tool is the fastest way to get the effective policy rules for a subject across every binding.
kubectl krew install rbac-tool
# Effective rules for one ServiceAccount, all bindings merged
kubectl rbac-tool policy-rules -e '^system:serviceaccount:payments:ledger-sync$' -o json
# Every subject with cluster-admin, and every rule with a wildcard
kubectl rbac-tool analysis -o json
The analysis subcommand ships with built-in checks for cluster-admin bindings, wildcard verbs and resources, and the escalation verbs bind, escalate, and impersonate. Those findings skip the usage join entirely and go straight to the report, because no 90-day window justifies verbs: ["*"] on resources: ["*"].
For the join, the agent needs a rule matcher that works the way the authorizer does. This is the only piece worth writing carefully, because the PR it produces is only as trustworthy as this function.
# rbac_match.py — does a PolicyRule cover an observed request?
def rule_covers(rule: dict, req: dict) -> bool:
def hit(allowed: list[str], value: str) -> bool:
return "*" in allowed or value in allowed
resource = req["resource"] + (f"/{req['subresource']}" if req.get("subresource") else "")
return (
hit(rule.get("verbs", []), req["verb"])
and hit(rule.get("apiGroups", []), req.get("apiGroup", ""))
and (hit(rule.get("resources", []), resource)
or (req.get("subresource") and "*/" + req["subresource"] in rule.get("resources", [])))
and (not rule.get("resourceNames") or req.get("name") in rule["resourceNames"])
)
def unused_rules(granted: list[dict], observed: list[dict]) -> list[dict]:
"""Return a per-rule breakdown: which (verb, resource) pairs were never exercised."""
out = []
for rule in granted:
pairs = [(v, r) for v in rule["verbs"] for r in rule["resources"]]
used = {
(v, r) for (v, r) in pairs
for req in observed
if rule_covers({**rule, "verbs": [v], "resources": [r]}, req)
}
out.append({"rule": rule, "unused": sorted(set(pairs) - used)})
return out
Wildcards are expanded against the discovery API before the join, so resources: ["*"] on apiGroups: ["apps"] becomes the real list from kubectl api-resources --api-group=apps. Without that expansion the agent would mark a wildcard as "used" because one Deployment get matched it, and the PR would leave the wildcard in place.
Step 3: The Model Writes the Narrower Role, the Matcher Checks It
The LLM gets one subject at a time: the granted rules, the unused pairs, the observed tuples with counts and last-seen dates, and the userAgent strings seen for the subject. It does not get a kubeconfig, and it does not get the audit log itself.
You right-size Kubernetes RBAC. Given a subject's granted PolicyRules and the
requests it made in the last 90 days, write the narrowest Role or ClusterRole
that still permits every observed request.
Rules:
- Never emit "*" in verbs, resources, or apiGroups.
- If every observed request is in one namespace and the current grant is a
ClusterRole, emit a namespaced Role and say so in the rationale.
- Keep "list" and "watch" together if either was observed; controllers use both.
- Keep "get" on a resource if "update" or "patch" on it was observed; a controller
that patches reads first and the sample may have missed it.
- If an unused verb is "delete" or an unused resource is "secrets", flag it as a
risk reduction in the rationale.
- Output JSON: {"manifest": "<yaml>", "rationale": "...", "removed": [...],
"confidence": "high|medium|low"}
The second and third rules exist because the sample is a sample. A 90-day window misses the quarterly job. The matcher takes the model's manifest and replays every observed request against it. Any request the new Role would deny blocks the PR and sends the model one more turn with the specific denied tuples. This is the same proof step the IAM agent gets from CheckNoNewAccess, done locally because Kubernetes has no equivalent API.
def proves_no_lockout(new_rules: list[dict], observed: list[dict]) -> list[dict]:
return [req for req in observed if not any(rule_covers(r, req) for r in new_rules)]
An empty list is the only result that lets the PR open.
Step 4: The PR Is a Two-Phase Change
A shrink that goes straight from the old Role to the new one has a blast radius of "everything this subject does." The PR the agent opens is structured so the old grant survives for one week.
- Phase one, this PR. Add the new Role as
ledger-sync-v2, and add a second RoleBinding to it. The subject is now bound to both old and new. Nothing can break. - Phase two, the follow-up PR the agent opens seven days later. Delete the old RoleBinding. The old Role object stays in git for one more release so a revert is a one-line change.
The PR body carries the usage table so the reviewer does not need to re-run anything.
## RBAC right-size: payments/ledger-sync
| Rule (current) | Observed in 90d | Proposed |
|---------------------------------------------|--------------------------|---------------------|
| "" secrets: get, list, watch | get x1204 (daily) | get, resourceNames: [ledger-db] |
| "" configmaps: get, list, watch, update | get x3190, list x90 | get, list, watch |
| apps deployments: * | get x412 | get |
| "" pods/exec: create | none | removed |
**Removed:** pods/exec create (never used), secrets list+watch (never used), deployments update/patch/delete (never used).
**Agent cannot prove:** nothing. Every observed request is permitted by the proposed Role (4806 of 4806 replayed).
**Evidence window:** 2026-08-03 to 2026-11-01. User agents seen: ledger-sync/v2.3.0, ledger-sync/v2.4.1.
**Phase 2 scheduled:** 2026-11-09, conditional on zero 403s from this subject.
The resourceNames narrowing on secrets is where the biggest risk reduction usually hides. A subject that reads one Secret by name every day does not need list, and without list it cannot enumerate the namespace's secrets if its token leaks.
This agent follows the GitOps rule for agents strictly: its own ServiceAccount has get and list on RBAC resources plus read access to the audit stream, and nothing else. It cannot bind, escalate, or impersonate, so a prompt injection in a userAgent string cannot talk it into granting itself anything.
Step 5: The 403 Watch Is the Rollback Trigger
After phase two merges, the agent polls the audit log for the subject once an hour for seven days.
fields @timestamp, verb, objectRef.resource, objectRef.subresource, objectRef.namespace
| filter @logStream like /kube-apiserver-audit/
| filter user.username = "system:serviceaccount:payments:ledger-sync"
| filter responseStatus.code = 403
| stats count(*) as denied by verb, objectRef.resource, objectRef.subresource
A single 403 from a verb the shrink removed opens a revert PR immediately, with the denied tuple in the title, and posts to the owning team's channel. A 403 on something the subject never had is not the agent's doing and is reported but not reverted. The revert PR restores the old RoleBinding, which is why the old Role stays in git through the watch window.
What It Found on a Real Cluster
Numbers from one 48-namespace EKS cluster, first run, 90-day window.
| Finding | Count |
|---|---|
| ServiceAccounts bound to cluster-admin | 6 |
| ClusterRoles with a wildcard verb or resource | 23 |
Subjects with secrets list or watch never exercised | 31 |
Subjects with pods/exec never exercised | 14 |
| Subjects with zero requests in 90 days | 19 |
The last row is the one that matters most and the one the agent will not act on. Nineteen bound ServiceAccounts made no request at all. Some are dead and should be deleted. Some belong to a disaster recovery job that runs twice a year. The agent lists them with the binding and the last-seen date and stops. Deleting a ServiceAccount that a quarterly job depends on is a decision for a human with the runbook open.
Honest Limits
- The window is a sample. Anything with a cadence longer than the window will be shrunk out. The two-phase PR and the 403 watch make that recoverable rather than impossible, but they do not make it free. Keep an allowlist of subjects the agent may not touch, and put the DR jobs on it.
- Aggregated APIs and CRDs. Requests to CRDs show up in the audit log with the custom
apiGroup, so they join fine. Wildcard expansion needs the discovery API to list them, which means the agent must run against the cluster, not just the log. - Impersonation hides the real caller. A request made with
--asis logged under the impersonated user with the impersonator inimpersonatedUser. The agent attributes usage to the impersonated subject, which is correct for RBAC but can surprise the reviewer. - Human users are a different problem. This agent targets ServiceAccounts, whose behaviour is a program. Humans use
kubectlinteractively and their 90-day history is not a contract. Report on them. Do not open PRs for them.
Run it monthly, after the security group audit and before the quarterly access review, and the "temporary" cluster-admin bindings finally get the follow-up ticket they were promised.