Build a Kubernetes CPU Throttling Agent: Find CFS-Throttled Pods, Prove the Throttling Hurts Latency, and Ship Limit Fixes as PRs

What This Agent Does

A Kubernetes CPU throttling agent reads two cAdvisor counters from Prometheus, ranks every container by how often the Linux CFS scheduler paused it for exceeding its CPU limit, checks whether that throttling lines up with the service's p99 latency, and opens one pull request per workload with the narrowest fix: a bigger limit, a Guaranteed QoS class with exclusive cores, a runtime thread-count setting, or nothing at all. It never writes to the cluster. It cannot lower a request, cannot remove a limit in a namespace whose policy requires one, and cannot propose a limit the namespace quota would reject.

It is the mirror image of the FinOps rightsizing agent. That one finds containers asking for far more CPU than they use. This one finds containers that were given too little and are paying for it in tail latency, which is the failure the cost agent quietly creates when a reviewer merges its PRs too eagerly. If you run one, run both.

Why Throttling Is Invisible Until You Look

CPU limits in Kubernetes are enforced by the CFS bandwidth controller. Each container gets a quota per 100 ms period. A container with limits.cpu: 500m may run for 50 ms of CPU time per period. The moment its threads use that up, every one of them is descheduled until the next period starts. The pod shows no restart, no event, no error in the logs, and a CPU graph that looks fine because average utilization is well below the limit. What the user sees is a request that takes 60 ms instead of 8 ms, because a 4-thread service burned its 50 ms quota in 12 ms of wall time and then slept for 88 ms.

The kube-prometheus stack ships a CPUThrottlingHigh alert at 25% throttled periods, and in most clusters it fires on dozens of containers nobody owns. That alert is the input to this agent, not its output. The agent exists to decide which of those dozens matter.

Step 1: Rank by Throttled Periods, Not by Throttled Seconds

Two counters matter: container_cpu_cfs_periods_total and container_cpu_cfs_throttled_periods_total. Their ratio is the fraction of 100 ms windows in which the container hit its quota. Rank on that, not on container_cpu_cfs_throttled_seconds_total, which rewards busy batch jobs that do not care about latency.

# throttle/miner.py
import requests

PROM = "http://prometheus-k8s.monitoring.svc:9090"

RATIO = '''
sum by (namespace, pod, container) (
  increase(container_cpu_cfs_throttled_periods_total{container!="", container!="POD"}[%s])
)
/
sum by (namespace, pod, container) (
  increase(container_cpu_cfs_periods_total{container!="", container!="POD"}[%s])
)
'''

def query(expr: str) -> list[dict]:
    r = requests.get(f"{PROM}/api/v1/query", params={"query": expr}, timeout=30)
    r.raise_for_status()
    return r.json()["data"]["result"]

def throttled_containers(window: str = "6h", min_ratio: float = 0.25) -> list[dict]:
    rows = []
    for s in query(RATIO % (window, window)):
        ratio = float(s["value"][1])
        if ratio < min_ratio:
            continue
        m = s["metric"]
        rows.append({"namespace": m["namespace"], "pod": m["pod"],
                     "container": m["container"], "throttle_ratio": round(ratio, 3)})
    return sorted(rows, key=lambda r: -r["throttle_ratio"])

Collapse pods to their owning workload before anything else, otherwise a 40-replica deployment produces 40 findings. The miner then enriches each workload with four more numbers: the current request and limit from kube_pod_container_resource_requests and kube_pod_container_resource_limits with resource="cpu", the p99 of per-second CPU usage over seven days, and the burst peak.

P99_USAGE = '''
quantile_over_time(0.99,
  sum by (namespace, pod, container) (
    rate(container_cpu_usage_seconds_total{namespace="%s", pod=~"%s-.*", container="%s"}[1m])
  )[7d:1m]
)
'''

The gap between the p99 of one-minute usage and the limit is the first clue. A container throttled 40% of the time while its one-minute average sits at half the limit is a burst problem: it needs its CPU in short, intense slices, which is exactly what per-period quota punishes.

Step 2: Prove It Hurts Before Proposing Anything

Throttling on a nightly report generator is a fact, not a problem. Throttling on a checkout API is an SLO problem. The wrapper joins the throttle series to the service's latency histogram over the same seven days at a five-minute step and computes a plain Pearson correlation. The agent does not get to claim a link that the numbers do not show.

# throttle/latency.py
import statistics

def correlation(prom_range_a: list[float], prom_range_b: list[float]) -> float:
    if len(prom_range_a) < 50 or len(prom_range_a) != len(prom_range_b):
        return 0.0
    return statistics.correlation(prom_range_a, prom_range_b)

P99_LATENCY = '''
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{namespace="%s", service="%s"}[5m])))
'''

The latency metric name comes from the service catalog, not from the model guessing a label. The Backstage MCP server supplies the owner, the tier, and the SLO definition for each workload. Tier-1 services with an SLO get ranked first. Workloads with no latency SLI at all are still reported, but the model is told so explicitly, and it is forbidden from inventing user impact for them.

Two more facts reach the model per finding, because they change the fix:

• Runtime and thread count. The wrapper reads the image labels and the GOMAXPROCS, JAVA_TOOL_OPTIONS, or UV_THREADPOOL_SIZE env vars from the pod spec. A Go binary built before Go 1.25 with no GOMAXPROCS set and a 1-core limit will schedule as many threads as the node has cores, then throttle. Newer Go runtimes read the cgroup limit, so the agent checks the version before blaming the runtime.
• Namespace policy. LimitRange maximums, ResourceQuota headroom for limits.cpu, and whether a Kyverno policy requires limits. The agent proposes nothing admission would reject, which is the same rule the manifest review agent enforces from the other direction.

Step 3: The Model Picks One of Five Verdicts

The model receives one workload at a time: throttle ratio, request, limit, p99 usage, peak usage, correlation with latency, tier, SLO, runtime facts, and policy constraints. It returns a strict schema.

VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string", "enum": [
            "raise_limit", "guaranteed_exclusive_cpu",
            "fix_runtime_threads", "leave_alone", "needs_human"]},
        "new_limit_millicores": {"type": "integer"},
        "new_request_millicores": {"type": "integer"},
        "evidence": {"type": "string"},
        "what_breaks_if_wrong": {"type": "string"},
    },
    "required": ["verdict", "evidence", "what_breaks_if_wrong"],
}

SYSTEM = (
    "You fix CPU throttling on Kubernetes for an SRE team.\n"
    "Only propose a change when throttling correlates with latency or the "
    "workload has a latency SLO. A throttled batch job with no SLO is leave_alone.\n"
    "Prefer raise_limit sized at the p99 of usage plus the burst peak, rounded "
    "up to the next 250m. Never lower a request. Never exceed the policy max.\n"
    "If the runtime schedules more threads than the limit allows, the fix is "
    "fix_runtime_threads first, and a limit change only if throttling persists.\n"
    "Use guaranteed_exclusive_cpu only for tier-1 services whose p99 usage "
    "already sits near an integer core count.\n"
    "Cite the numbers you were given. Do not invent user impact."
)

The what_breaks_if_wrong field is the one that keeps reviewers honest. A limit raise that is wrong costs a few cents of node capacity. A runtime thread change that is wrong can halve throughput. A move to Guaranteed QoS with the static CPU manager that is wrong strands cores nobody else can use. The model has to name that cost before the PR exists, and reviewers read that line first.

Step 4: One PR per Workload, With Before and After

Proposals ship through the repo, never through kubectl, following the GitOps-for-agents pattern. A typical limit PR is a four-line diff with a long body.

# apps/checkout-api/deployment.yaml
resources:
  requests:
    cpu: 500m          # unchanged
    memory: 512Mi
  limits:
    cpu: "1500m"       # was 500m. Agent PR #88: throttled 47% of periods over 6h,
                       # p99 usage 0.61 cores, burst peak 1.3 cores, r=0.71 with p99 latency
    memory: 512Mi

For the runtime case, the diff is an env var rather than a resource change:

env:
  - name: GOMAXPROCS
    value: "2"         # Agent PR #91: go1.22 image, 64-core nodes, 2-core limit

For the Guaranteed case the PR sets request equal to limit at an integer core count, and the body states the precondition: the node pool must run the kubelet with the static CPU manager policy, otherwise the change buys nothing except a bigger request. The kubelet does not apply CFS quota to containers that hold exclusive cores under that policy, which is the whole point of the move. Under the default policy the same spec is just a Guaranteed pod that still gets throttled.

Every PR carries the two throttle numbers from before the change and, after merge, a follow-up comment posting the same ratio one hour and one day later. If the ratio did not fall below 5% the wrapper reopens the finding, because the first guess was wrong and the next verdict should know that.

Guardrails That Live in Code

The prompt asks nicely. The wrapper refuses.

• Requests only go up or stay put. Any verdict that lowers a request is dropped before it becomes a PR, no matter how it was argued. Rightsizing downward belongs to the cost agent, with its own review path.
• Limit raises are capped at 3x the current limit and at the LimitRange max. Beyond 3x the problem is not a limit, it is a missing replica or a runaway loop, and a human should look.
• Quota headroom is checked at proposal time. If the namespace ResourceQuota would reject the new limits.cpu total, the verdict becomes needs_human with the quota math attached.
• Removing limits is not a verdict. Some teams run request-only workloads deliberately. That is a cluster policy decision made once, not a per-workload PR an agent opens on a Tuesday.
• Three open PRs at a time, one per workload. Twenty resource PRs get rubber-stamped, and a rubber-stamped limit raise across a whole namespace is how you discover node pressure.
• Read-only against the cluster and Prometheus. The agent's only PromQL path is the read-only Prometheus MCP server, and its ServiceAccount has get and list on pods and deployments and nothing else.

Run the first month in shadow mode. Findings land in a channel with their correlation and the proposed diff, and the team scores them. The score that matters is not agreement on the number. It is whether the leave_alone verdicts were right, because an agent that wants to raise every limit it sees is a cost agent's nightmare.

Honest Limits

Correlation between throttling and latency is a ranking signal, not a diagnosis. A service can be throttled and slow for an unrelated reason, like a downstream database, and the agent will happily propose a limit raise that fixes nothing. The after-merge check catches that within a day, but it cannot prevent it. Treat the first PR on any service as an experiment, and read the follow-up comment.

Per-period quota punishes bursts, and the cleanest fix for short bursts is a longer runway inside the period, not a bigger limit. The kubelet's --cpu-cfs-quota-period flag can shorten the period, and newer kernels support CFS burst, but neither is a per-workload knob the agent can set from a manifest, so it does not try. If most of your findings are low-average, high-burst services, that is a node-pool conversation, not an agent output.

The thread-count checks cover Go, the JVM, and Node, which is most of what runs in the clusters this series targets. A Rust service with a hand-tuned Tokio worker count, or a Python app forking workers from Gunicorn, needs a human to read the config, and the agent says so by returning needs_human with the env dump attached rather than guessing.

Finally, a limit raise is a request for node capacity you may not have. The capacity planning agent sees the same PRs and will tell you when enough of them have merged to need another node. Ignoring that message turns a latency fix into the next Pending pod, and the pod that ends up Pending is never the one you raised.