Build a Kubernetes HPA Tuning Agent: Catch Flapping, Pinned-at-Max, and Never-Scaling Autoscalers, Then Ship Stabilization Window and Replica PRs

What This Agent Does

A Kubernetes HPA tuning agent reads a week or two of HorizontalPodAutoscaler history out of Prometheus, sorts every autoscaler in the cluster into one of four behaviors (flapping, pinned at max, never scaling, or healthy), and opens one pull request per HPA with the narrowest fix: a longer scale-down stabilization window, a rate-limited scale-down policy, a lower minReplicas, a higher maxReplicas, or a note that the metric is wrong and the workload belongs on KEDA. It never patches the cluster. It cannot lower maxReplicas, cannot drop minReplicas below what the PodDisruptionBudget needs, and cannot touch an HPA that KEDA owns.

It exists because of how HPAs get written on a platform team. One engineer copies a working manifest with minReplicas: 3, maxReplicas: 10, and a 70% CPU target, and a year later there are 140 of them, most never tested against real load. The autoscaling explainer covers what the knobs mean. This post is about finding out which of the 140 are set wrong, without a human opening 140 Grafana panels.

The Four Behaviors and What Each Costs

Pinned at max and never scaling are mirror images, and both are invisible in a dashboard that only shows current replicas. The agent needs history, and kube-state-metrics has been exporting the right series for years.

Step 1: Mine kube-state-metrics for HPA History

Five series carry everything the classifier needs. All are labeled namespace and horizontalpodautoscaler.

# hpa/miner.py
import requests

PROM = "http://prometheus-k8s.monitoring.svc:9090"
WINDOW = "14d"

QUERIES = {
    # direction changes per hour, averaged over the window
    "flaps_per_hour": f'''
      sum by (namespace, horizontalpodautoscaler) (
        changes(kube_horizontalpodautoscaler_status_desired_replicas[{WINDOW}])
      ) / (14 * 24)''',
    # fraction of the window spent at maxReplicas
    "pinned_at_max_ratio": f'''
      avg_over_time((
        kube_horizontalpodautoscaler_status_desired_replicas
          == bool on (namespace, horizontalpodautoscaler)
        kube_horizontalpodautoscaler_spec_max_replicas
      )[{WINDOW}:1m])''',
    # highest desired count ever reached
    "peak_desired": f'''
      max_over_time(kube_horizontalpodautoscaler_status_desired_replicas[{WINDOW}])''',
    "min_replicas": "kube_horizontalpodautoscaler_spec_min_replicas",
    "max_replicas": "kube_horizontalpodautoscaler_spec_max_replicas",
    # how often the HPA could not read its metric at all
    "metric_unavailable_ratio": f'''
      avg_over_time(kube_horizontalpodautoscaler_status_condition{{
        condition="ScalingActive", status="false"}}[{WINDOW}:1m])''',
}

def query(expr: str) -> dict[tuple[str, str], float]:
    r = requests.get(f"{PROM}/api/v1/query", params={"query": expr}, timeout=60)
    r.raise_for_status()
    out = {}
    for s in r.json()["data"]["result"]:
        key = (s["metric"]["namespace"], s["metric"]["horizontalpodautoscaler"])
        out[key] = float(s["value"][1])
    return out

def collect() -> list[dict]:
    cols = {name: query(expr) for name, expr in QUERIES.items()}
    keys = set(cols["min_replicas"]) & set(cols["max_replicas"])
    return [{"namespace": ns, "hpa": name,
             **{c: cols[c].get((ns, name), 0.0) for c in cols}} for ns, name in keys]

The metric_unavailable_ratio column is the one people forget. An HPA whose metrics server went away looks like it flapped, because desired replicas jump when the metric comes back. Any HPA above 2% unavailability over the window is excluded from flapping and sent to the FailedGetResourceMetric runbook instead. Tuning a stabilization window on an HPA that cannot read CPU is tuning the wrong thing.

Two more facts come from the API server, read-only, per HPA: the current behavior block (most HPAs have none, which means a 300-second scale-down window and unlimited scale-up) and the metrics list, so the classifier knows whether the target is CPU utilization, memory, or something external.

Step 2: Classify in Code, Not in the Prompt

The thresholds are plain numbers a platform team can argue about in a code review. The model never sees an HPA that code has already decided is healthy.

# hpa/classify.py
def classify(h: dict, pdb_min_available: int | None, tier: str) -> str:
    if h["metric_unavailable_ratio"] > 0.02:
        return "metrics_broken"          # not ours, route to the runbook
    if h["min_replicas"] == h["max_replicas"]:
        return "static"                  # someone disabled autoscaling on purpose
    if h["flaps_per_hour"] >= 4:
        return "flapping"
    if h["pinned_at_max_ratio"] >= 0.30:
        return "pinned_at_max"
    if h["peak_desired"] <= h["min_replicas"] and h["min_replicas"] > 1:
        floor = max(pdb_min_available or 1, 2 if tier == "tier-1" else 1)
        if h["min_replicas"] > floor:
            return "never_scales"
    return "healthy"

Four direction changes an hour is the flapping line because scale-down cannot happen faster than once per stabilization window, so an HPA on the default 300 seconds that changes four times an hour is riding the sawtooth continuously: scale up, wait five minutes, scale down, repeat. A pinned-at-max ratio of 30% over two weeks means the autoscaler spends most business hours at its ceiling, which is where the ceiling stops being a safety margin and becomes a capacity decision nobody made.

The never-scales branch is the FinOps one. Its floor comes from two places: the PodDisruptionBudget's minAvailable (an HPA floor below it means every node drain violates the budget) and the service tier from the catalog. Tier-1 never goes below two replicas regardless of what fourteen days of traffic said, because the quiet fortnight is not the one that matters.

Step 3: The Model Picks the Narrowest Fix

For the three actionable classes, the model receives one HPA at a time with its numbers, its current behavior block, the resource requests of the target Deployment, the PDB, the tier, and the policy constraints. It returns a strict schema.

VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string", "enum": [
            "lengthen_scale_down_window", "rate_limit_scale_down",
            "lower_min_replicas", "raise_max_replicas",
            "wrong_metric_needs_keda", "leave_alone", "needs_human"]},
        "stabilization_window_seconds": {"type": "integer"},
        "scale_down_pods_per_minute": {"type": "integer"},
        "new_min_replicas": {"type": "integer"},
        "new_max_replicas": {"type": "integer"},
        "evidence": {"type": "string"},
        "what_breaks_if_wrong": {"type": "string"},
    },
    "required": ["verdict", "evidence", "what_breaks_if_wrong"],
}

SYSTEM = (
    "You tune HorizontalPodAutoscalers for a platform team.\n"
    "Flapping: prefer lengthen_scale_down_window to 600-900s. If the workload "
    "has a slow cold start (JVM, large model load) prefer rate_limit_scale_down "
    "at one pod per minute instead. Never shorten a window. Never touch scaleUp.\n"
    "Pinned at max: if the target Deployment's containers are CPU throttled or "
    "already at their CPU limit, the fix is not maxReplicas. Return needs_human "
    "and say so. Otherwise raise_max_replicas to peak_desired * 1.5, capped by "
    "namespace quota.\n"
    "Never scales: lower_min_replicas to the PDB floor or peak_desired, "
    "whichever is higher. Never below the floor you were given.\n"
    "If the metric is CPU but the workload is a queue consumer or a cron-shaped "
    "batch, return wrong_metric_needs_keda. Do not write a ScaledObject.\n"
    "Cite the numbers you were given. Do not invent traffic patterns."
)

The what_breaks_if_wrong field is read before anything else in review. A stabilization window that is too long costs money for ten extra minutes after each peak. A minReplicas that is too low means the first request after a quiet night hits a cold pod. A maxReplicas raise that is wrong is the expensive one, because it is a standing order for node capacity, and the capacity planning agent will come asking about it.

Step 4: One PR per HPA, Through Git

Proposals land in the manifests repository following the GitOps-for-agents pattern, never through kubectl. The flapping fix is the most common diff, and it is additive: most HPAs have no behavior block at all.

# apps/search-api/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: search-api
spec:
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:                      # Agent PR #112: 6.8 desired-replica changes/hour over 14d,
    scaleDown:                   # JVM cold start ~40s, p99 +180ms on each scale-down edge
      stabilizationWindowSeconds: 900
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60
      selectPolicy: Min

For a JVM or model-serving workload the agent pairs the longer window with a one-pod-per-minute policy and selectPolicy: Min, so a 20-replica service drains one pod a minute instead of dropping 10% of itself every 15 seconds. Scale-up is deliberately untouched. A flapping fix that slows scale-up turns a cost problem into an availability problem.

The never-scales diff is one line, and the PR body does the cost math from the Deployment's requests and the per-core price the cost allocation post derives per namespace:

spec:
  minReplicas: 2   # was 6. Agent PR #117: peak desired over 14d was 2, PDB minAvailable 1,
                   # tier-2. Frees 4 x (500m CPU, 1Gi) ≈ $58/month at this namespace's blended rate.

Every PR carries the before numbers, and after merge the wrapper posts a comment with the same three metrics at one day and seven days. A flapping HPA that still changes more than twice an hour after a week reopens the finding with the first verdict attached, so the next round knows the window alone did not fix it.

Guardrails That Live in Code

The prompt asks nicely. The wrapper refuses.

• maxReplicas only goes up. A verdict that lowers it is dropped before a PR exists. Capping an autoscaler is a capacity policy, not a tuning fix.
• minReplicas never crosses the floor. PDB minAvailable, the tier minimum, and the observed peak are all enforced in code. The model is told the floor; the wrapper checks it anyway.
• KEDA-owned HPAs are skipped. Any HPA with a ScaledObject owner reference (KEDA names them keda-hpa- plus the object name) is routed back to the ScaledObject, because editing the generated HPA is undone on the next reconcile.
• Recently changed HPAs are skipped. If the manifest changed in git within the window, the data is mixed. Fourteen days of clean history or no verdict.
• Quota headroom is checked for every maxReplicas raise. If the namespace ResourceQuota could not fit maxReplicas pods at the current request, the verdict becomes needs_human with the math attached.
• Three open PRs at a time, one per HPA. A rubber-stamped batch of minReplicas cuts across a namespace is how you find out which services share a cold-start dependency.
• Read-only everywhere. PromQL goes through the read-only Prometheus MCP server. The ServiceAccount has get and list on HPAs, Deployments, and PDBs, and nothing else.

Run the first month in shadow mode. The verdicts to score hardest are the leave_alone and needs_human ones on pinned-at-max HPAs. An agent that answers every ceiling with a higher ceiling is a cost agent's nightmare, and the correct answer is often that the pods are CPU throttled and the ceiling is a symptom. That case belongs to the CPU throttling agent, and the two should read each other's findings before either opens a PR on the same Deployment.

Honest Limits

Fourteen days is enough to catch flapping and never-scaling, and not enough to catch a monthly batch or a quarterly peak. The agent reports the window it used in every PR, and a minReplicas cut on a workload with a known month-end spike should be rejected by the human who knows that. The agent does not know it, and will not pretend to.

The HPA's global tolerance (10% by default, set on the controller manager) is not a per-workload knob the agent can change from a manifest. Newer Kubernetes releases add a per-HPA tolerance field behind the HPAConfigurableTolerance feature gate, and the agent proposes it only when the wrapper has confirmed the gate is on. If most of your flapping findings are services sitting a few percent either side of their target, that is a controller-level conversation, not an agent output.

Flapping on a memory-based HPA is a different animal. Memory utilization does not fall when load falls for most runtimes, so a memory HPA tends to pin at max rather than flap, and the honest fix is usually to stop scaling on memory at all. The agent returns needs_human for memory-target HPAs rather than guessing, with the usage curve attached.

Finally, the agent tunes HPAs it can see in git. An HPA created by a Helm chart with autoscaling.enabled: true has its real knobs in values.yaml, and a direct edit to the rendered manifest is overwritten on the next upgrade. The wrapper detects the Helm release label and proposes the values change instead, but that only works if the chart exposes a behavior value. Many do not, and for those the PR body says so and asks for a chart change.