Build a GPU Waste Agent: Find Idle and Underutilized GPUs on Kubernetes with DCGM, Then Reclaim Them with Time-Slicing, MIG, and Kueue Quotas

What This Agent Does

A GPU waste agent finds every GPU on your Kubernetes cluster that is allocated to a pod but doing nothing, proves the pod really is idle rather than between batches, and proposes the cheapest safe way to get the silicon back: shrink the request, time-slice the device, carve it into MIG slices, or move the workload behind a Kueue quota so it stops holding hardware while it waits. The finding and the proving are deterministic PromQL over DCGM and kube-state-metrics. The LLM only picks between remediations, and every remediation ships as a pull request or a Slack approval, never as a live kubectl call.

The reason this is its own agent and not a line in the generic idle-cost sweep is that GPUs break the usual rules. A CPU request is divisible and overcommittable; a nvidia.com/gpu: 1 request is a whole card, exclusively, for the life of the pod. A Jupyter notebook that loaded a model at 9 am and has been sitting in a browser tab since lunch holds an entire A100 for about four dollars an hour. On the shared ML platforms I have audited, between a third and a half of allocated GPU-hours were idle in this sense, and none of it showed up on the CPU efficiency dashboards because those pods also request almost no CPU.

Where GPU Waste Hides

Four buckets, each with a different fix, so the miner has to tell them apart.

The last row is the one that makes GPU waste political. Team A's idle notebook is Team B's Pending training job, and without a queue the only arbitration is a Slack message.

The Signals: DCGM Plus kube-state-metrics

The NVIDIA GPU Operator ships dcgm-exporter, which exposes per-GPU hardware counters and, when its Kubernetes mapping is enabled (it is by default in the operator), labels each series with the pod and namespace that holds the device. Three metrics carry almost all the information:

DCGM_FI_PROF_SM_ACTIVE     # fraction of time streaming multiprocessors ran a kernel (0-1)
DCGM_FI_DEV_FB_USED        # framebuffer (VRAM) in use, MiB
DCGM_FI_DEV_GPU_UTIL       # coarse "any kernel ran" percentage; misleading, see below

Do not rank on DCGM_FI_DEV_GPU_UTIL. It reports the share of time any kernel was running, so a pod doing tiny batch-1 decodes reads as 90% busy while the card mostly waits. DCGM_FI_PROF_SM_ACTIVE is the honest number. The cost-per-token post covers why this matters for inference serving in particular; here it is the difference between a card worth sharing and a card worth reclaiming.

One operational gotcha: when the Prometheus Operator scrapes dcgm-exporter, the exporter's own pod and namespace labels collide with the scrape target's, and Prometheus renames them to exported_pod and exported_namespace. Either set honorLabels: true on the ServiceMonitor or write your queries against the exported_ names. The queries below assume honorLabels: true.

The allocation side comes from kube-state-metrics, which reports extended resources with the slash replaced by an underscore:

# GPUs requested, per pod
kube_pod_container_resource_requests{resource="nvidia_com_gpu"}

# GPUs the node can hand out
kube_node_status_allocatable{resource="nvidia_com_gpu"}

The Miner: Four PromQL Queries, No Model

Ranking stays in code. These four queries, run over a seven-day window so a nightly training job is not mistaken for an idle one, produce the candidate list.

Bucket 1, allocated and idle. A pod that holds at least one GPU and whose SM activity has averaged under 2% for the window:

# Idle: holding a GPU, SM essentially never active over 7d
(
  avg_over_time(DCGM_FI_PROF_SM_ACTIVE[7d]) < 0.02
)
and on (namespace, pod)
(
  kube_pod_container_resource_requests{resource="nvidia_com_gpu"} > 0
)

Bucket 2, allocated and underutilized. Active, but never using enough of the card to justify exclusivity. The VRAM ceiling matters because a pod that peaks at 60 GiB cannot share an 80 GiB card no matter how low its SM activity is:

# Underutilized: busy sometimes, but small enough to share
(
  avg_over_time(DCGM_FI_PROF_SM_ACTIVE[7d]) < 0.20
  and
  max_over_time(DCGM_FI_DEV_FB_USED[7d]) < 20480      # under 20 GiB peak
)
and on (namespace, pod)
(
  kube_pod_container_resource_requests{resource="nvidia_com_gpu"} > 0
)

Bucket 3, allocatable but unallocated. Nodes paying for GPUs nobody requests:

# Per node: allocatable GPUs minus requested GPUs on that node
kube_node_status_allocatable{resource="nvidia_com_gpu"}
- on (node) group_left
  sum by (node) (
    kube_pod_container_resource_requests{resource="nvidia_com_gpu"}
    * on (namespace, pod) group_left
      (kube_pod_status_phase{phase="Running"} == 1)
  )

If this is nonzero for more than an hour on a node pool that Karpenter or the Cluster Autoscaler is supposed to manage, the usual cause is a taint the autoscaler does not understand or a minSize someone set during an incident and forgot.

Bucket 4, pending while idle exists. Count of GPU pods pending, surfaced next to bucket 1's total so the report can say "11 GPUs idle, 6 jobs waiting":

count(
  kube_pod_status_phase{phase="Pending"} == 1
  and on (namespace, pod)
  kube_pod_container_resource_requests{resource="nvidia_com_gpu"} > 0
)

Pricing the result is arithmetic. Pull the hourly node price from OpenCost if you run it, or hard-code per pool: roughly $4 per A100-class GPU-hour on demand at the big three clouds, closer to $1 for an A10G or L4, and north of $12 per H100-hour. Idle GPU-hours per week times that rate is the number that gets the agent funded.

The Decision: One Tool Call, Six Allowed Outcomes

The model receives one candidate at a time with the evidence attached, and it must answer through a single tool call whose schema enumerates every outcome the platform is willing to execute. Anything outside the enum is a parse failure and the candidate goes to a human.

{
  "name": "propose_gpu_remediation",
  "description": "Choose the remediation for one GPU-holding workload. Evidence is authoritative; do not infer utilization beyond it.",
  "input_schema": {
    "type": "object",
    "properties": {
      "action": {
        "type": "string",
        "enum": ["cull_idle_notebook", "scale_to_zero", "time_slice_node_pool",
                 "mig_partition_node_pool", "move_behind_kueue_queue", "leave_alone"]
      },
      "reason": { "type": "string", "maxLength": 400 },
      "evidence_cited": {
        "type": "array",
        "items": { "type": "string" },
        "description": "Metric names and values from the evidence block that justify the action"
      },
      "confidence": { "type": "number", "minimum": 0, "maximum": 1 }
    },
    "required": ["action", "reason", "evidence_cited", "confidence"]
  }
}

The system prompt is short and mostly prohibitions:

You are choosing a remediation for ONE Kubernetes workload that holds a GPU.
Rules:
- A pod whose owner is a Job or a training operator (PyTorchJob, RayJob, Argo
  Workflow) is NEVER culled or scaled to zero. Suggest move_behind_kueue_queue
  or leave_alone.
- time_slice_node_pool only if peak DCGM_FI_DEV_FB_USED is under 25% of the
  card and the workload is inference or notebooks, never training.
- mig_partition_node_pool only if the node's GPUs are MIG-capable (A100, H100,
  A30) and the evidence says the node pool is dedicated to small workloads.
- If the evidence is contradictory or incomplete, choose leave_alone.
Cite the exact metric values you relied on. Do not invent values.

The owner-kind rule is why the miner joins kube_pod_owner into the evidence block. A training job at 1% SM activity for six hours is usually stuck on data loading, which is a performance bug worth a ticket, not a candidate for deletion.

The Remediations, as Config

Each enum value maps to a concrete artifact the agent writes into a pull request. Here is what three of them look like.

Time-slicing for notebooks and small inference pods. One ConfigMap tells the NVIDIA device plugin to advertise each physical GPU as four schedulable devices, and a node label scopes it to one pool. Pods still request nvidia.com/gpu: 1; they just share the card with no memory isolation, which is fine for notebooks and dangerous for anything that can OOM a neighbour.

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  notebooks: |-
    version: v1
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4

# Reference the ConfigMap from the operator's ClusterPolicy, then opt nodes in
kubectl patch clusterpolicies.nvidia.com cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":""}}}}'
kubectl label node ip-10-0-41-7.ec2.internal nvidia.com/device-plugin.config=notebooks

MIG partitioning when you need hard isolation. On an A100 80GB, all-1g.10gb yields seven slices per card, each a separate schedulable resource with its own memory. The agent proposes this only for a node pool, never a single busy node, because mig-manager needs the GPUs empty to reconfigure them:

kubectl label node ip-10-0-41-7.ec2.internal nvidia.com/mig.config=all-1g.10gb --overwrite
# Pods then request the slice, not the whole card:
#   resources: { limits: { nvidia.com/mig-1g.10gb: 1 } }

Kueue for the pending-while-idle problem. A ClusterQueue with a nominal GPU quota per team and a borrowing limit means jobs queue instead of holding hardware, and an idle team's quota is lendable:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: ml-research
spec:
  namespaceSelector: {}
  cohort: gpu-pool
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: a100-spot
          resources:
            - name: "cpu"
              nominalQuota: 256
            - name: "memory"
              nominalQuota: 2Ti
            - name: "nvidia.com/gpu"
              nominalQuota: 16
              borrowingLimit: 8

The agent's PR adds the kueue.x-k8s.io/queue-name label to the Job templates in the offending namespace and leaves the quota numbers for the team to negotiate. Quota is a budget conversation, not something a model should set.

Guardrails That Are Not Optional

The agent is read-only against the cluster. It reads Prometheus through the same restricted path as the read-only PromQL MCP server, and its only write is to a Git repository. Beyond that:

• Never touch a pod directly. Culling a notebook means a PR to the JupyterHub or Kubeflow culling config, or a message to the owner, not kubectl delete. Owners found via kube_pod_owner and the namespace's team label.
• Never propose MIG or time-slicing for a node that currently runs a training job. The miner excludes those nodes before the model sees them.
• Price every proposal in GPU-hours per week and dollars, and refuse to open a PR under a threshold. A 0.3 GPU-hour saving is noise and burns reviewer goodwill.
• Every PR is gated the way the approval-gates post describes: the agent can open and update, a human merges, and a CODEOWNERS entry routes it to the platform team plus the workload's owner.
• Showback first. If GPU cost is not already attributed per namespace in your cost allocation report, do that before the agent, because a team that has never seen its GPU bill will not merge a PR that shrinks its sandbox.

What It Found on a Real Cluster

On a 48-GPU research cluster, the first weekly run reported 19 GPUs in bucket 1, all notebooks, averaging 31 idle hours each per week. The model chose cull_idle_notebook for all of them, with the PR tightening the JupyterHub idle culler from 24 hours to 4. Seven more GPUs landed in bucket 2, all small embedding and reranker services, where it proposed time-slicing at four replicas on their node pool and left training nodes alone. Bucket 4 showed six jobs pending at the same time, so the Kueue PR went to the two teams that owned them. The dollar figure in the PR description, at the cluster's blended rate, was a bit over $3,000 per week of reclaimable GPU time. Nothing was deleted by the agent; two of the three PRs merged within a day.

The things the agent gets wrong are predictable. A notebook used for an hour each morning looks idle on a seven-day average and gets flagged; the owner comments, you add a gpu-waste/keep annotation, and the miner skips it next week. A new team's training job spends its first day stuck at 0% SM activity because its dataset is downloading, and the model, correctly, picks leave_alone because the owner is a PyTorchJob. The failure mode to watch is the one every cost agent shares: nobody reads the fifth identical PR. Keep the threshold high, batch by node pool, and let the FinOps team own the cadence, the same discipline that made the Kubernetes FinOps agent survive past its first month.