Kubernetes Pod Evicted: How to Debug and Fix It

What Evicted actually means

A pod with status Evicted was killed by the kubelet because its node ran short of a resource — memory, disk, ephemeral storage, inodes, or PIDs. This is node-pressure eviction: the kubelet notices a resource crossing a threshold, picks victim pods, kills them, and leaves the pod object behind in phase Failed with reason Evicted so you can read why. If the pod belongs to a Deployment or StatefulSet, its controller immediately creates a replacement — which is why you often find dozens of Evicted corpses next to a perfectly healthy app.

The fix is always the same three moves: read the eviction message (it names the resource), fix the pod or the node depending on who's actually at fault, and clean up the leftovers. Unlike CrashLoopBackOff, the app did nothing wrong at the process level — the scheduler's math and the node's reality disagreed, and the kubelet resolved the argument by killing things.

One distinction before debugging: kubectl drain and the Eviction API also "evict" pods, but those are polite, respect PodDisruptionBudgets, and produce normal terminations. Node-pressure eviction is the impolite kind — it ignores PDBs completely. If your PDB "guaranteed" two replicas and you still lost them, this is how.

Step 1: Read the eviction message

The pod object tells you exactly which resource ran out. Don't skip this — the fix for ephemeral-storage is completely different from the fix for memory.

kubectl get pod api-6f7d8b9c4-tk2pw -o jsonpath='{.status.reason}: {.status.message}'

Evicted: The node was low on resource: ephemeral-storage. Threshold quantity: 10120387530, available: 8945672192. Container api was using 24717586432, which exceeds its request of 0.

If the pod is already garbage-collected, the same text lives in events:

kubectl get events -A --field-selector reason=Evicted \
  --sort-by=.lastTimestamp | tail -20

Three things in that message matter:

The resource: ephemeral-storage here. Could be memory, nodefs, imagefs, inodes, or pids.
Which container was using how much — the kubelet names the biggest offenders.
"exceeds its request of 0" — this pod declared no request for the resource it was devouring. That's why it was chosen, and it's the root cause more often than the node is.

Step 2: Understand why this pod was picked

When a threshold trips, the kubelet ranks every pod on the node and kills from the top:

Pods whose usage of the starved resource exceeds their request die first — and a pod with no request at all "exceeds" it at any usage. This is the ranking that kills you.
Among those, lower priorityClassName dies before higher.
Then whoever is furthest over its request.

Guaranteed-QoS pods (requests = limits for every container) never exceed their memory request by definition, so they're effectively last in line. BestEffort pods — no requests, no limits — are always first. If an "important" pod keeps getting evicted, the fix is almost never "get bigger nodes"; it's give the pod honest requests so it stops ranking as a freeloader:

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
    ephemeral-storage: "2Gi"
  limits:
    memory: "512Mi"
    ephemeral-storage: "4Gi"

Note ephemeral-storage gets requests and limits like memory does. Almost nobody sets it, which is exactly why ephemeral-storage evictions hit random victims.

Step 3: Fix by resource

Ephemeral-storage (the most common one)

Ephemeral storage is everything a pod writes outside a PersistentVolume: the container's writable layer, emptyDir volumes (disk-backed), and container logs. The classic offenders:

An app writing debug logs, temp files, or cache into the container filesystem.
An unbounded emptyDir used as a scratch dir for downloads or builds.
Log output so chatty the node-side log files balloon.

Confirm on the pod's node — since v1.21 you can query the kubelet's own accounting without SSH:

kubectl get --raw "/api/v1/nodes/ip-10-0-1-42/proxy/stats/summary" \
  | jq '.pods[] | {name: .podRef.name, disk: .["ephemeral-storage"].usedBytes}' \
  | jq -s 'sort_by(.disk) | reverse | .[0:5]'

Fixes, in order of correctness:

Stop writing it. Ship logs to stdout and let the runtime handle rotation; put real data on a PVC.
Cap emptyDir:

volumes:
  - name: scratch
    emptyDir:
      sizeLimit: 1Gi

A pod that fills a capped emptyDir is evicted individually for exceeding its own limit — annoying, but it no longer takes neighbors down with it.

Declare ephemeral-storage requests so the scheduler stops packing five disk-hungry pods onto one 20 GB node.

Memory

A memory eviction is the kubelet acting before the kernel OOM killer does: memory.available on the node fell under the threshold (default hard threshold: memory.available<100Mi), so the kubelet killed a pod to reclaim headroom. It's the softer sibling of OOMKilled / exit code 137 — same underlying problem, different executioner, and the same fix: set memory requests that reflect real usage, and limits equal to requests for anything you can't afford to lose. If evictions happen while pods are all within requests, the node itself is overcommitted — check that kube-reserved and system-reserved are set so the kubelet, container runtime, and OS have carved-out headroom the scheduler can't hand out.

Disk and inodes (nodefs / imagefs)

nodefs is the node's main filesystem; imagefs is where images and container layers live. Defaults trip at nodefs.available<10%, imagefs.available<15%, nodefs.inodesFree<5%. Before killing pods, the kubelet tries to reclaim by garbage-collecting dead containers and unused images — if you're still seeing evictions, that reclaim wasn't enough. On the node:

df -h /var/lib/kubelet /var/lib/containerd
df -i /var/lib/containerd          # inodes — millions of tiny files exhaust these first
crictl imagefsinfo

Frequent causes: enormous images accumulating across many deploys (every tag is a full copy on disk), a runaway pod writing into /var/log, or node disks simply sized for a workload profile from two years ago. Cheap wins: smaller images via multi-stage builds, fewer distinct tags per node (image locality), bigger root volumes on the node group.

Step 4: Check whether the node is the real patient

If evictions cluster on one node, describe it:

kubectl describe node ip-10-0-1-42 | grep -A8 Conditions:

  MemoryPressure   False
  DiskPressure     True    KubeletHasDiskPressure
  PIDPressure      False

A node under DiskPressure or MemoryPressure gets a matching taint (node.kubernetes.io/disk-pressure:NoSchedule), so new pods stop landing there — which is also why a wave of evictions is often followed by pods stuck in Pending with FailedScheduling: the evicted pods' replacements can't schedule anywhere because every node is tainted. That combination means cluster capacity, not a single bad pod.

Persistent pressure across the fleet is a capacity/rightsizing problem — solve it with honest requests and autoscaling rather than whack-a-mole, the same discipline that drives Kubernetes cost optimization from the other direction: requests that match reality keep nodes neither starved nor half-empty.

Step 5: Clean up the corpses

Evicted pods linger in Failed phase until the pod garbage collector trims them (only once the cluster passes terminated-pod-gc-threshold, default 12,500 — effectively never on small clusters). They're harmless but they pollute every kubectl get pods and hide real failures. Delete them:

kubectl delete pods -A --field-selector=status.phase=Failed

That selector also catches non-evicted failed pods (e.g. dead Jobs). To delete only evictions:

kubectl get pods -A -o json \
  | jq -r '.items[] | select(.status.reason=="Evicted")
      | .metadata.namespace + " " + .metadata.name' \
  | while read ns name; do kubectl delete pod -n "$ns" "$name"; done

Do read a couple of messages before bulk-deleting — the corpses are your only record of which resource and which container, and the pattern across ten of them is the diagnosis.

Alert before the kubelet acts

Two signals, both from defaults you likely already scrape in a Prometheus + Grafana setup:

# Evictions happening now (kube-state-metrics)
sum by (namespace) (kube_pod_status_reason{reason="Evicted"}) > 0

# Node ephemeral filesystem heading for the 10% threshold (node-exporter)
node_filesystem_avail_bytes{mountpoint="/"} 
  / node_filesystem_size_bytes{mountpoint="/"} < 0.15

The second one is the one that saves you: disk pressure builds over hours, and an alert at 15% free gives you time to act before the kubelet starts choosing victims at 10%.

A repeatable checklist

kubectl get pod <pod> -o jsonpath='{.status.message}' (or events) → which resource, which container, and did it exceed its request? (Step 1)
Pod had no request for the starved resource → add honest requests (and ephemeral-storage ones — Step 2).
Resource-specific fix: cap emptyDir with sizeLimit, move data to PVCs, shrink images, stop disk-chatty logging (Step 3).
kubectl describe node → pressure condition on one node vs the fleet? One node: clean it. Fleet: capacity and reserved-resource config (Step 4).
Delete Failed pods after reading a few messages; alert on kube_pod_status_reason{reason="Evicted"} so next time you're first to know (Step 5).