Kubernetes Pod Stuck in Pending (FailedScheduling): How to Fix It

What Pending actually means

A pod stuck in Pending has been accepted by the API server but has no node to run on. The kube-scheduler looked at every node in the cluster, none satisfied the pod's constraints, and it gave up for now — recording exactly why in a FailedScheduling event. Unlike a crash, nothing is wrong with your container yet. The image is never pulled, the process never starts. The problem is entirely about placement.

This is a different failure from the "my container won't stay up" family — ImagePullBackOff, CrashLoopBackOff, and OOMKilled all mean a node accepted the pod and then something broke. Pending means no node would take it in the first place. The good news: like the others, the scheduler tells you the exact reason. You never have to guess. This is the sequence I run to place a stuck pod, usually in a couple of minutes.

Step 1: Read the scheduler's verdict

Don't theorize from a Pending status. Read the event the scheduler already wrote:

kubectl describe pod payments-api-7d9f4c8b6-xk2mn

Scroll to Events at the bottom:

  Warning  FailedScheduling  default-scheduler
    0/5 nodes are available: 2 Insufficient cpu,
    2 node(s) had untolerated taint {dedicated: gpu},
    1 node(s) didn't match Pod's node affinity/selector.

That single line accounts for every node in the cluster and why each one rejected the pod. The scheduler evaluates nodes through filter predicates, and the message is the tally of which predicate failed where. Map the phrase to a root cause:

Read this first and everything below collapses to one path. Often you'll see several fragments at once — fix them in order of how many nodes each blocks.

Step 2: Insufficient CPU or memory

The most common cause. Insufficient cpu doesn't mean the node is using all its CPU — it means the sum of pod requests already scheduled there leaves less than your pod asks for. Scheduling is based on requests, not live usage. A node at 10% actual CPU can still reject a pod if its requests are already reserved.

First, see what the pod is asking for:

kubectl get pod payments-api-7d9f4c8b6-xk2mn \
  -o jsonpath='{.spec.containers[0].resources.requests}'

{"cpu":"2","memory":"4Gi"}

Then see what's actually free on the nodes:

kubectl describe node ip-10-0-1-42 | grep -A6 "Allocated resources"

  Resource   Requests      Limits
  cpu        3500m (87%)   6 (150%)
  memory     6Gi (78%)     10Gi

If free CPU is below your request on every node, you have two fixes:

• The request is oversized. A pod asking for 2 full cores that actually uses 150m is starving the scheduler of placements it could otherwise make. Right-size the request to real usage — the same discipline that keeps the Kubernetes bill down. The VPA in recommendation mode will suggest values from observed usage; see how it fits alongside HPA and KEDA in the pod autoscaling guide.
• The cluster is genuinely full. Every node is legitimately packed. You need more capacity — enable the cluster autoscaler (or Karpenter) so a Pending pod that can't fit triggers a new node instead of waiting forever. Confirm it's reacting:

kubectl -n kube-system logs -l app=cluster-autoscaler --tail=50 | grep -i scale

If the autoscaler logs pod didn't trigger scale-up: max node group size reached, you've hit your node-group ceiling — raise the max, or the pod stays Pending indefinitely.

Step 3: Untolerated taints

A message like node(s) had untolerated taint {node.kubernetes.io/unschedulable} means the target nodes are tainted and your pod carries no matching toleration. Taints repel pods unless the pod explicitly tolerates them. See the taint:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Two very different situations:

• The node is cordoned. node.kubernetes.io/unschedulable appears after kubectl cordon (or a drain during maintenance). If the cordon was intentional, that's expected — the pod schedules once you kubectl uncordon the node or new capacity arrives. A pod stuck Pending during a node upgrade is usually this.
• The node is deliberately reserved — GPU nodes, a dedicated=gpu:NoSchedule taint, spot-instance pools. If your pod should run there, add a toleration:

spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "gpu"
      effect: "NoSchedule"

A toleration lets a pod land on a tainted node; it does not force it there. Pair it with a nodeSelector or affinity (Step 4) when you want the pod to actually target that pool.

Step 4: Node affinity / selector mismatch

didn't match Pod's node affinity/selector means your nodeSelector or nodeAffinity names a label that no available node carries. Usually a typo, a decommissioned node pool, or a label that was never applied. Check what the pod demands:

kubectl get pod payments-api-7d9f4c8b6-xk2mn -o jsonpath='{.spec.nodeSelector}'

{"disktype":"ssd","topology.kubernetes.io/zone":"us-east-1a"}

Then check which nodes actually have those labels:

kubectl get nodes -l disktype=ssd

If that returns nothing, no node matches. Either the label is wrong on the pod, or it was never set on the nodes:

kubectl label node ip-10-0-1-42 disktype=ssd

A subtle trap: pinning a pod to a single zone with topology.kubernetes.io/zone means it can only schedule in that zone. If that zone is full and your autoscaler scales a different zone, the pod stays Pending forever. Prefer preferredDuringScheduling over requiredDuringScheduling unless the constraint is truly hard — a preference degrades gracefully instead of deadlocking.

Step 5: Unbound volumes and zone conflicts

pod has unbound immediate PersistentVolumeClaims or volume node affinity conflict means storage is blocking placement. Storage and scheduling are coupled: a pod can only run where its volume can attach. Check the PVC:

kubectl get pvc

NAME            STATUS    VOLUME   CAPACITY   STORAGECLASS
data-payments   Pending                       gp3

A Pending PVC means no PersistentVolume satisfied the claim. Common causes:

• No matching StorageClass or no dynamic provisioner. kubectl get storageclass — if there's no default class and the PVC names none, nothing provisions the volume. Set a default or name a valid class.
• volume node affinity conflict — the PV already exists in us-east-1a, but the only node with free capacity is in us-east-1b, and an EBS volume can't cross zones. Fix by setting volumeBindingMode: WaitForFirstConsumer on the StorageClass so the volume is provisioned after the scheduler picks the node, in the same zone — instead of binding first and pinning the pod to the wrong zone.

Step 6: Node is out of pod slots (too many pods)

too many pods means the node hit its max-pods limit even though it has spare CPU and memory. On AWS EKS with the VPC CNI, the ceiling is often driven by how many IP addresses the instance's ENIs can hold — a small instance type may cap at 8–17 pods regardless of how much RAM it has. Check the node's capacity and current count:

kubectl get node ip-10-0-1-42 -o jsonpath='{.status.capacity.pods}'
kubectl get pods --all-namespaces --field-selector spec.nodeName=ip-10-0-1-42 \
  --no-headers | wc -l

If the count equals capacity, the node is full of pods, not resources. Fixes: use larger instance types (more ENIs = more IPs = more pods), enable prefix delegation on the VPC CNI to pack far more IPs per node, or add nodes. This is an easy one to miss because kubectl top node shows the node as nearly idle.

Step 7: Topology spread and anti-affinity deadlocks

didn't match pod topology spread constraints or an anti-affinity rejection means your own placement rules can't be satisfied. A classic self-inflicted deadlock: a podAntiAffinity with requiredDuringScheduling that says "no two replicas on the same node" while you have 5 replicas and only 3 nodes. The 4th and 5th can never place.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:   # hard rule — deadlocks
      - topologyKey: kubernetes.io/hostname

Switch required to preferred so replicas spread when possible but still schedule when they can't, or add nodes so the hard rule is satisfiable. The same reasoning applies to topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule — that's a hard gate; ScheduleAnyway degrades gracefully.

A repeatable checklist

When a pod is stuck Pending, run this in order:

1. kubectl describe pod <pod> → read the FailedScheduling event; note every N node(s) ... fragment.
2. Insufficient cpu/memory → right-size requests or enable the cluster autoscaler; check it isn't at max node-group size (Step 2).
3. untolerated taint → uncordon the node, or add the matching toleration (Step 3).
4. didn't match node affinity/selector → fix the label on the pod or apply it to nodes; avoid hard zone pins (Step 4).
5. unbound PVC / volume node affinity conflict → fix the StorageClass; set WaitForFirstConsumer (Step 5).
6. too many pods → node hit its IP/pod cap; bigger instances, prefix delegation, or more nodes (Step 6).
7. Topology/anti-affinity rejection → relax required/DoNotSchedule to preferred/ScheduleAnyway (Step 7).

Don't let Pending pods hide

A pod that can't schedule is invisible to a health check that only watches running pods — it never crashes, it just quietly never runs. That's how a scaled-up deployment silently serves at half capacity. Alert on pods stuck Pending past a threshold:

kube_pod_status_phase{phase="Pending"} == 1
  and on(pod, namespace) (time() - kube_pod_created) > 300

Wire that into the Prometheus + Grafana monitoring setup so a five-minute-stuck pod pages before a rollout stalls, and bake the seven-step triage into your incident runbook. A cluster that's chronically full and can't place pods is one of the quiet Kubernetes mistakes that cost companies millions — cheap to catch, expensive to discover during a traffic spike.

Related Reading

• Kubernetes OOMKilled (Exit Code 137): How to Debug and Fix It — the mirror image of oversized requests: too-low limits get a running pod killed, while too-high requests keep it from scheduling at all. Right-sizing fixes both.
• Kubernetes Pod Autoscaling: HPA, VPA, and KEDA Explained — the VPA recommends the requests that decide whether a pod fits a node, and the cluster autoscaler is what turns a Pending pod into a new node.
• Kubernetes ImagePullBackOff: How to Debug and Fix It — once the scheduler does place your pod, the pull is the next thing that can fail; together with CrashLoopBackOff and OOMKilled these cover nearly every reason a pod fails to come up.