devops

Kubernetes PVC Stuck in Pending: How to Debug and Fix It (WaitForFirstConsumer, ProvisioningFailed, No StorageClass)

Fix a Kubernetes PVC stuck in Pending: read the claim events, then fix the missing StorageClass, WaitForFirstConsumer wait, dead CSI driver, or PV mismatch.

November 18, 2026·11 min read·
#kubernetes#sre#devops#containers#reliability#incident-response

What Pending means for a PVC

A PersistentVolumeClaim stuck in Pending has not been bound to a PersistentVolume, and the reason is always written in the claim's events. There are only six causes: no StorageClass applies to the claim, the named StorageClass does not exist, the class is waiting for a pod (WaitForFirstConsumer, which is not an error), the CSI provisioner is missing or broken, the cloud API rejected the volume, or a pre-created PV does not match the claim. Run kubectl describe pvc, read the last event, and match it against the table below.

The mistake that costs the most time is debugging the pod. The pod says pod has unbound immediate PersistentVolumeClaims and nothing else; that message only tells you to go look at the claim. The reverse also holds: with WaitForFirstConsumer, a Pending claim is frequently a symptom of a pod that cannot schedule, and the claim is innocent.

Step 1: Read the claim's events

kubectl -n orders describe pvc data-postgres-0
Name:          data-postgres-0
Namespace:     orders
StorageClass:  gp3
Status:        Pending
Annotations:   volume.kubernetes.io/selected-node: ip-10-0-41-17.ec2.internal
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Events:
  Type     Reason                Age                 From                         Message
  Normal   WaitForFirstConsumer  9m                  persistentvolume-controller  waiting for first consumer to be created before binding
  Normal   ExternalProvisioning  2m (x26 over 8m)    persistentvolume-controller  Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Warning  ProvisioningFailed    1m (x9 over 8m)     ebs.csi.aws.com_ebs-csi-controller-7d9f...  failed to provision volume with StorageClass "gp3": rpc error: code = Internal desc = Could not create volume "pvc-6f1c2a9e": could not create volume in EC2: api error UnauthorizedOperation: You are not authorized to perform this operation.

Three fields matter: StorageClass (empty or named), the selected-node annotation (present only after the scheduler has picked a node), and the newest event. Events expire after an hour by default, so on an old claim delete the consuming pod to make the controllers emit them again.

Last event on the claimCauseGo to
no persistent volumes available for this claim and no storage class is setNo default StorageClass, or a static PV does not matchStep 2, Step 6
storageclass.storage.k8s.io "fast-ssd" not foundTypo or missing classStep 2
waiting for first consumer to be created before bindingNormal for WaitForFirstConsumer; the pod is the problemStep 3
Waiting for a volume to be created either by the external provisioner ... and nothing after itCSI controller not installed or not runningStep 4
ProvisioningFailed with an rpc errorThe cloud API said no: IAM, quota, bad parameters, KMSStep 5
No events, no claim at allResourceQuota rejected the claim at admissionStep 7

Step 2: No StorageClass, or the wrong one

kubectl get storageclass
NAME   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2    kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  212d
gp3    ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   190d

Neither line says (default) after the name, so a claim that omits storageClassName gets no class and waits forever for a hand-made PV. This is the standard state of a fresh EKS cluster: from EKS 1.30 onward the built-in gp2 class is no longer annotated as default on new clusters, and Helm charts that assume a default silently produce Pending claims. Mark one:

kubectl patch storageclass gp3 -p \
  '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

On Kubernetes 1.28 and later, existing Pending claims with no class are assigned the new default retroactively and bind without being recreated. On older versions you must delete and recreate the claim.

Two details catch people here:

  • storageClassName: "" is not the same as omitting the field. An empty string means "never use dynamic provisioning, bind only to a PV that also has no class". Charts that render storageClassName: "" from an empty value will never pick up your default. Omit the key instead.
  • A wrong name cannot be patched. spec.storageClassName is immutable once set. Delete the claim and recreate it. For a StatefulSet, fix volumeClaimTemplates (which requires deleting the StatefulSet with --cascade=orphan and reapplying), then delete the Pending claim and its pod so the controller recreates both. A StatefulSet never updates or deletes claims it already created, which is why a fixed manifest keeps producing the same broken claim.

Step 3: WaitForFirstConsumer is waiting for a pod that can schedule

With volumeBindingMode: WaitForFirstConsumer, the claim stays Pending on purpose until a pod that uses it has been assigned a node. That way the volume is created in the node's zone instead of a random one. If no pod references the claim, Pending is the correct, permanent state, and there is nothing to fix.

If a pod does reference it and the claim has no selected-node annotation, the scheduler has not been able to place the pod. The claim is a bystander:

kubectl -n orders get pods -o json | jq -r '.items[]
  | select(.spec.volumes[]?.persistentVolumeClaim.claimName == "data-postgres-0")
  | "\(.metadata.name)\t\(.status.phase)\t\(.spec.nodeName // "unscheduled")"'
kubectl -n orders describe pod postgres-0 | sed -n '/Events/,$p'

Insufficient CPU, an untolerated taint, or a node selector that matches nothing will all hold the claim in Pending. Work through Kubernetes pod stuck in Pending and the claim binds seconds after the pod schedules.

The one genuinely stuck case is a stale annotation. The scheduler picked a node, wrote volume.kubernetes.io/selected-node, and then the node was removed before the volume was created. That is routine on clusters where Karpenter or a spot reclaim replaces nodes quickly (see EKS spot instances with Karpenter). Check whether the node still exists:

NODE=$(kubectl -n orders get pvc data-postgres-0 \
  -o jsonpath='{.metadata.annotations.volume\.kubernetes\.io/selected-node}')
kubectl get node "$NODE"

Current external-provisioner releases detect the missing node and clear the annotation so the scheduler tries again. If yours does not, and the claim has sat for minutes pointing at a node that returns NotFound, delete the pod and remove the annotation yourself. It is safe only while the claim is still unbound:

kubectl -n orders annotate pvc data-postgres-0 volume.kubernetes.io/selected-node-

Do not "fix" this by switching the class to Immediate. That creates the volume in an arbitrary zone, and you trade a Pending claim for a pod that can never schedule because of volume node affinity conflict.

Step 4: The provisioner is not running

The ExternalProvisioning event repeating with nothing after it means the controller handed the job to a CSI driver and nobody picked it up. The provisioner named in the StorageClass must exist as a running controller:

kubectl get csidrivers
kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-ebs-csi-driver

If ebs.csi.aws.com is missing from the first list, the driver was never installed. EKS does not ship it: the in-tree kubernetes.io/aws-ebs provisioner is redirected to the CSI driver by CSI migration, so even the legacy gp2 class needs the add-on.

aws eks describe-addon --cluster-name prod --addon-name aws-ebs-csi-driver \
  --query 'addon.{status:status,role:serviceAccountRoleArn,issues:health.issues}'

If the controller pods exist but are in CrashLoopBackOff or Pending themselves, fix them first. A frequent self-inflicted version: the controller Deployment has no toleration for the taint on your only node pool, so the thing that provisions storage cannot schedule.

Step 5: ProvisioningFailed means the cloud API refused

Now there is a real error, but the event truncates it. Get the full text from the provisioner sidecar and the driver:

kubectl -n kube-system logs deploy/ebs-csi-controller -c csi-provisioner --tail=30
kubectl -n kube-system logs deploy/ebs-csi-controller -c ebs-plugin --tail=30
Error fragmentMeaningFix
UnauthorizedOperation on CreateVolumeController role lacks EBS permissionsAttach AmazonEBSCSIDriverPolicy to the role bound to the ebs-csi-controller-sa ServiceAccount
AccessDenied ... sts:AssumeRoleWithWebIdentityIRSA trust policy or OIDC provider is wrongFix the trust policy's sub condition to system:serviceaccount:kube-system:ebs-csi-controller-sa
VolumeLimitExceeded / MaxIOPSLimitExceededRegional EBS quota hitRequest a Service Quotas increase, or reclaim unattached volumes
InvalidParameterValue / InvalidParameterCombinationStorageClass parameters invalid for the typeFor example iops set on gp2, or io2 IOPS above 1000 per GiB for the requested size
KMS ... AccessDenied or volume created then instantly deletedDriver role cannot use the kmsKeyId in the classAdd kms:CreateGrant, kms:GenerateDataKeyWithoutPlaintext, and kms:Decrypt on the key
InsufficientVolumeCapacityZone is out of that volume typeUsually transient; the provisioner retries with backoff

The provisioner retries with exponential backoff, so after you fix IAM the claim may sit for a few more minutes. Deleting the consuming pod forces a fresh attempt immediately. StorageClass parameters are immutable, so a bad parameter means deleting and recreating the class. Existing bound volumes are unaffected by that.

Quota exhaustion is often really waste: hundreds of unattached volumes from deleted claims with reclaimPolicy: Retain. The cleanup approach is in the AWS waste reclamation agent post.

Step 6: A static PV exists but does not match

For pre-created volumes (NFS, local disks with kubernetes.io/no-provisioner, restored EBS volumes), binding requires every one of these to line up:

  • PV capacity is greater than or equal to the claim's request
  • PV accessModes include every mode the claim asks for
  • storageClassName is identical on both, including both being empty
  • volumeMode matches (Filesystem versus Block)
  • The claim's selector, if any, matches the PV's labels
  • The PV's status.phase is Available
kubectl get pv -o custom-columns=NAME:.metadata.name,CAP:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName,STATUS:.status.phase,CLAIM:.spec.claimRef.name

The last condition is the usual culprit. A PV whose previous claim was deleted under Retain sits in Released, still holding a claimRef with the old claim's UID. A new claim with the same name has a new UID and will never bind to it. Once you have confirmed the data on it is what you want, clear the reference and it returns to Available:

kubectl patch pv pv-orders-restore --type=json \
  -p '[{"op":"remove","path":"/spec/claimRef"}]'

To guarantee the right claim gets it instead of whichever matching claim arrives first, set spec.volumeName on the claim to the PV's name.

Step 7: The claim does not exist at all

When a StatefulSet pod is Pending and kubectl get pvc shows no claim for it, a ResourceQuota rejected the claim at creation. Nothing is Pending because nothing was admitted. The evidence is on the StatefulSet, not the pod:

kubectl -n orders describe statefulset postgres | sed -n '/Events/,$p'
Warning  FailedCreate  statefulset-controller  create Claim data-postgres-2 for Pod postgres-2 in StatefulSet postgres failed error: persistentvolumeclaims "data-postgres-2" is forbidden: exceeded quota: storage, requested: requests.storage=100Gi, used: requests.storage=450Gi, limited: requests.storage=500Gi

Raise the quota or shrink what is already claimed. Requested storage is rarely what is actually used; the PVC right-sizing post shows how to find claims sitting at 8% utilization before you ask for more quota.

Prevention

  • Exactly one default StorageClass, with WaitForFirstConsumer. Assert it in your cluster bootstrap. With zero defaults, claims hang; with two, Kubernetes 1.26 and later silently uses the newest one.
  • Install the CSI driver as a managed add-on with its role in the same Terraform module as the cluster, so the driver and its IAM permissions cannot drift apart.
  • Alert on the state. kube-state-metrics exposes the phase:
- alert: PersistentVolumeClaimPending
  expr: kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} Pending for 15 minutes"

Expect this to fire for claims nobody consumes under WaitForFirstConsumer. That is useful signal too: an unconsumed claim is leftover config. To forecast quota and volume exhaustion before it blocks a rollout, see the capacity planning agent.

  • Know where Pending ends. Once the claim is Bound and the pod still will not start, the problem has moved to attach and mount, which is a different set of causes covered in FailedMount and FailedAttachVolume.

Quick reference

# 1. The answer is in the last event
kubectl -n NS describe pvc PVC
# 2. Is there a default class? Does the named class exist?
kubectl get storageclass
# 3. WaitForFirstConsumer: is the consuming pod scheduled? Is selected-node stale?
kubectl -n NS get pvc PVC -o jsonpath='{.metadata.annotations}'
# 4. Is the provisioner installed and running?
kubectl get csidrivers
kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-ebs-csi-driver
# 5. Full cloud error behind ProvisioningFailed
kubectl -n kube-system logs deploy/ebs-csi-controller -c csi-provisioner --tail=30
# 6. Static PVs: phase must be Available, class/size/modes must match
kubectl get pv
# 7. No claim at all: quota rejection is on the StatefulSet
kubectl -n NS describe statefulset NAME

A Pending claim is never silent. Kubernetes records which of the six causes it is on the claim itself; the only way to lose an hour is to look at the pod instead.

#kubernetes#sre#devops#containers#reliability#incident-response
D
DevToCashAuthor

Senior DevOps/SRE Engineer · 10+ years · Professional Trader (IDX, Crypto, US Equities)

I write about real infrastructure patterns and trading strategies I use in production and in live markets. No courses, no affiliate hype — just documentation of what actually works.

More about me →