What Pending means for a PVC
A PersistentVolumeClaim stuck in Pending has not been bound to a PersistentVolume, and the reason is always written in the claim's events. There are only six causes: no StorageClass applies to the claim, the named StorageClass does not exist, the class is waiting for a pod (WaitForFirstConsumer, which is not an error), the CSI provisioner is missing or broken, the cloud API rejected the volume, or a pre-created PV does not match the claim. Run kubectl describe pvc, read the last event, and match it against the table below.
The mistake that costs the most time is debugging the pod. The pod says pod has unbound immediate PersistentVolumeClaims and nothing else; that message only tells you to go look at the claim. The reverse also holds: with WaitForFirstConsumer, a Pending claim is frequently a symptom of a pod that cannot schedule, and the claim is innocent.
Step 1: Read the claim's events
kubectl -n orders describe pvc data-postgres-0
Name: data-postgres-0
Namespace: orders
StorageClass: gp3
Status: Pending
Annotations: volume.kubernetes.io/selected-node: ip-10-0-41-17.ec2.internal
volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Events:
Type Reason Age From Message
Normal WaitForFirstConsumer 9m persistentvolume-controller waiting for first consumer to be created before binding
Normal ExternalProvisioning 2m (x26 over 8m) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Warning ProvisioningFailed 1m (x9 over 8m) ebs.csi.aws.com_ebs-csi-controller-7d9f... failed to provision volume with StorageClass "gp3": rpc error: code = Internal desc = Could not create volume "pvc-6f1c2a9e": could not create volume in EC2: api error UnauthorizedOperation: You are not authorized to perform this operation.
Three fields matter: StorageClass (empty or named), the selected-node annotation (present only after the scheduler has picked a node), and the newest event. Events expire after an hour by default, so on an old claim delete the consuming pod to make the controllers emit them again.
| Last event on the claim | Cause | Go to |
|---|---|---|
no persistent volumes available for this claim and no storage class is set | No default StorageClass, or a static PV does not match | Step 2, Step 6 |
storageclass.storage.k8s.io "fast-ssd" not found | Typo or missing class | Step 2 |
waiting for first consumer to be created before binding | Normal for WaitForFirstConsumer; the pod is the problem | Step 3 |
Waiting for a volume to be created either by the external provisioner ... and nothing after it | CSI controller not installed or not running | Step 4 |
ProvisioningFailed with an rpc error | The cloud API said no: IAM, quota, bad parameters, KMS | Step 5 |
| No events, no claim at all | ResourceQuota rejected the claim at admission | Step 7 |
Step 2: No StorageClass, or the wrong one
kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 212d
gp3 ebs.csi.aws.com Delete WaitForFirstConsumer true 190d
Neither line says (default) after the name, so a claim that omits storageClassName gets no class and waits forever for a hand-made PV. This is the standard state of a fresh EKS cluster: from EKS 1.30 onward the built-in gp2 class is no longer annotated as default on new clusters, and Helm charts that assume a default silently produce Pending claims. Mark one:
kubectl patch storageclass gp3 -p \
'{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
On Kubernetes 1.28 and later, existing Pending claims with no class are assigned the new default retroactively and bind without being recreated. On older versions you must delete and recreate the claim.
Two details catch people here:
storageClassName: ""is not the same as omitting the field. An empty string means "never use dynamic provisioning, bind only to a PV that also has no class". Charts that renderstorageClassName: ""from an empty value will never pick up your default. Omit the key instead.- A wrong name cannot be patched.
spec.storageClassNameis immutable once set. Delete the claim and recreate it. For a StatefulSet, fixvolumeClaimTemplates(which requires deleting the StatefulSet with--cascade=orphanand reapplying), then delete the Pending claim and its pod so the controller recreates both. A StatefulSet never updates or deletes claims it already created, which is why a fixed manifest keeps producing the same broken claim.
Step 3: WaitForFirstConsumer is waiting for a pod that can schedule
With volumeBindingMode: WaitForFirstConsumer, the claim stays Pending on purpose until a pod that uses it has been assigned a node. That way the volume is created in the node's zone instead of a random one. If no pod references the claim, Pending is the correct, permanent state, and there is nothing to fix.
If a pod does reference it and the claim has no selected-node annotation, the scheduler has not been able to place the pod. The claim is a bystander:
kubectl -n orders get pods -o json | jq -r '.items[]
| select(.spec.volumes[]?.persistentVolumeClaim.claimName == "data-postgres-0")
| "\(.metadata.name)\t\(.status.phase)\t\(.spec.nodeName // "unscheduled")"'
kubectl -n orders describe pod postgres-0 | sed -n '/Events/,$p'
Insufficient CPU, an untolerated taint, or a node selector that matches nothing will all hold the claim in Pending. Work through Kubernetes pod stuck in Pending and the claim binds seconds after the pod schedules.
The one genuinely stuck case is a stale annotation. The scheduler picked a node, wrote volume.kubernetes.io/selected-node, and then the node was removed before the volume was created. That is routine on clusters where Karpenter or a spot reclaim replaces nodes quickly (see EKS spot instances with Karpenter). Check whether the node still exists:
NODE=$(kubectl -n orders get pvc data-postgres-0 \
-o jsonpath='{.metadata.annotations.volume\.kubernetes\.io/selected-node}')
kubectl get node "$NODE"
Current external-provisioner releases detect the missing node and clear the annotation so the scheduler tries again. If yours does not, and the claim has sat for minutes pointing at a node that returns NotFound, delete the pod and remove the annotation yourself. It is safe only while the claim is still unbound:
kubectl -n orders annotate pvc data-postgres-0 volume.kubernetes.io/selected-node-
Do not "fix" this by switching the class to Immediate. That creates the volume in an arbitrary zone, and you trade a Pending claim for a pod that can never schedule because of volume node affinity conflict.
Step 4: The provisioner is not running
The ExternalProvisioning event repeating with nothing after it means the controller handed the job to a CSI driver and nobody picked it up. The provisioner named in the StorageClass must exist as a running controller:
kubectl get csidrivers
kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-ebs-csi-driver
If ebs.csi.aws.com is missing from the first list, the driver was never installed. EKS does not ship it: the in-tree kubernetes.io/aws-ebs provisioner is redirected to the CSI driver by CSI migration, so even the legacy gp2 class needs the add-on.
aws eks describe-addon --cluster-name prod --addon-name aws-ebs-csi-driver \
--query 'addon.{status:status,role:serviceAccountRoleArn,issues:health.issues}'
If the controller pods exist but are in CrashLoopBackOff or Pending themselves, fix them first. A frequent self-inflicted version: the controller Deployment has no toleration for the taint on your only node pool, so the thing that provisions storage cannot schedule.
Step 5: ProvisioningFailed means the cloud API refused
Now there is a real error, but the event truncates it. Get the full text from the provisioner sidecar and the driver:
kubectl -n kube-system logs deploy/ebs-csi-controller -c csi-provisioner --tail=30
kubectl -n kube-system logs deploy/ebs-csi-controller -c ebs-plugin --tail=30
| Error fragment | Meaning | Fix |
|---|---|---|
UnauthorizedOperation on CreateVolume | Controller role lacks EBS permissions | Attach AmazonEBSCSIDriverPolicy to the role bound to the ebs-csi-controller-sa ServiceAccount |
AccessDenied ... sts:AssumeRoleWithWebIdentity | IRSA trust policy or OIDC provider is wrong | Fix the trust policy's sub condition to system:serviceaccount:kube-system:ebs-csi-controller-sa |
VolumeLimitExceeded / MaxIOPSLimitExceeded | Regional EBS quota hit | Request a Service Quotas increase, or reclaim unattached volumes |
InvalidParameterValue / InvalidParameterCombination | StorageClass parameters invalid for the type | For example iops set on gp2, or io2 IOPS above 1000 per GiB for the requested size |
KMS ... AccessDenied or volume created then instantly deleted | Driver role cannot use the kmsKeyId in the class | Add kms:CreateGrant, kms:GenerateDataKeyWithoutPlaintext, and kms:Decrypt on the key |
InsufficientVolumeCapacity | Zone is out of that volume type | Usually transient; the provisioner retries with backoff |
The provisioner retries with exponential backoff, so after you fix IAM the claim may sit for a few more minutes. Deleting the consuming pod forces a fresh attempt immediately. StorageClass parameters are immutable, so a bad parameter means deleting and recreating the class. Existing bound volumes are unaffected by that.
Quota exhaustion is often really waste: hundreds of unattached volumes from deleted claims with reclaimPolicy: Retain. The cleanup approach is in the AWS waste reclamation agent post.
Step 6: A static PV exists but does not match
For pre-created volumes (NFS, local disks with kubernetes.io/no-provisioner, restored EBS volumes), binding requires every one of these to line up:
- PV
capacityis greater than or equal to the claim's request - PV
accessModesinclude every mode the claim asks for storageClassNameis identical on both, including both being emptyvolumeModematches (FilesystemversusBlock)- The claim's
selector, if any, matches the PV's labels - The PV's
status.phaseisAvailable
kubectl get pv -o custom-columns=NAME:.metadata.name,CAP:.spec.capacity.storage,MODES:.spec.accessModes,CLASS:.spec.storageClassName,STATUS:.status.phase,CLAIM:.spec.claimRef.name
The last condition is the usual culprit. A PV whose previous claim was deleted under Retain sits in Released, still holding a claimRef with the old claim's UID. A new claim with the same name has a new UID and will never bind to it. Once you have confirmed the data on it is what you want, clear the reference and it returns to Available:
kubectl patch pv pv-orders-restore --type=json \
-p '[{"op":"remove","path":"/spec/claimRef"}]'
To guarantee the right claim gets it instead of whichever matching claim arrives first, set spec.volumeName on the claim to the PV's name.
Step 7: The claim does not exist at all
When a StatefulSet pod is Pending and kubectl get pvc shows no claim for it, a ResourceQuota rejected the claim at creation. Nothing is Pending because nothing was admitted. The evidence is on the StatefulSet, not the pod:
kubectl -n orders describe statefulset postgres | sed -n '/Events/,$p'
Warning FailedCreate statefulset-controller create Claim data-postgres-2 for Pod postgres-2 in StatefulSet postgres failed error: persistentvolumeclaims "data-postgres-2" is forbidden: exceeded quota: storage, requested: requests.storage=100Gi, used: requests.storage=450Gi, limited: requests.storage=500Gi
Raise the quota or shrink what is already claimed. Requested storage is rarely what is actually used; the PVC right-sizing post shows how to find claims sitting at 8% utilization before you ask for more quota.
Prevention
- Exactly one default StorageClass, with
WaitForFirstConsumer. Assert it in your cluster bootstrap. With zero defaults, claims hang; with two, Kubernetes 1.26 and later silently uses the newest one. - Install the CSI driver as a managed add-on with its role in the same Terraform module as the cluster, so the driver and its IAM permissions cannot drift apart.
- Alert on the state. kube-state-metrics exposes the phase:
- alert: PersistentVolumeClaimPending
expr: kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} Pending for 15 minutes"
Expect this to fire for claims nobody consumes under WaitForFirstConsumer. That is useful signal too: an unconsumed claim is leftover config. To forecast quota and volume exhaustion before it blocks a rollout, see the capacity planning agent.
- Know where Pending ends. Once the claim is
Boundand the pod still will not start, the problem has moved to attach and mount, which is a different set of causes covered in FailedMount and FailedAttachVolume.
Quick reference
# 1. The answer is in the last event
kubectl -n NS describe pvc PVC
# 2. Is there a default class? Does the named class exist?
kubectl get storageclass
# 3. WaitForFirstConsumer: is the consuming pod scheduled? Is selected-node stale?
kubectl -n NS get pvc PVC -o jsonpath='{.metadata.annotations}'
# 4. Is the provisioner installed and running?
kubectl get csidrivers
kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-ebs-csi-driver
# 5. Full cloud error behind ProvisioningFailed
kubectl -n kube-system logs deploy/ebs-csi-controller -c csi-provisioner --tail=30
# 6. Static PVs: phase must be Available, class/size/modes must match
kubectl get pv
# 7. No claim at all: quota rejection is on the StatefulSet
kubectl -n NS describe statefulset NAME
A Pending claim is never silent. Kubernetes records which of the six causes it is on the claim itself; the only way to lose an hour is to look at the pod instead.