Kubernetes FailedMount and FailedAttachVolume: How to Debug and Fix It

What FailedMount and FailedAttachVolume actually mean

Your pod is stuck in ContainerCreating, there are no logs, and kubectl describe shows FailedAttachVolume or FailedMount warnings. What happened: the pod scheduled fine, but its volume couldn't be attached to the node or mounted into the container, so the kubelet is waiting before it will even try to start your process.

The two events map to two distinct stages, owned by two different components, and knowing which stage failed is 80% of the debug:

Attach — the attach/detach controller (in kube-controller-manager) plus the CSI external-attacher connect the volume to the node, like plugging a disk into a server. Failures here emit FailedAttachVolume.
Mount — the kubelet on that node formats (if needed) and mounts the attached device into the pod's filesystem. Failures here emit FailedMount.

The trap: a genuine attach failure also produces a generic FailedMount ... timed out waiting for the condition event, because the kubelet gave up waiting for a device that never arrived. If you only read the last event, you'll debug the wrong stage. This is the same discipline as the rest of this series — like CreateContainerConfigError, the event text names the real problem, but here you have to find the first failure, not the loudest one.

Step 1: Read the events top to bottom, not bottom up

kubectl describe pod postgres-0 -n prod | grep -A2 -E "FailedAttachVolume|FailedMount"

The messages you'll actually meet, in rough order of frequency:

Warning  FailedAttachVolume  Multi-Attach error for volume "pvc-3fa8..."
  Volume is already used by pod(s) postgres-0

Warning  FailedMount  MountVolume.SetUp failed for volume "config" :
  configmap "app-config" not found

Warning  FailedAttachVolume  AttachVolume.Attach failed for volume "pvc-3fa8..." :
  rpc error: code = Internal desc = Could not attach volume "vol-0abc..." to node
  "i-0def...": operation error EC2: AttachVolume, api error UnauthorizedOperation

Warning  FailedMount  MountVolume.MountDevice failed for volume "pvc-3fa8..." :
  rpc error: code = Internal desc = format of disk "/dev/xvdba" failed:
  type:("ext4") errcode:(exit status 1)

Warning  FailedMount  Unable to attach or mount volumes: unmounted volumes=[data],
  unattached volumes=[data]: timed out waiting for the condition

That last one is the decoy. timed out waiting for the condition is never a root cause — scroll up for the FailedAttachVolume or the specific MountVolume.* failure that preceded it. If the events have already rotated out, check the kubelet journal on the node (journalctl -u kubelet | grep <pvc-id>).

Step 2: Check the PVC, PV, and VolumeAttachment chain

Three objects must all be healthy for a block volume to reach your pod:

kubectl get pvc data-postgres-0 -n prod        # must be Bound
kubectl get pv pvc-3fa8...                     # must be Bound, right storageclass
kubectl get volumeattachment | grep pvc-3fa8   # who holds it right now

VolumeAttachment is the object most people have never looked at, and it answers the single most important question in an attach failure: which node currently owns this volume:

NAME        ATTACHER          PV          NODE                 ATTACHED
csi-9c1e…   ebs.csi.aws.com   pvc-3fa8…   ip-10-0-2-114.ec2…   true

If ATTACHED is true on node A while your pod is scheduled to node B, you've found your Multi-Attach error. If it's false with an error in kubectl describe volumeattachment, the CSI attacher is failing and the message there (IAM denied, volume limit reached, volume in wrong AZ) is your root cause.

If the PVC itself is Pending rather than Bound, you have a provisioning problem, not an attach problem — and if the pod is Pending too, you're in FailedScheduling territory instead: a WaitForFirstConsumer StorageClass can't bind until the pod schedules, and a volume pinned to the wrong zone blocks scheduling entirely.

Step 3: Fix the actual cause

1. Multi-Attach error: an RWO volume, claimed from two nodes

The most common failure by far. ReadWriteOnce means one node at a time, and something is asking for a second node. Two ways you got here:

A rolling update moved the pod. A single-replica Deployment with an RWO PVC does a rolling update: the new pod starts on node B before the old pod on node A releases the volume. Deadlock — the new pod waits on a volume the old pod won't release until the new pod is ready. The fix is one line:

spec:
  strategy:
    type: Recreate    # kill the old pod first, then start the new one

For anything genuinely stateful, use a StatefulSet — it never runs two instances of the same ordinal, so it can't race itself. Reserve RollingUpdate + RWO for pods that don't hold their volume hostage.

The old node died. The pod's node went NotReady, the pod got rescheduled, but the dead kubelet never confirmed the unmount — so the attach/detach controller waits a full 6 minutes (its maxWaitForUnmountDuration) before force-detaching. If the replacement pod is stuck longer than that, the old pod is usually wedged in Terminating and blocking the detach:

kubectl delete pod postgres-0 --grace-period=0 --force   # only if the node is truly gone
kubectl delete volumeattachment csi-9c1e...              # last resort, forces CSI detach

Force-delete only when you're certain the node is dead, not partitioned — two kubelets writing one ext4 volume is how you corrupt data. On clusters with frequent spot interruptions, this 6-minute stall is a strong argument for the non-graceful node shutdown taint (node.kubernetes.io/out-of-service), which tells Kubernetes to skip the wait.

2. Missing ConfigMap or Secret volume

MountVolume.SetUp failed ... configmap "app-config" not found is the volume-flavored twin of CreateContainerConfigError — same dangling reference, different symptom, because volumes mount before container config resolves. The debug is identical to the env-var variant: confirm the object exists in the pod's namespace, fix key drift, or fix your GitOps sync ordering. optional: true on the volume exists but carries the same risk of booting an app with an empty config directory.

3. The CSI driver itself is broken on that node

Attach and mount RPCs are served by the CSI node plugin — a DaemonSet pod on every node. If it's not running where your pod landed, nothing mounts:

kubectl get pods -n kube-system -l app=ebs-csi-node \
  --field-selector spec.nodeName=ip-10-0-2-114.ec2.internal
kubectl get csinode ip-10-0-2-114.ec2.internal -o yaml   # is the driver registered?

An empty drivers: list in the CSINode object explains messages like driver name ebs.csi.aws.com not found in the list of registered CSI drivers — usually a crashed node plugin, a node the DaemonSet tolerations exclude, or a fresh node where registration is still in flight. For cloud-permission failures (UnauthorizedOperation in the attach error), the fix lives in the controller plugin's IAM role, not in Kubernetes at all — check the external-attacher container logs for the exact denied API call.

Also real: the node hit its attachment limit. An m5.large tops out around 25 EBS attachments; the scheduler tracks this via CSI limits, but mixed in-tree/CSI volumes or custom volume-attach-limit settings can lie to it, and the attach fails at the cloud API instead.

4. Filesystem mismatch on an existing volume

MountVolume.MountDevice failed ... wrong fs type, bad option, bad superblock means the volume already carries a filesystem that doesn't match what the mount asked for — an ext4 disk referenced by a StorageClass or PV that now says fsType: xfs, or a restored snapshot with unclean state. Kubernetes will format blank volumes to the requested fsType, but it will never reformat one with data (thankfully). Fix the fsType on the PV to match reality; for a dirty filesystem, attach the disk to a rescue instance and run fsck manually. Don't delete the PVC hoping for a clean start — that's your data.

5. Stale kubelet state after node restarts

If events show mounts failing on one specific node for multiple pods with is busy or orphaned-pod messages in the kubelet journal, the node's mount table is wedged. Cordon, drain, reboot — a fresh kubelet reconciles mounts from scratch. Chasing individual mountpoints by hand on a broken node is rarely worth it; cattle, not pets.

Alert on it — stuck ContainerCreating is silent by default

No restarts, no crash loops, nothing in error-rate dashboards — a pod waiting on a volume just waits. With kube-state-metrics in a standard Prometheus + Grafana setup:

sum by (namespace, pod) (
  kube_pod_container_status_waiting_reason{reason="ContainerCreating"}
) > 0

Hold it for: 15m — normal attach+mount finishes in seconds, and 15 minutes clears the 6-minute force-detach window plus retries, so it only fires on genuinely stuck pods. Pair it with kube_persistentvolumeclaim_status_phase{phase="Pending"} > 0 (for: 10m) to catch provisioning failures one stage earlier.

A repeatable checklist

kubectl describe pod and read events top to bottom — timed out waiting for the condition is a decoy; the first FailedAttachVolume or specific MountVolume.* error is the diagnosis. (Step 1)
Walk the chain: PVC Bound? PV healthy? kubectl get volumeattachment — which node owns the volume right now? (Step 2)
Multi-Attach: RWO volume wanted by two nodes. strategy: Recreate (or a StatefulSet) for the rolling-update race; for dead nodes, expect the 6-minute force-detach, then clear the Terminating pod or stale VolumeAttachment. (Step 3.1)
configmap/secret not found on a volume mount = dangling reference, same fix as CreateContainerConfigError. (Step 3.2)
CSI node plugin running on that node, driver registered in csinode, cloud IAM intact, attachment limit not exhausted. (Step 3.3)
wrong fs type = fsType drift on a data-bearing volume — fix the PV spec, never reformat. Node-wide mount weirdness = drain and reboot the node. (Steps 3.4–3.5)
Alert on ContainerCreating waiting 15+ minutes — this failure mode pages nobody unless you make it.