What Terminating actually means
A pod stuck in Terminating has been marked for deletion but not confirmed dead. When you run kubectl delete pod, the API server doesn't remove the object — it sets metadata.deletionTimestamp and waits. The kubelet on the pod's node then runs the shutdown sequence: execute the preStop hook, send SIGTERM to PID 1 of every container, wait up to terminationGracePeriodSeconds (default 30s), then SIGKILL whatever survived. Only when the kubelet reports all containers gone — and the pod's finalizer list is empty — does the API server actually delete the object.
Terminating forever means one of those confirmations never arrives. There are exactly three families of cause: a finalizer nobody is removing, a kubelet that can't report back (node dead or unreachable), or containers that genuinely won't die (hung volume unmount, D-state process). Unlike CrashLoopBackOff or Pending, there's no event that names the culprit for you — but the diagnosis is still mechanical. Here's the sequence I run, in order of likelihood.
Step 1: Check how long it's actually been
First, rule out the non-problem. A pod in Terminating for 40 seconds with a 30-second grace period is working as designed — preStop hooks and slow shutdowns take time. Check what the pod was granted:
kubectl get pod payments-api-7d9f4c8b6-xk2mn \
-o jsonpath='{.spec.terminationGracePeriodSeconds}'
Some workloads legitimately set this to 300 or 3600 (job runners draining a queue, databases flushing to disk). If the elapsed time is inside the grace period, wait. If the pod has been Terminating for minutes past its grace period, something is stuck — continue.
Step 2: Look for finalizers
The most common cause, and the only one kubectl describe won't shout about. A finalizer is a string on the pod that tells Kubernetes "do not delete this object until a controller removes me." If that controller is broken, uninstalled, or wedged, the pod stays Terminating forever — even though its containers exited long ago.
kubectl get pod payments-api-7d9f4c8b6-xk2mn \
-o jsonpath='{.metadata.finalizers}'
["example.com/mesh-cleanup"]
Anything in that list is your blocker. Typical sources: service meshes, backup operators, custom controllers from an operator that was uninstalled without cleaning up its CRDs and webhooks.
The correct fix is to figure out why the owning controller isn't doing its job — is the operator's pod running? Is its webhook Service still resolvable? An uninstalled operator that left finalizers behind is the classic case. If the controller is gone for good, remove the finalizer by hand:
kubectl patch pod payments-api-7d9f4c8b6-xk2mn \
-p '{"metadata":{"finalizers":null}}' --type=merge
The pod disappears immediately. Understand what you just did: you skipped whatever cleanup that finalizer guaranteed (deregistering from a mesh, releasing an external IP, snapshotting a volume). That's usually fine when the owning system is already gone — it's not fine when the controller is merely slow.
Step 3: Check the node
If there are no finalizers, the next suspect is the kubelet that's supposed to confirm the kill. A pod on a dead or partitioned node stays Terminating indefinitely, because the API server never hears "containers are gone."
kubectl get pod payments-api-7d9f4c8b6-xk2mn -o wide # note the NODE column
kubectl get node ip-10-0-1-42
NAME STATUS ROLES AGE VERSION
ip-10-0-1-42 NotReady <none> 94d v1.31.2
NotReady plus a stuck Terminating pod is an open-and-shut case. Your options, from cleanest to bluntest:
- Recover the node. If the kubelet is just down (OOM, disk full, crashed), restarting it lets the normal flow finish:
systemctl restart kubeleton the node. - Tell Kubernetes the node is really dead. Since v1.28 (stable), the non-graceful node shutdown feature lets you assert it — after you've confirmed the machine is actually off, e.g. terminated in your cloud console:
kubectl taint node ip-10-0-1-42 \
node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
Kubernetes then force-deletes the pods and — critically for StatefulSets — detaches their volumes so replacements can start elsewhere.
- Delete the node object. If the instance no longer exists (spot reclaim, scaled-down node group that didn't drain cleanly),
kubectl delete node ip-10-0-1-42lets the garbage collector clean up every pod that was on it.
Step 4: Check whether the containers are actually still running
No finalizers, node is Ready, still stuck? Now check what the container runtime sees. SSH to the node (or use your debug tooling) and ask:
crictl ps --name payments-api
Two very different outcomes:
The container is gone from the runtime but the pod object persists — that's an API-side problem after all; re-check finalizers on the pod and look for a stuck kubelet (journalctl -u kubelet --since "10 min ago" | grep -i <pod-name>).
The container is still alive. The process is refusing to die, which after SIGKILL should be impossible — unless it's in uninterruptible sleep (D state), which even SIGKILL cannot touch:
ps -eo pid,stat,wchan:30,cmd | awk '$2 ~ /D/'
D-state processes are almost always blocked on I/O. In Kubernetes, the usual culprit is a volume that won't unmount: an NFS server that went away under a hard mount, or a CSI driver whose node plugin pod is dead so unmount RPCs hang forever. Check the kubelet's view:
journalctl -u kubelet --since "15 min ago" | grep -iE "unmount|orphan"
kubelet: Orphaned pod "d4c7…" found, but volume paths are still present on disk
Fixes by cause: restart the CSI node plugin (kubectl -n kube-system rollout restart daemonset ebs-csi-node), restore reachability to the NFS server, or as a last resort reboot the node — a D-state process holding a dead NFS mount will survive anything less.
Step 5: Force delete — when it's safe and when it bites
The command everyone reaches for first should be the last resort:
kubectl delete pod payments-api-7d9f4c8b6-xk2mn --grace-period=0 --force
This removes the pod object from etcd without waiting for the kubelet's confirmation. The API forgets the pod; nobody has verified the process is dead. That distinction is the whole risk:
- Safe: the node is confirmed destroyed (terminated instance), or you've verified via
crictlthat the containers already exited and only the object is stuck. - Dangerous: the node is unreachable but possibly alive, and the pod belongs to a StatefulSet. The controller sees the name freed and starts a replacement — if the old pod is still running behind the partition, you now have two instances with the same identity writing to the same data. This is precisely the split-brain the StatefulSet contract exists to prevent, and why the out-of-service taint in Step 3 (which fences the node first) is the better tool.
For plain stateless Deployments the blast radius is small — worst case a zombie process wastes some CPU until the node dies. Force-delete freely there once you've glanced at the cause.
A repeatable checklist
- Elapsed time vs
terminationGracePeriodSeconds— inside the window? It's not stuck, it's draining (Step 1). kubectl get pod <pod> -o jsonpath='{.metadata.finalizers}'→ non-empty? Fix or remove the owning controller's finalizer (Step 2).kubectl get node <node>→NotReady? Recover the kubelet, apply theout-of-servicetaint, or delete the dead node (Step 3).crictl pson the node → container really alive? Look for D-state processes and stuck volume unmounts; restart the CSI node plugin (Step 4).--grace-period=0 --forceonly once you know which of the above you're overriding — and never on a StatefulSet pod whose node might still be alive (Step 5).
Prevent the slow-death variant
Most "stuck" pods that resolve after exactly 30 seconds aren't stuck at all — they're apps that ignore SIGTERM and ride the grace period to the SIGKILL. That inflates every rollout and every node drain. Two cheap fixes:
Make sure PID 1 actually receives the signal. A shell-form ENTRYPOINT wraps your app in sh -c, and the shell neither forwards SIGTERM nor exits. Use exec form, or exec in your entrypoint script:
# Bad: sh is PID 1, your app never sees SIGTERM
ENTRYPOINT node server.js
# Good: node is PID 1
ENTRYPOINT ["node", "server.js"]
Handle it, then drain. Catch SIGTERM, stop accepting new work, finish in-flight requests, exit. Pair it with a short preStop sleep so the endpoint controller removes the pod from Services before the app stops accepting — the trick that makes zero-downtime deployments actually zero-downtime, and the shutdown-side complement to the liveness and readiness probes that manage the startup side.
Finally, make stuck terminations page you instead of surprising you mid-rollout. kube-state-metrics exposes kube_pod_deletion_timestamp:
(time() - kube_pod_deletion_timestamp) > 600
Wire that into your Prometheus + Grafana setup and put the five-step triage above into the incident runbook — a pod ten minutes into Terminating is always one of the three families, and now you know which commands separate them.
Related Reading
- Kubernetes Pod Stuck in Pending (FailedScheduling): How to Fix It — the mirror image of this post: Pending is a pod that can't start, Terminating is one that can't die. A drain that strands pods in Pending while others hang in Terminating is the same maintenance gone wrong from both ends.
- Kubernetes OOMKilled (Exit Code 137): How to Debug and Fix It — exit code 137 is
SIGKILLtoo; there it's the OOM killer, here it's the kubelet ending the grace period. Reading exit codes correctly tells the two apart. - Kubernetes ImagePullBackOff: How to Debug and Fix It — completes the pod-lifecycle series: pull failures, crash loops, OOM kills, scheduling, and now teardown.