What NotReady actually means
A node shows NotReady when its Ready condition is anything other than True — and that single word hides two very different failures. Either the kubelet is alive and telling you it can't run pods (broken container runtime, uninitialized CNI, unhealthy PLEG), or the kubelet has stopped talking entirely and the control plane flipped the condition to Unknown after missing heartbeats for the node-monitor grace period (40s by default). The fix path splits immediately: the first case you debug on the node with journalctl -u kubelet; the second is a dead kubelet, a dead node, an expired certificate, or a network partition.
Meanwhile the control plane acts on its own: the node gets a NoExecute taint, and after 5 minutes every ordinary pod on it is marked for eviction. So a NotReady node is two incidents in one — the sick node, and the workloads it strands. This post walks both.
Step 1: Read the condition, don't guess
kubectl get nodes
kubectl describe node ip-10-0-1-42 | grep -A10 Conditions:
Ready False KubeletNotReady container runtime network not ready:
NetworkReady=false reason:NetworkPluginNotReady
message:Network plugin returns error: cni plugin not initialized
The Reason/Message pair on the Ready condition is the diagnosis, and it's one of two shapes:
Ready=Falsewith a concrete message — the kubelet is up and self-reporting. The message names the broken subsystem: container runtime, CNI, PLEG. Go to Step 3.Ready=UnknownwithNodeStatusUnknown/ "Kubelet stopped posting node status" — the kubelet hasn't heartbeated. The control plane knows nothing about the node except that it's silent. Go to Step 2.
Check when it went quiet — kubelets renew a Lease every 10 seconds, and its timestamp is the last confirmed sign of life:
kubectl get lease ip-10-0-1-42 -n kube-node-lease \
-o jsonpath='{.spec.renewTime}'
A renewTime from 30 seconds ago on a node marked Unknown means it's flapping (intermittent network, overloaded API server, kubelet restart loop) — a different problem from a node silent for an hour.
Step 2: Unknown — the kubelet went silent
Four causes account for nearly all of these, in rough order of frequency:
The kubelet process is dead. SSH (or SSM) to the node:
systemctl status kubelet
journalctl -u kubelet --since "30 min ago" | tail -50
Look at why it died. OOM-killed (journalctl -k | grep -i "killed process" naming kubelet) means the node was memory-overcommitted so badly the OS killed its own agent — the real fix is honest pod requests and kube-reserved/system-reserved carve-outs, the same overcommit story as pod evictions, just one level deeper. Crash-looping with a config error means a bad kubelet flag or a mangled /var/lib/kubelet/config.yaml from a recent change.
The kubelet's client certificate expired. The signature in the kubelet log is unmistakable:
part of the existing bootstrap client certificate is expired
... x509: certificate has expired or is not yet valid
Verify directly:
openssl x509 -enddate -noout \
-in /var/lib/kubelet/pki/kubelet-client-current.pem
# kubeadm clusters: check everything at once
kubeadm certs check-expiration
Kubelet certs auto-rotate — but only while the kubelet is running. A node powered off for weeks past its cert expiry comes back permanently NotReady until you re-bootstrap it (kubeadm certs renew on control-plane certs, or delete and rejoin the worker).
The whole node is down or unreachable. Cloud console says stopped/failed, or the node dropped off the network. On managed platforms, check the health of the underlying instance before anything Kubernetes-level.
Network partition between node and API server. Kubelet log fills with Failed to update node status ... dial tcp <apiserver>:6443: i/o timeout. Security-group change, NACL, overloaded API server, expired proxy in between. The node itself is healthy — its pods are usually still running the whole time, which matters in Step 4.
Step 3: Ready=False — the kubelet names the culprit
Container runtime not ready
container runtime is down
... connect: no such file or directory: /run/containerd/containerd.sock
systemctl status containerd
journalctl -u containerd --since "30 min ago" | tail -30
crictl info | jq '.status.conditions'
containerd most often dies from a full disk (/var/lib/containerd on the root volume — df -h first, always) or crashes after an unattended package upgrade. If disk is the cause, you'll usually see DiskPressure=True in the same conditions block; clear space (image GC, log truncation, bigger volume) before restarting the runtime, or it just falls over again.
CNI plugin not initialized
NetworkReady=false reason:NetworkPluginNotReady
message:Network plugin returns error: cni plugin not initialized
The kubelet is waiting for a CNI config in /etc/cni/net.d/ that the network plugin's DaemonSet is supposed to install. So the node's error is really a pod's error — find the CNI pod (Calico, Cilium, aws-node, flannel) for that node:
kubectl get pods -n kube-system -o wide \
--field-selector spec.nodeName=ip-10-0-1-42 | grep -Ei 'calico|cilium|aws-node|flannel|cni'
ls /etc/cni/net.d/ # on the node: empty = plugin never wrote its config
If that pod is in CrashLoopBackOff, debug it — its crash reason (often an image pull failure, an IPAM pool exhausted, or an incompatible version after a cluster upgrade) is the node's root cause. This is also the classic state of a freshly joined node: NotReady for a minute or two while the CNI DaemonSet lands is normal; NotReady for ten minutes is not.
PLEG is not healthy
PLEG is not healthy: pleg was last seen active 5m32s ago; threshold is 3m
The Pod Lifecycle Event Generator is the kubelet's loop that asks the runtime "what changed?" — when a single relist takes over 3 minutes, the kubelet declares itself NotReady. It means the container runtime is responding slowly, not that it's down: extreme container churn (a tight CrashLoop across many pods), disk I/O saturation, or very high pod density. Check crictl ps -a | wc -l for a corpse pile-up and node I/O metrics. A runtime restart clears the symptom; the churn source is the cure.
Step 4: The stranded pods — what the control plane does meanwhile
The moment the Ready condition leaves True, the node controller taints the node node.kubernetes.io/not-ready:NoExecute (or unreachable for Unknown). Every pod carries a default toleration for these taints with tolerationSeconds: 300 — so after 5 minutes, pods on the node are marked for deletion and controllers create replacements. Three consequences worth knowing before you touch anything:
- A surge of Pending pods elsewhere — replacements need somewhere to land, and if the cluster can't fit them you get a wave of FailedScheduling right after the node event. Losing a node is also a capacity test.
- On an unreachable node, pods get stuck Terminating — the API can't confirm the containers stopped, so deletion never completes. That's the "node down" case of pods stuck in Terminating, and it's why StatefulSet replacements don't start: the controller refuses to risk two copies of pod-0 writing to the same volume. Only force-delete (
kubectl delete pod --force --grace-period=0) once you have out-of-band proof the node is actually dead — cloud console says terminated — never on a mere partition, where the containers are still happily running. - Your maintenance window is shorter than 5 minutes only on paper — pods on a flapping node bounce every flap. If you're doing planned node work,
kubectl cordon+drainfirst so evictions are orderly and respect PodDisruptionBudgets.
On managed node groups, the pragmatic fix for a single bad node is often replacement, not repair: cordon it, drain what's reachable, terminate the instance, and let the autoscaler — Cluster Autoscaler or Karpenter — bring a fresh one. Repair the pattern (why did it die?), replace the instance.
Alert before pods start moving
Both signals come free with kube-state-metrics in a standard Prometheus + Grafana setup:
# Any node not Ready for 2 minutes — fires before the 5-minute eviction clock
kube_node_status_condition{condition="Ready", status="true"} == 0
# Flap detector: Ready condition changed more than twice in 15m
changes(kube_node_status_condition{condition="Ready", status="true"}[15m]) > 2
The two-minute for: clause matters: it beats the tolerationSeconds: 300 deadline, so a human sees the node before the control plane starts evicting — the difference between fixing one node and explaining a reshuffle of every workload on it.
A repeatable checklist
kubectl describe node→Ready=Falsewith a message (kubelet self-reporting) orReady=Unknown(kubelet silent)? Check the Lease renewTime for last sign of life (Step 1).- Unknown → on the node:
systemctl status kubelet,journalctl -u kubelet, cert expiry viaopenssl x509 -enddate, instance health, API-server reachability (Step 2). - False → the message names it: containerd down (check disk first),
cni plugin not initialized(debug the CNI DaemonSet pod, not the node),PLEG is not healthy(runtime overload/churn) (Step 3). - Mind the 5-minute eviction clock; force-delete stuck pods only with out-of-band proof the node is dead; expect a Pending wave if capacity is tight (Step 4).
- Managed nodes: cordon, drain, replace the instance; investigate the pattern separately. Alert on
kube_node_status_conditionwith a 2mfor:so you beat the eviction clock next time.
Related Reading
- Kubernetes Pod Evicted: How to Debug and Fix It — node pressure conditions (Disk/Memory/PIDPressure) are the milder siblings of NotReady: the node stays schedulable-ish but the kubelet starts choosing victims.
- Kubernetes Pod Stuck in Terminating: How to Debug and Fix It — the full story of deletion that can't complete, including the unreachable-node case and when force-delete is actually safe.
- Kubernetes Pod Stuck in Pending (FailedScheduling): How to Fix It — where the evicted workloads go next; a NotReady node and a Pending wave arriving together is a capacity problem wearing two hats.