What ImagePullBackOff actually means
ImagePullBackOff means the kubelet tried to pull your container image, failed, and is now waiting — with an exponential backoff — before trying again. Like CrashLoopBackOff, it is a symptom, not a root cause. The real failure is the pull itself, and the kubelet already recorded exactly why in the pod's events. You never have to guess.
The important distinction: ErrImagePull is the first failed attempt, and ImagePullBackOff is what you see after the kubelet starts backing off (10s, 20s, 40s, capped at 5 minutes). Same underlying problem. Deleting the pod does nothing — a new pod pulls the same broken reference and lands in the same state. This is the exact sequence I run to find the cause in about a minute.
Step 1: Read the actual pull error
Never start from theory. Start from describe, because the Events block quotes the container runtime's error verbatim:
kubectl describe pod payments-api-7d9f4c8b6-xk2mn
Scroll to the bottom:
Warning Failed kubelet Failed to pull image
"myregistry/payments-api:1.4.2": failed to resolve reference:
unexpected status: 401 Unauthorized
Warning Failed kubelet Error: ErrImagePull
Normal BackOff kubelet Back-off pulling image
Warning Failed kubelet Error: ImagePullBackOff
That one line — 401 Unauthorized — routes the entire investigation. The runtime error text maps cleanly to a root cause:
| Error text | Root cause | Go to |
|---|---|---|
not found / manifest unknown | Wrong image name or tag | Step 2 |
401 Unauthorized / denied | Missing or wrong registry credentials | Step 3 |
429 Too Many Requests / toomanyrequests | Docker Hub anonymous rate limit | Step 4 |
no such host / i/o timeout / connection refused | Node can't reach the registry | Step 5 |
no match for platform | Architecture mismatch (arm64 vs amd64) | Step 6 |
Read this first and the rest collapses to a single path.
Step 2: Wrong image name or tag (manifest unknown)
The most common cause is the simplest: the image reference doesn't exist. A typo in the repository, a tag that was never pushed, or a CI pipeline that pushed 1.4.2 while your manifest still says 1.4.1.
Confirm exactly what the pod is asking for:
kubectl get pod payments-api-7d9f4c8b6-xk2mn \
-o jsonpath='{.spec.containers[0].image}'
Then verify that reference actually exists in the registry from your workstation:
docker manifest inspect myregistry/payments-api:1.4.2
If that returns manifest unknown, the tag isn't there. Two things to check:
- A CI race. Your deploy ran before the image push finished. This is common when build and deploy are separate jobs — pin the deploy to the image digest (
@sha256:...) instead of a moving tag so a deploy can never reference a non-existent image. - The
:latesttrap. If you use:latestwithimagePullPolicy: IfNotPresent, a node that already cached an oldlatestwill silently run stale code instead of failing. Always tag immutably (1.4.2, a git SHA) in production. This is one of the Kubernetes mistakes that quietly cost companies money — a "successful" deploy that shipped nothing.
Step 3: Private registry authentication (401 / denied)
If the image exists but the pull returns 401 Unauthorized or denied, the kubelet has no valid credentials for that registry. Kubernetes pulls images using an imagePullSecrets reference, not your local docker login. The node never sees your laptop's ~/.docker/config.json.
Create the pull secret:
kubectl create secret docker-registry regcred \
--docker-server=myregistry.example.com \
--docker-username=deploy-bot \
--docker-password="$REGISTRY_TOKEN" \
--namespace=prod
Then attach it. Either reference it on the pod spec:
spec:
imagePullSecrets:
- name: regcred
containers:
- name: payments-api
image: myregistry.example.com/payments-api:1.4.2
Or — cleaner for a whole namespace — patch the default ServiceAccount so every pod inherits it:
kubectl patch serviceaccount default -n prod \
-p '{"imagePullSecrets":[{"name":"regcred"}]}'
Two gotchas that eat hours:
- Namespace scope. A pull secret only works in the namespace it was created in. A pod in
prodcannot use a secret indefault. If you have 12 namespaces, you need the secret in each one that pulls private images. docker-servermust match the image host exactly. If your image ismyregistry.example.com/..., the secret's server must bemyregistry.example.com— nothttps://..., not a trailing slash. A mismatch means the kubelet holds a valid credential it never applies.
Because these secrets carry registry write tokens in some setups, scope them tightly and rotate them — the least-privilege reasoning in the Kubernetes RBAC deep dive and the broader Kubernetes security best practices both apply directly to pull credentials.
Step 4: Docker Hub rate limits (429 Too Many Requests)
If you pull public images from Docker Hub anonymously, you're capped at 100 pulls per 6 hours per IP. On a busy cluster where many nodes share one NAT egress IP, you hit that ceiling fast, and pulls start failing with toomanyrequests:
Failed to pull image "nginx:1.27": toomanyrequests: You have reached
your pull rate limit.
The fix is to authenticate even for public pulls, which raises the limit substantially. Create a Docker Hub pull secret exactly as in Step 3 (server docker.io) and attach it to the ServiceAccount. Better still, run a pull-through cache (Harbor, or the registry mirror built into most managed clusters) so each image is fetched from Docker Hub once and served internally forever after — that also cuts pull latency on scale-ups and reduces your dependency on an external service during an incident.
Step 5: The node can't reach the registry (no such host / timeout)
i/o timeout, no such host, or connection refused means DNS or network — the pull never got far enough to check credentials. The problem is on the node, so debug from the node's perspective, not your laptop's.
Find which node the pod landed on, then test resolution and reachability with a throwaway pod scheduled there:
kubectl get pod payments-api-7d9f4c8b6-xk2mn -o wide # note the NODE
kubectl run netcheck --rm -it --image=nicolaka/netshoot \
--restart=Never -- \
sh -c "nslookup myregistry.example.com && \
curl -sSv https://myregistry.example.com/v2/ 2>&1 | head"
Common findings:
- CoreDNS is down or misconfigured — every pull and every service lookup fails cluster-wide. Check
kubectl get pods -n kube-system -l k8s-app=kube-dns. - A private registry behind a VPC endpoint where the node's security group or NAT route was changed. The registry is reachable from your office VPN but not from the node subnet.
- A self-hosted registry with an untrusted TLS cert —
x509: certificate signed by unknown authority. The node's container runtime needs the CA added to its trust store.
Persistent pull failures across many nodes are exactly the kind of signal worth alerting on before a scale-up stalls a rollout — the Prometheus + Grafana monitoring setup can watch kubelet_image_pull errors and page you before users notice.
Step 6: Architecture mismatch (no match for platform)
If you build on an Apple Silicon laptop and deploy to amd64 nodes, an image built only for arm64 fails with no match for platform in manifest. The pull "succeeds" in finding the manifest but has no layer for the node's CPU. Build multi-arch images explicitly:
docker buildx build --platform linux/amd64,linux/arm64 \
-t myregistry.example.com/payments-api:1.4.2 --push .
This is the pull-time twin of the exec format error you'd see at runtime in a crash loop — same architecture root cause, caught one stage earlier.
A repeatable checklist
When a pod is stuck in ImagePullBackOff, run this in order:
kubectl describe pod→ read theFailed to pull imageevent text.manifest unknown→ wrong name/tag; verify withdocker manifest inspect(Step 2).401/denied→ create and attach animagePullSecrets, right namespace, exact server (Step 3).429/toomanyrequests→ authenticate to Docker Hub or run a pull-through cache (Step 4).no such host/ timeout → debug DNS and network from the node (Step 5).no match for platform→ build a multi-arch image (Step 6).
ImagePullBackOff looks intimidating because the pod never even starts, but the container runtime always quotes the exact reason in the pod's events. Read that line first and the fix is almost always mechanical. Pin immutable tags, put pull secrets on the ServiceAccount in every namespace, and mirror public images — and most of these loops never reach production. Pair this with the liveness, readiness, and startup probes guide, and you've covered nearly every reason a Kubernetes pod fails to come up.