devops

Kubernetes ImagePullBackOff: How to Debug and Fix It (2026)

A step-by-step ImagePullBackOff and ErrImagePull playbook: read the pull error, fix wrong tags, registry auth, Docker Hub rate limits, and node network issues.

July 13, 2026·8 min read·
#kubernetes#sre#devops#containers#reliability

What ImagePullBackOff actually means

ImagePullBackOff means the kubelet tried to pull your container image, failed, and is now waiting — with an exponential backoff — before trying again. Like CrashLoopBackOff, it is a symptom, not a root cause. The real failure is the pull itself, and the kubelet already recorded exactly why in the pod's events. You never have to guess.

The important distinction: ErrImagePull is the first failed attempt, and ImagePullBackOff is what you see after the kubelet starts backing off (10s, 20s, 40s, capped at 5 minutes). Same underlying problem. Deleting the pod does nothing — a new pod pulls the same broken reference and lands in the same state. This is the exact sequence I run to find the cause in about a minute.

Step 1: Read the actual pull error

Never start from theory. Start from describe, because the Events block quotes the container runtime's error verbatim:

kubectl describe pod payments-api-7d9f4c8b6-xk2mn

Scroll to the bottom:

  Warning  Failed     kubelet  Failed to pull image
           "myregistry/payments-api:1.4.2": failed to resolve reference:
           unexpected status: 401 Unauthorized
  Warning  Failed     kubelet  Error: ErrImagePull
  Normal   BackOff    kubelet  Back-off pulling image
  Warning  Failed     kubelet  Error: ImagePullBackOff

That one line — 401 Unauthorized — routes the entire investigation. The runtime error text maps cleanly to a root cause:

Error textRoot causeGo to
not found / manifest unknownWrong image name or tagStep 2
401 Unauthorized / deniedMissing or wrong registry credentialsStep 3
429 Too Many Requests / toomanyrequestsDocker Hub anonymous rate limitStep 4
no such host / i/o timeout / connection refusedNode can't reach the registryStep 5
no match for platformArchitecture mismatch (arm64 vs amd64)Step 6

Read this first and the rest collapses to a single path.

Step 2: Wrong image name or tag (manifest unknown)

The most common cause is the simplest: the image reference doesn't exist. A typo in the repository, a tag that was never pushed, or a CI pipeline that pushed 1.4.2 while your manifest still says 1.4.1.

Confirm exactly what the pod is asking for:

kubectl get pod payments-api-7d9f4c8b6-xk2mn \
  -o jsonpath='{.spec.containers[0].image}'

Then verify that reference actually exists in the registry from your workstation:

docker manifest inspect myregistry/payments-api:1.4.2

If that returns manifest unknown, the tag isn't there. Two things to check:

  • A CI race. Your deploy ran before the image push finished. This is common when build and deploy are separate jobs — pin the deploy to the image digest (@sha256:...) instead of a moving tag so a deploy can never reference a non-existent image.
  • The :latest trap. If you use :latest with imagePullPolicy: IfNotPresent, a node that already cached an old latest will silently run stale code instead of failing. Always tag immutably (1.4.2, a git SHA) in production. This is one of the Kubernetes mistakes that quietly cost companies money — a "successful" deploy that shipped nothing.

Step 3: Private registry authentication (401 / denied)

If the image exists but the pull returns 401 Unauthorized or denied, the kubelet has no valid credentials for that registry. Kubernetes pulls images using an imagePullSecrets reference, not your local docker login. The node never sees your laptop's ~/.docker/config.json.

Create the pull secret:

kubectl create secret docker-registry regcred \
  --docker-server=myregistry.example.com \
  --docker-username=deploy-bot \
  --docker-password="$REGISTRY_TOKEN" \
  --namespace=prod

Then attach it. Either reference it on the pod spec:

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: payments-api
      image: myregistry.example.com/payments-api:1.4.2

Or — cleaner for a whole namespace — patch the default ServiceAccount so every pod inherits it:

kubectl patch serviceaccount default -n prod \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

Two gotchas that eat hours:

  1. Namespace scope. A pull secret only works in the namespace it was created in. A pod in prod cannot use a secret in default. If you have 12 namespaces, you need the secret in each one that pulls private images.
  2. docker-server must match the image host exactly. If your image is myregistry.example.com/..., the secret's server must be myregistry.example.com — not https://..., not a trailing slash. A mismatch means the kubelet holds a valid credential it never applies.

Because these secrets carry registry write tokens in some setups, scope them tightly and rotate them — the least-privilege reasoning in the Kubernetes RBAC deep dive and the broader Kubernetes security best practices both apply directly to pull credentials.

Step 4: Docker Hub rate limits (429 Too Many Requests)

If you pull public images from Docker Hub anonymously, you're capped at 100 pulls per 6 hours per IP. On a busy cluster where many nodes share one NAT egress IP, you hit that ceiling fast, and pulls start failing with toomanyrequests:

Failed to pull image "nginx:1.27": toomanyrequests: You have reached
your pull rate limit.

The fix is to authenticate even for public pulls, which raises the limit substantially. Create a Docker Hub pull secret exactly as in Step 3 (server docker.io) and attach it to the ServiceAccount. Better still, run a pull-through cache (Harbor, or the registry mirror built into most managed clusters) so each image is fetched from Docker Hub once and served internally forever after — that also cuts pull latency on scale-ups and reduces your dependency on an external service during an incident.

Step 5: The node can't reach the registry (no such host / timeout)

i/o timeout, no such host, or connection refused means DNS or network — the pull never got far enough to check credentials. The problem is on the node, so debug from the node's perspective, not your laptop's.

Find which node the pod landed on, then test resolution and reachability with a throwaway pod scheduled there:

kubectl get pod payments-api-7d9f4c8b6-xk2mn -o wide   # note the NODE
kubectl run netcheck --rm -it --image=nicolaka/netshoot \
  --restart=Never -- \
  sh -c "nslookup myregistry.example.com && \
         curl -sSv https://myregistry.example.com/v2/ 2>&1 | head"

Common findings:

  • CoreDNS is down or misconfigured — every pull and every service lookup fails cluster-wide. Check kubectl get pods -n kube-system -l k8s-app=kube-dns.
  • A private registry behind a VPC endpoint where the node's security group or NAT route was changed. The registry is reachable from your office VPN but not from the node subnet.
  • A self-hosted registry with an untrusted TLS certx509: certificate signed by unknown authority. The node's container runtime needs the CA added to its trust store.

Persistent pull failures across many nodes are exactly the kind of signal worth alerting on before a scale-up stalls a rollout — the Prometheus + Grafana monitoring setup can watch kubelet_image_pull errors and page you before users notice.

Step 6: Architecture mismatch (no match for platform)

If you build on an Apple Silicon laptop and deploy to amd64 nodes, an image built only for arm64 fails with no match for platform in manifest. The pull "succeeds" in finding the manifest but has no layer for the node's CPU. Build multi-arch images explicitly:

docker buildx build --platform linux/amd64,linux/arm64 \
  -t myregistry.example.com/payments-api:1.4.2 --push .

This is the pull-time twin of the exec format error you'd see at runtime in a crash loop — same architecture root cause, caught one stage earlier.

A repeatable checklist

When a pod is stuck in ImagePullBackOff, run this in order:

  1. kubectl describe pod → read the Failed to pull image event text.
  2. manifest unknown → wrong name/tag; verify with docker manifest inspect (Step 2).
  3. 401 / denied → create and attach an imagePullSecrets, right namespace, exact server (Step 3).
  4. 429 / toomanyrequests → authenticate to Docker Hub or run a pull-through cache (Step 4).
  5. no such host / timeout → debug DNS and network from the node (Step 5).
  6. no match for platform → build a multi-arch image (Step 6).

ImagePullBackOff looks intimidating because the pod never even starts, but the container runtime always quotes the exact reason in the pod's events. Read that line first and the fix is almost always mechanical. Pin immutable tags, put pull secrets on the ServiceAccount in every namespace, and mirror public images — and most of these loops never reach production. Pair this with the liveness, readiness, and startup probes guide, and you've covered nearly every reason a Kubernetes pod fails to come up.

#kubernetes#sre#devops#containers#reliability
D
DevToCashAuthor

Senior DevOps/SRE Engineer · 10+ years · Professional Trader (IDX, Crypto, US Equities)

I write about real infrastructure patterns and trading strategies I use in production and in live markets. No courses, no affiliate hype — just documentation of what actually works.

More about me →