devops

Kubernetes CrashLoopBackOff: How to Debug and Fix It (2026)

A step-by-step CrashLoopBackOff playbook: read crash logs, decode exit codes 1/137/127, and fix OOMKills, bad config, failing probes, and dependencies.

July 14, 2026·10 min read·
#kubernetes#sre#devops#containers#reliability

What CrashLoopBackOff actually means

CrashLoopBackOff means your container started, ran, and then exited — and the kubelet has restarted it enough times that it's now waiting, with an exponential backoff, before trying again. Unlike ImagePullBackOff, the image pulled fine and the process actually executed. Something inside the container is dying, and Kubernetes is doing exactly what you told it to: restart a failed container.

The word BackOff is the important part. The kubelet restarts a crashed container immediately, then after 10s, 20s, 40s, doubling up to a 5-minute cap. So a pod that's been crashing for an hour might only restart once every five minutes — the RESTARTS count climbs slowly even though the loop is constant. Deleting the pod does nothing useful: the replacement runs the same broken code or config and lands in the same state. This is the exact sequence I run to find the cause, usually in a couple of minutes.

Step 1: Read the crash logs — including the previous container

Never theorize. The crashed process almost always printed why it died before exiting. The catch: by the time you look, the kubelet may have already started a new container, so plain logs shows the fresh (and often empty) attempt. You want the logs from the instance that actually crashed:

kubectl logs payments-api-7d9f4c8b6-xk2mn --previous

The --previous (or -p) flag is the single most useful thing in this entire playbook. It dumps stdout/stderr from the last terminated container, which is where the stack trace, the panic:, or the Error: connect ECONNREFUSED lives. Nine times out of ten the answer is right there:

Error: Missing required environment variable DATABASE_URL
    at loadConfig (/app/config.js:14:11)
    at Object.<anonymous> (/app/server.js:3:16)

If the log is empty, the process died before it could log — a missing binary, a bad entrypoint, or an instant OOM kill. That's what Step 2 is for.

Step 2: Decode the exit code from describe

Every terminated container records an exit code, and the exit code narrows the cause immediately:

kubectl describe pod payments-api-7d9f4c8b6-xk2mn

Look at the Last State block under the container:

    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      ...
      Finished:     ...

Map the exit code to a root cause:

Exit code / ReasonWhat it meansGo to
1 (Reason: Error)Application threw and exited — bad config, missing env, startup exceptionStep 3
137 + Reason OOMKilledKernel killed the container for exceeding its memory limitStep 4
137 / 143 (Reason: Error)Process got SIGKILL/SIGTERM — usually a failing liveness probeStep 5
127command not found — bad command/entrypoint or missing binaryStep 6
126Command found but not executable (bad permissions / wrong arch)Step 6
0 (still looping)Process exits cleanly but has nothing to keep it aliveStep 7

Read the exit code first and the rest of the investigation collapses to a single branch.

Step 3: The app crashes on startup (exit code 1)

Exit 1 with a stack trace is the friendliest case — the app told you exactly what's wrong. In practice it's almost always one of three things:

  • A missing or wrong environment variable. DATABASE_URL, an API key, a feature flag the code assumes is set. Check what's actually injected:
kubectl set env pod/payments-api-7d9f4c8b6-xk2mn --list
  • A missing ConfigMap or Secret key. If the pod references a key that doesn't exist, the container may not even start; if it starts but the value is empty, it crashes on first use. Verify the source object exists and has the key:
kubectl get secret app-secrets -o jsonpath='{.data}' | tr ',' '\n'
  • A bad migration or unreachable dependency at boot. Many apps run DB migrations or open a connection pool during startup. If the database isn't reachable yet, they exit non-zero — and Kubernetes crash-loops them until the dependency comes up. The fix is either an initContainer that waits for the dependency, or making the app retry with backoff instead of exiting. Hard-exiting on a transient dependency failure is one of the quiet Kubernetes mistakes that cost companies money — a single slow database restart cascades into every dependent service crash-looping at once.

Step 4: OOMKilled (exit code 137)

If describe shows Reason: OOMKilled and exit 137, the container tried to use more memory than its limit and the kernel killed it. This one is treacherous because the app logs are often empty — the process is killed with SIGKILL and gets no chance to write anything.

Confirm it and see the limit that was breached:

kubectl describe pod payments-api-7d9f4c8b6-xk2mn | grep -A5 "Last State"
kubectl get pod payments-api-7d9f4c8b6-xk2mn \
  -o jsonpath='{.spec.containers[0].resources}'

There are two distinct fixes, and picking the wrong one wastes a day:

  1. The limit is genuinely too low. The app needs more memory than you granted. Raise the resources.limits.memory (and requests to match, so the scheduler places it correctly). If a JVM or Node process, remember the runtime's heap must fit inside the container limit with headroom — set -Xmx or --max-old-space-size to roughly 75% of the container limit, not equal to it.
  2. There's a memory leak. The app grows until it hits any limit you set. Raising the limit just delays the crash. Watch usage over time with kubectl top pod and fix the leak — a bigger box only buys hours.

Right-sizing these limits is also a cost lever, not just a stability one: over-provisioned memory requests reserve capacity you pay for and never use. The same discipline shows up in Kubernetes cost optimization and FinOps for Kubernetes — right-sized limits keep pods alive and keep the bill down.

Step 5: A failing liveness probe is killing a healthy app

This is the most misdiagnosed CrashLoopBackOff of all. Your app is fine — but the liveness probe is failing, so the kubelet kills the container (SIGTERM, then SIGKILL → exit 143/137) on a schedule, and the pod loops forever. The tell is in the events:

Warning  Unhealthy  kubelet  Liveness probe failed: HTTP probe failed
                              with statuscode: 500
Normal   Killing    kubelet  Container failed liveness probe, will be
                              restarted

Common causes:

  • The probe hits an endpoint that depends on a slow downstream (a /health that checks the database). If the DB blips, the probe fails and Kubernetes kills an otherwise-serving app. Liveness should test the process, not its dependencies — use readiness for dependency checks.
  • initialDelaySeconds is too short for an app with a slow boot (JVM warmup, large cache load). The probe starts failing before the app has finished starting, so it never gets a chance to become healthy. Use a startupProbe for slow starters so the liveness clock doesn't start until the app is actually up.
  • Wrong port or path — the probe checks :8080/healthz but the app serves :3000/health.

The full breakdown of which probe does what — and why conflating liveness and readiness causes exactly this loop — is in the liveness, readiness, and startup probes guide. Fixing the probe config, not the app, is the fix here.

Step 6: Bad command or wrong architecture (exit 127 / 126)

Exit 127 means the container's entrypoint or command pointed at something that doesn't exist — a typo'd binary path, a script that isn't in the image, or a shell form that assumes /bin/sh in a distroless image that has none. Exit 126 means the file is there but isn't executable, or is built for the wrong CPU (exec format error — an amd64 binary on an arm64 node, the runtime twin of the platform mismatch you'd catch at pull time).

Check what the container is actually told to run, and confirm the binary exists:

kubectl get pod payments-api-7d9f4c8b6-xk2mn \
  -o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'

If it's an architecture mismatch, rebuild multi-arch (docker buildx build --platform linux/amd64,linux/arm64 ...). If it's a missing shell in a distroless base, switch the probe/command to exec form with a real binary rather than sh -c.

Step 7: The container exits 0 and loops anyway

Sometimes the container exits cleanly (code 0) and still crash-loops. That happens when the main process finishes and there's nothing to keep PID 1 alive — a script that runs once and returns, or a web server started in the background while the foreground command exits. Kubernetes treats "container finished" as "restart it" under the default restartPolicy: Always. The fix is to run the long-lived process in the foreground as PID 1. For genuine run-once workloads, use a Job or CronJob instead of a Deployment — those are designed to complete.

A repeatable checklist

When a pod is stuck in CrashLoopBackOff, run this in order:

  1. kubectl logs <pod> --previous → read the crash output from the container that actually died.
  2. kubectl describe pod <pod> → read the exit code and Reason in Last State.
  3. Exit 1 → app-level crash; check env vars, ConfigMaps/Secrets, and startup dependencies (Step 3).
  4. OOMKilled / 137 → raise the memory limit or fix the leak — decide which with kubectl top (Step 4).
  5. Exit 143/137 with a Liveness probe failed event → fix the probe, not the app (Step 5).
  6. Exit 127/126 → bad command path or wrong architecture (Step 6).
  7. Exit 0 looping → run the process in the foreground, or use a Job (Step 7).

CrashLoopBackOff looks alarming because the pod visibly refuses to stay up, but the container runtime records the exit code and the process almost always logs its own cause. Read --previous logs first, decode the exit code second, and the fix is nearly always mechanical. A rising kube_pod_container_status_restarts_total is worth alerting on before a loop takes out a whole service — wire it into the Prometheus + Grafana monitoring setup and capture the diagnosis path in your incident runbook so the next on-call engineer solves it in minutes, not hours. Pair this with the ImagePullBackOff playbook and you've covered nearly every reason a Kubernetes pod fails to run.

Related Reading

#kubernetes#sre#devops#containers#reliability
D
DevToCashAuthor

Senior DevOps/SRE Engineer · 10+ years · Professional Trader (IDX, Crypto, US Equities)

I write about real infrastructure patterns and trading strategies I use in production and in live markets. No courses, no affiliate hype — just documentation of what actually works.

More about me →