What CrashLoopBackOff actually means
CrashLoopBackOff means your container started, ran, and then exited — and the kubelet has restarted it enough times that it's now waiting, with an exponential backoff, before trying again. Unlike ImagePullBackOff, the image pulled fine and the process actually executed. Something inside the container is dying, and Kubernetes is doing exactly what you told it to: restart a failed container.
The word BackOff is the important part. The kubelet restarts a crashed container immediately, then after 10s, 20s, 40s, doubling up to a 5-minute cap. So a pod that's been crashing for an hour might only restart once every five minutes — the RESTARTS count climbs slowly even though the loop is constant. Deleting the pod does nothing useful: the replacement runs the same broken code or config and lands in the same state. This is the exact sequence I run to find the cause, usually in a couple of minutes.
Step 1: Read the crash logs — including the previous container
Never theorize. The crashed process almost always printed why it died before exiting. The catch: by the time you look, the kubelet may have already started a new container, so plain logs shows the fresh (and often empty) attempt. You want the logs from the instance that actually crashed:
kubectl logs payments-api-7d9f4c8b6-xk2mn --previous
The --previous (or -p) flag is the single most useful thing in this entire playbook. It dumps stdout/stderr from the last terminated container, which is where the stack trace, the panic:, or the Error: connect ECONNREFUSED lives. Nine times out of ten the answer is right there:
Error: Missing required environment variable DATABASE_URL
at loadConfig (/app/config.js:14:11)
at Object.<anonymous> (/app/server.js:3:16)
If the log is empty, the process died before it could log — a missing binary, a bad entrypoint, or an instant OOM kill. That's what Step 2 is for.
Step 2: Decode the exit code from describe
Every terminated container records an exit code, and the exit code narrows the cause immediately:
kubectl describe pod payments-api-7d9f4c8b6-xk2mn
Look at the Last State block under the container:
Last State: Terminated
Reason: Error
Exit Code: 1
Started: ...
Finished: ...
Map the exit code to a root cause:
| Exit code / Reason | What it means | Go to |
|---|---|---|
1 (Reason: Error) | Application threw and exited — bad config, missing env, startup exception | Step 3 |
137 + Reason OOMKilled | Kernel killed the container for exceeding its memory limit | Step 4 |
137 / 143 (Reason: Error) | Process got SIGKILL/SIGTERM — usually a failing liveness probe | Step 5 |
127 | command not found — bad command/entrypoint or missing binary | Step 6 |
126 | Command found but not executable (bad permissions / wrong arch) | Step 6 |
0 (still looping) | Process exits cleanly but has nothing to keep it alive | Step 7 |
Read the exit code first and the rest of the investigation collapses to a single branch.
Step 3: The app crashes on startup (exit code 1)
Exit 1 with a stack trace is the friendliest case — the app told you exactly what's wrong. In practice it's almost always one of three things:
- A missing or wrong environment variable.
DATABASE_URL, an API key, a feature flag the code assumes is set. Check what's actually injected:
kubectl set env pod/payments-api-7d9f4c8b6-xk2mn --list
- A missing ConfigMap or Secret key. If the pod references a key that doesn't exist, the container may not even start; if it starts but the value is empty, it crashes on first use. Verify the source object exists and has the key:
kubectl get secret app-secrets -o jsonpath='{.data}' | tr ',' '\n'
- A bad migration or unreachable dependency at boot. Many apps run DB migrations or open a connection pool during startup. If the database isn't reachable yet, they exit non-zero — and Kubernetes crash-loops them until the dependency comes up. The fix is either an
initContainerthat waits for the dependency, or making the app retry with backoff instead of exiting. Hard-exiting on a transient dependency failure is one of the quiet Kubernetes mistakes that cost companies money — a single slow database restart cascades into every dependent service crash-looping at once.
Step 4: OOMKilled (exit code 137)
If describe shows Reason: OOMKilled and exit 137, the container tried to use more memory than its limit and the kernel killed it. This one is treacherous because the app logs are often empty — the process is killed with SIGKILL and gets no chance to write anything.
Confirm it and see the limit that was breached:
kubectl describe pod payments-api-7d9f4c8b6-xk2mn | grep -A5 "Last State"
kubectl get pod payments-api-7d9f4c8b6-xk2mn \
-o jsonpath='{.spec.containers[0].resources}'
There are two distinct fixes, and picking the wrong one wastes a day:
- The limit is genuinely too low. The app needs more memory than you granted. Raise the
resources.limits.memory(andrequeststo match, so the scheduler places it correctly). If a JVM or Node process, remember the runtime's heap must fit inside the container limit with headroom — set-Xmxor--max-old-space-sizeto roughly 75% of the container limit, not equal to it. - There's a memory leak. The app grows until it hits any limit you set. Raising the limit just delays the crash. Watch usage over time with
kubectl top podand fix the leak — a bigger box only buys hours.
Right-sizing these limits is also a cost lever, not just a stability one: over-provisioned memory requests reserve capacity you pay for and never use. The same discipline shows up in Kubernetes cost optimization and FinOps for Kubernetes — right-sized limits keep pods alive and keep the bill down.
Step 5: A failing liveness probe is killing a healthy app
This is the most misdiagnosed CrashLoopBackOff of all. Your app is fine — but the liveness probe is failing, so the kubelet kills the container (SIGTERM, then SIGKILL → exit 143/137) on a schedule, and the pod loops forever. The tell is in the events:
Warning Unhealthy kubelet Liveness probe failed: HTTP probe failed
with statuscode: 500
Normal Killing kubelet Container failed liveness probe, will be
restarted
Common causes:
- The probe hits an endpoint that depends on a slow downstream (a
/healththat checks the database). If the DB blips, the probe fails and Kubernetes kills an otherwise-serving app. Liveness should test the process, not its dependencies — use readiness for dependency checks. initialDelaySecondsis too short for an app with a slow boot (JVM warmup, large cache load). The probe starts failing before the app has finished starting, so it never gets a chance to become healthy. Use astartupProbefor slow starters so the liveness clock doesn't start until the app is actually up.- Wrong port or path — the probe checks
:8080/healthzbut the app serves:3000/health.
The full breakdown of which probe does what — and why conflating liveness and readiness causes exactly this loop — is in the liveness, readiness, and startup probes guide. Fixing the probe config, not the app, is the fix here.
Step 6: Bad command or wrong architecture (exit 127 / 126)
Exit 127 means the container's entrypoint or command pointed at something that doesn't exist — a typo'd binary path, a script that isn't in the image, or a shell form that assumes /bin/sh in a distroless image that has none. Exit 126 means the file is there but isn't executable, or is built for the wrong CPU (exec format error — an amd64 binary on an arm64 node, the runtime twin of the platform mismatch you'd catch at pull time).
Check what the container is actually told to run, and confirm the binary exists:
kubectl get pod payments-api-7d9f4c8b6-xk2mn \
-o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'
If it's an architecture mismatch, rebuild multi-arch (docker buildx build --platform linux/amd64,linux/arm64 ...). If it's a missing shell in a distroless base, switch the probe/command to exec form with a real binary rather than sh -c.
Step 7: The container exits 0 and loops anyway
Sometimes the container exits cleanly (code 0) and still crash-loops. That happens when the main process finishes and there's nothing to keep PID 1 alive — a script that runs once and returns, or a web server started in the background while the foreground command exits. Kubernetes treats "container finished" as "restart it" under the default restartPolicy: Always. The fix is to run the long-lived process in the foreground as PID 1. For genuine run-once workloads, use a Job or CronJob instead of a Deployment — those are designed to complete.
A repeatable checklist
When a pod is stuck in CrashLoopBackOff, run this in order:
kubectl logs <pod> --previous→ read the crash output from the container that actually died.kubectl describe pod <pod>→ read the exit code and Reason inLast State.- Exit
1→ app-level crash; check env vars, ConfigMaps/Secrets, and startup dependencies (Step 3). OOMKilled/137→ raise the memory limit or fix the leak — decide which withkubectl top(Step 4).- Exit
143/137with aLiveness probe failedevent → fix the probe, not the app (Step 5). - Exit
127/126→ bad command path or wrong architecture (Step 6). - Exit
0looping → run the process in the foreground, or use aJob(Step 7).
CrashLoopBackOff looks alarming because the pod visibly refuses to stay up, but the container runtime records the exit code and the process almost always logs its own cause. Read --previous logs first, decode the exit code second, and the fix is nearly always mechanical. A rising kube_pod_container_status_restarts_total is worth alerting on before a loop takes out a whole service — wire it into the Prometheus + Grafana monitoring setup and capture the diagnosis path in your incident runbook so the next on-call engineer solves it in minutes, not hours. Pair this with the ImagePullBackOff playbook and you've covered nearly every reason a Kubernetes pod fails to run.
Related Reading
- Kubernetes ImagePullBackOff: How to Debug and Fix It (2026) — the other half of "my pod won't start." ImagePullBackOff means the image never pulled; CrashLoopBackOff means it pulled and the container keeps dying. Same
describe-first discipline, different root causes. - Kubernetes Liveness, Readiness, and Startup Probes: A Practical Guide — misconfigured liveness probes are the single most common cause of a healthy app crash-looping; this explains exactly which probe to use where.
- 10 Kubernetes Mistakes That Cost Companies Millions — hard-exiting on transient dependency failures and mis-set resource limits both show up here as expensive, avoidable crash loops.