Kubernetes OOMKilled (Exit Code 137): How to Debug and Fix It

What OOMKilled actually means

OOMKilled with exit code 137 means the Linux kernel killed your container because it tried to use more memory than it was allowed. The 137 is 128 + 9 — the process received signal 9 (SIGKILL). It gets no warning, no chance to flush, no chance to log. One moment it's serving traffic; the next it's gone, and the pod restarts into a CrashLoopBackOff if the same thing keeps happening.

There are two completely different situations that both surface as OOMKilled, and the entire fix depends on telling them apart:

1. Container-level OOM (cgroup): the container exceeded its own resources.limits.memory. Only that container dies. This is 90% of what you'll see.
2. Node-level OOM: the whole node ran out of physical memory, and the kernel's OOM killer picked a victim — sometimes not even the pod that caused the pressure.

This playbook is the exact sequence I run to confirm which one you have and fix it for good, usually in a few minutes.

Step 1: Confirm it's actually OOMKilled

Don't guess from a restarting pod — read the terminated container's state:

kubectl describe pod payments-api-7d9f4c8b6-xk2mn

Look at the Last State block:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Wed, 15 Jul 2026 09:14:02
      Finished:     Wed, 15 Jul 2026 09:14:48

Reason: OOMKilled plus Exit Code: 137 is the signature. Note that a bare 137 without Reason: OOMKilled is different — that's usually a SIGKILL from a failing liveness probe, which is a probe problem, not a memory one. If you see that, the fix is in the liveness and readiness probes guide, not here.

The app logs are almost always useless for OOM because SIGKILL can't be caught — the process never logs its own death. So don't waste time in kubectl logs. The evidence lives in the container state and, for node-level kills, the kernel log.

Step 2: See the limit that was breached

You can't reason about a memory kill without knowing the ceiling it hit:

kubectl get pod payments-api-7d9f4c8b6-xk2mn \
  -o jsonpath='{.spec.containers[0].resources}'

{"limits":{"memory":"256Mi"},"requests":{"memory":"128Mi"}}

Now watch what the container actually consumes over time. You need metrics-server installed for this:

kubectl top pod payments-api-7d9f4c8b6-xk2mn --containers

POD                          NAME          CPU    MEMORY
payments-api-7d9f4c8b6-xk2mn payments-api  120m   248Mi

Sitting at 248Mi against a 256Mi limit is the whole story: this container lives one request away from death. The next question is why — and that's the fork that decides everything.

Step 3: The decisive fork — too-low limit or a leak?

Watch memory over a few minutes, not a single snapshot:

watch -n 5 'kubectl top pod payments-api-7d9f4c8b6-xk2mn --containers'

• Flat plateau just under the limit → the limit is genuinely too low. The app has a stable working set that simply doesn't fit. Fix: raise the limit (Step 4).
• Steady climb that never comes down, even under constant load → a memory leak. The app grows until it hits any ceiling you set. Fix: a bigger limit only buys hours; you have to find the leak (Step 5).

Getting this wrong wastes a day. If you raise the limit on a leaking app, it OOMs again the next night at 2 a.m. — same crash, bigger blast radius, because now it took more of the node down with it before dying.

Step 4: Right-size requests and limits

If the working set is legitimately larger than the limit, raise it — but set requests and limits deliberately, because they mean different things:

resources:
  requests:
    memory: "512Mi"   # what the scheduler reserves; guarantees placement
  limits:
    memory: "512Mi"   # the hard ceiling; exceed it and you're OOMKilled

Two rules that save real incidents:

• For memory, set requests == limits. Memory isn't compressible like CPU — you can't throttle it. When requests equals limits, the pod gets the Guaranteed QoS class and is the last thing the kubelet evicts under node pressure. If requests is much lower than limits (the Burstable class), the scheduler may overcommit the node, and your pod becomes a prime eviction target the moment the node gets tight.
• Leave headroom for the runtime, not just the heap. A JVM or Node process needs the container limit to hold the heap plus metaspace, thread stacks, and off-heap buffers. Set the runtime heap to roughly 75% of the container limit:

# Node.js: heap capped below the container limit
node --max-old-space-size=384 server.js      # ~384Mi heap under a 512Mi limit

# JVM: let it read the cgroup limit and cap the heap as a percentage
java -XX:MaxRAMPercentage=75.0 -jar app.jar

Setting -Xmx equal to the container limit is a classic self-inflicted OOM: the heap fills to the limit, then the first thread stack allocation pushes the container past its ceiling and the kernel kills it — even though the JVM heap wasn't technically full.

Right-sizing is also a cost lever, not just a stability one. Over-provisioned memory requests reserve capacity you pay for and never use, and that reservation is what pushes you onto extra nodes. The same discipline runs through Kubernetes cost optimization and FinOps for Kubernetes — right-sized limits keep pods alive and keep the bill down. If you'd rather not tune by hand, the VPA in recommendation mode will watch actual usage and suggest values; see how it fits with HPA and KEDA in the pod autoscaling guide.

Step 5: Hunt the leak

If memory climbs and never recovers, raising the limit is a stopgap. Confirm the trend first, then attack the source.

Confirm with the raw cgroup metric Prometheus exposes. container_memory_working_set_bytes is the number the kernel actually compares against the limit (it excludes reclaimable page cache), so it's the honest signal:

container_memory_working_set_bytes{pod=~"payments-api-.*", container="payments-api"}

A sawtooth that resets on each restart, with the peak creeping up run over run, is a leak's fingerprint. Common culprits by runtime:

• Node.js: unbounded caches, listeners added in a loop, closures holding large buffers. Capture a heap snapshot with node --inspect and diff two snapshots taken minutes apart in Chrome DevTools — the objects that grew are your leak.
• JVM: rising heap after full GC. Trigger a heap dump on OOM so the next kill leaves evidence:

java -XX:+HeapDumpOnOutOfMemoryError \
     -XX:HeapDumpPath=/dumps/heap.hprof -jar app.jar

Then open the .hprof in Eclipse MAT and run the leak-suspects report.

• Go: goroutine leaks (each holds a stack) or slices that keep a reference to a huge backing array. Hit /debug/pprof/heap with go tool pprof and look at inuse_space.

The point is the same everywhere: a leak is a code bug. Kubernetes can only contain it, not cure it, and every limit you set is just a delay timer.

Step 6: When the whole node runs out of memory

If describe shows OOMKilled but the container was nowhere near its own limit, you have node-level OOM. The node ran out of physical RAM — often because too many Burstable pods were overcommitted — and the kernel's OOM killer chose a victim by oom_score, which can be a bystander pod. Check the node:

kubectl describe node ip-10-0-1-42 | grep -A6 "Allocated resources"

If total memory Limits far exceed the node's allocatable RAM, you're overcommitted. You may also see the kubelet evicting pods before the kernel even acts — an Evicted status with The node was low on resource: memory. That's the kubelet trying to reclaim memory gracefully, and Guaranteed QoS pods survive it while BestEffort (no requests/limits at all) pods die first.

Fixes:

• Set memory requests on every pod so the scheduler stops overcommitting the node.
• Give critical workloads Guaranteed QoS (requests == limits) so they're evicted last.
• Add capacity or enable the cluster autoscaler so pressure triggers a scale-out instead of a massacre.

For the deepest confirmation, the kernel logs the kill on the node itself: dmesg -T | grep -i oom shows the Out of memory: Killed process ... line with the exact PID and RSS the kernel objected to.

A repeatable checklist

When a pod is OOMKilled, run this in order:

1. kubectl describe pod <pod> → confirm Reason: OOMKilled + Exit Code: 137 in Last State.
2. kubectl get pod <pod> -o jsonpath='{.spec.containers[0].resources}' → read the limit that was breached.
3. watch kubectl top pod <pod> --containers → flat plateau = raise the limit; steady climb = leak.
4. Too-low limit → set requests == limits, leave runtime headroom (MaxRAMPercentage=75, --max-old-space-size).
5. Leak → confirm with container_memory_working_set_bytes, then heap-dump the runtime and fix the code.
6. Container under its limit but still killed → node-level OOM; check overcommit and QoS, add requests, scale out.

Don't wait for the kill — alert on the approach

OOMKilled is one of the few pod failures you can see coming. When a container's working set is riding at 90% of its limit, it's a scheduled outage, not a surprise. Wire that into monitoring so it pages before the kernel acts:

container_memory_working_set_bytes{container!=""}
  / on(pod, container) kube_pod_container_resource_limits{resource="memory"}
  > 0.9

Alert on that ratio in the Prometheus + Grafana monitoring setup and you convert a 2 a.m. crash loop into a business-hours ticket. Bake the six-step diagnosis into your incident runbook so the next on-call engineer fixes it in minutes. Mis-set memory limits and hard OOMs are among the quiet Kubernetes mistakes that cost companies millions — cheap to prevent, expensive to sleep through.

Related Reading

• Kubernetes CrashLoopBackOff: How to Debug and Fix It — OOMKilled is one of several reasons a container crash-loops; this covers the full exit-code decision tree (1, 127, 143, 0) when the cause isn't memory.
• Kubernetes ImagePullBackOff: How to Debug and Fix It — the third member of the "my pod won't run" trilogy: the image never pulled, so the container never even started.
• Kubernetes Pod Autoscaling: HPA, VPA, and KEDA Explained — let the VPA recommend memory requests from real usage instead of tuning limits by hand every time.