Fix Kubernetes OOMKilled and Exit Code 137: Step-by-Step Guide (2026)

This guide is for anyone staring at a pod that keeps restarting with OOMKilled or exit code 137 and wondering whether to just double the memory limit. By the end you will know which of the three different "out of memory" paths you are on (container limit, node OOM, or kubelet eviction), how to measure memory the way the kernel counts it, how to size the JVM, Node.js, Go and Python runtimes to the container limit, and how to stop it recurring with requests, VPA and alerts. Commands and field names are checked against the Kubernetes v1.35 to v1.37 documentation on cgroup v2 nodes (cgroup v1 is no longer the default path since v1.35). You need kubectl v1.32 or newer and read access to the namespace; node-level steps need permission to run kubectl debug node. If you want the theory behind requests, limits and QoS first, read the companion explainer, Kubernetes requests and limits explained.

Step 1: Confirm it really is an OOMKill

Exit code 137 means the process received SIGKILL (128 + 9). The kernel OOM killer sends SIGKILL, but so does the kubelet when a container ignores SIGTERM past its grace period, for example after a failed liveness probe. So read the reason, not just the code.

# Last termination reason and exit code for every container in the pod
NS=default
POD=api-7d9c8-abcde
kubectl get pod -n "$NS" "$POD" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.restartCount}{"\n"}{end}'

Expected output for a genuine container OOM:

api	OOMKilled	137	5

kubectl describe pod "$POD" shows the same under Last State: Terminated, together with the limits, as in the official troubleshooting example. If the reason is Error with 137, skip to the liveness and grace period rows of the troubleshooting table. If the pod's status.reason is Evicted, you are on the eviction path in Step 2, not an OOM kill.

Step 2: Container limit OOM, node OOM, or eviction?

These three look similar in a dashboard and need different fixes. The decision flow:

Container limit OOM. The container's cgroup reached memory.max and the kernel could not reclaim enough. On cgroup v2 nodes the kubelet defaults singleProcessOOMKill to false and sets memory.oom.group, so every process in the container is killed together, per the Kubernetes blog's cgroup v2 post. The reason shows OOMKilled.

Node OOM. The whole node ran out before the kubelet could evict anything, and the kernel's global OOM killer picked the process with the highest oom_score. The kubelet biases that with oom_score_adj: -997 for Guaranteed, 1000 for BestEffort, and 2 to 999 for Burstable depending on the memory request (node-pressure eviction docs). The kubelet records a SystemOOM event on the Node object.

Eviction. The kubelet noticed memory.available crossing a threshold (default hard threshold memory.available<100Mi on Linux) and terminated pods, starting with those whose usage exceeds their requests. Evicted pods stay in Failed phase with reason Evicted.

# Evictions in the namespace, and SystemOOM events on nodes
kubectl get events -n "$NS" --field-selector reason=Evicted
kubectl get events -A --field-selector involvedObject.kind=Node,reason=SystemOOM

# Kernel log on the node where the pod ran (needs node debug permission)
NODE=$(kubectl get pod "$POD" -o jsonpath='{.spec.nodeName}')
kubectl debug node/"$NODE" -i --profile=sysadmin --image=busybox:1.37 -- \
  chroot /host journalctl -k --since "2 hours ago" | grep -iE "oom|out of memory|memory cgroup"

In the kernel log, a line mentioning memory cgroup or memcg means a container limit OOM; a plain Out of memory: Killed process without a cgroup reference means the node ran out, as Google's GKE OOM troubleshooting page describes. The kubectl debug docs note that the node's root filesystem is mounted at /host.

Step 3: Measure memory the way the kernel counts it

Your application's idea of memory (heap used) and the kernel's (everything charged to the cgroup: anonymous memory, page cache, kernel memory, tmpfs) are different numbers. Kubernetes evicts based on working set, which is usage minus inactive file cache, and kubectl top reports working set too.

# Per-container working set right now (needs metrics-server)
kubectl top pod "$POD" --containers

# Inside the container: the cgroup v2 files the kernel actually uses
kubectl exec "$POD" -c api -- sh -c '
  echo "max:     $(cat /sys/fs/cgroup/memory.max)"
  echo "current: $(cat /sys/fs/cgroup/memory.current)"
  echo "peak:    $(cat /sys/fs/cgroup/memory.peak 2>/dev/null || echo n/a)"
  grep -E "^(anon|file|shmem|kernel) " /sys/fs/cgroup/memory.stat
  cat /sys/fs/cgroup/memory.events'

In memory.events, the kernel cgroup v2 docs define oom as the number of times the cgroup hit its own limit and oom_kill as processes killed by any kind of OOM killer. A non-zero oom therefore points at the container limit, not the node. The counters reset when the container restarts into a new cgroup, so check soon after a kill. A big shmem number usually means a memory-backed emptyDir, which is charged to the container.

For history, use Prometheus with cAdvisor and kube-state-metrics (v2.20.0 at the time of writing):

# Working set as a fraction of the memory limit, per container
max by (namespace, pod, container) (container_memory_working_set_bytes{container!="", container!="POD"})
/ on (namespace, pod, container)
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory", unit="byte"})

A sawtooth that climbs to 1.0 and drops on each restart is a leak or an undersized limit. A flat line well below 1.0 followed by a sudden kill is a spike: a big request, a batch job, or a burst of concurrent work.

Step 4: Size the runtime to the container limit

Most "mystery" OOMKills are a runtime that does not know, or does not respect, its container limit.

Java. The JVM is container-aware by default (UseContainerSupport), and its default max heap is 25% of the limit, per the JDK 25 java man page. Checked locally on OpenJDK 21.0.7:

$ java -XX:+PrintFlagsFinal -version | grep -E "MaxRAMPercentage|ExitOnOutOfMemoryError"
     bool ExitOnOutOfMemoryError                   = false                                     {product} {default}
   double MaxRAMPercentage                         = 25.000000                                 {product} {default}

Heap is not the only consumer: metaspace, thread stacks, direct buffers and the JIT live outside it. A reasonable starting point for a service is 60 to 75% heap, then measure:

env:
- name: JAVA_TOOL_OPTIONS
  value: >-
    -XX:MaxRAMPercentage=70
    -XX:MaxMetaspaceSize=256m
    -XX:+ExitOnOutOfMemoryError
    -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps
    -XX:NativeMemoryTracking=summary
# Then compare heap vs native: kubectl exec $POD -- jcmd 1 VM.native_memory summary

Node.js. V8's heap limit is set at start. Node's CLI now offers --max-old-space-size-percentage, documented as a percentage of available system memory (Node CLI docs); the older --max-old-space-size takes megabytes. Always verify what you got, because the default on a laptop is not the default in a container. On my machine, Node v20.17.0 outside any container reports:

$ node -e "console.log(process.version, Math.round(require('v8').getHeapStatistics().heap_size_limit/1048576)+' MiB')"
v20.17.0 4144 MiB

Run the same one-liner via kubectl exec and make sure the number sits comfortably below the container limit. For a 1 GiB limit, NODE_OPTIONS=--max-old-space-size=700 leaves room for buffers and native modules.

Go. Set GOMEMLIMIT (available since Go 1.19, per the Go GC guide) to roughly 90% of the limit so the garbage collector works harder before the kernel steps in:

env:
- name: GOMEMLIMIT
  value: "900MiB"   # about 90% of a 1Gi memory limit; Go accepts B, KiB, MiB, GiB suffixes

Python. There is no heap cap to set. Memory scales with worker processes, so count them: four Gunicorn workers each holding a 300 MiB model is 1.2 GiB before a single request. Use fewer workers per pod and more pods, and for slow leaks Gunicorn's --max-requests with --max-requests-jitter recycles workers.

Step 5: Decide whether it is a leak or a sizing problem

Raise the limit only after you know which one you have. If working set plateaus under load and the kill happens at a predictable peak, it is sizing; raise request and limit together. If it climbs steadily across hours regardless of traffic, it is a leak, and a bigger limit only moves the restart further apart.

# Java: heap dump from a running pod (pod must have a writable volume at /dumps)
kubectl exec "$POD" -c api -- jcmd 1 GC.heap_dump /dumps/heap.hprof
kubectl cp "$NS/$POD:/dumps/heap.hprof" ./heap.hprof -c api

# Node.js: start with --heapsnapshot-signal=SIGUSR2, then trigger a snapshot
kubectl exec "$POD" -c api -- kill -USR2 1

# Go: if net/http/pprof is enabled on :6060
kubectl port-forward "$POD" 6060:6060 &
go tool pprof -top http://localhost:6060/debug/pprof/heap

# Python: compare tracemalloc snapshots taken minutes apart (stdlib, no install)
# import tracemalloc; tracemalloc.start(); ... snapshot.compare_to(old, "lineno")[:10]

Take two snapshots some minutes apart and diff them; the growing allocation site is your bug. Node's --heapsnapshot-signal has existed since v12, and --heapsnapshot-near-heap-limit can capture one automatically as the heap fills.

Step 6: Right-size requests and limits

Once the runtime is configured and leaks are ruled out, set memory honestly:

resources:
  requests:
    memory: "768Mi"   # p95 working set over a week, plus headroom
  limits:
    memory: "768Mi"   # equal to the request: no overcommit, predictable eviction rank

Equal request and limit keeps the pod out of the "usage exceeds request" group that the kubelet evicts first. For recommendations, run the Vertical Pod Autoscaler with updateMode: "Off" and read its target; its InPlaceOrRecreate mode is GA since VPA 1.6.0 if you want it to apply changes, per the VPA features doc. To adjust a running pod without recreating it, in-place resize is GA since Kubernetes 1.35:

kubectl patch pod "$POD" --subresource resize --patch \
  '{"spec":{"containers":[{"name":"api","resources":{"requests":{"memory":"1Gi"},"limits":{"memory":"1Gi"}}}]}}'
kubectl get pod "$POD" -o jsonpath='{.status.conditions[?(@.type=="PodResizePending")]}'

The resize docs warn that memory limit decreases are best-effort and the QoS class cannot change. Also update the Deployment, or the next rollout reverts your fix. Cost data says most teams pad memory heavily rather than tightly; Cast AI's 2026 report puts average memory utilization at 20%, so check whether the pod that keeps dying is the exception before padding everything else further.

Step 7: Reproduce it and alert on it

The Kubernetes docs ship a reproducer: a container with a 100Mi limit that tries to allocate 250M.

kubectl create namespace mem-example
kubectl apply -n mem-example -f https://k8s.io/examples/pods/resource/memory-request-limit-2.yaml
kubectl get pod memory-demo-2 -n mem-example -w   # expect OOMKilled, then CrashLoopBackOff
kubectl delete namespace mem-example

Then alert on OOM kills, not just restarts:

# PrometheusRule fragment (prometheus-operator CRD monitoring.coreos.com/v1)
- alert: ContainerOOMKilled
  expr: |
    increase(kube_pod_container_status_restarts_total[15m]) > 0
    and on (namespace, pod, container)
    kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
  for: 0m
  labels: { severity: warning }
  annotations:
    summary: "{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) was OOMKilled"

Repeated OOMKills turn into CrashLoopBackOff; my CrashLoopBackOff guide covers the backoff mechanics. For general node memory triage, the 60-second Linux checklist is the first thing I run on the node.

Troubleshooting

Symptom	Likely cause	Fix
OOMKilled, exit 137, working set climbs steadily for hours	Memory leak	Heap dump or pprof diff (Step 5); fix code before raising limit
OOMKilled right after start	Limit below startup footprint, or JVM/Node heap set above the limit	Measure startup peak (memory.peak), lower heap percentage or raise limit
Java OOMKilled but heap looks fine	Native memory: metaspace, threads, direct buffers	-XX:NativeMemoryTracking=summary, cap metaspace, reduce heap to 60 to 70%
Exit 137 with reason Error	Kubelet SIGKILL after failed liveness probe or long shutdown	kubectl describe pod events; fix probe timeouts or terminationGracePeriodSeconds
Pod Failed with reason Evicted	Node memory pressure, pod using more than its request	Set request near real usage; priority classes; check node reserves
Several unrelated pods die at once, SystemOOM node event	Node OOM before kubelet eviction	Add system-reserved/kube-reserved, raise eviction thresholds, stop overcommitting limits
Whole multi-process container restarts after upgrade	cgroup v2 memory.oom.group kills all processes together	Expected default; fix the memory hog, or set kubelet singleProcessOOMKill: true node-wide
shmem large in memory.stat	Memory-backed emptyDir counted against the limit	Set sizeLimit, move to disk-backed volume, or raise limit
OOMKills started after node image change	Different cgroup version or runtime on new nodes	Confirm cgroup v2 (stat -fc %T /sys/fs/cgroup shows cgroup2fs) and runtime versions
Resize to more memory stuck	PodResizePending with reason Deferred or Infeasible	Free node capacity, raise priority, or recreate the pod on a larger node

FAQ

What does exit code 137 mean in Kubernetes?

Exit code 137 means the container's main process was killed with SIGKILL (128 plus signal 9). If lastState.terminated.reason is OOMKilled, the kernel killed it for exceeding memory; if the reason is Error, the kubelet most likely killed it after a failed liveness probe or an expired grace period.

What is the difference between OOMKilled and Evicted?

OOMKilled is the kernel enforcing a memory limit (or a node-wide OOM) and the kubelet restarts the container in the same pod. Evicted is the kubelet proactively terminating a whole pod because the node crossed a memory threshold; the pod is not restarted and its controller creates a replacement elsewhere.

Should I just increase the memory limit?

Only after you know it is a sizing problem rather than a leak. If working set plateaus near the limit under normal load, raise request and limit together. If it grows without bound, a bigger limit only delays the next kill.

Why does my Java app get OOMKilled when the heap is not full?

Because the container limit covers all memory, not just the heap: metaspace, thread stacks, direct buffers, the JIT and native libraries all count. Leave 25 to 40% of the limit for non-heap memory and use Native Memory Tracking to see where it goes.

When an OOMKill pages me, I check two things before anything else: the lastState reason, to know which of the three paths I am on, and the working-set graph against the limit over the past day, to tell a leak from a spike. Most of the time the fix is a runtime flag or an honest request, not more memory. The official references are the Kubernetes pages on managing container resources, node-pressure eviction, Pod QoS classes and resizing container resources.

Related: Kubernetes Requests and Limits Explained, Observability in 2026 and Kubernetes as the Agent Control Plane in 2026.