How to Fix Kubernetes CrashLoopBackOff: Causes and Commands (2026)
This guide is for anyone staring at CrashLoopBackOff in kubectl get pods and wondering where to start. By the end you will be able to read the restart state, get the logs from the crash that actually happened, decode the exit code, match it to one of six common causes, and debug a container that dies before you can exec into it. Commands and field names are checked against the Kubernetes v1.37 documentation (the current release, with 1.35 and 1.36 also supported) and work with kubectl 1.32 or newer. You need kubectl access to the namespace with permission to read Pods, logs and events, and to create ephemeral containers for the debugging step.
What CrashLoopBackOff means
CrashLoopBackOff is not an error. It is the kubelet telling you it is waiting before restarting a container that keeps exiting. The Pod lifecycle docs describe the sequence: the first crash gets an immediate restart, later crashes get an exponential delay of 10s, 20s, 40s and so on, capped at 300 seconds (5 minutes), and once a container runs for 10 minutes without a problem the timer resets. While the kubelet is sitting in that delay, the container's state is Waiting with reason CrashLoopBackOff.
So the real question is always "why did the container exit?", and the answer lives in the previous container instance, not the current one.
Two newer knobs change the timing, and they matter if you are wondering why one cluster retries faster than another:
KubeletCrashLoopBackOffMaxwas alpha in 1.32 to 1.34 and has been beta and enabled by default since 1.35. It lets a node administrator lower the 5 minute cap per node withcrashLoopBackOff.maxContainerRestartPeriodin the kubelet configuration, anywhere from"1s"to"300s".ReduceDefaultCrashLoopBackOffDecayhas been alpha and off by default since 1.33. When enabled it changes the cluster default to start at 1s and cap at 60s, as proposed in KEP-4603.
On managed clusters (EKS, GKE, AKS) you usually cannot change kubelet configuration on managed nodes, so assume the 10s to 300s default unless you run your own node pools.
Step 1: Read the Pod's last state with kubectl describe
Start by confirming which container is crashing and how many times it has restarted.
kubectl get pods -n shop
# NAME READY STATUS RESTARTS AGE
# checkout-7d9f8b6c5-x2kqp 0/1 CrashLoopBackOff 6 (82s ago) 9m
Then describe it. Ignore most of the output and look for three things under the container: State, Last State and the Events at the bottom.
kubectl describe pod checkout-7d9f8b6c5-x2kqp -n shop
The part you care about looks like this:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 20 Oct 2026 09:14:02 +0000
Finished: Tue, 20 Oct 2026 09:14:03 +0000
Restart Count: 6
...
Events:
Warning BackOff 12s (x25 over 5m) kubelet Back-off restarting failed container checkout in pod checkout-7d9f8b6c5-x2kqp_shop(...)
Last State is the crash. Reason and Exit Code tell you which branch of the diagram you are on. Started and Finished tell you how long it lived: one second means it died during startup, several minutes usually means a probe or memory limit killed it.
For scripts or multi-container Pods, pull the same data with JSONPath:
kubectl get pod checkout-7d9f8b6c5-x2kqp -n shop -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'
# checkout 6 Error 1
Check initContainerStatuses the same way if the Pod status shows Init:CrashLoopBackOff: the failing container is an init container, and the main container has not started at all.
Step 2: Get the crashed container's logs with --previous
Plain kubectl logs shows the current instance, which during backoff is either not running or has just started and printed nothing useful. You want the previous instance, as the debugging docs show:
# Logs from the instance that crashed
kubectl logs checkout-7d9f8b6c5-x2kqp -n shop -c checkout --previous
# Last 50 lines only, with timestamps
kubectl logs checkout-7d9f8b6c5-x2kqp -n shop -c checkout --previous --tail=50 --timestamps
# Every container in the Pod, including sidecars
kubectl logs checkout-7d9f8b6c5-x2kqp -n shop --all-containers --previous
If the app writes its fatal error to a file rather than stdout, or exits before flushing logs, set terminationMessagePolicy: FallbackToLogsOnError on the container. Kubernetes then copies the last chunk of the log into the termination message, which shows up in describe as Message: under Last State (docs).
containers:
- name: checkout
image: ghcr.io/example/checkout:2.4.1
terminationMessagePolicy: FallbackToLogsOnError
Step 3: Decode the exit code
Exit codes above 128 mean the process was killed by a signal: subtract 128 to get the signal number.
| Exit code | Meaning | Usual Kubernetes cause |
|---|---|---|
| 0 | Clean exit | Process finished; with restartPolicy: Always it restarts forever |
| 1 | Generic application error | Unhandled exception, missing config, failed dependency |
| 126 | Command found but not executable | Wrong permissions, script without exec bit, wrong architecture |
| 127 | Command not found | Typo in command, binary missing from a slim image |
| 137 | SIGKILL (128 + 9) | OOMKilled, or liveness probe kill after grace period |
| 139 | SIGSEGV (128 + 11) | Native crash, often image built for the wrong CPU |
| 143 | SIGTERM (128 + 15) | Graceful kill, usually by a failed liveness probe |
The reason field disambiguates 137: OOMKilled means the kernel killed it for exceeding the memory limit; Error with a Liveness probe failed event means the kubelet killed it.
Step 4: Check events for probe and scheduling clues
Events expire (the API server keeps them for one hour by default), so check them early, or ship them to your log pipeline as I describe in Observability in 2026.
kubectl events -n shop --for pod/checkout-7d9f8b6c5-x2kqp --types=Warning
# LAST SEEN TYPE REASON OBJECT MESSAGE
# 40s Warning Unhealthy Pod/checkout-7d9f8b6c5-x2kqp Liveness probe failed: Get "http://10.0.3.17:8080/healthz": dial tcp 10.0.3.17:8080: connect: connection refused
# 40s Warning BackOff Pod/checkout-7d9f8b6c5-x2kqp Back-off restarting failed container checkout ...
Unhealthy with Liveness probe failed before every restart is the strongest single signal you will get. If you see FailedMount or CreateContainerConfigError instead, you do not have a crash loop: the container never started. That is a different problem, covered below.
Step 5: Match the cause and apply the fix
Missing config, secrets or environment variables
The most common exit-code-1 crash is an app that validates its config at startup and exits. The log line will say so (DATABASE_URL is not set, no such file /etc/app/config.yaml). Compare what the Pod gets with what the app expects:
# Environment as rendered in the Pod spec (values from Secrets show as references)
kubectl get pod checkout-7d9f8b6c5-x2kqp -n shop -o jsonpath='{.spec.containers[0].env}' | jq .
# Does the referenced Secret and key exist?
kubectl get secret checkout-db -n shop -o jsonpath='{.data}' | jq 'keys'
A useful distinction: if a referenced Secret or key is missing entirely, the kubelet usually refuses to start the container and the Pod shows CreateContainerConfigError, not CrashLoopBackOff. A crash loop means the variable existed but had the wrong value, or the app needed something you never wired up.
Liveness probe kills a slow-starting app
If the app takes 40 seconds to warm up and the liveness probe starts failing after 30, the kubelet kills it, it restarts, and it never finishes starting. The fix is a startup probe, which disables the liveness probe until it succeeds:
containers:
- name: checkout
image: ghcr.io/example/checkout:2.4.1
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30 # 30 x 10s = up to 300s to start
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
failureThreshold: 3
Also check the liveness endpoint does not call your database. A liveness probe that fails when a dependency is down turns a dependency outage into a restart storm across every replica.
OOMKilled (exit code 137)
Reason: OOMKilled means the container exceeded its memory limit. Check what it was allowed and what it used just before dying:
kubectl get pod checkout-7d9f8b6c5-x2kqp -n shop -o jsonpath='{.spec.containers[0].resources}'
kubectl top pod checkout-7d9f8b6c5-x2kqp -n shop --containers # requires metrics-server
Raise the limit if the workload genuinely needs it, or cap the runtime's heap (JVM -XX:MaxRAMPercentage, Node --max-old-space-size) so it stays under the limit. The dedicated OOMKilled exit code 137 guide covers this in depth, and requests and limits explained covers how to size them.
Wrong command, entrypoint or architecture (126, 127, 139)
These die within a second and often print a single line such as exec /app/server: no such file or directory or exec format error. Check what the Pod actually runs versus what the image expects:
kubectl get pod checkout-7d9f8b6c5-x2kqp -n shop -o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'
# Inspect the image's own entrypoint and platform without running it (crane v0.22)
crane config ghcr.io/example/checkout:2.4.1 | jq '{entrypoint: .config.Entrypoint, cmd: .config.Cmd, arch: .architecture}'
exec format error almost always means an arm64 image on amd64 nodes or the reverse. Build multi-arch images or pin node affinity.
The process exits 0 (Completed)
A container that does its work and exits cleanly, under a Deployment's restartPolicy: Always, will be restarted forever and land in CrashLoopBackOff with Reason: Completed. Either it should be a Job or CronJob, or the process is daemonising into the background (a shell script that starts a server with & and then ends). Run the server in the foreground with exec.
Permissions and security context
Hardened Pods (runAsNonRoot, readOnlyRootFilesystem: true) break images that write to /tmp or their own directory, with logs such as permission denied or read-only file system. Mount an emptyDir at the path the app writes to rather than loosening the security context.
Step 6: Debug a container that crashes too fast to exec into
kubectl exec needs a running container, which a crash-looping one rarely is. Ephemeral containers attach to a live Pod, so they help when the container stays up for a while between crashes:
# Add a busybox shell that shares the target container's process namespace
kubectl debug -it checkout-7d9f8b6c5-x2kqp -n shop --image=busybox:1.37 --target=checkout --profile=general
When it dies on startup, make a copy of the Pod with a different command, so the container stays up and you can run the real entrypoint by hand:
# Copy the Pod, replace the checkout container's command with a shell
kubectl debug checkout-7d9f8b6c5-x2kqp -n shop -it --copy-to=checkout-debug --container=checkout -- sh
# Inside the copy: check env, files and run the binary to see the real error
env | sort
ls -la /app /etc/app
/app/server --config /etc/app/config.yaml
# Distroless image with no shell? Add a debug container to a copy that shares
# the process namespace, then browse the app's filesystem via /proc/<pid>/root
kubectl debug checkout-7d9f8b6c5-x2kqp -n shop -it --image=busybox:1.37 --share-processes --copy-to=checkout-debug2
# Clean up the copies afterwards
kubectl delete pod checkout-debug checkout-debug2 -n shop
Note that --copy-to creates a standalone Pod with the same spec, Secrets and service account. Treat it like production, and delete it when done.
Since 1.35, container restart rules are beta and on by default: a container can set restartPolicy plus restartPolicyRules to restart only on specific exit codes. That is useful for batch work where exit code 42 means "retry" and anything else means "stop", but it does not fix a crash loop; it only changes which exits trigger one.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Exit code 1, log says a variable or file is missing | Config not wired into the Pod | Add the env var, ConfigMap or Secret key; check spelling and namespace |
Exit 137, Reason OOMKilled |
Memory limit too low or a leak | Raise resources.limits.memory or cap the runtime heap |
Exit 137 or 143, Liveness probe failed events |
Probe too aggressive for startup time | Add a startupProbe; keep liveness checks local to the process |
Exit 127, executable file not found |
Wrong command or missing binary |
Fix command/args; check the image entrypoint with crane config |
exec format error |
Image architecture does not match node | Build a multi-arch image or set node affinity |
Reason Completed, exit 0 |
Short-lived process under a Deployment | Use a Job/CronJob, or run the server in the foreground with exec |
Init:CrashLoopBackOff |
An init container is failing | kubectl logs POD -c INIT_NAME --previous |
--previous logs are empty |
Crash before logging, or logs written to a file | Set terminationMessagePolicy: FallbackToLogsOnError; use kubectl debug --copy-to |
permission denied or read-only file system |
runAsNonRoot or read-only root filesystem |
Mount an emptyDir where the app writes; fix file ownership in the image |
Shows CreateContainerConfigError, not a crash loop |
Referenced Secret or ConfigMap key does not exist | Create the object or key; then the Pod starts without a restart |
If the Pod never gets to a node at all, see Pod Pending troubleshooting; if the image cannot be pulled, see ImagePullBackOff and ErrImagePull.
FAQ
How long does CrashLoopBackOff wait between restarts?
By default the kubelet waits 10 seconds after the second crash and doubles the delay each time up to 5 minutes. The delay resets after the container runs cleanly for 10 minutes. Since Kubernetes 1.35 a node administrator can lower the cap with crashLoopBackOff.maxContainerRestartPeriod in the kubelet configuration.
How do I see logs from a container in CrashLoopBackOff?
Use kubectl logs POD -c CONTAINER --previous. The --previous flag returns the output of the instance that crashed rather than the one currently waiting to start. If that is empty, the process died before logging, so use kubectl debug --copy-to with a shell.
Will deleting the Pod fix CrashLoopBackOff?
Usually not. The controller creates a new Pod from the same spec, which crashes for the same reason, though it does reset the backoff timer. It only helps when the cause was transient, such as a dependency that has since recovered.
What is the difference between CrashLoopBackOff and Error?
Error is the reason recorded for a single non-zero exit. CrashLoopBackOff means the container has exited repeatedly and the kubelet is now waiting before the next restart. You will usually see Error under Last State and CrashLoopBackOff under State on the same container.
In production, the first thing I check is not the logs but Last State: the exit code and how long the container lived narrow it to one or two causes before I read a line of output. The second is whether the liveness probe touches anything outside the process, because that is the crash loop that takes down every replica at once. If you are weighing whether all of this operational surface is worth it for your team, my sibling post on whether Kubernetes is still worth it in 2026 covers that. Official references: Pod lifecycle and container restarts, Debug running Pods, Determine the reason for Pod failure and Configure liveness, readiness and startup probes.
Related: Kubernetes as the Agent Control Plane in 2026, Agent Observability in 2026 and It's 2 AM. Do You Know What Your AI Agent Is Doing?.
I'm Ajay Kumar — I build and operate PandaStack, an open-source Firecracker microVM cloud for AI agents, and write about the infrastructure and tooling around it.
Need this kind of infrastructure work? See what I do or email hello@ajayk.sh.
Related
How to Run Chaos Experiments on Kubernetes with Chaos Mesh (2026)
Step-by-step Chaos Mesh 2.8.5 guide: install with Helm on containerd, opt namespaces in, write PodChaos, NetworkChaos and StressChaos YAML, gate experiments on a Prometheus SLO, schedule them, and run the same test with a LitmusChaos promProbe, plus a troubleshooting table.
16 minOct 20, 2026How to Set Up Grafana Loki on Kubernetes with Alloy (2026)
Step-by-step Grafana Loki 3.7 setup on Kubernetes: the new grafana-community Helm chart, deployment modes, S3 storage, Grafana Alloy replacing end-of-life Promtail, labels vs structured metadata, LogQL, retention, and fixes for 429 and too many outstanding requests.
12 minOct 20, 2026How to Write a Blameless Postmortem: Template and Examples (2026)
A step-by-step guide to writing a blameless postmortem: when one is required, building the timeline from evidence, contributing factors instead of one root cause, why the 5 Whys misleads, owned action items, the review meeting, and a copy-paste markdown template.
15 min