How to Fix Kubernetes Readiness and Liveness Probe Failed Errors (2026)

This guide is for anyone looking at a Pod that shows 0/1 READY, or a restart count that keeps climbing, with Readiness probe failed or Liveness probe failed in its events. By the end you will be able to read the exact probe error, map it to one of four causes (connection refused, timeout, a bad HTTP status, or a slow start), reproduce the check from inside the Pod, and fix the probe or the app with the right timing fields. Field names and behaviour are checked against the Kubernetes v1.37 documentation (current patch v1.37.2); the commands work with kubectl 1.32 or newer. You need kubectl access to read Pods and events in the namespace, plus permission to exec or create ephemeral containers for the testing step. If you want the background on what each probe is for first, read the companion explainer on liveness vs readiness vs startup probes.

Step 1: Read the probe failure event

Start with the Pod's state and its events. A readiness failure leaves the Pod Running but not ready; a liveness or startup failure adds restarts.

# READY 0/1 with RESTARTS 0 is a readiness problem; climbing RESTARTS is liveness or startup
kubectl get pods -n shop -l app=api

# Events for one Pod (kubectl 1.32+); --types narrows to warnings
kubectl events -n shop --for pod/api-7c9d8b6f4-x2kqp --types=Warning

# The classic view, which also shows the probe configuration the kubelet is using
kubectl describe pod -n shop api-7c9d8b6f4-x2kqp

The kubelet records probe failures as Warning events with reason Unhealthy, and the message is <Probe type> probe failed: <output> (prober.go in v1.37.2). When liveness or startup gives up, it adds a Killing event saying Container api failed liveness probe, will be restarted. A typical block looks like this (illustrative Pod, real message formats):

Events:
  Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Warning  Unhealthy  12s (x9 over 92s)   kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  Unhealthy  40s (x3 over 60s)   kubelet  Liveness probe failed: Get "http://10.42.1.17:8080/livez": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    40s                 kubelet  Container api failed liveness probe, will be restarted

Also print the probe as the API server stored it, because what is in Git is not always what is running:

kubectl get pod -n shop api-7c9d8b6f4-x2kqp \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{.readinessProbe}{"\n"}{.livenessProbe}{"\n"}{.startupProbe}{"\n"}{end}'

Fields you leave out take their defaults: periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3, successThreshold: 1, initialDelaySeconds: 0 (configuration fields). A one second timeout is the default that bites most often.

Step 2: Match the error message to the cause

For HTTP and TCP probes, the kubelet passes the Go error string straight into the event (http.go). Any status outside 200 to 399 becomes HTTP probe failed with statuscode: N, and the response body is deliberately left out of the message. I reproduced the three network cases with a small Go 1.26 program that uses the same http.Client timeout and TCP dialer settings as the kubelet; this is the real output:

STATUS: 503
ERR: Get "http://127.0.0.1:18080/slow": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
ERR: Get "http://127.0.0.1:18081/ready": dial tcp 127.0.0.1:18081: connect: connection refused
TCP: dial tcp 10.255.255.1:8080: i/o timeout

So the event text tells you which family you are in:

• connect: connection refused: nothing is listening on that IP and port. Go to Cause 1.
• context deadline exceeded (Client.Timeout exceeded while awaiting headers) or i/o timeout: something is listening, or packets vanish, but no answer arrived within timeoutSeconds. Go to Cause 2.
• HTTP probe failed with statuscode: 503 (or 404, 401, 500): the app answered and said no. Go to Cause 3.
• Restarts that happen only in the first minute or two after start, with any of the above: the app is being killed while it boots. Go to Cause 4.

Exec probes report their command output, or command timed out when they exceed timeoutSeconds. gRPC probes have their own messages, such as service unhealthy (responded with "NOT_SERVING") or timeout: failed to connect service, defined in grpc.go.

Step 3: Test the probe from inside the Pod

The kubelet probes the Pod IP, not localhost and not the Service. That one fact explains a lot of "it works when I curl it" confusion. Reproduce what the kubelet does:

# Get the Pod IP and the port the probe targets
POD=api-7c9d8b6f4-x2kqp
kubectl get pod -n shop $POD -o jsonpath='{.status.podIP}{"\n"}'

# If the image has a shell and wget/curl, test against the Pod IP with a 1s timeout like the kubelet
kubectl exec -n shop $POD -c api -- sh -c 'wget -S -T 1 -O- http://$(hostname -i):8080/readyz'

# Distroless or no shell: attach an ephemeral debug container to the same network namespace
kubectl debug -n shop -it $POD --image=busybox:1.38.0 --target=api -- sh
# inside the debug shell:
#   wget -S -T 1 -O- http://POD_IP:8080/readyz     # HTTP status and timing
#   nc -w 1 POD_IP 8080 </dev/null && echo open    # TCP probe equivalent
#   netstat -tln                                    # what is actually listening, and on which address

If netstat shows the app listening on 127.0.0.1:8080 rather than 0.0.0.0:8080 or :::8080, you have found a connection refused cause: the kubelet cannot reach a loopback-only listener. If the request takes longer than a second, time it a few times; an answer that takes 800 ms when idle will fail a 1 second timeout under load.

NetworkPolicies are rarely the cause. The NetworkPolicy docs state that traffic to and from the node a Pod runs on is always allowed, and the kubelet probes from the node. CNI plugins can behave differently in edge cases, so if everything else checks out, look at your CNI's documentation next.

Cause 1: connection refused

Nothing accepted the connection on the Pod IP and the probe's port. Work through these in order:

1. Wrong port. Compare the probe's port with what the process actually listens on. Named ports must match a containerPort name in the same container, and gRPC probes cannot use named ports at all.
2. Loopback-only listener. Many frameworks default to 127.0.0.1 in development mode. Bind to 0.0.0.0 (or set the framework's host variable).
3. Not started yet. If refusals appear only in the first seconds after start and then stop, the app simply is not listening yet. For readiness that is harmless noise. For liveness it is Cause 4.

Cause 2: timeouts (context deadline exceeded, i/o timeout)

The connection was accepted, or packets went nowhere, and no response arrived within timeoutSeconds. The default is 1 second, and the kubelet counts a slow answer as a failure.

The usual culprits are a health handler that calls a database or downstream service, a container at its CPU limit being throttled (see requests and limits explained), garbage collection pauses, or a single-threaded server busy with real requests so the probe waits in the queue. i/o timeout on a TCP probe means the connection attempt itself got no answer, which more often points at the network path dropping packets or a listen backlog that is full.

Fix the handler first, then the numbers:

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 3        # default is 1; set from measured p99 of the handler, with headroom
  failureThreshold: 3      # ~15s of consecutive failures before leaving endpoints
livenessProbe:
  httpGet:
    path: /livez           # process-only check, no dependency calls
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 5      # tolerate ~50s before a restart; restarts are expensive

Raising timeoutSeconds without making the handler cheaper only moves the failure to a busier day.

Cause 3: HTTP probe failed with statuscode 503, 404 or 401

The app answered, and the status was outside 200 to 399.

• 404: the path is wrong, often a framework prefix such as /api/healthz or a trailing slash. Check with the wget command from Step 3.
• 401 or 403: authentication middleware is applied to the health route. Exempt the probe paths. The kubelet sends User-Agent: kube-probe/1.37 and Accept: */*, and you can add headers with httpHeaders, but you should not need credentials for a health check.
• 503: the app is reporting itself not ready. That is the readiness probe working as designed, so the question is why. Read what the handler checks. If a shared dependency is down, every replica goes unready at once, and the Service has no endpoints. If that is happening on the liveness probe, remove the dependency check from it; the probes concept page warns that liveness probes done wrong cause cascading failures.

Two side effects to know: a redirect to a different hostname counts as success with a ProbeWarning event, and the kubelet stops reading the body after 10KiB, which can leave connection reset by peer lines in your app logs if the health endpoint returns a large payload.

Cause 4: the app is killed while it starts

If restarts cluster in the first minutes after each start, liveness (or a too-short startup probe) is firing before the app is ready. The docs' rule: if startup can take longer than initialDelaySeconds + failureThreshold x periodSeconds, add a startup probe on the same endpoint and leave liveness at normal values. Liveness and readiness do not run until it succeeds.

startupProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  failureThreshold: 30     # 30 x 10s = 300s worst-case startup budget
livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  failureThreshold: 3      # unchanged; fast deadlock detection after startup

Measure how long startup really takes before picking the number:

# Container start time vs the time the Pod became Ready
kubectl get pod -n shop $POD -o jsonpath='{.status.containerStatuses[0].state.running.startedAt}{"\n"}{.status.conditions[?(@.type=="Ready")].lastTransitionTime}{"\n"}'

If the startup probe itself fails, you will see Startup probe failed: events and then Container api failed startup probe, will be restarted. Raise failureThreshold until it covers the slowest start you have measured. Repeated kills from either probe lead to CrashLoopBackOff, which the CrashLoopBackOff guide covers.

Exec and gRPC probes

For exec probes, run the exact command yourself and check the exit code. The timeoutSeconds limit applies (enforced since the ExecProbeTimeout behaviour became the default in v1.20), so a script that takes two seconds fails a default probe with command timed out.

kubectl exec -n shop $POD -c api -- sh -c '/bin/healthcheck.sh; echo "exit=$?"'

For gRPC probes, the server must implement grpc.health.v1.Health and listen on the Pod IP over plaintext unless you are on a v1.37 cluster with the alpha GRPCContainerProbeTLS gate. Test with grpcurl from an ephemeral container (the alpine image has a shell):

kubectl debug -n shop -it $POD --image=fullstorydev/grpcurl:v1.9.3-alpine --target=api -- sh
# inside: service name must match the probe's "service" field, if set
grpcurl -plaintext -d '{"service":"readiness"}' POD_IP:9090 grpc.health.v1.Health/Check

Troubleshooting

Symptom	Likely cause	Fix
connect: connection refused	Wrong port, or app listens on 127.0.0.1	Match probe port to the real listener; bind to 0.0.0.0
Refused only for the first seconds	App not listening yet	Harmless on readiness; add a startupProbe if liveness restarts it
Client.Timeout exceeded while awaiting headers	Handler slower than timeoutSeconds (default 1)	Make the handler cheap; set timeoutSeconds from measured latency
Timeouts only under load	CPU throttling or request queueing	Raise CPU request or remove the limit; give health checks their own thread
statuscode: 404	Wrong path or prefix	Fix path; test with wget against the Pod IP
statuscode: 401 or 403	Auth middleware on health route	Exempt /livez and /readyz from auth
statuscode: 503 on every replica at once	Readiness depends on a shared dependency that is down	Decide if that is intended; never put it in liveness
Restarts only during boot	Liveness fires before startup finishes	Add startupProbe with failureThreshold x periodSeconds above worst-case start
Pod 0/2 READY but the app is fine	A sidecar's readiness probe is failing	kubectl describe the sidecar container; sidecar readiness counts toward Pod Ready
command timed out on exec probe	Script exceeds timeoutSeconds	Speed up the script or raise timeoutSeconds

FAQ

Why does my Pod show Running but 0/1 READY?

The container is up but its readiness probe is failing, so Kubernetes removed the Pod from Service endpoints without restarting it. Run kubectl events --for pod/NAME to see the probe output, then test the same request against the Pod IP from inside the Pod.

Does a failed readiness probe restart the container?

No. Only liveness and startup probe failures cause restarts. A readiness failure sets the Pod's Ready condition to false and keeps probing; traffic returns once it passes successThreshold times.

What timeoutSeconds should I use for probes?

The default of 1 second is tight for anything that does real work. Measure the health handler's latency under load and set the timeout comfortably above its p99, typically 2 to 5 seconds, while keeping the handler itself cheap.

Why does the probe fail when curl inside the container works?

You probably tested localhost. The kubelet probes the Pod IP, so an app listening only on 127.0.0.1 passes your curl and fails the probe. Test against $(hostname -i) or the Pod IP instead.

In production, the first thing I check is whether the restarts are coming from readiness (they cannot), liveness or startup, and the second is whether the liveness handler makes any network call, because that is the version of this problem that takes every replica down together. After that it is almost always the 1 second timeout. Official references: Configure liveness, readiness and startup probes, Liveness, readiness and startup probes, Pod lifecycle and Sidecar containers.

Related: How to Fix Kubernetes CrashLoopBackOff: Causes and Commands (2026), Fix Kubernetes OOMKilled and Exit Code 137: Step-by-Step Guide (2026) and How to Troubleshoot Kubernetes DNS and CoreDNS: Step by Step (2026).