Kubernetes Probes Explained: Liveness vs Readiness vs Startup (2026)
One of the more embarrassing ways to take down a service on Kubernetes is to have a perfectly healthy application killed by its own health checks. Kubernetes probes look like three copies of the same YAML block, but liveness, readiness and startup probes answer different questions and do very different things when they fail: one restarts your container, one takes it out of the load balancer, and one holds the other two off while the app boots. This post explains the three Kubernetes probes as they work in v1.37, the four check mechanisms (including gRPC and the two new alpha options), the timing arithmetic that decides how fast Kubernetes reacts, the anti-patterns that cause cascading restarts, and how probes interact with sidecars and graceful shutdown. If you are already looking at a Readiness probe failed event, the sibling guide on fixing readiness and liveness probe failures is the faster read.
What do liveness, readiness and startup probes actually do?
A probe is a diagnostic the kubelet runs against a container on a schedule. The probes concept page is precise about what each one controls, and the differences are the whole point:
| Probe | Question it answers | What happens on failure | When it runs | Source |
|---|---|---|---|---|
startupProbe |
Has the app finished starting? | kubelet kills the container; restart policy applies | Only at startup; liveness and readiness wait until it succeeds | Probes concept |
livenessProbe |
Is the process stuck beyond recovery? | kubelet restarts the container | Periodically, for the container's life | Probes concept |
readinessProbe |
Should this Pod get traffic right now? | Pod Ready becomes false; IP removed from EndpointSlices; no restart |
Periodically, for the container's whole life | Probes concept |
| No probe defined | n/a | kubelet treats the result as Success |
n/a | Probe results |
Two details trip people up. First, liveness and readiness do not wait for each other. A container can be failing readiness and still be probed for liveness, and the docs say plainly that if you want to delay either, you use initialDelaySeconds or a startup probe. Second, readiness is not just a boot-time gate. It runs for the life of the container, so a Pod can drop out of a Service at 3 PM on a Tuesday because its readiness endpoint started returning 503, and nothing restarts.
The docs also make a point that sounds obvious but is widely ignored: if your process already crashes on its own when it hits an unrecoverable problem, you may not need a liveness probe at all. The kubelet restarts crashed containers according to restartPolicy anyway. A liveness probe earns its place only when the process can be alive and useless at the same time, the classic case being a deadlock.
httpGet, tcpSocket, exec and gRPC: which check mechanism should you use?
Each probe must define exactly one of four mechanisms. The check mechanisms section defines success for each:
httpGetsends a GET to the Pod IP. Any status from 200 up to 399 is success. The kubelet addsUser-Agent: kube-probe/<version>andAccept: */*, follows redirects only to the same host, and treats a redirect to a different host as success with aProbeWarningevent. It also stops reading the body after 10KiB, which can show up as "connection reset by peer" in your app logs if your health endpoint returns a large payload.tcpSocketsucceeds if the port accepts a connection. That proves something is listening, nothing more.execruns a command in the container and succeeds on exit code 0. The docs warn that exec probes fork processes on every run, which adds node CPU overhead on dense nodes with short periods.grpccalls the standard gRPC Health Checking Protocol and succeeds when the response status isSERVING. It has been stable since v1.27, so the old pattern of shippinggrpc_health_probein the image and calling it from an exec probe is no longer necessary.
gRPC probes have a few sharp edges. You cannot use a named port, you cannot set a custom hostname, and the probe connects to the Pod IP, so a server listening only on localhost fails. The service field lets one health endpoint answer differently per probe type, for example liveness and readiness.
Kubernetes v1.37, released in August 2026 and currently at v1.37.2, added two alpha options. H2CContainerProbe adds protocol: HTTP2 to httpGet for apps that serve health checks only over HTTP/2 cleartext, and GRPCContainerProbeTLS adds mode: TLS to grpc probes. Both are off by default and need the gate on the API server and the kubelet. The gRPC TLS mode skips certificate verification, as HTTPS probes always have.
My default is httpGet for HTTP services, grpc for gRPC services, and exec only when there is no network endpoint to check. tcpSocket is fine for a database sidecar or a proxy where "the port is open" really is the useful signal.
Probe timing math: how long until Kubernetes acts?
Every probe shares the same timing fields, and the defaults from the configuration fields reference are worth memorising:
| Field | Default | Minimum | Notes |
|---|---|---|---|
initialDelaySeconds |
0 | 0 | Delay after container start before the first probe |
periodSeconds |
10 | 1 | Readiness may run off-schedule while the Pod is not Ready |
timeoutSeconds |
1 | 1 | A slow answer is a failure, not a pass |
successThreshold |
1 | 1 | Must be 1 for liveness and startup |
failureThreshold |
3 | 1 | Consecutive failures before action |
terminationGracePeriodSeconds |
inherits Pod value (30) | 1 | Probe-level override, liveness and startup only |
The reaction time once an app breaks is roughly failureThreshold x periodSeconds, plus up to timeoutSeconds for the final attempt. With defaults that is about 30 seconds for liveness and the same for readiness. The startup budget is the formula the docs give: if your container usually needs more than initialDelaySeconds + failureThreshold x periodSeconds to start, add a startup probe. The task page example uses failureThreshold: 30 and periodSeconds: 10, a 300 second budget, then lets a strict failureThreshold: 1 liveness probe take over.
What people forget is what happens after liveness fires. The kubelet stops the container using the grace period (30 seconds by default unless you set a probe-level terminationGracePeriodSeconds), then the restart is subject to the CrashLoopBackOff delay, which doubles from 10 seconds to a cap of 300 seconds. A trigger-happy liveness probe does not give you a fast restart; it gives you a slow, exponentially growing outage.
Why your liveness probe should not check the database
The probes page carries a caution box that says incorrect liveness probes can lead to cascading failures: restarts under high load, failed requests, and more load on the Pods that survive. The commonest way to get there is a /healthz handler that pings the database, the cache and two downstream APIs.
Walk it through. The database has a 20 second hiccup. Every replica's liveness probe fails at the same time, because they all depend on the same thing. Every replica restarts. The database recovers into a connection storm from every Pod booting at once, while your Service has zero ready endpoints. A restart cannot fix a dependency outage; it can only add one of your own.
The rule I use, which matches what the docs recommend for apps with strict back-end dependencies:
- Liveness checks only the process itself. Can the HTTP server accept a request and return? Is the event loop or worker pool alive? No network calls.
- Readiness may check the dependencies this instance cannot serve without, because the right reaction to "my database is gone" is "stop sending me traffic", not "kill me".
- Both should be cheap. A health handler that takes a lock, runs a query or allocates a lot will time out under exactly the load where you need it to answer, and
timeoutSecondsdefaults to 1.
Even readiness dependency checks need thought: if every replica checks the same shared database, one outage empties the Service, and clients get load balancer errors instead of a fast 503. Often it is better to stay ready and fail requests quickly.
Other anti-patterns I see regularly: identical liveness and readiness probes with identical thresholds (the docs suggest the same cheap endpoint is fine, but with a higher failureThreshold on liveness so the Pod goes unready before it gets killed); a huge initialDelaySeconds on liveness to cover a slow boot, which delays deadlock detection for the life of the container; and probes that time out because the container is CPU-throttled at its limit, which is a resources problem rather than a probe problem (my post on Kubernetes requests and limits covers that side).
When to use a startup probe instead of initialDelaySeconds
initialDelaySeconds is a fixed wait. If you set it to 120 because the app sometimes takes two minutes to warm its caches, liveness does nothing for the first two minutes of every start, even on the days it boots in eight seconds. A startup probe is a budget instead: it checks every periodSeconds, succeeds as soon as the app answers, and only then hands over to liveness and readiness.
The docs' guidance is to point the startup probe at the same endpoint as liveness, keep the liveness probe at its normal values, and size the startup probe's failureThreshold to cover the worst-case start. That gives a slow JVM or model server room to boot without weakening deadlock detection. If you have ever traced a CrashLoopBackOff to a liveness probe firing during boot, this is the fix, and the CrashLoopBackOff guide shows how to confirm that from the events.
How probes interact with sidecars and graceful shutdown
Native sidecar containers (init containers with restartPolicy: Always) have been stable since v1.33, and they change probe behaviour in two ways. A sidecar's readinessProbe counts toward the Pod's Ready state, so a proxy sidecar that is not ready keeps the whole Pod out of the Service. And a sidecar's startupProbe gates the next init container: per the sidecar docs, the kubelet moves on once the sidecar's startup probe succeeds, which is how you make "start the proxy, wait until it works, then start the app" reliable without sleep loops.
Shutdown is where readiness gets misunderstood. You do not need a readiness probe for Pod deletion to drain traffic. When a Pod starts terminating, the termination flow marks its endpoints as terminating with ready: false while the kubelet runs any preStop hook and then sends SIGTERM. The EndpointSlice conditions also expose serving, which still tracks the Pod's readiness during termination so that load balancers can drain properly.
The catch is that these updates are asynchronous. kube-proxy, your ingress controller and any external load balancer learn about the change some time after the kubelet starts shutting the Pod down. If the app exits on SIGTERM immediately, requests routed during that gap fail. That gap is the motivation behind KEP-3960, the native sleep action for preStop, which went stable in v1.34. It replaces the old exec: ["sleep", "10"] trick, which needed a sleep binary in the image. Size terminationGracePeriodSeconds to cover the sleep plus your app's own drain time, because the grace period clock starts when deletion starts, not when SIGTERM arrives.
Liveness kills also honour the grace period, so a queue worker with an hour-long grace period would sit stuck for an hour before restarting. The probe-level terminationGracePeriodSeconds (stable since v1.28) exists for exactly this, and it cannot be set on readiness probes.
A probe configuration I would start from
This is the shape I would put on a typical HTTP service and then tune from metrics. It uses only stable fields, so it works on any supported release (1.34 to 1.37, all patched in October 2026).
# Deployment pod template excerpt; Kubernetes 1.34+ for preStop sleep
spec:
terminationGracePeriodSeconds: 45 # covers preStop sleep + app drain
containers:
- name: api
image: ghcr.io/example/api:1.8.2 # pin your own tag or digest
ports:
- name: http
containerPort: 8080
startupProbe: # boot budget: 30 x 5s = 150s
httpGet: { path: /livez, port: http }
periodSeconds: 5
failureThreshold: 30
livenessProbe: # process only, no dependency calls
httpGet: { path: /livez, port: http }
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3 # about 30s of failure before restart
terminationGracePeriodSeconds: 10 # do not wait 45s to kill a stuck process
readinessProbe: # may check required dependencies
httpGet: { path: /readyz, port: http }
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2 # leave endpoints after about 10s
lifecycle:
preStop:
sleep: { seconds: 10 } # let endpoint removal propagate
The split endpoints matter more than the numbers. /livez should be trivially cheap; /readyz is where you put "can I serve right now". If your cluster enforces NetworkPolicies, note that the docs state traffic from the node a Pod runs on is always allowed, which is where kubelet probes come from, so a default-deny policy from my NetworkPolicy guide does not block them.
Model servers and agent runtimes that load weights for minutes and spike CPU during inference are exactly where a 1 second timeout restarts a healthy Pod. Measure startup and handler latency, then set the numbers from the measurements.
FAQ
What is the difference between liveness and readiness probes?
A failed liveness probe makes the kubelet restart the container. A failed readiness probe only marks the Pod not ready, which removes its IP from Service EndpointSlices until the probe passes again; nothing restarts. Use liveness for stuck processes and readiness for temporary inability to serve.
Do I need a liveness probe at all?
Not always. If the process exits by itself when it hits an unrecoverable error, the kubelet already restarts it according to restartPolicy. Add a liveness probe when the process can hang while staying alive, such as a deadlock, and keep it free of dependency checks.
What are the default probe timings in Kubernetes?
periodSeconds 10, timeoutSeconds 1, failureThreshold 3, successThreshold 1 and initialDelaySeconds 0. That means a probe needs about 30 seconds of consecutive failures before Kubernetes acts. The probe-level grace period inherits the Pod's terminationGracePeriodSeconds, which defaults to 30.
Should I use startupProbe or initialDelaySeconds?
Use a startup probe for slow or variable startup. It succeeds as soon as the app is up and then hands over to liveness, while initialDelaySeconds waits the full time on every start. Size it so failureThreshold x periodSeconds covers your worst-case boot.
This quarter I would grep every Deployment in the cluster for liveness probes whose handler makes a network call, split them into /livez and /readyz, replace any long liveness initialDelaySeconds with a startup probe, and add a preStop sleep to anything behind an ingress. None of that is glamorous, and all of it removes a class of self-inflicted outage. When something does fail, the probe failure troubleshooting guide walks through the events and commands.
Related: How to Fix Kubernetes CrashLoopBackOff: Causes and Commands (2026), Kubernetes Requests and Limits Explained: 8% CPU Used, Resize GA in 1.35 and Is Kubernetes Still Worth It in 2026? v1.37, 82% in Production, Real Costs.
I'm Ajay Kumar — I build and operate PandaStack, an open-source Firecracker microVM cloud for AI agents, and write about the infrastructure and tooling around it.
Need this kind of infrastructure work? See what I do or email hello@ajayk.sh.
Related
Amazon EKS in 2026: 1.37, $0.60 Extended Support and Auto Mode Explained
Amazon EKS in October 2026: Kubernetes 1.34 to 1.37 in standard support at $0.10 an hour, 1.31 to 1.33 in extended support at $0.60, what the Auto Mode fee buys, Pod Identity vs IRSA, Hybrid Nodes pricing, the new 7-day rollback, and when EKS is the right choice.
13 minOct 31, 2026Docker Compose in Production (2026): Is v5.6 Enough, or Do You Need K8s?
Is Docker Compose good enough for production in 2026? What Compose v5 changed (pre_start init containers, reconciliation, unsupported-key warnings), what a single host still cannot do, how Kamal, Coolify and Swarm compare, and when to move to Kubernetes.
13 minOct 31, 2026How to Upgrade an EKS Cluster to 1.37 Safely: Step by Step (2026)
Upgrade Amazon EKS one minor version at a time without an outage: refresh upgrade insights, scan for removed APIs with Pluto, upgrade the control plane with eksctl or the AWS CLI, then VPC CNI, CoreDNS and kube-proxy, then node groups and Karpenter, with the 7-day rollback.
15 min