Liveness vs Readiness Probes: Kubernetes Health Checks Done Right

Liveness vs Readiness Probes is the choice between restarting a container and only removing it from service. When you swap those meanings, Kubernetes will crash loop a healthy process or send users to a dead one. Also, startup time, drain, and probe timeouts all change the result. Therefore, set each probe for the action you want the kubelet to take.

If you copy a sample manifest, you often get a live check that hits a dependency. Still, a failed live probe kills the container, which will not fix a down database. In my experience, the stable clusters use startup probes for slow boots and keep liveness boring.

What Each Probe Means and Why It Fails.

A liveness probe asks whether the container should be killed and started again. When it fails past the threshold, the kubelet restarts the container. A readiness probe asks whether the pod should receive Service traffic. If readiness fails, the pod stays up and leaves the endpoints list.

A startup probe covers the boot window. While it has not succeeded, liveness and readiness stay out of the way. Then the normal probes take over. As a result, a slow first start does not get killed for missing an early live check.

The failure mode I see most is a live probe pointed at a remote call. For example, the handler checks a database and returns an error when the query is slow. Then every pod restarts together, caches go cold, and the database gets a stampede. Consequently, a dependency incident becomes a cluster incident.

The other common failure is a ready probe that always succeeds. Endpoints include pods that cannot do the work. Since the Service has no other signal, traffic keeps flowing and user errors rise. Although the pod looks ready, the process is only listening.

These probes are a specific form of health checks. The platform details matter because the defaults restart or unroute your pods on a timer. If you ignore the timer, a correct handler can still fail the pod.

How Kubernetes Applies the Probes.

The kubelet runs the probe on a period you set. First, it waits initialDelaySeconds, unless a startup probe replaces that wait. Next, it tries the probe.

Then it counts failures against failureThreshold. Finally, it restarts on live failure or edits endpoints on ready failure.

Startup, Then Live and Ready.

Use a startup probe when boot can take longer than a live failure budget. Image pull is outside the probe, but migrations, cache warm, and JIT time are inside. If those steps vary, give startup enough failures times period to cover the slow case. Also, once startup succeeds, keep later live checks cheap so a long boot is not required again.

Readiness can fail during startup too, and that is fine. The pod should not join the Service until it can serve. When you also have a startup probe, you still want ready to mean real traffic readiness after boot. Specifically, do not make ready a copy of the startup command if startup only checks that the process exists.

Timing Fields That Cause Incidents.

timeoutSeconds must be shorter than periodSeconds, or probes pile up. The probe configuration page defines these fields and the HTTP, TCP, and exec forms. Because an exec probe starts a process in the container, it costs more than an HTTP handler on localhost. Also, a TCP probe only proves a port is open, which is a weak ready signal.

The pod lifecycle docs explain that a container can be running while the pod is not ready. However, many dashboards count running pods and hide the endpoints gap. In addition, a deployment will not finish while new pods miss the ready gate, which is what you want during rolling deployments.

Set your application timeouts below the probe timeout. If the handler waits longer than timeoutSeconds, the kubelet records a failure even when the app would have succeeded. Since repeated failures restart or unroute the pod, a slow handler is a false negative. Therefore, fail the probe locally before the kubelet gives up.

Trade-offs in Probe Design.

You want fast removal of dead pods and few restarts of live ones. Those goals fight when the same probe does both jobs. When the check is deep, readiness is the safer place for it. If the check is only a heartbeat, liveness is enough.

Exec probes can test a local file or a loop heartbeat without opening a port. They are also easier to get wrong, because a shell script can hang. HTTP on a localhost port is usually easier to observe in access logs. After you pick a form, use it the same way in every service so on call engineers know what failure means.

A common mistake I have seen is failureThreshold set to one. A single slow probe then restarts the container. Instead, allow a small number of failures so one blip is noise. Also, do not set the period so long that a dead pod serves traffic for many minutes.

The Kubernetes deployment controller uses readiness to decide whether the roll can continue. However, a live restart during the roll creates extra churn and can trip the progress deadline. In addition, a bad probe on the old pods can make a rollback strategies attempt stall, because the previous build also fails the new check you added.

Pitfalls and Failure Modes.

Probe incidents look like crash loops, stuck rollouts, or endpoints that never fill. If you read the kubelet events, the reason is usually timing or the wrong check. After you fix one service, search the cluster for the same pattern.

1. Point liveness at a dependency, then a blip restart loops every replica.
2. Leave initial delay at zero on a slow JVM or a migration boot.
3. Use a TCP ready probe and call the pod healthy when the router is not up.
4. Forget preStop, so the pod is killed while it still has open requests.
5. Require a new header or path that older images do not serve, then rollback fails.
6. Run an exec probe that sleeps or waits on a lock in the container.

Shutdown needs a ready flip before the process exits. Kubernetes can set the pod unready when deletion starts, but your app should also stop taking work. Then wait for in flight requests inside a termination grace period.

Finally, exit. Although preStop can sleep, a blind sleep is weaker than a real drain.

Sidecars change the meaning of ready. If the app is ready and the proxy is not, the Service will still send traffic into a pod that cannot leave the node. Since the kubelet probes containers you configure, probe the proxy or use a readiness gate. Still, do not let the sidecar live probe depend on the app database.

High restart counts hide in plain sight. A pod can stay available enough that pages stay quiet while churn burns CPU and cold caches. Because each restart looks local, nobody ties it to the probe. If restarts correlate with probe failures, fix the probe before you scale the deployment.

A Manifest You Can Explain.

Write the three probes so a reviewer can say what each failure does. First, startup allows a slow boot. Next, liveness checks a local heartbeat only.

Then readiness checks drain state and a cached dependency flag. Finally, preStop marks the app draining.

Keep the numbers tied to measurements. If boot usually takes twenty seconds and sometimes forty, startup must allow more than forty. When the heartbeat should move every second, a live budget of several seconds is enough. Also, document the reason in a comment next to any number that looks generous.

startupProbe:
  httpGet: { path: /startup, port: http }
  periodSeconds: 5
  failureThreshold: 12
livenessProbe:
  httpGet: { path: /live, port: http }
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /ready, port: http }
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 2
lifecycle:
  preStop:
    httpGet: { path: /drain, port: http }
terminationGracePeriodSeconds: 45

Here startup can spend about a minute before a restart. Live failure needs three local misses, and it never calls the database. Ready can drop the pod from the Service after two misses, which is the right place for a dependency or a drain. If your boot is faster, shrink the startup threshold instead of deleting the probe.

Performance, Scale, and Cost.

Each probe is a request from the kubelet to the pod. A short period on a huge fleet means a steady load on every node and on any dependency you foolishly call. If the handler is heavy, you stole CPU from user traffic. Therefore, answer live and ready from memory.

Restarts are the costly outcome. They repeat startup, drop connections, and can fail a rollout that was otherwise fine. Since liveness is the restart switch, keep it stable under dependency pain. Also, a crash loop backoff will slow recovery if you let the probe flap for a long time.

An illustrative production range is a few probe requests per pod per minute, which is noise for a normal API and loud if each one runs a query. Therefore, cache dependency status inside the app and refresh it on a timer with jitter. Overall, the probe interval should match how quickly you need traffic to move, not how anxious the manifest looks.

Watch four numbers per deployment. First, container restart count. Second, ready replicas versus desired replicas. Third, probe failure reasons in events.

Fourth, user error rate while ready replicas look full. If the fourth moves alone, the ready handler is too shallow.

Do not remove probes to make a rollout look green. Although a missing ready probe lets the deployment finish fast, the Service will send traffic to pods that cannot serve. When boot is slow, add a startup probe and headroom. When the check is noisy, fix the signal.

Key Takeaways

• Liveness vs Readiness Probes maps to restart versus remove from Service, so do not point both at the same deep check.
• Use a startup probe for slow boots, then keep liveness on a local heartbeat.
• Put dependency health and drain state on readiness, with a timeout shorter than the kubelet timeout.
• Allow more than one failure before you kill a container, and do not use TCP as proof of real work.
• Drain in preStop and keep the grace period long enough for in flight requests.
• Treat restart spikes and green ready counts during user errors as probe bugs.

FAQ

Do I need all three probes?

Use readiness whenever a Service sends traffic to the pod. Add liveness when a stuck process is a real failure mode you can fix with a restart. If boot can exceed the live failure budget, add a startup probe. When the process is a short job, probes for a Service do not apply, and a restart policy is a different choice.

Should liveness and readiness call the same handler?

No, because they trigger different repairs. If they share a handler that checks a database, a database blip will restart the fleet. Also, a live handler should stay local even if the ready handler reads a cached flag. Therefore, use separate paths and test them separately.

What initial delay should we set?

Prefer a startup probe over a long initial delay, because delay waits even when boot is fast. If you cannot use startup yet, set the delay from the slowest normal boot, not from the average. However, a delay that is too long leaves a dead container running with no live check. Finally, measure boot in the cluster, since laptop times will lie.

Why does rollback fail the probes?

The old image may not serve the new probe path or port. When that happens, the reverse rollout can never go ready. Still, you can keep probe paths stable across versions so old and new images both answer. In addition, test the previous image against the probe spec before you need it in an incident.

Open the manifest for one production deployment and name what a live failure does versus a ready failure. Then move any dependency call off liveness and add a startup probe if boot is slower than the live budget. After the next roll, check restart count and endpoints, not only the deployment status. That review is how Liveness vs Readiness Probes stay correct as the service changes.

Last updated on 18 September 2026.