Health Checks in Distributed Systems: Liveness, Readiness, and Dependency Probes

Health Checks in Distributed Systems answer a narrow question: should this task receive work right now. When the answer is wrong, you either drop a good task or send users to a dead one. Also, platforms will restart or replace whatever you call unhealthy. Therefore, the check must match the action you want the platform to take.

If every check hits every dependency, a brief network blip can drain the whole fleet. Still, a check that always returns success will keep a broken task in rotation. In my experience, the useful design splits process death from traffic readiness, then probes dependencies with a budget.

What Health Checks Are and Why They Fail.

A health check is a signal from a task, or a probe against it, that a controller can trust. When the signal says ready, a load balancer may send traffic. When the signal says not alive, a supervisor may restart the process. If you mix those meanings, the controller does the wrong repair.

Checks fail in production because they test the wrong thing. For example, a process can open a socket and still be unable to reach its database. Then the balancer sends traffic and every request times out. Consequently, the dashboard looks green while users see errors.

The opposite failure is a deep check that is too deep. If the ready endpoint runs a heavy query, the check itself causes the failure it reports. Also, when the database slows, every task flips unready together and capacity vanishes at once. Since the remedy was a restart or a removal, you made a dependency incident into a fleet incident.

Distributed systems add skew. One task can be ready while its neighbor is not, and callers must handle both. When a deploy rolls forward, old and new tasks report health for different code. Although that is normal, a check that requires a new schema will fail the old tasks during rolling deployments.

The pod lifecycle docs separate running from ready for this reason. However, the same split matters on virtual machines and in service meshes. If your platform has one health URL, you still need to decide which failure should remove traffic and which should restart the process.

Architecture of Useful Probes.

Use three different questions, even if they share code. First, is the process alive and not deadlocked. Next, can it accept new work right now.

Then, did startup finish, including warm caches and one time setup. Finally, do not let a dependency probe answer the first question.

Liveness, Readiness, and Startup.

Liveness should fail only when a restart is the best fix. A stuck event loop or a fatal internal error qualifies. A down database does not, because a restart will not bring the database back. Also, a slow dependency should trip readiness, not a crash loop.

Readiness should fail when the task must not receive traffic. That includes a missing required connection, an overloaded pool, or a process that is draining for shutdown. Since the task stays up, it can become ready again without a restart. The details of probe timing belong with liveness and readiness probes, but the meaning should stay stable across platforms.

Dependency Probes and Budgets.

Split dependencies into required and optional. If the task cannot do its job without a dependency, readiness should include a cheap check. When the dependency is only used by a side path, do not let it pull the whole task out of rotation. Specifically, a failed metrics endpoint is not a reason to drop user traffic.

Put a tight budget on every probe. The check needs its own timeouts, shorter than the platform probe timeout. If the check hangs, the platform will treat the task as failed anyway, but slower. Also, cache the dependency result for a brief window so a probe storm does not become extra load.

The probe configuration guidance shows failure thresholds and delays. You should set them from startup time and dependency latency, not from defaults you never measured. Meanwhile, the release engineering material is a reminder that health is part of the release, because a bad probe blocks or accelerates a bad build.

Trade-offs Between Shallow and Deep Checks.

Shallow checks are fast and stable. They tell you the process can run code. Deep checks tell you a real dependency answered.

When you only use shallow checks, broken dependencies stay in rotation. If you only use deep checks, a shared outage empties the fleet.

A practical middle path is a shallow liveness check and a ready check with cached, budgeted dependency probes. You then alert on dependency failure without restarting every task. After the dependency returns, readiness recovers. Also, keep a manual or automatic drain for deploys so shutdown is not a crash.

A common mistake I have seen is a ready check that writes a row to prove the database. The probe then fills a table and takes locks. Instead, use a cheap read or a connection validation that the driver already supports. Also, cap concurrency so probes cannot pile up behind a slow query.

Load balancer checks and app checks can disagree. However, both must use the same meaning of ready, or the balancer will route to a task the app calls unready. In addition, mesh sidecars need their own ready signal so you do not send traffic before the proxy can forward.

Pitfalls and Failure Modes.

Probe bugs are quiet until a deploy or a dependency incident. If you test the failure path, you see them sooner. After a false restart storm, fix the classification before you tune the threshold.

1. Use one endpoint for live and ready, so a dependency blip restart loops the fleet.
2. Forget a startup delay, then slow boots get killed before they can pass.
3. Let the probe call the same heavy handler as a user request.
4. Return success while the process is draining and still has no replacement.
5. Ignore clock and timeout skew, so the platform gives up before your check returns.
6. Require a new downstream that old tasks do not have during a rollback.

Graceful shutdown is part of health. When a task should leave rotation, mark it unready first. Then finish in flight work.

Finally, exit. Although a hard kill is faster, users see failed requests and the next task may not be ready yet.

Authentication on the probe path will confuse you. If the check needs a token, a token outage looks like an app outage. Since the platform probe is usually unauthenticated, keep the local health port private and simple. Still, do not expose a deep status page that leaks dependency names to the public internet.

Partial health needs a product decision. A task that can read but not write might still serve some routes. When your platform only understands ready or not, pick the safer bit. If writes are the job, not ready is the honest answer.

A Practical Check Layout.

Keep the handlers local and cheap. First, liveness runs an in memory check that the loop is making progress. Next, readiness reads cached flags for required dependencies and for drain state.

Then a background loop refreshes those flags with short timeouts. Finally, startup stays unready until the first successful refresh.

Use the same definitions in the deploy gate. If readiness is red, rollback strategies should treat the new tasks as bad. When the old tasks are the ones failing, you may be looking at a dependency, not a bad build. Also, log the reason code so the probe is debuggable without a shell on the host.

live:  loop_heartbeat_age < 2s
ready: not draining
       and db_ok age < 5s
       and cache_ok age < 5s
startup_ready_after: first_refresh
probe_timeout: 1s
refresh_timeout: 300ms
on_shutdown: set draining, wait in_flight, then exit

The live rule does not mention the database. When the database fails, ready flips and the process stays up. If the heartbeat stops, the supervisor restarts the process. That split is the whole design, and the numbers should match your latency budget.

Performance, Scale, and Cost.

Probes are traffic. A large fleet with a short interval can generate more health calls than user calls to a small dependency. If that dependency is the database, you just built a denial of service against yourself. Therefore, cache results and keep the interval only as fast as your failover needs.

Restart storms are the expensive failure. Each restart drops work, thrashes caches, and can stampede a dependency when tasks boot together. Since a bad liveness rule causes that storm, a cheap CPU saving in the check is not the goal. Also, add jitter to refreshes so tasks do not all probe on the same tick.

An illustrative production range is a ready interval of a few seconds, with a failure threshold of two or three, for stateless request tasks. Therefore, time to remove a bad task is on the order of those intervals, not instant. Overall, pick the threshold from how long you can send traffic to a dead task, and from how often a single slow probe is noise.

Watch the checks as a product. First, how often ready flips. Second, how often a restart follows a live failure. Third, probe latency against the probe timeout.

Fourth, whether user errors rise while ready still looks green. If ready is green during user errors, the check is too shallow for that failure mode.

Do not add a new remote call to the probe to make it feel complete. Although a full synthetic request can be useful on a schedule, it is a poor fit for every task every few seconds. When you want a synthetic, run it as a job with its own alert, and keep the per task probe small.

Key Takeaways

• Health Checks in Distributed Systems must say whether to restart a task or only remove it from rotation.
• Keep liveness local, and put required dependencies on readiness with a cache and a short timeout.
• Do not let an optional dependency or a public status page decide the fate of the fleet.
• Mark tasks unready before shutdown so in flight work can finish without new traffic.
• Budget probe load, because a deep check on a short interval can overwhelm the dependency.
• Treat a green probe during user errors as a defect in the check, then fix the signal.

FAQ

Should the load balancer and the app share one health URL?

They should share one meaning of ready, and the app URL is usually the source. When the balancer uses a different, weaker check, it will send traffic the app would refuse. Also, keep liveness off the public balancer path so a platform restart is not tied to a public route. If a mesh sidecar exists, include it in ready before you open traffic.

How deep should a dependency probe go?

Deep enough to know a required dependency can do the cheap operation you actually need. If a connection ping matches real use, stop there. However, a probe that runs a full business transaction is too deep for every interval. Therefore, put heavy synthetics on a slower schedule and keep the task probe cached and small.

What happens if all tasks become unready?

The balancer has nowhere honest to send work, and users see failures. Since that is safer than sending work to tasks that cannot succeed, the next step is to fix the dependency or the bad rule. Still, a rule that flips the whole fleet on a brief blip is too strict. Finally, use thresholds and caches so one slow sample does not drain capacity.

Do background workers need the same checks?

They need the same split, but readiness means safe to pull work rather than safe for a balancer. When the queue is the job, a worker that cannot reach the queue should stop pulling. Also, liveness still means the loop is stuck and a restart can help. In addition, do not restart workers only because a downstream API is slow.

Separate live, ready, and startup on the next service you touch, and keep dependency probes off the restart path. Then add a timeout and a cache around each required check, and log a reason code when ready is false. After the next deploy, confirm that a drained task leaves rotation before it exits. That is the health model the rest of the platform can trust.

Last updated on 09 September 2026.