Load Balancing in System Design: Types, Algorithms, and Health Checks

Ten application servers, one problem: which one receives each request? Without an answer, half the fleet idles while the other half melts, a dead server keeps receiving traffic, and adding capacity changes nothing. A load balancer is the component that answers the question, and keeps answering it correctly as servers join, leave, and fail.

Load balancing is the distribution of incoming network traffic across multiple servers so that no single server becomes a bottleneck or a single point of failure. A load balancer sits in front of a fleet, decides which server receives each request, checks continuously whether servers are healthy, and stops routing to nodes that fail. It is the piece of infrastructure that makes horizontal scaling real: many commodity machines behaving as one scalable, fault-tolerant service.

Why load balancing exists

A load balancer does three jobs at once, and each answers a different failure of an unscaled fleet:

• Distribution. Requests are spread across all nodes, so capacity is actually used. The queueing math that makes even distribution matter (a node near saturation slows down while its neighbors idle) is in latency vs throughput.
• Health detection. A node that starts failing is pulled from rotation before it fails users. Routing around dead nodes is what converts redundant hardware into measured uptime; the definitions of availability and fault tolerance used across this library are in availability vs reliability vs durability.
• A single, stable entry point. Clients resolve one address, while the fleet behind it changes constantly (servers are added, removed, deployed, and replaced) without any client ever noticing.

Where the load balancer sits

In a typical web request path, the load balancer is the entry point to your own infrastructure:

Client
   ↓
DNS
   ↓
CDN                  ← static content served at the edge
   ↓
Load balancer        ← entry point to your fleet
   ↓
App servers (N)     ← stateless; any node can serve any request
   ↓
Cache / Database

Static assets are usually served before this point, from a CDN‘s edge servers. Inside the perimeter, the balancer can be a dedicated hardware appliance, a managed cloud service, or plain software: nginx, HAProxy, and Envoy all appear in this role, often doubling as the reverse proxy at the same time.

L4 vs L7: two layers of balancing

A load balancer operates at one of two levels of the network stack, and the choice determines what it can see and therefore what it can decide.

• Layer 4 (transport layer). The balancer routes on IP addresses and TCP/UDP ports. It sees connections, not content: fast, simple, protocol-agnostic, but blind to the request itself.
• Layer 7 (application layer). The balancer terminates the connection, including TLS, typically, and reads the actual HTTP request: path, headers, cookies, method. It can route /api/ to one fleet and static paths to another, pin sessions by cookie, and apply policy per request, at the cost of parsing every request.

Real stacks often chain both: an L4 balancer spreads connections across a set of L7 balancers, which in turn make request-aware decisions. The division of labor between the balancer’s job and the proxy’s job is drawn in reverse proxy explained.

Load balancing algorithms

The algorithm is the rule that answers “which server gets this request?” The right choice depends on one question above all: do your requests cost roughly the same, or wildly different amounts?

Round robin

Requests go to servers in order (A, B, C, A, B, C) and wrap around. It is simple, stateless, and perfectly fair when two assumptions hold: requests cost about the same, and servers have about the same capacity. When either breaks, fairness breaks with it, one heavy usage pattern pins one server while the others idle. Weighted round robin extends the idea: each server receives a share of traffic proportional to an assigned weight, so bigger machines get proportionally more.

Least connections

Each new request goes to the server currently holding the fewest active requests. It is cost-aware without measuring cost: long-running requests accumulate on a node and steer new traffic elsewhere automatically. Whenever request cost varies (and real web traffic almost always varies) least connections distributes load better than round robin. Variants fold in recent response time as well, preferring servers that are both idle and fast.

Source IP hash

The balancer hashes the client’s IP address to pick a server, so the same client consistently lands on the same node. It provides session affinity without the balancer recording any state, with two weaknesses: a few dominant clients can unevenly load one node, and large populations behind NAT or mobile carriers change addresses constantly, breaking the mapping.

Consistent hashing

Instead of the client’s address, hash a meaningful key (user ID, session, or object name) onto a ring of servers. The same key always routes to the same server, and (the property that earns consistent hashing its name) adding or removing a server relocates only a fraction of keys rather than nearly all of them. It is the default for requests that must land where their data lives: distributed caches, sharded stores. The ring mechanics, and why minimal remapping matters, are covered in consistent hashing.

Choosing an algorithm

• Similar request costs, similar servers: round robin; weighted, if hardware differs.
• Variable request costs: least connections, or a response-time variant.
• Client affinity required: source IP hash, or better, fix the state problem (next section).
• Data-dependent routing: consistent hashing on the data key.

The honest default for stateless HTTP traffic is least connections; round robin is fine for cheap, uniform work; consistent hashing belongs wherever the request must meet its data.

Health checks: routing around failures

Health checks are how the balancer learns a node has died, without waiting for users to report it. Two mechanisms, usually combined:

• Active checks. The balancer probes each node on a health endpoint every few seconds; a threshold of consecutive failures marks the node down and traffic stops. What the endpoint measures matters more than how often it is probed, see below.
• Passive checks. Real traffic is the probe: repeated connection failures or timeouts from a node demote it from rotation immediately, without waiting for the next probe cycle. Faster to react, but a node is only caught after users have already hit the failure.

What the check measures decides what “healthy” means:

• Depth. A TCP connect check proves the process has an open port, nothing else. A dependency check asks whether the node can actually serve: is the database reachable, is the local disk full, is a required sidecar up? A node that passes shallow checks while failing users is worse than no check at all, because the balancer keeps feeding it traffic.
• Flapping. Overly deep checks swing the other way: a node that is merely degraded (one slow dependency) fails, recovers, fails again, oscillating in and out of rotation. Checks should fail on exactly the conditions that make a node unfit to serve, and no more.

Failover time is the sum of three intervals: the probe interval, the failure threshold, and the detection latency of the health endpoint itself. Under a node failure, that sum is the gap between “node died” and “users stop feeling it”, the MTTR term in the availability formula in availability vs reliability vs durability.

Health checks also power zero-downtime operations. On a rolling deploy, the balancer marks a node as draining (no new requests, in-flight requests finish) then the node updates, passes its check, and rejoins rotation. This is the machinery behind the rolling upgrade path in horizontal scaling.

Session persistence: the sticky problem

Sticky sessions pin a user to one server: the first request lands wherever the algorithm says, and every later request from that user follows. The balancer implements this with a cookie or an IP hash. Affinity is the right tool when per-user state lives in a node’s memory, and it quietly costs the fleet two things it depends on: even distribution (a few heavy users can pin a node) and failover (when the node dies, the sessions die with it).

The robust alternative is to externalize session state (into a database or a distributed cache) so any node can serve any request and the balancer stays free to route on cost alone. Treat affinity as a transition step; if you must keep it, cookie-based affinity survives the mobile-network IP churn that defeats IP-based pinning.

Overflow: queueing and load shedding

What happens when traffic exceeds what the fleet can absorb? An unmanaged balancer does one of two things by accident: queues every request until latency grows past timeouts (the queueing math is in latency vs throughput) or drops connections under pressure. Both outcomes are felt by every user at once.

Load shedding is the deliberate version: reject a bounded, chosen subset of traffic early (a fast 503, a shorter queue for non-critical paths, a reserved lane for health checks and payment traffic) so the rest completes quickly. Shedding converts a slow, universal failure into a fast, partial one. Deciding which traffic never even reaches the balancer is the sibling mechanism, rate limiting; under sustained overload the two work together; limits keep the offered load sane, shedding absorbs what slips through.

Failure modes

• The balancer itself. A fleet of redundant servers behind one balancer still has one failure domain: the balancer. Production setups run HA pairs (active-passive with a virtual IP that fails over, or active-active peers) for exactly this reason.
• Thundering recovery. A node that passes its health check gets full traffic instantly and dies again mid-request. Ramping recovered nodes back into rotation, slowly increasing their share, absorbs the shock.
• Lying health checks. Passing TCP checks while dependencies are down keeps dead nodes fed; overly deep checks that flap starve healthy ones. The check must test the request path, not the machine.
• Hot keys under hashing. A few dominant keys can concentrate load on one node even with consistent hashing: visible only in per-node metrics, never in fleet averages.

Common mistakes

• Balancing stateful services. The balancer distributes requests; it cannot distribute in-memory sessions. Until state is externalized, sticky sessions cap both fairness and failover.
• Health checks that check nothing. “TCP connect succeeded” is not “the node can serve.” Check the dependency path, and design the check so it cannot flap.
• One balancer, no failover. The entry point without redundancy caps the availability of everything behind it.
• Picking algorithms by folklore. Watch per-node connection counts and latency percentiles (latency vs throughput); the data picks the algorithm.
• Forgetting the balancer’s own limits. Balancers saturate too: connection limits, packets per second, TLS handshakes per second. Monitor the balancer like any other tier.

FAQ

What is load balancing in system design?

A load balancer distributes incoming traffic across a fleet of servers so that no single server becomes a bottleneck or a single point of failure. It decides which server receives each request, uses health checks to route around failed nodes, and gives clients one stable entry point while the fleet changes underneath. It is the infrastructure layer that turns horizontal scaling into a working system.

What is the difference between L4 and L7 load balancing?

L4 routes on IP address and port, per connection, without seeing request content, fast and protocol-agnostic. L7 terminates the connection (including TLS) and reads the HTTP request; paths, headers, cookies, so it can route each request individually, at higher cost. Many production stacks use both, chained.

Which is better, round robin or least connections?

When requests cost about the same, round robin is fine. When costs vary (most real web traffic) least connections adapts automatically, because expensive requests accumulate on a node and steer new traffic elsewhere. The deciding question is whether your request cost is uniform; per-node connection counts answer it.

Can a load balancer be a single point of failure?

Yes, one balancer in front of a redundant fleet caps the whole system’s availability at the balancer’s own. Production balancers run as HA pairs with automatic failover, so the entry point itself is redundant.

Is a load balancer the same as a reverse proxy?

The jobs overlap but differ: a load balancer distributes traffic across many servers; algorithms and health checks are its core. A reverse proxy fronts servers; TLS termination, routing, security, whether there is one server or many. The same software often does both; the comparison is drawn in reverse proxy explained.

Related articles

• Next read: reverse proxy explained; the server in front of your servers, and the other job the front machine does.
• how a CDN works; serving content from the edge, before traffic ever reaches your balancer.
• API gateway in system design: routing, authentication, and policy one level above the balancer.
• rate limiting in system design; deciding who gets to send traffic at all.
• vertical vs horizontal scaling; the scaling decision a load balancer makes real.

Last updated on 8 September 2026