Load Shedding: Dropping Traffic to Stay Alive

Load shedding is the deliberate rejection of a chosen fraction of offered work while a system is overloaded, done early and fast, so that the work kept is served successfully instead of everything queueing toward shared failure. It is overload survival by dropping: the admission decision that says “not this, not now,” protects capacity for what remains, and reports the refusal honestly instead of accepting work it cannot finish.

The boundaries are drawn, so the comparisons will be links, not sections. Rate limiting is admission control by policy, per-client budgets enforced at the front door in normal operation, owned by its own article; shedding is survival under aggregate overload, when capacity is short even for polite callers. Circuit breaking is the dependency-side sibling (one broken dependency stopped, not a drowning fleet) and the breaker article owns it. Backpressure is the pipeline-side sibling (slowing producers rather than dropping requests) and its article follows this one in the series. Shedding is what remains when the high availability design has done its redundancy work and the day still arrives with more demand than fleet: the availability question answered with a partial yes instead of a total no.

What is load shedding

Overload has a physics, and it ends the same way every time. Work arrives faster than workers finish it; queues grow; latency climbs past the point where callers have given up; memory and connections pile up behind the queues; and the workers spend their remaining capacity reporting their own exhaustion. A fleet in this state does not fail cleanly; it fails slowly and universally, serving nothing while pretending to serve everything. The retry machinery of well-meaning clients adds amplification on top: every timeout becomes another request, so the fleet is drowning and being splashed at the same time.

Shedding is the counter-move, and load balancing wrote its thesis plainly: “reject a bounded, chosen subset of traffic early (a fast 503, a shorter queue for non-critical paths, a reserved lane for health checks and payment traffic) so the rest completes quickly.” The conversion is the entire value: a slow, universal failure becomes a fast, partial one. Users who are dropped know immediately and can retry later or elsewhere; users who are kept get real service, not a queue position. The cost is paid in honesty up front (somebody is told no) instead of in universal lying that ends in an outage nobody chose.

The decision space has three moves, and choosing is the design work. Queue; absorb the excess as a backlog, betting it is a burst that will drain; the honest bet, and the one that fails badly when the surge is sustained, because the queue is memory that fills and latency that compounds. Degrade, keep serving, minus the expensive parts: stale instead of fresh, cached instead of computed, core instead of complete. Drop: refuse some work outright, chosen by rule, told the truth fast. Real overload responses mix all three, and the sections that follow take them in order: what degrading means, how the protection is wired, and how the dropping chooses its targets.

Graceful degradation

Degradation is a ladder, and the rungs are decided long before the emergency. Fault-tolerant systems fixed the discipline: “the system that already knows which features shed first (recommendations before checkout, previews before search) keeps its core promise while the degraded one is still holding a meeting.” The ordering principle is the product’s promise, not its feature list: checkout, payment, login, and the read path of record are the load-bearing walls; recommendations, previews, counts computed exactly, and anything enriched by a third-party call are the decorative finishes. The ladder is written in a design review with product at the table, because the alternative is the mid-incident meeting; the one where availability is being negotiated by whoever shouts loudest.

The rungs have names, and most are caching decisions in disguise. Serve stale instead of fresh; the cached result the caching series already knows how to hold, accepted with a shorter TTL than usual during the emergency. Serve static instead of computed: yesterday’s aggregated counts instead of live rollups, the published price instead of the personalized one. Serve partial instead of complete: the list without the per-item enrichment, the page without the third-party widgets. And defer instead of doing; batch jobs, emails, reindexing, anything that is already asynchronous, pushed further from the peak. The common thread: each rung trades a quality dimension the caller can tolerate losing for capacity the fleet cannot spare.

When a cache fleet goes cold all at once (the avalanche failure mode) the ladder is forced on you without warning, and the article’s warning stands: the choices are degraded responses, queuing, and dropping, “the same decision space as load shedding, forced on you by a component that was sold as an optimization.” The difference between a graceful incident and a catastrophic one is usually whether the degraded responses were rehearsed. A team that has served the degraded variant on purpose (in a game day, a staging drill, or a feature flag) owns a ladder it can climb in the dark; a team that never has is designing its degradation live, with users holding the stopwatch.

Server overload protection

Protection lives at three depths, and each sees a different signal. At the edge (the balancer or the API gateway) the visible quantity is concurrency and queue depth: how many requests are waiting, how old the oldest is, and whether the waiting line is growing faster than the serving rate. Edge shedding is blunt and fast, and it is the layer that can enforce the reserved lanes load balancing named: health checks and payment traffic must not queue behind previews and recommendations, because the machinery that recovers the fleet and the traffic that pays for it are the last things a healthy degradation ever drops.

Inside the service, the honest signals are saturation and the tail: worker utilization, connection-pool pressure, and the latency percentiles from latency vs throughput that show overload long before averages admit it. The trigger needs hysteresis; the same discipline circuit breaking applies to flapping: shed when a sustained threshold is crossed, restore gradually, never let one noisy minute toggle the system between full service and half service. A trigger without hysteresis is an oscillator; a fleet that sheds at 90% and restores at 89% will spend the incident vibrating.

The third depth is outbound, and it is the one teams forget: your fleet can be the flood, not just the flood’s victim. One user request that fans out into dozens of downstream calls can saturate a dependency all by itself, so protection here means per-dependency concurrency limits (the bulkhead discipline the circuit breaker article carries) query budgets that cap how much one request may spend downstream, and admission control into dependencies that are already slow. When the downstream is not merely busy but failing, the tool changes: shedding protects the fleet from overload, the breaker stops calls to a dependency that has stopped answering, and the two compose rather than compete.

What the dropped request sees is part of the mechanism, not an afterthought. The honest drop is a fast 503 with a Retry-After header; the status code fork rate limiting drew precisely: 429 tells one caller its personal budget is spent, 503 tells everyone the system itself is struggling, which is transient and not the caller’s fault. Fast is the point: a drop that costs the fleet a full request’s worth of work before answering protects nothing, and a queue that holds the request until the caller has given up is the dishonest version of the same drop. The client side of the contract belongs to retry discipline (back off by the indicated interval, add jitter, do not come back as an amplifier) which is why the drop carries the Retry-After in the first place: the fleet that refuses honestly is also telling its clients how to help.

Shedding requests

Choosing whom to drop is a policy, and the policy axes are few. By class: background and batch work sheds before interactive traffic, crawlers before users, previews before checkout: the ladder from the degradation section, enforced at the door. By tenant: under scarcity, per-tenant quotas decide who is protected and who waits, which turns a technical decision into a contractual one that should exist before the incident. By age: the oldest requests in the queue are usually already lost (their callers have timed out) so dropping them first is not even a refusal, just the truth arriving sooner. By cost: the expensive endpoints, the ones that fan out and aggregate, shed earlier than the cheap reads. Random dropping is the fair baseline when no policy fits, and a policy that has never been written defaults to random with none of the fairness.

Many drops are deferrals wearing a different word, and separating them is honest design. Deferral moves work in time: the job queued for the off-peak window, the request answered with “come back later,” the batch postponed until the fleet breathes. Dropping ends the work; it will not run, and the caller or the operator must decide whether it matters. The boundary is exactly the bounded-queue question: message queues already warned that when producers outpace consumers persistently, “the honest answer is not a bigger queue but backpressure”, for pipelines, that answer is the consumer pacing the producer, which is the next article’s subject; for request-response, it is the fast refusal this article owns. What deferral and dropping share is the admission that capacity is finite: one spends queue memory to buy time, the other spends honesty to buy capacity, and neither spends the fleet.

The last piece is composition, because overload is the incident that exercises every resilience tool at once. Rate limiting at the door keeps offered load sane in normal operation; shedding inside converts what slips through into fast, partial failure; circuit breakers stop the calls that were doomed before they were made; backpressure paces the pipelines so they slow instead of dying; retry discipline keeps the clients of a struggling fleet from amplifying it; and idempotency makes the shed-retry-resume cycle safe to run. Each mechanism owns one direction of the overload problem, and the design review that walks the whole list (who is limited, who is dropped, who is slowed, who is stopped, what is safe to repeat) is the difference between a fleet that degrades and a fleet that disappears.

Common mistakes

• Shedding at the wrong layer. The balancer starts refusing while the application behind it is still accepting work its own health checks depend on, or the app sheds requests but the balancer’s queue hides the refusals behind its own patience. Edge and service must agree on the lanes: the balancer can enforce them fast, but only the service knows which paths keep recovery alive.
• No ladder chosen in advance. The degradation ladder decided mid-incident is not a ladder; it is a meeting, and the meeting happens while the fleet is down. “Recommendations before checkout” is a one-line decision in a design review and an unsolvable argument during an outage.
• Dropping without telling. A bare 503 with no Retry-After invites every dropped client to retry immediately, which means the drop manufactures the retry storm it was meant to prevent. The refusal is half the mechanism; the other half is the instruction that makes the refusal survivable.
• Uniform fairness theater. Dropping the same fraction of checkout traffic and crawler traffic is fair to machines and cruel to everyone else. Priority classes exist precisely so degradation can be unfair on purpose, in writing, before the incident.
• Shedding only inbound. The fleet that carefully drops what arrives may still be flooding its own dependencies, the fan-out problem again. Outbound budgets and per-dependency limits are load shedding too: the version where you are the flood, and the refusal is the favor you do the system next door.

FAQ

What is the difference between load shedding and rate limiting?
Rate limiting is admission control by policy: per-client budgets enforced at the front door in normal operation, answering “has this caller spent too much?” Load shedding is overload survival: dropping a chosen fraction of work because the fleet cannot serve what has already been admitted, answering “what must go so the rest lives?” A system needs both, limits keep offered load sane; shedding handles the day the arithmetic turns against you anyway.

Should the balancer or the application do the shedding?
Both, at different depths. The edge sees queue depth and can enforce priority lanes fast, without knowing the application’s internals. The service sees saturation and knows which paths are load-bearing and which are decorative. Edge shedding protects the fleet; service shedding protects the promise, and the lanes the two layers enforce must be agreed in advance, or each will undo the other’s decisions.

What status code should a shed request return?
A fast 503 with a Retry-After header. The 429 belongs to rate limiting, it names the caller’s exhausted personal budget. The 503 names the system: over here, struggling, not your fault, come back in the indicated interval. The distinction is a contract with every client, and the Retry-After is what turns the refusal into pacing instead of amplification.

How is load shedding different from circuit breaking?
The breaker protects the fleet from one failing dependency and is owned by its own article; shedding protects the fleet from aggregate overload. Breakers answer “is this dependency worth calling right now?” and decide per dependency; shedding answers “can we serve what has arrived at all?” and decides across the whole request population. The two compose: break the doomed calls, shed the excess, serve the rest.

Does queueing solve overload without dropping anything?
For bursts, yes; a queue is the right shock absorber, and the queue article owns that case. For sustained overload, no: the queue is memory that fills and latency that compounds, and the backlog outlives the patience of everyone waiting in it. Queueing buys time; shedding spends honesty; only adding capacity actually changes the arithmetic.

Related articles

• Next read: backpressure, the pipeline-side sibling and the next rung of the series: bounded queues and the consumer pacing the producer; the overload answer for pipelines, where this article is the answer for front doors.
• rate limiting, the front-door policy: token buckets, sliding windows, and per-client budgets, the boundary this article draws its own job against.
• load balancing in system design, the overflow question’s first home: queueing at the edge, the deliberate version, and the reserved lanes that keep recovery machinery alive.
• the circuit breaker pattern; the dependency-side sibling: failing fast against a doomed dependency while shedding handles a drowning fleet; the two compose rather than compete.
• idempotency; the base layer: the shed-retry-resume cycle is only safe because the operation underneath survives repetition.
• fault-tolerant systems, the wider family: redundancy, failover, and the degradation ladder this article’s rungs hang from.
• high availability; the pillar: the redundancy and failover design that makes overload rare enough for shedding to be the exception rather than the architecture.