Rate Limiting in System Design: Algorithms and Distributed Enforcement

A service sized for five thousand requests per second stays fast right up until one client sends fifty thousand: a script with a tight loop, a cron job that fired twice, a retry that feeds itself. Capacity is finite, and past saturation every request gets worse, not just the excess. Something must decide who gets in. Rate limiting is that decision, made as deliberate policy instead of reflexive failure.

Rate limiting admits a client’s requests up to a defined budget (so many per second or per minute) and rejects or defers the rest. It is admission control by policy: a deliberate rule for how much of the system each caller may use, applied identically whether the fleet is idle or saturated.

Why rate limiting exists

Four problems converge on one control:

• Fairness. Without budgets, the loudest client takes the capacity; one misbehaving caller becomes every other caller’s latency problem. Limits are how a shared system stays shared.
• Abuse and accidents. Scrapers, credential-stuffing attempts, and buggy integrations all arrive as request volume. Budgets bound the blast radius of each.
• Stability. Past saturation, queues form and latency climbs; the mechanics of which are in latency vs throughput. A limit set below saturation keeps the system on the flat part of its curve.
• Cost. Compute, egress, and downstream calls are billed per unit. An unbounded client is an unbounded bill.

Rate limiting is one of three tools that control load, and they compose:

• Rate limiting; admission by policy, applied before capacity is reached. This article.
• Load shedding; survival: dropping work when overload has already arrived, covered in load shedding.
• Backpressure: flow control where the consumer paces the producer through the pipeline itself, covered in backpressure.

The three are not alternatives; a production system uses all three. Policy rejects the excess in calm weather; shedding drops work in the storm; backpressure keeps pipelines from bursting in between.

Rate limiting algorithms

Every algorithm answers the same four questions: does it tolerate bursts, what does it cost in memory per client, what does it cost to coordinate, and how exact is the budget? The differences matter because each one is right for a different place in the architecture.

Fixed window

Keep one counter per client, reset every window, 100 requests per minute, say. The implementation is a single atomic counter, which is why it survives as the default where memory is scarce: one INCR plus an EXPIRE. Its flaw is the window edge: 100 requests at 11:59:59 and 100 more at 12:00:01 are both within budget, and the system served 200 requests in two seconds. The limit is honest per window and dishonest at every boundary.

Token bucket

The bucket holds up to B tokens and refills at R tokens per second; each request consumes one, and an empty bucket rejects. Two knobs, two decisions: R caps the average rate, and B caps the burst. A client sending steadily gets R per second; a client bursting gets at most B at once and then waits for the refill. This is the usual choice for public APIs; steady average with tolerated bursts, and rejection that fails fast.

Leaky bucket

Requests pour into a bucket that drains at a constant rate: bursts queue, and the outflow is always smooth. Where the token bucket treats a burst as inventory to spend, the leaky bucket treats it as a backlog to work through, which means latency grows with queue depth, and the queue needs a bound. Its strength is protecting something downstream that needs even flow: a fragile dependency, a partner integration with a strict contract. Rate shaping, not just rate limiting.

Sliding window rate limiting

The window slides instead of resetting. The exact version stores the timestamp of every request in the trailing window and counts them: honest at every instant, at a memory cost proportional to requests per window. The counter version keeps two counters: previous window and current, and estimates the trailing count by weighting the overlap; approximate, but a fixed two counters per client, which is why high-volume systems usually choose it.

Token bucket vs leaky bucket

The two most common algorithms differ in one thing: what happens to a burst. The token bucket spends it instantly, then rejects; the leaky bucket queues it and serves it at a fixed pace. Inventory versus queue.

The decision rule: front doors fail fast, token bucket. Pipelines toward fragile dependencies flow evenly, leaky bucket.

Where to enforce limits

Limiting works as defense in depth, coarse at the outside and fine toward the inside:

• At the edge. A CDN or edge layer can apply blunt per-IP caps long before traffic reaches origin infrastructure: cheap, early, and impersonal.
• At the gateway. The API gateway is the natural home for per-client, per-endpoint budgets: it has the caller’s identity and the route, which is exactly what a budget is keyed on.
• At the proxy. A reverse proxy tier can enforce coarse limits even without a full gateway; protection that does not wait for identity infrastructure.
• In the service. Fine-grained budgets per resource or per operation; the only tier that knows a report request costs a thousand times a status request.
• At the data store. Per-row or per-tenant caps as the last line of defense, because everything upstream can be misconfigured.

Keying decides what a limit actually limits. IP-based keys are the easiest and the worst behaved: behind carrier-grade NAT and corporate proxies thousands of innocent users share one address, and behind your own proxies the address is wrong unless the forwarded headers are trusted. API keys and user identities are stable and fair. The remaining choice is granularity; one budget per client, or per client per endpoint, because a cheap read and an expensive export are not the same unit of capacity.

Distributed rate limiting

A limiter on one instance is trivial and wrong the moment the fleet grows: ten instances each enforcing 100 per minute admit a thousand, unevenly. Distributed rate limiting has to coordinate one budget across a fleet, and everything interesting follows from how it does.

The exact approach shares counters in a store such as Redis: every instance increments the same key, so every instance sees the same total. The subtlety is atomicity; an implementation that reads the counter and then increments it in two steps lets two instances both see 99 and both admit request 100. The check and the increment must be one operation: a single atomic command, or a small server-side script that runs as one. Non-atomic limiters leak requests per window per race; atomic ones do not.

Shared counters buy exactness and charge two prices. The first is latency: a round trip to the counter store on the hot path of every request, paid in the same currency as every other hop. The second is a new failure domain: when the counter store is down, the limiter is down, and the choice is stark. Fail open (let requests through uncounted) protects the service’s availability and loses the policy; fail closed (reject everything) protects the policy and turns the limiter’s outage into the service’s. The definitions of availability that frame this trade are in availability vs reliability vs durability.

The approximate approach skips the shared store: each instance enforces the global budget divided by fleet size, and instances occasionally reconcile. No hot-path dependency, no new failure domain, but the true total wanders with autoscaling and uneven distribution, so a burst concentrated by the balancer can overshoot. Most abuse-prevention limits should be approximate: exactness is a cost decision, not a virtue.

Answering with HTTP 429

A rejection is part of an API’s contract, and HTTP has a status code for it: 429 Too Many Requests, standardized in RFC 6585. Two headers turn a rejection from an insult into information:

• Retry-After, how long to wait: delay-seconds or an HTTP-date. This one header is the difference between clients that back off politely and clients that treat the 429 as a reason to retry immediately.
• Rate limit headers: conventions like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. Not standardized anywhere, but common enough that SDKs parse them.

What a well-behaved client does with a 429 (wait the indicated interval, back off with jitter, avoid a synchronized retry) is its own discipline, covered in retry with backoff. One distinction matters for the caller: 429 means the caller’s budget is spent, so back off and come back; 503 means the system itself is struggling, which is transient and not about the caller. Answering overload for the whole system is the load shedding story, not the rate limiting one.

Failure modes

• The fail-open window. A limiter that fails open is silently unprotected exactly when things are breaking; the outage is invisible until abuse arrives and nothing stops it. If failing open, alert on the limiter’s health as loudly as on any dependency.
• The fail-closed outage. A limiter that fails closed converts its own maintenance window into the service’s. Either way, the failure behavior is a designed decision, not an accident of code order.
• Hot keys. One viral client turns its single counter into the hottest item in the shared store. Coarse per-endpoint counters distribute the load; per-client keys concentrate it.
• IP limits in the proxy era. Thousands of users behind one NAT address discover your limit together, and it behaves like a shared outage for them.
• Retry storms. Clients that retry immediately on a 429 (without honoring Retry-After, without jitter) convert the limit into an amplifier: each rejection becomes extra traffic at the worst moment.

Common mistakes

• Limits without visibility. No 429 metrics per endpoint and per key means customers discover the limits before you do. Dashboards for rejected volume are as important as the limits themselves.
• One number for everything. A single global limit protects the cheap endpoints not at all and the expensive ones barely. Budgets are per client and per endpoint.
• Limiting only the front door. East-west traffic (one internal service hammering another) sees no limits unless it has them. Internal callers get budgets too.
• Rejecting without Retry-After. An undocumented 429 is an invitation to a retry storm; the header costs nothing and changes client behavior entirely.
• Paying for exactness nobody needs. Shared atomic counters on the hot path to enforce a marketing number precisely. Approximate local budgets cover most abuse-prevention cases at a fraction of the cost.

FAQ

What is rate limiting in system design?

Admitting each client’s requests up to a defined budget and rejecting or deferring the rest. It is admission control by policy; a deliberate rule for how much of the system each caller may use, applied before capacity is reached. It differs from load shedding, which survives overload by dropping work, and from backpressure, where consumers pace producers through a pipeline.

Which is better, token bucket or leaky bucket?

They answer different problems. A token bucket spends bursts instantly up to a fixed size, then rejects; right for admission at a front door, where excess should fail fast. A leaky bucket queues bursts and drains at a constant rate; right for shaping traffic toward a dependency that needs even flow, at the cost of latency that grows with the queue.

What is the difference between fixed window and sliding window rate limiting?

A fixed window resets periodically, so the boundary can admit up to twice the budget in a short span. Sliding windows count the actual trailing interval: exactly, by storing timestamps, or approximately, by weighting the previous and current window counters. The exact version costs memory proportional to the request rate; the counter approximation is the usual production choice.

How does distributed rate limiting work?

Either every instance shares atomic counters in a store such as Redis (exact, at the cost of a hot-path round trip and a new failure domain with a fail-open or fail-closed decision) or each instance enforces an approximate local share of the global budget and reconciles periodically, which is cheap and dependency-free but inexact under autoscaling and uneven distribution.

What does HTTP 429 mean?

Too Many Requests, defined in RFC 6585: the caller’s budget is spent. A Retry-After header tells the client how long to wait (delay-seconds or an HTTP-date) and well-behaved clients honor it with jittered backoff rather than immediate retries.

How do you choose the actual limit?

Measure what the endpoint costs at an acceptable latency (the p99 under load, not the average) and set the per-client budget well below the saturation point, per endpoint rather than globally. Then watch the 429 data: where legitimate clients hit limits is a product decision, and where nobody does is budget wasted.

Related articles

• Next read: load shedding, what a system does when the limits were not enough and overload arrives anyway.
• API gateway in system design: the natural place to enforce per-client, per-endpoint budgets.
• backpressure, the consumer-paced counterpart for pipelines instead of front doors.
• retry with backoff, how a well-behaved client answers a 429.
• load balancing in system design, spreading the traffic the limits admit across the fleet.

Last updated on 1 September 2026