Latency vs Throughput: The Two Numbers That Define Performance

A payment API is benchmarked at 20,000 requests per second (impressive) while its slowest 1% of requests quietly take 1.8 seconds each. Another service answers every request in 12 milliseconds but collapses above 200 requests per second. Which one has a performance problem? Both. Latency and throughput measure different properties of the same system, and a system can be excellent on one axis while failing on the other.

Latency is the time a system takes to complete one operation: one request, one query, one message. Throughput is the number of operations the system completes per unit of time. Latency describes the experience of a single request; throughput describes the capacity of the system as a whole. Neither number alone describes performance; together they define it.

Two different questions

Latency answers: how long does one unit of work take? Throughput answers: how many units of work per second can the system finish?

They are not two ways of measuring the same thing, and one does not determine the other:

• A single-threaded in-memory key-value store can serve each request in microseconds (excellent latency) but use one CPU core and max out at a modest request rate (a poor throughput ceiling).
• A batch analytics job processing 500 MB chunks achieves enormous throughput while every individual record waits minutes (terrible latency, irrelevant for batch).

Which number matters depends on the workload, which is why the first design question for any system is: is it latency-bound or throughput-bound? Before answering, it helps to separate throughput from a term it is often confused with.

Throughput vs bandwidth

Bandwidth is the theoretical maximum rate a link can carry, the width of the pipe. Throughput is the rate actually achieved. A 10 Gbps network link with an inefficient protocol and small packets may deliver a fraction of its bandwidth as throughput. Bandwidth is a property of the infrastructure; throughput is what your system actually gets out of it.

Little’s Law: the relationship you cannot escape

Latency and throughput are not independent. The connection is a queueing theory result so fundamental it gets its own name. Little’s Law says: in any stable system, the average number of requests inside the system equals the arrival rate multiplied by the average time each one spends inside:

L = λ × W

L  (average requests in the system
λ ) average arrival rate (requests per second)
W ; average time spent in the system (seconds)

If a service receives 100 requests per second and each request spends 250 milliseconds inside; queueing, processing, responding, then on average 25 requests are in flight at any moment (100 × 0.25). The law holds for any system in steady state, however it is built, and it turns three practical tricks:

• Capacity planning. If one instance comfortably holds 100 concurrent requests and the latency target is 200 ms, one instance absorbs about 500 requests per second (100 ÷ 0.2). Required fleet size follows from demand.
• Bottleneck diagnosis. Request rate unchanged but in-flight requests climbing? Latency is rising somewhere. The dashboards must agree with each other.
• The shape of latency work. To reduce latency, either process faster or process more at once, which is why every latency problem eventually becomes an optimization problem or a parallelism problem.

The queueing trap

Little’s Law hides the cruelest fact of systems under load: queueing delay is not linear. As utilization approaches 100%, wait times grow disproportionately. A server at 50% utilization has slack in every interval; a server near saturation has almost none, so requests pile up behind stragglers and every hiccup becomes a queue. The practical consequences:

• Latency targets, not average capacity, size production fleets. A system sized exactly to its average demand queues during every small spike.
• Throughput can look fine while latency falls apart. Past a knee that varies by system, every additional request admitted makes all requests wait longer. Watching p99 latency is how you notice a saturated system before it breaks.

Percentiles: why the average lies

The average of a latency distribution is convenient and misleading. A mean of 80 ms can mean 70 ms for 99% of requests and 800 ms for 1%, and nothing about the mean tells you which world you are in. This is why latency is reported in percentiles:

• p50 (median): half of requests are faster.
• p90: 90% of requests are faster.
• p99: 99% are faster; the slowest 1 in 100 requests sits beyond this line.
• p99.9 and beyond: the territory of large fleets and high-stakes systems.

The gap between p50 and p99 is called tail latency, and it is where user experience actually lives:

• A page load that touches 50 backend calls waits for the slowest of them. If each backend is slow just 1% of the time, the probability that at least one of 50 is slow is roughly 40% (1 − 0.99⁵⁰). User-facing latency is dominated by the tail, not the average.
• The users who hit the tail are disproportionately the heaviest ones: the accounts with the most data, the most cache misses, and the most concurrent activity. Google’s The Tail at Scale (Dean and Barroso, 2013) argued that tail latency is a systems-design problem rather than a fact of life, and it changed how large services are engineered.

Throughput has no equivalent subtlety (requests per second is what it is) but the two numbers interact: past saturation, throughput plateaus while p99 latency climbs toward timeouts. Watching p99 latency is how you notice a saturated system before it breaks.

Where latency actually comes from

Latency is the sum of everywhere a request goes, and the components differ by orders of magnitude. Rough figures that have held as orders of magnitude for years (intuition, not spec sheets):

Three design conclusions fall out of that table:

• Memory is about 1,000× faster than SSD, and SSD is about 100× faster than a disk seek. Whether data sits in RAM, on flash, or on spinning disk is the single biggest latency decision a data system makes; it is why caches exist (caching in system design).
• Network hops are not free, but they are predictable. Within a datacenter, a hop costs a fraction of a millisecond; across continents, it costs human-noticeable time. Keeping latency-critical workloads near users is why CDNs push content to the edge (how a CDN works).
• The software-made latencies are the ones under your control. Serialization, GC pauses, lock contention, cold caches, and chatty protocols do not appear in the table, and they routinely dominate real response times.

When latency and throughput fight each other

You can often improve either number by spending the other. Recognizing the trade is a core design skill:

• Batching raises throughput and hurts latency. Grouping writes, aggregating messages, or pipelining requests amortizes overhead across many items, but the first item in a batch now waits for the last to be ready. Message queues exist partly to make this trade explicit and tunable.
• Synchronous replication costs latency. Acknowledging a write only after a replica confirms it adds at least one network round trip per write: durability and consistency bought with milliseconds (database replication, the CAP theorem).
• Caching trades consistency work for latency. Serving from memory is fast; keeping it correct is the cost (distributed caching).
• Parallelism can improve both. Sharding a hot service across machines improves latency and throughput at the same time, if the distribution is sound (consistent hashing).
• Latency-hiding techniques; asynchronous writes, read replicas (database replication), speculative execution, make latency invisible to the user without making it smaller. Sometimes that is the right answer: the user’s experience is the target, not the stopwatch.

Which number to prioritize is a product question with an engineering answer:

Measuring both honestly

• Measure under load. An idle system has no queues, and its latency says nothing about behavior at peak. Load-test through the whole path (balancer, service, data tier) or the numbers are fiction.
• Report percentiles, not averages. p50, p95, p99, per endpoint. If only the mean is available, ask what it is hiding.
• Measure where the user is. Server-side latency excludes the network, DNS, and connection setup the user actually experiences. Client-side measurement catches what the server cannot see.
• Beware coordinated omission. A benchmark that fires the next request only after the previous one completes undercounts exactly the spikes that matter, because the load generator politely waits out the stalls. Real clients do not wait; honest benchmarks should not either.
• Throughput needs a latency guardrail. “We served 40,000 requests per second” means nothing if p99 was 8 seconds. The only honest capacity number a system has is the pair: throughput at a stated latency target.

Common mistakes

• Reporting averages only. The mean hides the tail, and the tail is the product.
• Benchmarking with a single client. One connection serializes requests and never builds a queue. The system looks far faster than it will behave in production.
• Optimizing throughput on a latency-bound path. Adding batch size to a user-facing API raises capacity on paper and page-load times in reality.
• Treating bandwidth as throughput. A 25 Gbps NIC delivers far less to the application; protocol overhead, packet sizes, and round trips decide the real number.
• Ignoring the fan-out multiplier. One request that fans out to 30 backends inherits 30 tail latencies. Extra hops through gateways and proxies add up too; API gateways are worth their cost, but their latency belongs in the budget.

FAQ

What is the difference between latency and throughput?

Latency is how long one operation takes; throughput is how many operations complete per second. A system can have excellent latency and poor throughput (a fast single core), or excellent throughput and poor latency (a large batch pipeline). Any performance claim that quotes one without the other is incomplete.

What is p99 latency?

The duration below which 99% of requests complete. If p99 is 400 ms, then 1 request in 100 takes longer than 400 ms. Percentiles exist because latency averages hide the slow requests that dominate user experience.

Does adding more servers reduce latency?

Not directly. More servers reduce queueing, which reduces latency under load, but every hop between services adds its own round trip. Latency improves when you remove queueing and distance (caching, locality, faster storage), not merely when you add capacity. The queueing dynamics are in Little’s Law above.

What is a good latency target?

There is no universal number. Set targets from product needs: interactive paths usually need responses in tens to low hundreds of milliseconds; batch work can tolerate minutes. The specific target matters less than having one and reporting against it in percentiles.

Why does latency increase when throughput is high?

Queueing. As a system approaches capacity, requests wait in line behind each other, and wait time grows disproportionately as utilization nears 100%. Latency degrades before throughput does, which is why p99 latency is the earliest warning of saturation.

Related articles

• Next read: availability vs reliability vs durability; speed says nothing about whether the system is dependable.
• vertical vs horizontal scaling, how systems absorb more load.
• caching in system design and how a CDN works, the two standard attacks on latency.
• message queues, throughput smoothing with explicit latency trade-offs.

Last updated on 12 September 2026