P99 Latency Explained: Measurement, Debugging, and SLOs

P99 Latency Explained in one line: in a set of calls, 99 percent finish at or below that time, and one percent finish slower. It matters because that one percent is where fan-out, queues, and bad shards show up. An average will not name them. You should measure P99 in a histogram, debug it with a trace, and only then put it in an SLO.

What P99 means

Sort the durations. The P99 value is the smallest time that still covers 99 of each 100 calls. It is not the average of the slow calls.

It is not a promise that every user is happy 99 percent of the time. A user who sends many calls, or a page that fans out, can miss that promise even when the service chart looks calm.

P99 is also not P999. The last tenth of a percent is often one tenant, one key, or one pause. If you set an SLO at P999 before you understand P99, you will page on noise or you will ignore a broad slowdown.

Start with the mark that matches the user budget. Then watch a higher mark when you hunt a single outlier.

Low traffic lies. If you only have 50 calls in a window, one slow call moves P99 a lot. The number jumps even when nothing structural changed.

Use a longer window, or require a minimum count, before you page. Still, do not hide a real regression behind a huge window. A short window for humans and a longer window for the page is a fair split.

Why the number fails in production

In my experience, the P99 page fires and the first graph is a single number for the whole fleet. That number cannot tell you which shard, which zone, or which build is slow. You acknowledge the page and then lose the hour in guesses. After you break the histogram down by the dimensions you can act on, the same page is a ten minute check.

A common mistake I have seen is to compute P99 inside each pod and then average those P99 values. The average of percentiles is not the percentile of the whole. A quiet pod with a tiny sample can pull the chart down while a hot pod misses the SLO.

Aggregate the histogram first. Read P99 from the aggregate. Prometheus histogram guidance walks through why the buckets must be combined before the quantile.

Another failure is the wrong clock boundary. Server time can look fine while the client waits on a stalled connection. If the SLO is about the user, measure where the user waits.

If the SLO is about a leaf you own, say so in the SLI. The SLI you publish should name the hop, the status codes you include, and the window. Vague SLIs make honest P99 look like a fight.

Buckets and what they hide

A histogram counts calls into buckets. The P99 you read is an estimate inside the bucket that holds the 99th percent. If your buckets jump from 100 ms to 1 second, you cannot see a move from 200 ms to 800 ms.

That move is the whole incident. Pick buckets around the SLO, tighter near the target and wider in the far tail.

Buckets are also a cardinality bill. A bucket set per method is useful. A bucket set per user id is how you blow up the time series database.

Keep high cardinality in traces and logs, sampled on the slow path. Keep the histogram labels boring: method, status class, and maybe shard group. When you need one user, jump to a trace, not to a new label.

Architecture for measurement

Record a histogram at the client that holds the user budget, and record another at the server that does the work. The gap between them is network, queue, and client overhead. When P99 moves, the gap tells you which side to open. One histogram for the whole process is not enough once you have more than one dependency.

Histograms, summaries, and traces

A summary computes quantiles on the process and exports the quantile. You cannot sum those quantiles across processes into a fleet P99. A histogram exports buckets, and you can sum buckets.

Therefore use histograms for a fleet SLO. Use a summary only when the process is the whole service and you will never aggregate it.

Traces explain a single slow call. They do not replace the histogram. OpenTelemetry traces show spans, and exemplars can jump from a histogram bucket to a trace.

Sample the slow bucket more heavily than the fast one. If you sample at random only, you will store thousands of boring fast calls and miss the P99 offender.

Tie the number to an SLO

The Google SRE chapter on objectives frames the idea: pick a target the user needs, and an error budget that lets you ship. P99 as an SLI means the fraction of calls at or under the target, or the P99 value versus a threshold. Be explicit which one you alert on. A burn rate on the fraction is usually calmer than a raw P99 that jitters.

Write the objective in user terms, then derive the leaf target. If the page budget is 300 ms and the page waits for this leaf, the leaf cannot also be 300 ms at P99. Leave room for the rest of the path.

SLOs for backend engineers are the place to record that split. If you copy one number onto every service, the leaves will look green and the page will not.

Trade-offs

Finer buckets make a better P99 estimate and cost more series. More labels find the culprit faster and cost more series. Full tracing explains everything and costs the most.

A raw P99 alert is simple and noisy. A burn rate is calmer and slower to design. Choose the cheapest view that still tells you what to fix.

There is also a product trade. A strict P99 forces you to shed, cache, or cut fan-out. A loose P99 is cheaper and will show up as complaints.

Read why the tail matters before you pick a number you cannot buy capacity for. An SLO you miss every day trains people to ignore the page.

Pitfalls and failure modes

Status codes change the meaning. If you drop timeouts from the histogram, P99 improves while users suffer. If you include client cancels that were not your fault, P99 looks worse than the service.

Decide in the SLI. Document it next to the chart so the next on call does not relitigate it at night.

• Buckets are too wide near the SLO, so a real regression stays inside one bucket.
• Someone averages P99 across pods and calls the result the fleet number.
• The window is shorter than the traffic, so one call pages the team.
• Timeouts are excluded, so the chart rewards failure.
• Labels include user id, and the metrics store falls over.
• Traces are sampled only at random, so the slow path is missing.

Deadlines interact with the metric. A call that hits timeouts in distributed systems should count as slow or as an error, in a way you chose on purpose. If the client retries and both attempts sit in the histogram, you can double count one user wait. Prefer the user request as the SLI event, and keep the leaf attempts in a separate chart for debugging.

How to debug a P99 jump

1. Confirm the jump is in the aggregate histogram, not in an average of quantiles.
2. Split by build, zone, and shard group.
3. Open exemplars from the slow bucket and read the longest span.
4. Check queue depth and whether hedges or retries added load.
5. Compare client P99 with server P99 to place the wait.
6. Only then change a timeout, a pool, or a deploy.

A histogram you can scrape

The buckets below cluster around a 100 ms regional target. They are an illustrative set, not a universal standard. Put more buckets near your SLO, and keep a wide last bucket so nothing is silent.

Scrape the histogram, sum it across pods, then estimate P99. Do not alert on a quantile computed inside one pod.

histogram_buckets_ms:
  - 5
  - 10
  - 25
  - 50
  - 75
  - 100
  - 150
  - 250
  - 500
  - 1000
labels:
  - method
  - status_class
slo:
  target_ms: 100
  percentile: 99
  min_calls_per_window: 500

The minimum call count stops a quiet window from paging. Five hundred is an illustrative floor for a busy method. A rare admin method needs a different rule, often a longer window instead of a page. Also, keep status class as a label so you can decide later whether errors sit inside the latency SLO or beside it.

When a zone cannot reach a replica, the tail is often the timeout, not the CPU. Treat that as a partition signal as well as a latency signal. The notes on network partitions explain why one unreachable replica pins every call that still waits for it. Stop waiting for a full set when a quorum is enough, and the P99 chart will show the change.

Performance, scale, and cost

Histograms are cheap if the labels stay small. Each bucket is a series per label set. Ten buckets and two labels with a few values is fine.

Ten buckets times thousands of tenants is a bill and a scrape problem. In an illustrative range, teams that keep labels to method and status class scrape latency metrics without a special cluster. Teams that add tenant id then spend the quarter on the metrics tier.

Traces cost more per event, so sample them. A higher sample rate on the slow bucket gives you debug power without storing every fast call. Exemplars are the bridge.

They point from the bucket that missed the SLO to one concrete trace. Without that bridge, on call guesses which request to open. With it, the first trace is already in the bad bucket.

The compute cost of a bad P99 is usually larger than the metrics cost. A queue that holds requests for a second ties up threads, memory, and connection slots. Those slots are the capacity you could have used for fast calls.

Fixing P99 often returns capacity. Buying capacity first, without a histogram, often feeds the queue. Measure, then spend.

When you tighten the target, tighten the buckets too. Otherwise a real move stays inside one bucket and looks flat. Mark the bucket change on the graph so a dip is not a fake win.

Key Takeaways

• P99 is a rank in the call set, not the average of the slow calls.
• Sum histogram buckets across pods before you read the fleet percentile.
• Place tight buckets around the SLO or the regression will hide inside one bucket.
• Do not average per pod percentiles and call that number P99.
• Page on burn rate and a minimum call count, not on a tiny sample.
• Jump from a slow bucket to a trace when you need the cause.
• Name the hop and the status rules in the SLI so the number stays honest.

FAQ

Is P99 the same as 99 percent of users?

No. P99 is about calls, or about whatever event you put in the histogram. A user with many calls, or a page that fans out, can have a worse experience than the call percentile suggests.

If the promise is about users, define the SLI on a user event. Do not rename a call metric and hope.

Why did P99 move when the mean did not?

The slow slice changed and the common path did not. A pause, a hot shard, or a retry pile can move the 99th rank while most calls stay fast. That is normal. Trust the percentile for user pain, and use the mean only as a capacity hint.

Can I alert on P99 directly?

You can, and it will be noisy when traffic is low. Prefer the fraction of calls over the target, with a burn rate and a minimum count. Keep raw P99 on the dashboard for humans. The page should fire when the budget is going, not when one window had 30 calls.

Which hop should own the SLO number?

The hop that matches the promise. A user facing SLO belongs on the client that starts the user request. A dependency SLO belongs on the calls to that dependency.

Publish both if you need both, and do not expect them to be the same number. The leaf must be tighter if the page waits for it.

Add a fleet histogram for one user facing method, with buckets packed around the target you claim. Then stop averaging per pod quantiles, and page only when the call count is high enough to trust. Next, attach one exemplar from the slow bucket to a trace and confirm the longest span is a cause you can change. If you cannot name that span, the SLO is ahead of the telemetry.

Last updated on 12 September 2026.