SLIs Explained: Choosing the Right Service Level Indicators

SLIs Explained in practice means one thing: count the events users care about, then say which of those events were good. That ratio is the indicator. If you pick a signal users cannot feel, later objectives and pages will be busy and useless. Choose the indicator before you choose the target.

What It Is and Why It Fails

A service level indicator is a measurement. It is not the target, because the target comes later. Service level objectives sit on top of the indicator and say how high the ratio must stay.

Teams skip this split and argue about nines while the events are wrong. Fix the events first.

The usual shape is good events divided by valid events. Good means the user got a correct and timely result. Valid means the call was a real attempt you agreed to serve.

A bad request the client sent may be valid or not, depending on the promise. Therefore, write that down. Ambiguity here becomes a monthly argument.

Indicators fail when they are averages. An average latency can look fine while a large minority of users wait. A common mistake I have seen is an SLI on mean CPU.

However, users do not buy CPU. They buy a finished action. Use a threshold on the user action, and count how many events fall inside it.

They also fail at the wrong vantage point. Server logs miss calls that died in the client, the network, or the edge. Client metrics miss calls from other services.

Pick the point that matches the promise. If you cannot measure there yet, say so next to the chart. Do not imply coverage you do not have.

Classes of Indicator

Availability asks whether the service returned a successful result. Latency asks whether it was fast enough. Correctness asks whether the result was right. Freshness asks whether the data was new enough.

Durability asks whether stored data is still there later. Each class needs its own events. Do not mash them into one ratio and hope.

Availability is the easiest and the most abused. A 200 with a wrong price is available and still a failure. If correctness matters, count it.

You may sample correctness if a full check is expensive. Also, label the sample. A biased sample will hide the worst tenants.

Freshness matters for feeds, search, and replicas. A successful read of stale data can be a bad event if the product promise is near real time. Define the age limit in the indicator. Then a slow pipeline shows up even when HTTP status looks perfect.

Valid Events and Excludes

Exclude events that are not the service’s job. Health checks from a load balancer are a common exclude. So are requests you rejected on purpose because the caller was over quota, if the promise is to reject them.

Also, do not exclude 500s. Also, do not exclude timeouts. Those are the failures when users feel them.

Be careful with client errors. A 404 on a missing optional asset may be valid and good. A 404 on the primary checkout document is bad.

Status code alone is a weak definition. Pair it with the route class. Health checks should stay out of the user ratio, or a probe storm will paint a fake outage or hide a real one.

Architecture and Implementation

Emit counters, not only logs. You want a good counter and a valid counter, or a histogram you can turn into a threshold count. Monitoring distributed systems favors a small set of stable metrics.

Therefore, follow that. An indicator that depends on a fragile log query will break during the incident you need it for.

Put the threshold in the metric definition when you can. For latency, count requests faster than the chosen limit, and count all valid requests. Charts of percentiles still help humans. The indicator used by SLOs for backend engineers should be the ratio, because a budget needs a count of bad events.

Standardize labels. Service, route class, and status class are enough for most indicators. A label per customer will explode cost and slow queries.

If one tenant needs a view, aggregate in a separate, coarser metric. Do not put the high-cardinality label on the SLI itself.

Document the query next to the service. When someone changes a status code, the indicator should be in the same review. I keep a short spec: vantage point, good rule, valid rule, and known blind spots. If the spec and the dashboard disagree, the spec wins until you change it on purpose.

How to Choose One

1. Write the user sentence, including a time or freshness limit.
2. List the events that prove the sentence, and where you can observe them.
3. Define good and valid in one paragraph a new teammate can test.
4. Drop any event that is a probe, a retry duplicate, or not a user attempt.
5. Replay a known bad hour and confirm the ratio falls.
6. Only then attach an objective and an alert.

Trade-offs You Should Name

A client-side indicator sees the truth and is noisy, sampled, and late. A server-side indicator is complete for what the server saw and blind past the socket. A strict correctness check is honest and expensive.

A status-code check is cheap and can bless a wrong body. Pick the blind spot you can live with, and name it.

Retries complicate the count. If the client retries and the second call works, the user may be fine while the server saw a failure. Decide whether the indicator is about the attempt or the outcome. For user pain, outcome is better.

For capacity, attempts matter. Use two indicators if you need both. Do not average them.

Pitfalls and Failure Modes

The ratio lies when the denominator changes meaning. A deploy that starts rejecting bad input with 400s can move the ratio a lot if 400s were valid failures yesterday and excluded today. Then version the rule.

Then annotate the chart. Otherwise you will celebrate a definition change as a reliability win.

• Using averages or CPU as if they were user events.
• Mixing probe traffic with user traffic in one ratio.
• Letting one rare route dominate because volume is tiny.
• Counting each retry as a separate user failure.
• Hiding partial data loss behind a successful HTTP status.
• Building the indicator as a dashboard that nobody can recompute.

Low volume is a math trap. Two failures in ten calls look like a disaster. Two failures in ten million do not.

An indicator can still report the ratio, but alerting that works must demand a minimum count. Also avoid a single global SLI that lets a noisy test route drown production.

Clocks and freshness checks fail in odd ways. If the writer and the reader disagree on time, every object looks stale or every object looks fresh. Use a timestamp from the source of truth, not the cache clock. We once hit a bottleneck when a replica’s clock jumped and the freshness SLI paged all night while the data was fine.

Error budgets inherit every lie in the indicator. A bad SLI spends budget on the wrong hours and saves it during real harm. If the budget feels random, do not tune the policy first.

Replay the events. The indicator is wrong until a known incident shows up as bad events.

A Spec You Can Start From

The spec below is illustrative for a read API. It counts a request as good when the status is successful and the body is under a latency limit. It drops health checks.

It does not claim to see client timeouts. Copy the shape, not the numbers, unless they match your promise.

indicator: catalog_read
vantage: server
good:
  status_class: 2xx
  latency_ms_under: 300
valid:
  route_class: user_read
  exclude:
    - user_agent: health-probe
blind_spots:
  - client timeouts before the request is logged
  - wrong body that still returns 200
unit: good_events / valid_events

Implement the counters in the request path or at the edge proxy. Test them with a fixture: ten good, two slow, one probe. The probe must not change the ratio.

The slow calls must. If the unit test is hard to write, the definition is too vague to operate. Simplify it before you build a dashboard.

Performance, Scale, and Cost

Indicators should be the cheapest metrics you own, because you query them forever. A histogram with a modest bucket set is enough for a latency ratio. A histogram with hundreds of buckets, times thousands of routes, will dominate the bill. Pre-aggregate to route class.

Sampling is acceptable for expensive correctness checks. It is a poor default for availability, because the failures are the rare events you most need. If you sample, sample failures at a higher rate than successes, and record the weights. An unweighted sample will undercount the bad minutes.

At high scale, compute the ratio in a streaming job, not in the browser. Dashboards that sum raw series on load will time out during the incident. Store a rolled-up good and valid per minute.

Keep raw series for a short debug window. The SLI only needs the rollup.

Cardinality budgets are reliability tools. When a team adds a label, the SLI query gets slower and sometimes silently drops series. Block new labels on the indicator metric in review.

Put debug labels on a different metric with a short retention. The indicator stays boring on purpose.

Also plan for missing data. A gap should not look like perfect reliability. If the counter stops, mark the window as unknown, not as zero bad events.

Unknown time should not mint free budget. That single rule prevents a scraper outage from looking like your best month.

Key Takeaways

• An SLI is good events over valid events, not a target and not a host metric.
• Match the class to the promise: success, speed, correctness, freshness, or durability.
• Say where you measure and which failures you cannot see.
• Keep probes and test traffic out of the user ratio.
• Replay a known bad hour before you alert on the indicator.
• Treat missing data as unknown, not as a perfect score.

FAQ

Is a percentile an SLI?

A percentile is a view of latency, not yet an indicator you can budget. Turn it into a ratio by counting events under a threshold. You can still graph the percentile for humans. The objective should use the count so bad events add up in a straight line.

Where should you measure?

Measure at the point of the promise. For a public API, the edge or the client sees more of the truth. For an internal queue worker, the worker’s own success and age metrics are the right place. If you measure only in the database, you will miss the app errors above it.

How do you handle batch jobs?

Use a deadline and a completeness check. A run is good when it finishes on time and processes the expected input. A run that exits zero after skipping half the file is not good. HTTP-style ratios are the wrong shape for a nightly job.

Can one service publish several SLIs?

Yes. A read path and a write path often need different indicators. Keep each one simple.

Do not multiply them until you have an objective and an owner for each. Unused indicators rot and then mislead the next incident.

Write the user sentence for one route. Define good and valid in a short spec, and implement counters you can test with a fixture.

Replay a bad hour from last month. If the ratio does not move, change the indicator before you set an objective or a page.

Last updated on 07 September 2026.