Error Budgets: Balancing Reliability, Velocity, and Product Risk

Error Budgets are the unreliability you are allowed to spend. If the objective says the service must be good almost all of the time, the budget is the small share that may be bad. You spend it on launches, failures, and experiments.

When it is gone, you slow down. When it is full, you can take more product risk.

What It Is and Why It Fails

The budget is one minus the objective, applied to the window. Embracing risk frames this as a product choice, not a moral score. A tighter objective means a smaller budget and less room to ship risky changes.

A looser objective means more room and more user pain. The failure is pretending you chose one while you operate the other.

Budgets fail when nobody may spend them and nobody must stop. A chart of remaining budget, with deploys that ignore it, is decoration. A hard stop with no named exception becomes a side channel.

People ship from a branch the policy does not see. The policy has to be real and short enough to follow at 2 in the morning.

They also fail when the indicator is wrong. SLIs explained as good over valid events feed the budget. If probes or a tiny route dominate, you will freeze the team for a fake burn.

If the indicator misses client timeouts, you will keep shipping while users are stuck. Fix the events before you enforce the spend.

A common mistake I have seen is a monthly budget that resets on a calendar day. Teams then rush risky launches on the first of the month and refuse all change on the last day. A rolling window smooths that game.

Use the same window as SLOs for backend engineers already published. However, two windows means two arguments.

What May Spend the Budget

Incidents spend budget whether you like it or not. Launches should spend it on purpose. Experiments, load tests, and failover drills spend it too.

If a drill is free, you will hide a scary test inside a so-called maintenance window. Count user-visible harm. Do not build a list of excuses that zero the chart.

Not every failure should spend the user budget. A broken admin page with three users is a different promise. Give it a separate budget or none at all.

If you pour every blip into one pool, the checkout team will pay for an internal tool. Therefore, split pools by journey.

Planned risk needs a receipt. Write who is spending, how much of the window you might burn, and how you will stop. An illustrative range is a small slice of the monthly budget for one launch, not the entire quarter. If the launch needs the whole budget, the launch is the reliability project.

Burn Rate

Remaining budget is a level. Burn rate is the speed. A full budget with a fast burn is an emergency.

An empty budget with a flat line is a policy problem, not a page. Alerting that works should page on fast burn and ticket on slow burn. The level alone is too late and too jumpy.

Fast burn means you will exhaust a month of budget in hours if the rate holds. Slow burn means you will miss the window if the week continues. You need both alerts. One threshold cannot see a short outage and a long limp at the same time.

Architecture and Policy

Compute two numbers. First, bad events allowed in the window. Second, bad events already seen. The difference is what you have left.

Publish it where deploys happen. A budget that lives in a slide deck will not change a Friday release. Put the number on the pipeline.

Error budget policy is the human half. It says what the team does at certain levels. For example, above a healthy reserve, ship normally. Below a reserve, skip optional risk.

At zero, stop risky launches and staff reliability work. The exact lines are yours, however. The point is to decide them before the incident.

Name the roles. The service owner owns the indicator. The product owner accepts the target, because it is a user promise.

A named approver can break glass when a fix is more important than the policy. If everyone can break glass, the policy does not exist. If nobody can, you will ship the fix in secret.

Record spend. A launch note should say whether it burned budget and how much. An incident review should say the same.

Over a quarter you can see whether you spend on purpose or only on surprises. If surprises take the whole budget, your reliability work is more urgent than the next feature theme.

A Policy You Can Operate

1. Use one rolling window shared with the SLO.
2. Show remaining budget and burn on the service page.
3. Page humans on fast burn, and open a ticket on slow burn.
4. Below the reserve, defer launches that are not reversible.
5. At zero, switch the next sprint toward reliability until the budget returns.
6. Allow break-glass only with a name, a reason, and a follow-up.

Trade-offs You Should Name

A small budget protects users and clusters work into safe changes. A large budget speeds experiments and will show up as pain. A strict freeze is clear and easy to game.

A soft review is flexible and easy to ignore. Pick the version your team will still follow when a launch is late.

Shared platforms need a shared rule. If the platform burns its budget, every product on top is affected. The platform objective should be tighter than the products, or the products cannot meet their promises.

Still, do not let every product page the platform for a local bug. Attribute spend before you halt the world.

Pitfalls and Failure Modes

Budget theater is the main failure. Leaders quote the policy and then demand the launch anyway, with no break-glass note. The team learns the real policy in the meeting, not in the doc.

Write exceptions down. If exceptions are constant, change the objective. You are aiming at the wrong target.

• Resetting the budget on a calendar day so teams game the boundary.
• Excluding deploys and maintenance until the budget never moves.
• Freezing all work, including safe rollbacks and reliability fixes.
• Using one budget for a critical path and a tiny admin tool.
• Paging on the level of the budget instead of on the burn rate.
• Hiding a drill or a load test outside the indicator.

Another failure is punishing the on-call for spend they did not choose. The budget is a system limit, not a personal score. Use it to pick work, not to blame a person for a dependency outage. If the same dependency burns you every month, the next project is that dependency, or a fallback.

Chaos engineering should spend budget in a planned way. A drill that takes the whole window is too big. A drill that does not show up in the indicator did not test the user path.

Schedule drills when you have reserve. Abort if burn runs hotter than the plan. That is how you learn without emptying the month.

We once hit a bottleneck when three teams shared one budget and no owner. Each team assumed another would stop. The launch train continued through a red chart.

Assign one owner for the number. Others may comment. One person decides whether the next risky change waits.

A Policy Record You Can Start From

The record below is illustrative. It assumes a 28 day window and a small reserve before you tighten launches. Map the actions to your real release tool. The numbers are a starting point for a discussion, not a standard you must copy.

service: checkout
window_days: 28
budget:
  bad_events_allowed: derived from objective
  reserve_fraction: one fifth of the window budget
actions:
  above_reserve: ship normally
  below_reserve: reversible changes only
  at_zero: pause risky launches, staff reliability work
break_glass:
  approver: service owner
  require: reason, expiry, follow-up note
alerts:
  fast_burn: page
  slow_burn: ticket

Derive allowed bad events from the objective and the expected valid events. Do not hard-code a count you copied from another service. Recompute when traffic grows, or the same raw error count becomes a smaller share and the policy goes soft without a decision. Review the reserve line each quarter with product, not only with on-call.

Performance, Scale, and Cost

The math is cheap. The cost is the reliability you hold in reserve and the features you delay. Price both.

A team that never spends budget is paying for unused reliability. A team that always sits at zero is paying in incidents and rework. The useful zone is a budget that sometimes gets low and often recovers.

At many services, a manual spreadsheet will rot. Compute remaining budget in the same pipeline as the SLO. Alert from that pipeline.

A weekly human review is still useful, but the number should not depend on someone updating a cell. If the job fails, show unknown, not a full budget.

Scale the policy with templates. Each service can set its own objective. The actions at reserve and at zero can be shared.

Local exceptions belong in the service record. If every team invents a new freeze ritual, nobody will remember the steps during an incident. Boring policy is a feature.

Watch the cost of slow burn that never quite pages. It eats the budget by the end of the window and surprises a launch. The ticket on slow burn exists for this.

If tickets are ignored, the policy is already failing. Escalate the ticket, or accept a looser objective in public.

Finally, connect budget spend to unit cost. A reliability project that doubles capacity to protect a tiny budget may be the wrong trade. Sometimes the honest move is a slightly looser objective and a cheaper design. Product risk includes the bill, not only the error rate.

Key Takeaways

• The budget is the allowed bad share of the SLO window.
• Use a rolling window so teams cannot game a calendar reset.
• Decide actions at healthy, low, and zero before you need them.
• Page on burn rate, not only on an empty budget.
• Count user-visible harm, including drills and launches.
• Assign one owner, and write down every break-glass.

FAQ

Does a full budget mean we should take more risk?

It means you can afford more risk than a team that is already empty. It does not mean you should create outages for sport. Spend on launches you already want. If the budget stays full for a long time, ask whether the objective is tighter than users need.

Should reliability fixes stop when the budget is zero?

No. A zero budget should stop risky feature work. Rollbacks, safeguards, and fixes are how the budget returns.

If the policy blocks the repair, people will route around it. Write that exception in the policy so nobody has to improvise.

How do dependencies affect our budget?

If users feel the dependency failure, your journey budget burns. That is fair. Also track the dependency’s own budget so you know where to invest.

Do not ignore your burn because another team is at fault. The user does not care which team it was.

Can we borrow budget from the next window?

A rolling window already blends days. Do not add a loan program on top. Borrowing sounds flexible and becomes permanent debt.

If you need more room, change the objective in a review with product. Then live with the new promise.

Write the actions for healthy, low, and zero budget on the same page as the SLO. Name the approver for break-glass.

Then put remaining budget on the deploy pipeline and review last month of spend. If the policy and the launches disagree, change one of them this week, in writing.

Last updated on 17 September 2026.