Chaos Engineering: Principles, Experiments, and Production Safety

Chaos Engineering is how you learn whether the system survives failure, by causing a controlled failure on purpose. When you only test the happy path, the first real outage is the experiment. If you inject faults with no hypothesis and no stop button, you are not learning.

You are breaking production. Therefore treat each drill as a small, reversible test with an owner.

What It Is and Why It Fails

The practice starts from a steady state you can measure, then varies real conditions, such as latency, loss, or a dead dependency. Principles of Chaos Engineering put it simply: form a hypothesis, vary real events, run in production when you can do it safely, automate, and limit the blast radius. When teams skip the hypothesis, they collect stories instead of evidence.

Experiments fail when steady state is a vibe. You need a number. Error ratio, latency under a limit, or queue age can work.

SLOs for backend engineers are a good steady state because they already describe user harm. If the drill does not move a user signal, you tested a host metric, not the product.

They also fail when the blast radius is the whole company. A common mistake I have seen is a first game day that kills the primary database. The team learns nothing new and spends the day restoring.

Start with one dependency call, one percent of traffic, or one shard. Also write the abort rule before you start. If you invent the stop condition after users complain, you started too big.

Addressing cascading failures explains how one slow hop can take down its callers. Chaos is how you check that your limits actually stop that chain. AWS Fault Injection Service is one way to add latency or stop instances with an audit trail.

The tool is optional. The abort switch is not.

Hypothesis Before Faults

Write the hypothesis in one sentence. If payments add a few hundred milliseconds, checkout still stays inside the objective because the timeout and the fallback are in place. Then the result is yes or no.

Also record what you will watch. If you watch everything, you will see patterns that do not answer the question.

A good hypothesis can fail. When the result is obvious, the drill is a demo. When you truly do not know whether the fallback works, the drill is worth the risk.

Still keep the risk small. You can widen it after the small run matches the hypothesis.

Production Versus a Copy

A staging copy misses the traffic shape, the data size, and the third parties that make production special. When the question is about user impact, production is the honest place, with a tiny radius. If you cannot limit the radius, run in a shadow or a single cell first. Also do not use a toy environment to claim you are ready for a region loss.

Some faults should stay out of production until the small ones are dull. Disk fill on a singleton, or dropping the only leader with no fence, is how drills become incidents. When the safety tools are new, practice the abort on a harmless fault. Then raise the stakes.

Architecture of an Experiment

Each experiment needs an owner, a window, a fault, a steady state, and a stop. The owner can halt it without a committee. The window avoids your peak unless the hypothesis is about the peak.

The fault is one change, so you know what caused the result. The steady state is measured before, during, and after. The stop is automatic and manual.

Inject as close to the real failure as you can without skipping safety. Extra latency on one client call is safer than powering off a building. Packet loss on one host is safer than a bad route for the region.

When you need failover architecture evidence, you will eventually move a real path. Do that only after timeouts, alerts, and the fence have passed smaller tests.

Alerting that works should fire for the user symptom, not only for the chaos tool’s own event. If the drill burns the objective and nobody is paged, the alert is the bug you found. Also silence the pages that you know the drill will cause, with an expiry. A silence with no expiry hides the next real fault.

Automate the boring drills once they are safe. A weekly latency injection that always passes will rot unless it still checks the hypothesis. When it starts failing, that is a gift.

Someone removed a timeout or a fallback. The scheduled run is how you notice before a real dependency gets slow.

How to Run the First Drill

1. Pick one dependency and write the hypothesis in a sentence.
2. Choose a steady state tied to users, and note the baseline.
3. Limit traffic, duration, and which cluster can see the fault.
4. Set an automatic abort on fast error burn and a manual stop.
5. Run during a staffed window, with the owner watching.
6. Write what happened, including a failed hypothesis, before you widen the radius.

Trade-offs You Should Name

Smaller blasts teach less about full failure and are much safer. Larger blasts teach more and can become the outage. Production is realistic and risky.

A copy is safer and often wrong. Automatic drills catch regressions and can fire at a bad time if the guard is weak. Also, a game day builds shared skill and does not scale to every service every week. Mix a few automated checks with rare, staffed game days.

Timeouts in distributed systems are a common first hypothesis. If you add delay and the caller waits forever, you found a cascading failure before customers did. If you add delay and retries multiply the load, you found a retry bug.

Both results are useful. Neither requires you to delete data.

Pitfalls and Failure Modes

The drill becomes an incident when abort does not work. Test the stop on a no-op fault. When the stop depends on the same network you are breaking, it will not save you.

Also keep a second path, such as a local flag on the injector, that does not need the broken dependency. If you have one stop button and it is inside the blast radius, you do not have a stop button.

• Starting with data loss or a full database kill.
• Running without a hypothesis or a steady state.
• Forgetting to expire alert silences after the window.
• Injecting faults during an already burning objective.
• Letting the tool’s success mean the user path was fine.
• Widening the radius before the small run is understood.

Error budget matters. A drill spends user harm on purpose. When the budget is already gone, do not add a game day.

Schedule it when you have reserve, and abort if burn runs hotter than the plan. A team that chaos-tests through an empty budget is choosing a second incident.

Observability gaps waste the drill. If you cannot see the dependency latency and the user ratio, you will argue afterward. Add those charts before the fault.

We once hit a bottleneck when the only proof of the drill was a chat scroll. The next day nobody could say whether the fallback had served stale data or errors.

People fail too. A surprise drill on an unprepared on-call is a trust problem. Tell the primary that a window exists, even if you do not reveal the minute.

Also invite the owners of the dependency. A drill that surprises a partner team becomes a political incident and a technical one.

An Experiment Record You Can Start From

The record below is illustrative. It delays one dependency for a small share of traffic and aborts on fast burn. Copy the shape.

Do not copy the delay if your normal timeout is shorter or longer. The fault should sit near a real slowness, not at a random extreme, for the first run.

experiment: payments_latency
hypothesis: checkout stays inside the objective when payments slow down
steady_state: checkout_good_ratio
blast_radius:
  traffic: small_slice
  duration_minutes: 10
fault:
  type: latency
  target: payments_client
abort_when:
  - fast_burn_on_checkout
  - owner_stops
window: staffed_off_peak
silence:
  expire_with_experiment: true

After the run, keep the note short. State the hypothesis, the result, and the next change. If the result matched, schedule a slightly wider run or automate this one.

If it did not, fix the timeout or the fallback before any larger fault. A library of failed hypotheses you ignored is not a program. It is a warning label.

Performance, Scale, and Cost

The injector should be cheap and boring. A sidecar that adds delay must not become the bottleneck you are trying to study. Cap its CPU.

When the injector falls over, you get a fake result. Also measure the cost of game days in engineering time. A monthly all-hands drill is expensive. A weekly automated check on one call is usually the better scale path.

Do not inject faults that multiply load without a limit. Retry storms are a result you may want to see once, at tiny scale. At full scale they become a denial of service you caused.

Therefore keep a global stop on the number of extra calls. If you cannot cap retries, do not test them in production yet.

At many services, centralize the platform and decentralize the hypothesis. One team can own the injector, the abort, and the audit log. Each service team owns what good looks like.

If every team builds a private fault tool, someone will ship a tool with no abort. Shared guardrails scale better than shared curiosity.

Store results where the next on-call can find them. A wiki novel will not be read. A one page note next to the service, plus a chart link, is enough.

Also track how many critical dependencies have never been delayed on purpose. That list is your real backlog, not a catalog of tools.

Finally, stop when the practice becomes theater. Green drill badges with no failed hypothesis mean the tests are too soft. Add one harder question per quarter.

When a drill cannot fail, it cannot teach. Spend the budget on a question you are afraid to answer.

Key Takeaways

• Start from a user-facing steady state and a one-sentence hypothesis.
• Keep the first blast radius tiny, and grow only after a clear result.
• Put an automatic abort and a manual stop outside the fault path.
• Do not run drills when the error budget is already gone.
• Silence only the alerts you expect, and expire that silence.
• Automate the dull checks, and use game days for rare, wide faults.

FAQ

Should the first experiment run in production?

Yes, if you can limit who is affected and you can stop quickly. Also use a non-production cell when you cannot limit the radius. The goal is evidence about real behavior, not a dare. If abort is untested, you are not ready for production.

What fault should you start with?

Add latency to one downstream call. It is reversible and it reveals timeout and retry bugs. Also avoid deleting data, filling every disk, or killing the only database on day one. Those drills come after the small ones pass.

Who should be allowed to stop an experiment?

The owner, the on-call, and the automatic burn rule. When any one of them stops it, it stays stopped. Do not require a meeting. Also practice the stop before you need it, so the button is not folklore.

How often should you run game days?

Run small automated faults often, and wide game days rarely. A quarterly regional drill is enough for many teams if the small checks run every week. When a game day finds the same bug twice, automate that check instead of repeating the meeting.

Write one hypothesis about a single dependency your users need. Set a small traffic limit, a ten minute cap, and an abort on fast burn.

Run it while the owner is watching. If the objective holds, automate that check. If it does not, fix the timeout or fallback before you plan a bigger drill.

Last updated on 06 September 2026.