Circuit Breaker Pattern: States, Failing Fast, and Bulkheads

This article is the architecture cluster’s fifth stop, and it arrives on a bridge the last one built. The service discovery article chased the failure-detection window, how fast a fleet can tell a dead instance from a slow one at the registry level; this article applies the same question to the call itself: when a dependency is up but failing, who decides the calling should stop, and for how long? The hub, microservices architecture, and the fault-tolerant systems article that owns the wider family both pointed here for the same reason: surviving failure is not only fault tolerance; it is refusing to volunteer for it, and the breaker is the mechanism of the refusal.

The problem it solves does not fail politely. A dying dependency does not return errors quickly; it hangs until timeouts expire, while every caller holds a thread, a connection, and a grudge; retries multiply the load on the thing already dying, which is how a slow outage becomes a cascading one. The message queues article flagged the retry end of that spiral and pointed here; the retry mechanics themselves (backoff, jitter, the storm math) are the retry with exponential backoff article’s to own, and this article borrows their conclusion: when retries make things worse, something has to stop the calling.

The breaker is one tool in a trio, not a category of its own. Timeouts bound a single call; retries express hope with discipline; the breaker decides the dependency is not worth calling at all, for now. What follows defines the pattern, walks its state machine honestly (including the thresholds that are judgment calls and the states that are easy to fake) then spends its remaining length on what failing fast actually requires, and on the bulkheads that keep one failure from flooding the whole hull.

What is the circuit breaker pattern

The circuit breaker pattern is a client-side mechanism that stops a fleet from calling a failing dependency: it counts failures, trips open when a threshold is crossed, fails fast instead of calling while open, and lets trial calls in half-open decide when to close again.

The name is the electrical one, and the metaphor is exact. A physical breaker does not fix the fault; it trips, stopping the current before the fault burns the house down, and a human resets it once there is reason to believe. The software breaker makes the same trade deliberately: while a dependency is failing, the fastest call to it is the call not made, and “error, immediately, with a fallback” beats “hope, held for thirty seconds” on every axis the caller can feel.

Scope is where implementations get honest or lie. A breaker is per dependency (one for the payments API, another for the inventory service, another for the database) because “the fleet is broken” is never one truth; it is a set of per-dependency truths, and a single global breaker treats them as one, tripping on everything when one thing fails and on nothing when everything half-fails. What the breaker does not do is retry, and it does not time out: those are the adjacent tools, with their own article, and the breaker sits above them; observing what the timeouts and the retries report, and deciding what the calling policy should be.

Circuit breaker states

Circuit breaker states are the three positions the mechanism holds: closed, where calls flow and failures are counted; open, where calls are refused before they are made; and half-open, where a limited number of trial calls decide which of the other two comes next.

Closed is normal traffic with a ledger. Every call through the breaker is scored: a timeout, an error, a rejection each counts as a failure, and successes reset or dilute the count depending on the policy. The counting policy is the first judgment call: consecutive failures trip fast but let a single unlucky request open the breaker, while a sliding window over the last N calls or the last T seconds tolerates one-offs but needs a threshold high enough to matter. What counts as failure is the second judgment call, and the one implementers get wrong most often; counting every non-success response as a failure opens the breaker on the application’s own valid error answers, while counting only timeouts leaves the breaker blind to the dependency that responds instantly with nonsense.

Open is the refusal: calls fail before any network happens, in microseconds, with the breaker’s error rather than the dependency’s hang. The open state lives on a timer (the recovery window) sized to the failure being recovered from: long enough that the dependency has had time to clear its backlog, restart, or scale, and short enough that the fleet does not avoid a healthy dependency out of habit. Too short, and the breaker probes a still-dying service straight back into failure; too long, and an outage that ended at minute two keeps costing revenue at minute ten.

Half-open is the experiment that ends the open state. When the timer expires, the breaker lets a small number of trial calls through (one, or a handful, never the full flood) and watches them: successes close the breaker and traffic resumes; a failure re-opens it and the timer starts again. The probe’s blast radius is why the trial population stays small, and its honesty is why probes are real calls made for real requests rather than synthetic pings that can succeed while the actual workload still cannot. A dependency that passes probes but fails under load is the half-open state’s classic blind spot, and sizing the probe traffic (not the threshold) is where most breaker tuning actually lives.

Failing fast

Failing fast is the open breaker’s side of the contract: when the calling stops, the failing starts, deliberately, immediately, and into something the caller chose in advance, rather than into a hang the caller can neither predict nor afford.

The fast failure only beats the slow one if it is spent on something. A breaker that opens into a bare exception has traded a dependency’s thirty-second hang for the user’s thirty-second shrug; the failure must land in a fallback, and fallbacks are designed per call, not chosen from a menu. The ladder runs from cheapest to most honest: serve last-known-good data and label it stale; compute a default that is wrong in acceptable ways; degrade the feature and say so; queue the work for later through the message queues article’s asynchronous backbone and tell the user it is on its way; or refuse cleanly, with an error the caller’s code was written to handle. Which rung is valid depends on what the call means; an approximate stock count is a fallback; an approximate payment is a lawsuit.

What failing fast buys is the caller’s own resources. Every refused call returns in microseconds with zero threads held, zero connections consumed, zero latency budget spent, which means the fleet that fails fast keeps the capacity to serve everything that does not depend on the broken thing. That arithmetic has a sibling in the resilience family: when the whole fleet is beyond what it can serve, choosing what to drop is the load shedding article’s subject, and the two compose: shedding protects the fleet from overload, the breaker protects it from a doomed dependency, and neither substitutes for the other.

An open breaker that nobody can see is a silent partial outage, which is why failing fast includes telling somebody. Every state transition (open, close, half-open flapping) is an operational event worth a metric and, for opens that persist, an alert; the telemetry that makes breaker state legible is monitoring and observability territory. A fleet that cannot answer “what is open right now, and for how long” has not deployed a resilience mechanism; it has deployed a new way to be confused.

Bulkhead pattern

The bulkhead pattern is resource isolation: partitions, per-dependency thread pools, connection pools, concurrency limits; sized so that one failing or slow dependency can consume only its own compartment, never the resources every other call needs.

The name comes from ships, where a bulkhead is the wall that keeps one flooding compartment from sinking the vessel. The software version answers a question the breaker cannot: even while a dependency is still technically working (before thresholds trip, before anything opens) a slow dependency holds resources it does not return on time. The classic casualty is the shared connection pool: a dependency that used to answer in fifty milliseconds starts answering in five seconds, and a pool sized generously for the good case drains to zero in the bad one. Every other dependency, however healthy, now competes for connections the slow one is hoarding, and the dependency’s problem becomes the caller’s outage. The bulkhead prevents the conversion: per-dependency pools mean the flooding compartment holds only its own share, and the fleet keeps serving everything that does not route through it.

The trio therefore divides the work cleanly. A timeout bounds a single call’s duration; a bulkhead bounds how many calls may be exposed at once; a breaker decides whether calls happen at all. Bulkheads are the passive wall, breakers the active decision, timeouts the unit of measurement, and a fleet that deploys breakers without bulkheads still loses its pools while the breaker’s window counts its way to open.

Sizing the compartments is the pattern’s real work, and it is queueing theory’s oldest arithmetic: concurrency is throughput multiplied by latency. A dependency serving a thousand requests per second at a hundred milliseconds each needs a hundred slots of exposure; the same throughput against a one-second dependency needs a thousand, and a pool sized for the fast case starves the slow one into timeout cascades it did not cause. Bulkheads sized without that arithmetic are guesses, and guesses drift, which is why the sizing belongs to the same review as the latency budgets, revisited whenever either side of the multiplication changes.

Common mistakes

• Opening into a bare exception. The breaker trips, every call throws, and the caller has a faster outage instead of a shorter one. An open breaker without a designed fallback is failure with extra steps; the mechanism exists to redirect capacity somewhere useful, and “somewhere useful” is a design decision made before the trip, not an exception handler improvised during it. If no rung of the fallback ladder is honest for a call, the honest design is the clean refusal, not the shrug.
• One global breaker for all dependencies. A single breaker shared across every outbound call turns one truth into a false one: the database slows, the breaker opens, and the fleet now refuses to call the cache, the edge service, and the two dependencies that never misbehaved. Per-dependency breakers are the pattern: “the fleet is broken” is a set of separate stories, and a global breaker welds them into a single coordination failure the pattern exists to avoid.
• Using the breaker as retry logic. Half-open invites it: the timer expires, so the fleet hammers the dependency with full traffic as an experiment, and a half-open probe indistinguishable from a retry storm re-kills the thing the breaker was protecting. Probes are small by design, and the machinery for trying again with discipline (backoff, jitter, budgets) is the retry with exponential backoff article’s to own. The breaker decides whether; retries decide how.
• Thresholds copied from folklore. A failure threshold of “fifty failures opens it” means nothing without the traffic level, the error profile, and the flapping risk behind it; a default borrowed from a blog post was sized for someone else’s dependency. Thresholds, windows, and recovery timers are empirical per dependency: start from its normal error rate, set the trip meaningfully above it, and tune on the flapping, because the only universal constant in breaker tuning is that the first numbers are wrong.
• Fallbacks that share the dependency’s fate. The breaker opens, the caller serves stale data, from the same database the breaker just stopped calling. Or queues the work; onto the same downstream that is dying. A fallback must be audited for its own dependencies: the point is capacity redirected to something that works, and a fallback that calls into the failure converts the pattern into a slower way to fail. Every rung on the ladder owes an answer to “what does this fallback itself depend on?”: asked in design, not discovered mid-incident.

FAQ

How is a circuit breaker different from a retry?
A retry is one caller’s second chance on one call; a breaker is the calling policy for the whole dependency. Retries answer “was that failure transient?” and belong to the moment; breakers answer “is this dependency worth calling right now?” and belong to the pattern across many calls. They compose (disciplined retries inside a breaker’s closed state) and the full comparison, with backoff, jitter, and the storm math, is what the retry with exponential backoff article exists to settle.

How is a circuit breaker different from a timeout?
A timeout bounds one call; a breaker reacts to the pattern across many. The timeout is actually the breaker’s main sensor; it is the timeouts, counted over a window, that tell the breaker a dependency is hanging rather than merely busy. A fleet with timeouts and no breakers still loses its pools one slow call at a time; a fleet with breakers but no honest timeouts is running a pattern detector on noise.

How long should a circuit breaker stay open?
Long enough for the failure to have plausibly ended, and no longer, which means the recovery window is sized from the failure mode, not from a universal constant. A dependency that recovers by restarting wants at least its worst-case restart time; one that recovers by draining a backlog wants the drain time, which scales with how long it was broken. Start conservative, let the half-open probes gather evidence, and tune the window on the flapping; an open time that keeps re-killing the dependency or keeps avoiding a healthy one is telling you it is wrong.

Where does the circuit breaker live, in the client, in a library, or in the mesh?
In the caller, conceptually; the party whose capacity is at risk is the party that stops calling. In practice the breaker is a library inside each service instance, a sidecar applied by the mesh, or a platform feature the fleet inherits; each placement trades per-language maintenance against uniform policy. What no placement changes is the state’s shape: breaker truth is per-instance and fleet-scattered, which is why aggregation (which instances see which dependencies open) is part of deploying breakers, not an afterthought.

What is the difference between a circuit breaker and a bulkhead?
The breaker stops calling; the bulkhead caps exposure while the calling continues. A breaker is a decision about traffic: informed by failure patterns, enforced by refusing calls; a bulkhead is a partition of resources: enforced by pool and concurrency limits, no decision required. They solve different halves of the same failure: bulkheads keep a slow dependency from drowning the caller’s shared resources, breakers keep the fleet from feeding a failing dependency its full traffic. Fleets that mean it deploy both.

Related articles

• Next read: retry with exponential backoff, the dependency graph’s own next step: backoff, jitter, retry budgets, and the storm math, plus the circuit breaker vs retry comparison this article kept deferring, settled where the two mechanisms meet.
• fault-tolerant systems, the wider family that pointed here: redundancy, failover, failure detection, and the place the breaker occupies as one rung among the fault tolerance techniques.
• microservices architecture, the cluster hub: the trade-off ledger that explains why a fleet of services owes every dependency a calling policy in the first place.
• message queues, the asynchronous fallback backbone: delivery guarantees, dead letter queues, and the honest way to defer the work a broken dependency cannot take right now.
• load shedding; the sibling mechanism: choosing what to drop when the whole fleet is overloaded, where the breaker chooses what to stop calling when one dependency is failing.

Last updated on 21 September 2026.