High Availability: Designing Away Single Points of Failure

High availability is the discipline of designing a system so that component failures reduce capacity instead of stopping service: keeping measured uptime inside a stated budget by engineering away single points of failure, sizing redundancy for the failures that will happen, and building failover that does not become its own incident.

What is high availability

Availability is measured as the fraction of time a system performs correctly, and the industry quotes it in nines. The budgets are worth memorizing because they are what the number actually is: 99% allows 3.65 days of downtime a year, a maintenance window and a couple of bad nights; 99.9% allows 8.76 hours; 99.99% allows 52.6 minutes; 99.999% allows 5.26 minutes, less than one restart of a large process. A nine is not a quality badge; it is a downtime budget, spent by failures, maintenance, and mistakes, and the architecture’s job is to keep the spending inside it.

Budgets compound, which is where the design lives. Dependencies in series multiply their unavailability: a path through two tiers that are each 99% available is 0.99 times 0.99, 98.01%, about a week of downtime a year, from components that individually look respectable. Redundancy in parallel multiplies availability instead: two independent 99% paths are available 1 minus 0.01 times 0.01, 99.99%, because both must be down at once for the customer to notice. Every availability architecture is these two operations applied over and over: shorten the series chains, and parallelize the links whose failure would otherwise spend the whole budget alone.

The budget is also a cost curve. Each additional nine costs more than the last (in hardware, in engineering, and in the operational maturity to run what was built) while the incremental downtime avoided shrinks by a factor of ten per nine. The honest design question is never “how many nines?” but “what does an hour of downtime cost, what does the next nine cost, and where is the crossover?” Most systems overbuy nines they never measure and underbuy the operational discipline that keeps the ones they have; the budget, written down and revisited, is what keeps that trade honest.

The single point of failure hunt

The audit that gives this discipline its name is a walk, not a meeting. Take one real user request and trace it end to end (DNS, balancer, gateway, service, cache, database, and every external dependency any of them calls) and at each hop ask one question: “what happens tonight when this dies?” A component whose death stops the request is a single point of failure, and the document it produces (the path with its death answers written next to each hop) is the availability architecture’s source of truth. The hunt is systematic precisely because intuition is not: everyone knows the database is critical and almost no one remembers the one DNS record, the one config value, or the one external API that everything else quietly depends on.

The finds sort into families, each with a standard fix. A single node dies: make it a redundant pair behind a load balancer: “a load balancer plus a second node is the cheapest high availability money buys,” and the purchase holds at every tier of the fleet. A shared dependency fails and takes the request with it: pool it, replicate it, or decouple the caller with backpressure and queues so the dependency’s bad minute does not become the fleet’s. A failure domain (one rack, one zone, one power feed) kills the redundant pair together: spread the replicas the way fault-tolerant systems teaches, because redundancy that shares a domain is redundancy in name only. And the quietest family, the human and process single points of failure: the one person who knows the failover procedure, the release that ships to everything at once, the family that the deployment strategies exist to retire.

The hunt is also periodic, because single points of failure grow back. Every new dependency adds a candidate; every “temporary” one-box utility is a future outage with a friendly name; every optimization that funnels a workload through one component re-creates the shape on purpose. Mature teams schedule the audit, and then they do the part the document cannot: they test what they claim. A redundant tier that has never had a node killed on schedule is a hypothesis, and the game-day discipline (kill the primary in production, on purpose, in daylight) is what converts the hypothesis into the architecture the budget actually rests on.

n+1 redundancy

N+1 redundancy is the sizing rule that makes redundancy real instead of decorative: run N nodes worth of capacity plus one, at a utilization where losing any single node still leaves the survivors carrying the full load. The test is arithmetic, not sentiment, if one node dies tonight, does the rest of the tier hold the peak, or does the failure become a capacity incident with a redundant-looking prefix? A service registry is the canonical example precisely because it must be simple to reason about: “three or five nodes, sized like any redundant tier by the high availability math,” with the odd number there so a quorum has something to disagree about and survive. The price is idle capacity (machines you pay for every day and use only on the worst one) and it is the honest cost of the insurance: utilization headroom is availability, purchased in advance.

The rule lands differently on different tiers, because what must survive the failure changes. Stateless tiers are the easy case: any instance is as good as any other, the balancer spreads the load, and n+1 is a sizing exercise. Stateful tiers inherit the harder problem: the data must survive the node, so redundancy becomes replication; a leader with followers, a promoted survivor, and the replication lag between them deciding what the failover costs in lost writes. Coordination tiers take it one step further: a quorum must survive, which is why the honest size is three or five (the numbers distributed consensus works with) and the ladder a real system climbs is the one the Redis internals article traced: single-instance replication, then Sentinel to supervise the failover, then Cluster to partition the data and the failure surface at the same time.

What n+1 does not cover is everything correlated. The pair that shares a rack dies together; the fleet that shares a deploy shares its bugs; the tier sized for last year’s peak discovers that capacity drift has quietly spent the headroom. The rule is necessary and it is not sufficient; it is the component-level answer inside an architecture that still has to hunt failure domains, stagger rollouts, and re-run the arithmetic whenever the peak moves. Redundancy in one dimension is a good start; availability is the discipline of noticing how many dimensions remain.

Failover design

Failover is the architecture’s answer to the question the hunt cannot prevent: “the component died anyway; now what?”, and it runs in three stages, each with its own failure mode. Detection: health must be observed before it can be acted on, and the detection machinery (heartbeats, timeouts, health checks) is the layer fault-tolerant systems owns; the architecture decision is what the detector is allowed to conclude and how fast. Decision: who promotes the survivor, and by what authority. Takeover: the new primary starts serving, the balancer or the client finds it, and the budget stops bleeding. The design goal is to convert a component failure into minutes of degraded service instead of hours of heroics: mean time to repair, bought in advance.

Two classic ways the cure becomes the disease. The first is split-brain: the detection layer concludes the primary is dead when it is merely slow, promotes a survivor, and the fleet now has two primaries spending the same writes, the failure replication spent its design avoiding. The standard defense is fencing: a promoted component must be able to prove its authority and to make the deposed one’s actions ignorable; the lease-and-token discipline distributed locks owns. The second is the failover storm: a promotion that triggers a client-wide reconnect, which triggers load that triggers the next tier’s detection, which promotes again; an incident paced retries exist to prevent. Both are reasons service discovery asked for “failover that does not become its own incident”; the request is not paranoia; it is the failure mode’s actual shape.

And the discipline that makes any of it real: failover is a rehearsed behavior, not a document. A promotion path that has never run in production is a hypothesis with a pager number, and the game-day habit (kill the primary on purpose, in daylight, with the team watching the clock) is what converts it into architecture. The drill also settles the questions the design cannot: how much in-flight work replays, and whether the replay is safe, which is why the failover tier leans on idempotent operations underneath, so the requests the failover re-runs land as confirmations instead of duplicates. Practice is the part of failover design that does not fit in the diagram, and it is the part the budget believes in.

The pillar and its mechanics

This article is the pillar of its series because the mechanics underneath it all face the same direction. Idempotency exists so that the replays failover causes land as confirmations, the series “assume this property the way a building assumes its foundation.” Retries with backoff and circuit breakers are the caller’s response to the failures the architecture made survivable, and the point of the architecture is that they stay “the exception rather than the mechanism.” Load shedding is “what remains when the high availability design has done its redundancy work and the day still arrives with more demand than fleet” (the pressure valve for the overload the n+1 sizing could not buy away) with backpressure its pipeline-side sibling. The division of labor is clean: the architecture makes single points of failure rare and failures into capacity events; the mechanics make the surviving failures cheap to retry, safe to replay, and survivable to shed.

Where the architecture’s coverage ends, two later articles extend it, and neither repeats this one. Disaster recovery takes the budget to the scale this page deliberately stops short of: the region-level failure, where the question stops being “which node survives?” and becomes “how long until we are serving again, and how much data did the gap cost?”, the RTO and RPO conversation. The deployment strategies take on the outage cause the hunt keeps finding in its human family: the release that ships to everything at once, retired with blue-green fleets and canary rollouts. Availability is built here, recovered there, and protected from its own operators in the third place: one discipline in three articles, each assuming the last.

FAQ

How many nines does my system actually need?
As many as the downtime arithmetic justifies, and no more. Write down what an hour of downtime costs the business, price the next nine in hardware, engineering, and operational maturity, and stop where the lines cross. 99.9% is a respectable target for most products; 99.99% and beyond are bought for systems whose downtime is measured in revenue per minute or in trust that never comes back, and the purchase is usually less another nine than it is the game-day discipline that makes the existing ones real.

What does n+1 redundancy mean in practice?
Run one more node than the workload needs, at a utilization where losing any single node leaves the survivors carrying the peak. Three registry nodes when two would carry the load; five when four would. The arithmetic is the design review: kill one in your head tonight, and either the tier holds or the “+1” was decoration. The cost is idle capacity in normal operation: the insurance premium, paid in machines you only value on the worst day.

What is the difference between high availability and fault tolerance?
Fault tolerance is a mechanism: redundancy, failover, and failure detection keeping a system behaving while components break; high availability is the architecture and the measurable outcome those mechanisms are assembled into: uptime inside a stated budget. The definitions live in the foundations article; the mechanics live in fault-tolerant systems; this page is the assembly. You build the former to earn the latter.

Does high availability mean the system never goes down?
No; it means downtime stays inside the budget, and the budget is honest about its members: hardware failures, maintenance windows, and the releases and mistakes that spend more of it than the hardware ever does. A system with 99.9% availability is expected to be down nearly nine hours a year by design; the architecture’s job is to spend those hours deliberately (in scheduled windows and rehearsed failovers) instead of accidentally at 3 a.m.

How often should we hunt for single points of failure?
On a schedule, not once. Every new dependency is a new candidate; every one-box “temporary” utility is a future outage with a friendly name; every optimization that concentrates load re-creates the shape on purpose. Walk one real request end to end quarterly, ask “what happens when this dies?” at every hop, and convert each claim of redundancy into a demonstrated one with a game-day kill. The hunt is cheap; the finding is always cheaper than the outage.

Related articles

• Next read: disaster recovery, where the series goes when the whole zone is the failure: RTO, RPO, and getting back online when the question is no longer “which node survives” but “how long until we serve again.”
• fault-tolerant systems, the mechanisms this architecture assembles: redundancy, failover, and failure detection as component-level tools, including the common-mode failures redundancy cannot see.
• availability, reliability, and durability, the definitions anchor: the three outcomes and fault tolerance, precisely separated, so the budget measures what it claims to.
• idempotency; the property that makes failover’s replays safe: re-run requests land as confirmations instead of duplicates, which is what lets the architecture promote survivors aggressively.
• load shedding, what remains when the redundancy work is done and demand still outruns fleet: degradation ladders and priority lanes for the overload that could not be bought away.
• retry with backoff and jitter; the pacing discipline that keeps a recovering fleet from being crushed by its own retries, and the caller’s half of the survivable failure.
• blue-green vs canary deployment, the release strategies that retire the human-family single point of failure: shipping without betting the whole fleet on one release.