By Imran M ·
· 10 min read
Failover Architecture: Active-Passive, Active-Active, and DNS-Based Failover
Failover Architecture moves traffic when a site dies. Learn active-passive, active-active, and DNS failover, plus the checks that keep each cutover safe.
Failover Architecture is how you move work when a site, a zone, or a database primary stops serving. When the plan is only a diagram, the first real cutover becomes the test. If health checks lie or data lags, you fail over into a second outage. Therefore design the promotion, the traffic shift, and the failback as one system.
What It Is and Why It Fails
Failover means a standby becomes the place that serves, or another active site takes the full load. It is not the same as a restart on one host. When an instance dies, a load balancer can drop it.
When a region dies, you need a path that already has capacity, data, and a way to steer clients. Also you need a human or a controller that knows when to act.
The design fails when detection is late or wrong. A single bad probe can swing a global site. A probe that only checks the local process can stay green while users cannot connect.
Health checks should prove the path a user takes, or at least the dependency that path needs. If the check is shallow, you will flap.
It also fails when data is behind. You can shift traffic in seconds and still serve stale or empty writes. When the standby is too far behind, promotion breaks the point objective you promised.
A common mistake I have seen is a DNS change that succeeds while the database refuses to promote because replication lag is large. Users get a new address and an error.
Route 53 DNS failover can steer records based on health. DNS is simple and widely understood. It is also slow, because caches ignore your time to live more often than you hope.
The AWS reliability pillar treats failover as a tested recovery path, not as a record edit. Follow that spirit even if you use another cloud.
Active-Passive
In active-passive, one site serves and the other waits. When you fail over, you promote the waiter and move traffic. The capacity on the passive side must already be warm enough, or the time objective includes a cold start.
Also decide if the passive site takes read traffic. A warm reader is easier to promote than a cold cluster.
This mode is easier to reason about because one site owns writes. Split brain is still possible if both sides accept writes after a bad cut. Therefore use a fence.
Stop the old primary, or make it read-only, before the new one takes writes. If you cannot reach the old primary, wait for a quorum rule you wrote in advance. Do not improvise that rule in chat.
Active-Active
In active-active, more than one site serves at once. When one site dies, the others already have traffic and capacity. You avoid a cold promotion.
You pay with conflict handling, data movement, and a higher bill. If two sites write the same key, you need a rule: last write, merge, or reject. Without a rule, failover is how you corrupt data.
Active-active is a good fit when reads dominate and writes can stick to one home. It is a poor fit for a single hot row that every site updates. When you are unsure, start with active-passive.
You can add a second writer after the promotion drill is boring. Boring is the goal.
Architecture and the Cutover
Split the problem into detect, decide, promote, and steer. Detect uses checks from outside the failing site, so a local network partition does not hide the truth. Decide uses lag, error rate, and a minimum failure duration.
Promote makes data safe to write. Steer sends clients to the new place. If you steer before you promote, you create the outage you hoped to avoid.
DNS failover changes a record and then waits for caches. A short time to live helps and does not guarantee speed. Also keep a direct load balancer or anycast path if the time objective is shorter than DNS reality.
When clients pin an old address, you need a way to drain them. Connection draining belongs in the plan.
Timeouts in distributed systems must be longer than a normal blip and shorter than the time you will wait before failover. If every call waits for minutes, detection is late. If every call gives up instantly, you flap. Set the health failure count so one timeout does not move the world.
Multi-region deployments supply the sites failover moves between. Failover is the control action. Multi-region is the footprint.
You can run two regions and still have a bad failover if promotion is manual and untested. Also document failback. Returning to the old site is a second failover, and it fails when people treat it as a casual cleanup.
Steps for a Safe Promotion
- Confirm the user symptom from outside the affected site.
- Check replication lag against the maximum you will accept.
- Fence the old primary so it cannot keep taking writes.
- Promote the standby and wait until it accepts writes.
- Shift traffic, then watch errors and lag on the new primary.
- Keep the old site read-only until you choose a planned failback.
Trade-offs You Should Name
Active-passive is simpler and slower to recover if the passive side is cold. Active-active is faster and harder to make correct. DNS failover is easy to explain and slow to converge.
A load balancer cutover is faster and depends on clients already using that balancer. Also, automatic failover saves minutes and can cause split brain. Manual failover is safer and may miss the time objective. Pick the risk you will rehearse.
| Mode. | Best when. | Main risk. | Recovery speed. |
|---|---|---|---|
| Active-passive. | Use it when one writer is enough. | Cold start if the standby is tiny. | Minutes, if it is warm. |
| Active-active. | Use it when reads are global. | Write conflicts. | Fast, because capacity is live. |
| DNS failover. | Use it when clients can retry. | Caches keep the old address. | Often slower than the TTL. |
| Manual cutover. | Use it when split brain is worse than delay. | People are slow at night. | Depends on the drill. |
Automation should stop if the checks disagree. When lag is high and probes are red, do not promote. When probes are red and a second vantage point is green, do not promote.
Although a human is slower, a human can see a bad deploy that looks like a dead region. Let automation handle the clear case. Page a human for the ambiguous one.
Pitfalls and Failure Modes
Split brain is the failure that corrupts data. Both sites take writes, then you merge two histories. When you cannot fence the old primary, refuse promotion or switch the app to read-only.
Also make the app reject writes if it cannot see the leader lease. A database flag the app ignores is not a fence.
- Failing over on one probe from inside the same broken network.
- Promoting a standby that is minutes behind a strict data promise.
- Changing DNS and forgetting long-lived clients and mobile caches.
- Leaving background jobs writing to the old region after the cut.
- Testing failover only in a slide, never on production-sized data.
- Failing back while the old site is still missing writes.
Jobs and caches are easy to forget. A worker pool in the old region can keep dequeueing after DNS moves. Then you double-charge a customer or send two emails.
When you fail over, stop writers first. Also purge or bypass edge caching only if cached bodies point at the dead origin. If the body is still valid, leave the cache alone so you do not stampede the new site.
Health checks that depend on the database you are promoting will go red during promotion. If that red triggers another failover, you oscillate. Therefore freeze automation during the cut.
Use a maintenance lease. Chaos engineering is how you learn whether that lease actually holds. A game day that stops at the diagram will not show the oscillation.
We once hit a bottleneck when DNS failed over and the new region had no warm connections to a third-party API. The database was fine. The payment calls timed out.
Capacity is more than CPU. Warm the dependencies, or the cutover only moves the error.
A Cutover Config You Can Start From
The record below is an illustrative active-passive plan. It requires three failed checks, a lag limit, and a fence before DNS changes. It is not a vendor export.
Map it to your load balancer and your database tools. If you cannot implement the fence, do not enable automatic promotion.
mode: active-passive
primary: region-a
standby: region-b
health:
vantage: outside
path: /ready
interval_seconds: 10
unhealthy_count: 3
promote_when:
max_lag_seconds: 10
fence_old_primary: true
dns:
ttl_seconds: 30
change_after_promote: true
failback:
manual_only: true
require_caught_up: true
abort_if:
- lag_above_limit
- fence_fails
Run this in a drill before you trust it. Shift a test name first, not the apex record. Then watch client errors, replication, and job runners.
If any writer remains on the old site, stop and fix the fence. Backup strategies still matter after failover works, because a bad write can replicate to both sites.
Performance, Scale, and Cost
Passive capacity is a standing cost. A tiny standby is cheap and misses the time objective when it must grow during the incident. A full twin is expensive and fast.
An illustrative middle path is a warm standby that can take the load with a small, prebuilt headroom. Measure the scale-up time. If it is longer than the objective, the standby is too small.
Active-active costs more every day and wastes less during the event. You also pay for cross-region traffic. When writes are chatty, that bill dominates.
Route writes to one home if you want active-active reads without a full write mesh. The latency win on reads is real. The write conflict cost is optional if you refuse it.
DNS performance is about tail latency of convergence, not about query speed. Some clients will use the old site for a long time. Also design the old site to reject or redirect once it is fenced, so stragglers fail fast and retry.
A polite error beats a silent timeout. Timeouts pile up and make the incident look larger than the region loss.
At large scale, a thundering herd follows failover. Every client reconnects at once. The new site then falls over.
Therefore shed or stagger. Raise connection limits before the drill, and keep backoff in the clients. If the drill never generates the herd, it did not test the real failure.
Finally, cost the drill itself. A failover test can double load or break a dependency quota. Schedule it when you have error budget, and cap the blast radius.
A test that takes the company down is not proof of reliability. It is an outage with a name.
Key Takeaways
- Detect from outside, then fence, then promote, then steer traffic.
- Prefer active-passive until promotion drills are boring.
- Treat DNS time to live as a hint, not as a stopwatch.
- Refuse promotion when replication lag breaks the data promise.
- Stop old writers, including jobs, before the new site takes traffic.
- Practice failback as a separate, planned failover.
FAQ
Should failover be automatic?
Automate the clear case: probes agree, lag is low, and the fence works. Also page a human when signals disagree. Full automation without a fence is how you get two writers. If you cannot fence, keep the decision manual and drill it.
How small can the passive site be?
It must reach the needed capacity inside the time objective. When scale-up takes longer than that, it is too small. Also keep dependencies warm. A database that promotes beside a cold cache can still miss the user promise.
What is a safe DNS time to live?
Shorter records converge faster and increase DNS load. A value on the order of half a minute is a common starting point, not a law. Also assume some clients will ignore it. Build a server-side reject so old addresses do not keep serving writes.
How do you avoid split brain?
Allow one writer. Fence the old one before promotion, or use a lease the app checks. When the fence fails, do not promote. Also test the case where the old site is unreachable, because that is the usual disaster.
Write the detect, fence, promote, and steer steps for one critical service. Then run a drill that moves a real client path, not only a health endpoint.
If lag, jobs, or DNS caches surprise you, fix those before you enable automation. When the drill is dull, you can shorten the detection window.
Last updated on 12 September 2026.
