Network Partitions: Detection, Mitigation, and CAP in the Real World

Network Partitions are a normal fault in any system that spans hosts, racks, or regions. They matter because both sides can look healthy and still disagree about the data. If you plan only for dead nodes, the first real cut will write two histories. You should decide, before the incident, which side may accept a write.

What a partition is

A partition is a loss of messages, not a loss of process. Some hosts can still talk to each other. Others cannot.

CPU, disk, and local clients often keep working on both sides. Because each side sees local success, each side may try to lead.

The cut can be clean or messy. A clean cut splits the group into two sets with no path between them. A partial cut is worse.

Host A can reach B, and B can reach C, but A cannot reach C. Also, the path can be one way. A hears B, while B never hears A.

You should not treat a slow link as a healthy link. When delay exceeds your budget, the effect on the user is the same as a drop. Packets may still arrive later.

Then they confuse a leader that already moved on. Therefore your protocol needs a term, an epoch, or a fence, not only a retry.

Why this fails in production

In my experience, the outage report names the database. The trigger is often a cut between zones. After clients time out, they retry.

Then the side that can still form a quorum takes a storm of work. Meanwhile the other side may still accept writes if you failed to require a majority.

As a result, a short blip becomes a repair job. We once hit a bottleneck when both sides accepted writes for half a minute. The merge took hours, because every conflict needed a human rule. Watch rejected writes, leader changes, and retry rate, not packet loss alone.

False detection

A dead peer and a slow peer look alike at first. A heartbeat that stops is only a clue. When a garbage collection pause lasts longer than your timeout, a live node looks gone.

Also, CPU steal on a noisy neighbor can delay the same probe. Therefore a detector should wait for a threshold before it ejects a member.

Still, waiting forever is also a bug. If you wait too long, a real cut lets a stale leader keep the lease. You need a bound that is longer than a normal pause and shorter than the harm you will accept. Measure your pause times, then set the window from that evidence.

Architecture and implementation

Use a protocol that already has a proof, such as Raft or Paxos, for the metadata that picks a leader. Do not invent a vote rule during the incident. Then keep the data path boring and strict.

Quorum as a fence

Treat quorum as a write fence, not as a hope. With five replicas, a write that needs three votes can land on only one side of a clean split. The other side cannot gather three votes, so it must refuse.

However, placement matters as much as the count. If three of five replicas sit in one zone, that zone is a silent majority.

When the zone fails, the other two replicas cannot lead. You kept a single writer during a cut, and you also lost the service during a zone loss. Instead, spread votes so no single zone holds a majority.

Then a zone loss still leaves a quorum elsewhere. Specifically, put the odd replica in a third zone if you can pay for the latency.

Epochs, leases, and clocks

A new leader must prove it is newer than the old one. An epoch, a term, or a fencing token does that. Every write carries the epoch. Storage rejects a write from an older epoch.

After the new leader wins, the old leader can still be alive. Its writes fail closed. That is the point.

Leases add a time bound so a paused leader cannot act forever. The bound is only as honest as the clock. If clocks jump, a lease can outlive the owner.

Before you trust a lease, read how clock skew in distributed systems changes the fence. If you cannot bound the clock, do not use wall time as your only lock.

What clients must do

Clients are part of the design. A client that retries without a budget will crush the healthy side. Set timeouts in distributed systems so a stuck call ends, and so the retry has a smaller budget than the user request.

Also, make the write idempotent. Then a retry does not apply the same payment twice.

Reads need a rule too. A stale read can be fine for a cache. It is not fine for a balance check.

If the read must be current, it should touch a quorum or a leader, and it should carry the epoch. When the minority cannot do that, it should fail the read. Users prefer a clear error over a silent wrong answer.

Trade-offs you actually choose

CAP is a choice during a cut, not a slogan for the whole company. If you answer every request on every side, you may split the history. If you refuse the minority, that side loses uptime.

You can choose per operation. Some keys can require a quorum. Some caches can serve a stale local copy.

Even when the network is healthy, you still trade delay for a stronger read. A quorum read costs more than a local read. That cost shows up as tail latency on the slowest replica. Therefore pick the strong path only where a wrong answer is expensive.

Static stability is the option many teams skip. The AWS note on static stability argues that a zone should keep serving with the capacity it already has. If your plan needs a cross zone call to survive a zone cut, the plan will fail when you need it. Pre-provision the local path instead.

Pitfalls and failure modes

Most bad partitions are partial. A total two way split is the picture in textbooks. Production gives you a gray failure. Some ports work.

Health checks pass. The data port does not. If your detector only pings the health port, it will keep a sick node in the group.

• Heartbeats stay inside one rack, so a top of rack fault looks like a dead fleet.
• A majority of votes sits in one zone, so one zone loss ends leadership.
• Clients retry with no budget, and the healthy side falls over next.
• Reads ignore the epoch, so users see a stale primary after a failover.
• The detector ejects a node during a long pause, then flaps when it returns.
• Repair tools assume a total order that neither side actually has.

The Google SRE chapter on cascading failures shows how a small cut grows. Retries, shared pools, and slow timeouts move the pain to the next tier. Shed load on purpose.

Do not let every client become a retry loop. Also, cap in flight work so a partition cannot queue without bound.

Databases have the same shape. PostgreSQL warm standby can promote a replica, but a partition can tempt you to promote too soon. If the old primary still accepts writes, you have two primaries.

Fence the old primary first. Then promote. Since promotion is a human or a controller action, practice it before the cut.

Steps when you suspect a split

1. Probe from outside the cluster, not only from a peer inside it.
2. Stop writes on any side that cannot see a quorum.
3. Fence the old leader with a new epoch before you promote.
4. Shed retries that have used their budget.
5. Repair divergence only after one side is the sole primary.
6. Write down the choice you made, so the next cut is not a debate.

A policy you can run

Put the rules in config, not in an incident chat. The sample below is a starting point for a five replica group. It is not a measured standard.

Change the times after you measure pause, disk stall, and cross zone delay. If you copy the numbers with no measurement, you will either flap or fence too late.

failure_detector:
  interval_ms: 200
  suspect_after: 8
  phi_threshold: 8
quorum:
  replicas: 5
  read_votes: 3
  write_votes: 3
fencing:
  require_epoch: true
  reject_stale_epoch: true
on_partition:
  minority_writes: false
  shed_client_retries: true
  request_budget_ms: 80

Read votes and write votes both require three. That overlap means a successful read sees the latest successful write. If you drop read votes to one, you gain speed and you can serve a stale value during a cut.

Say that trade in the design doc. Do not discover it from a user ticket.

The request budget is the client side of the same policy. Eighty milliseconds is only an illustrative budget for a call inside one region. A cross region call needs a larger budget. When you design multi-region deployments, set the budget from the far path, not from the lab path on one laptop.

Performance, scale, and cost

Heartbeats are cheap until the mesh grows. A full probe from every node is fine at tens of peers. At hundreds, gossip or a hierarchy keeps the cost flat.

Still, the detector must feed one membership view. Two views are a split brain you built on purpose.

Quorum writes wait for the slower votes, not for the fastest disk. That wait is why a strong write has a worse tail than a local write. If a hot key can take a local write plus async copy, do that. If it cannot, pay for the quorum and cap the wait.

A cross region quorum adds delay. In an illustrative range, a regional round trip is a few milliseconds, and a cross region round trip is tens or hundreds. Place a local quorum on keys that can stay in one region. Use a global quorum only when one worldwide history is required.

The human cost is larger than the packet cost. A forked history burns the error budget behind your SLOs and nights of repair. A refused write on the minority is a short, loud error.

Prefer that loud error when the data is money, stock, or identity. Then drill a dropped port, not only a process kill.

Key Takeaways

• A partition leaves both sides up, so liveness checks alone will not save you.
• Require a quorum and an epoch before you accept a write that must not fork.
• Place votes so one zone cannot form a majority by itself.
• Bound leases with a clock you understand, or do not use leases as the only fence.
• Give clients a budget and an idempotent write, or retries will extend the outage.
• Choose per operation: strong where a fork is costly, local where stale data is fine.
• Drill a partial cut, not only a process kill, before you trust the design.

FAQ

Is a partition the same as a dead node?

No. A dead node stops. A partitioned node keeps serving local clients.

Because it is still alive, it can accept writes if you let it. You must fence it, not only mark it down. If you only remove it from a load balancer, direct clients can still reach it.

Can a better network remove the choice?

A better network makes cuts rarer. It does not make them impossible. When the cut happens, you still choose between a single history and full answers on every side.

Redundant links help. They are not a proof. Plan the refuse path anyway.

Should the minority keep serving reads?

Sometimes. If the product allows a stale read, a local copy can serve. If the read gates a write, a payment, or a permission, the minority should fail.

Label the read path in the API so callers know which rule they get. Do not mix the two modes in one method.

How fast should you fence a leader?

Faster than the harm of a stale writer, and slower than a normal pause. If you fence on the first missed heartbeat, you will flap. If you wait for a human, the old leader may keep writing.

Measure pause time, then set the suspect window above it. Revisit the window after each large pause incident.

Decide the write rule now, and put it in config. Then run one drill that drops packets, not only one drill that kills a process. After the drill, check that the minority refused writes and that clients backed off. Next, list the keys that must use a quorum, and test that list before you add another region.

Last updated on 21 September 2026.