Database Replication: Leader-Follower, Multi-Leader, and Leaderless

A single database server is a convenient assumption, and it expires on schedule. The machine restarts and the site goes dark with it. The disk dies and the data goes with it. Read traffic doubles and there is nowhere to send it. Replication is the oldest answer, and it works precisely because the copies live on machines that fail independently, the defining property of a distributed system.

Database replication is the practice of keeping multiple copies of the same data on different machines, with every copy receiving every change, so that the dataset survives the loss of any single machine and reads can be served from more than one place.

What replication buys and what it does not

The gift list is short and valuable. A replica means the dataset survives a machine loss: availability is the observable outcome of that redundancy and durability is the property of not losing the data; the two terms are defined precisely, and kept apart, in availability vs reliability vs durability. A replica also reads: every added copy is read capacity, absorbed the way horizontal scaling absorbs load, by adding machines rather than buying a bigger one. And a copy can live near its readers: a replica in another region answers queries without the cross-ocean hop.

What replication does not do is make writes faster or bigger. Every replica performs every write, so write throughput stays pinned at what one copy can absorb, and storage multiplies by the copy count. A write shipped to three replicas is not three times a write anywhere; it is one write, paid for three times. Splitting the dataset itself is a different mechanism, sharding, and the two combine in production more often than either stands alone.

Replication also forces the trade-off the CAP theorem names. Two replicas that cannot talk to each other must choose between refusing updates (staying consistent) or accepting them on both sides and diverging. Every topology below is a different way of living with that choice, and the synchronous-versus-asynchronous decision is its sharpest edge.

Synchronous vs asynchronous replication

The split runs underneath every topology. Synchronous replication makes the leader wait: a write is acknowledged only after a follower has committed it too, so an acknowledged write survives the leader dying a moment later. The price is latency (every write now includes the round trip to the slowest participating follower) and availability: a follower that stalls stalls every write with it.

Asynchronous replication acknowledges the write when the leader commits locally, and ships the log in the background. Writes stay fast, and the leader keeps accepting them even while followers are unreachable, the mode most fleets actually run. PostgreSQL’s streaming replication and MySQL’s binlog replication both default to asynchronous; both can be tightened. PostgreSQL gains a synchronous-commit mode, and MySQL gains semi-synchronous replication, where the leader waits for at least one follower’s acknowledgment. Semi-synchronous buys most of the durability guarantee at less of the latency cost, and both engines let you choose where on that line to stand.

The line is the durability-versus-availability trade. Synchronous favors the acknowledged write surviving any single failure, at the cost of writes stalling when a follower is slow or down. Asynchronous favors a leader that never stops writing, at the cost of a window in which a crash loses recent writes and followers fall behind. Neither default is wrong; they answer different questions about which failure hurts more: silently losing the last few writes in a crash, or refusing writes while a replica is being repaired.

Leader-follower replication

Leader-follower replication (single-leader, master-slave in older vocabulary) gives exactly one node the right to accept writes. The leader appends each change to its log and ships that log to the followers, which replay the same changes in the same order. A follower’s state is the leader’s state, a bounded distance behind.

Three mechanics do the real work:

• The log. Replication happens by replaying an ordered log: physical write-ahead-log shipping, row-based change streams, or statement replay. Order is the whole guarantee: the same changes, in the same sequence, produce the same state.
• The snapshot. A new follower starts from a consistent point-in-time snapshot of the leader, then catches up by replaying everything after it. Joining the fleet is a routine operation, not a heroic one.
• Read spread. Followers serve reads while the leader serves writes. Which reads may go to a follower, and which must not, is its own discipline; read/write separation has its own article.

Writes remain pinned at the leader. Leader-follower scales reads cleanly and writes not at all: the ceiling that motivates both sharding and the decision article that separates the mechanisms, sharding vs replication.

Failover, promotion, and split-brain

The leader is a single point of failure with an escape hatch. When it dies, the fleet detects the failure, promotes a follower, and repoints the clients; the mechanics are those of leader election. The hard parts are the boundaries: how long to wait before declaring the leader dead (too short, and a slow-but-healthy leader gains multiple successors; too long, and the outage stretches), and which follower to promote (the most caught-up one loses the least data).

With asynchronous replication, the promoted follower is missing every write the dead leader had not yet shipped; those acknowledged writes are silently gone. A subtler failure waits behind it: the old leader returns, still holding the writes its successors never saw, still willing to accept more. Two live leaders is split-brain, and the standard defense is fencing: tokens that make a stale leader’s commands safely ignorable, the lease-and-token discipline covered under distributed locks.

Failover is also a rehearsal discipline. A promotion path that has never been exercised will fail in production in a creative and entirely preventable way, so the first promotion a fleet performs should be a drill, not an emergency.

Replication lag

Replication lag is the distance between the leader’s state and a follower’s, measured in time or in bytes of un-replayed log. Asynchronous replication makes lag the normal condition: zero lag is a special case that lasts until the next write. Under load or a slow network, seconds of lag turn follower reads into a look into the recent past.

Lag is invisible until it isn’t. The user updates a profile, the write succeeds, the reload shows the old value; the data is safe on the leader and the read went to a follower still catching up. Three mitigations exist, in rising order of cost:

• Read-your-writes. A session that just wrote reads from the leader, or from a follower only after the session’s last write has reached it. Users always see their own changes.
• Monotonic reads. A session is pinned to one replica, or routed with the guarantee that it never moves backward in time. Being consistently slightly stale is tolerable; time-traveling is not.
• Causal consistency. A read that logically happens after a given write sees that write, from any session, the strictest and costliest of the three.

The same mathematics governs other tiers: a distributed cache invalidating across nodes is running a replication-lag problem with a smaller payload. Lag also doubles as a safety parameter; the failover timeout is set against expected lag, because promoting before the writes arrive converts lag into loss.

Multi-leader replication

Multi-leader replication lets several nodes accept writes and exchange them with each other. The classic case is multi-datacenter: each site runs its own leader, writes complete locally without a cross-continent round trip, and the leaders reconcile in the background. The same shape powers offline-first applications; a phone editing a document on a plane is a leader that will sync later.

The cost is the conflict. Two leaders accept changes to the same row while disconnected, and both arrive at reconciliation time. Resolution strategies exist, each with a bill: last-write-wins is simple and silently discards data, and its timestamps cannot truly order events, because the machines writing them do not agree on the time; conflict avoidance routes all writes for a given row to one leader deterministically, which restores a single writer at the cost of remote latency for that row; and explicit application-level merges (including conflict-free replicated data types, whose merges commute by construction) push the decision into the data model, which is why CRDTs appear in collaborative editors and offline stores.

Multi-leader trades a property away: while partitions last, there is no single definition of the write order. Systems that can express conflicts as mergeable data tolerate this well; systems whose rows are plain last-value-wins do not.

Leaderless replication

Leaderless replication (the Dynamo style, popularized by Amazon’s 2007 paper on its internal key-value store) removes the leader entirely. Any replica can accept a write. A client writes to several of the n replicas that hold the key, in parallel, and reads from several of them, choosing the newest version it sees.

The arithmetic that makes this work is the quorum. A write succeeds when w of the n replicas confirm; a read consults r of them. If r + w > n, the read set and the write set must overlap, at least one replica the read touches holds the newest version, so a read cannot miss a completed write. n = 3 with w = 2 and r = 2 is the common shape; raising w buys durability at the cost of write latency, lowering r buys read latency at the cost of freshness. Cassandra, Riak, and DynamoDB run variants of this design.

Two repairs keep the fleet converging. Read repair fixes a stale replica the moment a read exposes it. An anti-entropy process compares replicas in the background and heals differences; Dynamo did it with Merkle trees, so the comparison only descends into subtrees that differ. During real partitions, sloppy quorums accept writes on reachable nodes outside the key’s home set and hand them back later, availability chosen over placement purity.

Which nodes hold a key is a placement question: consistent hashing distributes keys around a ring, which is why leaderless stores and distributed caches share the same placement machinery. And leaderless designs still do not hand out strong consistency, quorums narrow the stale-read window; they do not close it. The guarantees and their names are the subject of the consistency models this article keeps leaning on.

Choosing a topology

No column wins universally. Leader-follower is the default answer; a single writer is a simplification most systems can afford, and its failure modes are understood everywhere. Multi-leader pays its way when geography forces multiple write sites or devices must work disconnected. Leaderless pays its way when staying writable through partial failure matters more than the last word in consistency. Most teams meet these designs as defaults their database already ships; the decision is more often acknowledging which one you are running than picking one from a menu.

Common mistakes

• Treating replicas as backups. Replication copies DELETE FROM orders as faithfully as any other change. A mistaken mass deletion replicates perfectly; backups are a different and unskippable discipline.
• Promoting the wrong follower. Failing over to a lagging asynchronous follower converts lag into data loss. Promote the most caught-up node, or pay for synchronous replication and stop worrying.
• Reading the leader “to be safe.” Routing every read to the leader re-creates the bottleneck replication was meant to relieve. Safe routing is deliberate, not reflexive.
• Last-write-wins everywhere. Clock skew makes timestamps a poor ordering across machines. LWW is fine for caches and overwrite-tolerant fields, and silent loss for everything else.
• Not watching lag. Lag is a distribution with a tail, not a boolean. The first sign of trouble should be a dashboard, not a user ticket about stale data.
• An unrehearsed failover path. The promotion that has never been run will misbehave in production in a boring and entirely preventable way.

FAQ

What is database replication?

The practice of keeping multiple copies of the same dataset on different machines, with every copy receiving every change, so the data survives the loss of any single machine, and reads can be served from more than one place.

What is the difference between synchronous and asynchronous replication?

It is the moment of acknowledgment. Synchronous replication acknowledges a write only after a follower has committed it, so an acknowledged write survives the leader’s immediate failure, at the cost of write latency and writes that stall when a follower does. Asynchronous replication acknowledges at the leader and ships changes in the background; fast and always writable, with a window in which a crash loses recent writes.

What is leader-follower replication?

A topology in which exactly one node, the leader, accepts writes and ships an ordered log; followers replay it and serve reads. It scales reads across the fleet and leaves writes pinned at the leader.

What is replication lag?

The distance between a leader’s state and a follower’s, measured in time or un-replayed log bytes. It is the source of stale reads, and it is managed with read-your-writes routing, monotonic reads, and (where the budget allows) causal consistency.

What is multi-leader replication?

A topology in which several nodes accept writes and exchange them with each other. It enables multi-datacenter writes and offline-first clients, at the cost of write conflicts that must be resolved by rule or by the application.

What is leaderless replication?

A topology with no leader: any replica accepts writes, which are sent to w of the n replicas holding the key; reads consult r of them. When r + w > n the sets overlap, so reads see completed writes. Read repair and anti-entropy keep replicas converging.

Does replication improve write throughput?

No; every replica performs every write, so write capacity stays at what a single copy can absorb. Replication scales availability, durability, and read throughput; write throughput is what sharding vs replication separates, and sharding is the mechanism that moves it.

Where this cluster goes

This article anchors the Databases & Data Scaling cluster, together with its sibling pillar. The rest of the cluster:

• database sharding; the complementary pillar: splitting the dataset itself, which replication never does.
• sharding vs replication, the decision article between the two mechanisms.
• database indexing; the first lever to pull before either one: reads that stop doing full scans.
• SQL vs NoSQL; the data-model decision that changes what replication means in practice.
• blob and object storage; the file tier, where objects are their own unit of copying.
• search engines in system design, when search deserves a system of its own, and what that system must keep consistent.
• read/write separation; routing reads across the replicas this article creates.
• data partitioning: the general concept underneath sharding, and the vocabulary around it.
• Elasticsearch architecture; a full stack in which every index is both sharded and replicated.

Related articles

• Next read: database sharding; the other scaling mechanism: splitting the dataset so writes and storage grow beyond one machine.
• sharding vs replication, which one your workload actually needs, and the usual answer: both.
• CAP theorem and consistency models, the framework behind every trade-off in this article.
• availability, reliability, and durability; the precise vocabulary replication exists to serve.
• leader election; the machinery underneath failover, and the split-brain question it answers.
• distributed caching: the same lag mathematics, one tier up, at cache speed.