Read/Write Separation: Scaling Reads with Replicas

This article arrives with the cluster’s longest paper trail. Database replication created the replicas and pointed here twice: once for “which reads may go to a follower, and which must not,” and once for the routing discipline across the copies it makes. The sharding vs replication comparison fixed the ladder (indexing first, caching second, replicas third, sharding last) and named this article the first concrete step of the replication path, “with lag handled.” Database sharding added the fleet view: each shard keeps its own replicas, and every one of them needs the same routing rules. This article pays all four debts in order: what separation adds, the staleness audit, the routing mechanics, and lag as the governing number.

Read/write separation is the practice of sending writes to the primary and reads to replicas, with each read assigned by its tolerance for staleness rather than by habit: queries that can miss the last few writes go to the replicas and their extra capacity; queries that must see the current truth stay on the primary. The mechanics of how replicas are made and kept (topologies, snapshots, catch-up, failover) are replication’s subject; this article owns the routing discipline built on top of them.

The third rung: what separation adds

What separation adds is arithmetic: each replica is a full copy that can serve reads, so read capacity grows with the replica count, and the growth is operationally routine: replication made joining the fleet a snapshot-and-catch-up procedure, not a heroic one. What it does not add is equally important, because the costs travel with the copies: write capacity is unchanged (every replica performs every write, so the commit path is not relieved by a single added copy) and storage grows with each one, because copies do not shrink data. Separation scales the read half of the workload and leaves the write half exactly where it was.

The separation also isolates memory, which is half the win and rarely the half that gets named. Each replica carries its own buffer pool and its own working set, so replica reads do not compete with the primary’s cache; a fleet of read replicas is a fleet of independent memory budgets, each warming the queries it serves. The primary, relieved of the tolerant read volume, keeps its pool for the write path and the current reads. What looks like a routing change is also a memory architecture: the working set stops fighting itself.

The ladder position explains the sequencing. Indexing and caching come first because they reduce the volume of reads that reach the database at all; separation spreads whatever volume survives across more machines. And each rung has a price the next one avoids: indexes tax writes, caches introduce staleness by design, and separation introduces a staleness source the lower rungs never had, the replica’s lag behind the primary. That price is why the discipline this article exists for comes before the mechanics: the audit of which reads may pay it.

The staleness audit: which reads may go to a replica

The audit is a three-way sort, and it answers the question replication left here. First, lag-tolerant reads: catalog listings, feeds, search summaries, analytics; data seconds or minutes stale that no user can detect, which is most reads in most systems and exactly the traffic the replicas were built to carry. Second, read-your-own-writes: the profile just updated, the order just placed, the comment just posted; a second stale read is survivable, but the same user seeing their own change disappear is not. Third, strictly current reads: balances, inventory at checkout, authorization decisions; reads that must see the committed truth now, which stay on the primary always, under the same rule the caching series set: strictly fresh data is not cached, and strictly current reads are not routed away from the source.

The audit is per-read and explicit, which is the part teams skip. The routing decision is made by each query’s declared tolerance, not by connection habit; lag-tolerant reads marked as such, write-then-read flows marked sticky, current reads unmarked and therefore primary-bound. The failure mode is classification by laziness: with replicas sitting there, everything drifts into “tolerant,” and the first user who updates a profile and sees the old one files the bug that the audit was supposed to prevent. Naming each class in code is the design; the routing layer is just the enforcement.

A worked audit shows the sort in action. The product listing page: rendered from data seconds stale at worst, and the first page of traffic: class one, replica-bound, always. The settings page after the user clicks save: the next render must show the change: class two, sticky, pinned to the primary or lag-gated for that session. The checkout total: inventory and price as of this instant; class three, primary, no exceptions, under the same rule that keeps this data out of caches. Three reads, three destinations, one audit, and the bug pattern that the audit prevents is always the same shape: a class-two or class-three read that drifted into class one because the replica was there and the routing was default.

Replica routing: where the decision lives

The routing decision needs a home, and there are three candidates. The client library: drivers and ORMs split connections by role (read-marked sessions to replicas, everything else to the primary) which puts the decision next to the code that knows each query’s tolerance, the most honest placement and the easiest to get wrong per team. The proxy tier: a router in front of the database that speaks its protocol and routes by query type: the reverse-proxy idea applied one layer up, buying uniform enforcement and one more component to operate. Or per-transaction markers: the application declares a read-only transaction and the database’s own machinery sends it to a replica, the strongest guarantee and the least flexible. Most real stacks use the first with the second behind it, and the audit’s per-read classes as the vocabulary.

The placement tension is worth naming, because it decides whether the enforcement can work at all. The proxy sees query text and protocol, but tolerance is intent, the router cannot know that this particular catalog query backs a page that just wrote. So the declaration must travel with the request: read-only hints on the session, endpoint-level conventions the router trusts, or explicit staleness parameters the application sets. A routing layer installed without a declaration protocol routes by guesswork with good telemetry; the audit’s vocabulary is what turns the proxy from a load balancer into a separation design.

Sticky reads solve the read-your-own-writes class, and the three mechanisms trade exactly as expected. Session pinning: after a write, the user’s session stays on the primary for a short window: simple, common, and a capacity cost paid precisely for the users the system just made wait. Lag-aware routing: route the user’s reads only to replicas whose measured lag is under a threshold, which needs lag visibility at routing time and tolerates the small window the threshold allows. And version tokens: the write returns its position in the log, and the routing layer will not serve the user’s next read from a replica that has not applied that position, the strongest and the most machinery. The choice is per-workload; the requirement is not negotiable, because the profile-update bug arrives within hours of classifying that read as tolerant.

Failover is routing’s bad day, and the mechanics stay in replication’s article: the promotion, the quorum of who notices, the timeline. What belongs here is the client’s half: connections re-home to the promoted replica, in-flight reads fail honestly and retry, and the routing layer learns the new topology rather than caching the old one. A separation design that has never rehearsed its own failover has not finished the routing layer; it has only written the happy path.

Replication lag: the number that governs everything

Lag is the distance between the primary’s position in the log and the position a replica has applied; a measured, per-replica number, and the budget every decision in this article spends. It is not steady: it spikes with write bursts and large transactions, grows when a replica’s apply rate falls behind the commit rate, and settles when the stream drains. The mean matters less than the tail; a replica averaging a second and spiking to thirty is a replica whose staleness budget is thirty seconds, and tail latency is the discipline that says so.

The biggest lag events are written, not caused. A bulk update (a backfill, a migration, a mass price change) is one transaction on the primary and a long, single stream on the replica, and the lag it produces outlasts the operation by however long the apply takes to drain. The discipline is chunking: big writes shipped as bounded batches keep the replica’s apply interleaved with its read traffic, and engines where heavy queries and the apply stream share execution resources make the same point from the other side; an analytical query can slow the copy it is reading, so the analytical replica is often a dedicated one. Lag is a shared budget; what spends it fastest is a big write sent without thinking about it.

The budget makes lag actionable. For each read class from the audit, the design declares tolerable staleness; monitoring alerts when any replica’s lag crosses the class’s budget, not some generic threshold. When the budget breaks, the honest responses are the three the article has kept in view: route that class back to the primary and pay its capacity, serve the read with the staleness named, or hold the read until the replica catches up and pay the latency. What the budget prevents is the silent fourth option; serve stale data to a class that was never allowed to receive it, which is the failure mode the audit exists to prevent.

The discipline generalizes, and the cluster has a cousin to prove it: the search pipeline’s lag (the distance between a committed change and a searchable document) is this same number on a different copy, and search system design monitors it with the same budget logic. A source of truth plus derived copies is the recurring shape of this entire cluster, and lag is what the shape costs.

When separation stops being enough

The ceiling is the write path, and separation never touches it. Every replica performs every write, so a saturating commit rate saturates on every copy simultaneously, and no replica count relieves it; the exact asymmetry the sharding vs replication comparison fixed: replication scales reads, sharding scales writes. Storage follows: copies do not shrink the dataset, so a table that has outgrown one machine has outgrown every replica of itself. When either ceiling arrives (the commit rate or the working set) the next mechanism is sharding, and everything this article built survives it: each shard keeps its own replicas, with the same audit and the same lag budgets applied per shard.

Replica count itself has diminishing returns worth naming. Each replica taxes the primary’s write path (the replication stream is load on the source of the copies) so a fleet of replicas is also a fleet of consumers competing for the primary’s attention. The mature shape is a handful of replicas, well-used, standing behind the lower rungs that reduced the read volume in the first place; the immature shape is replica count as a reflex; adding copies to absorb traffic that an index or a cache would have removed entirely.

The operational rhythm completes the design. Adding a replica is the snapshot-and-catch-up routine replication documented; capacity arrives in hours, not quarters. Failover is rehearsed, because the routing layer’s bad day is not the day to learn its topology. And the audit is treated as living documentation: product changes reclassify reads, the search-summary that never mattered becomes the account page that does, and the class labels in code are cheaper to keep honest than the bugs their drift produces.

Capacity arithmetic closes the section. Replica count is sized by three numbers: the tolerant read volume per replica, the lag budget the class-one reads can spend, and the loss the design must survive; one dead replica must not route every tolerant read back to the primary, or the rung that was bought for resilience becomes the outage that proves it. A common shape is a small fleet behind the lower rungs: volume already reduced by indexes and caches, spread over enough replicas that losing one degrades instead of concentrates. The count is small because the ladder, not the replica fleet, is what scales.

FAQ

What is read/write separation?

Routing reads to replicas and writes to the primary, with each read assigned by its tolerance for staleness rather than by connection habit. Queries that can miss the last few writes use the replicas’ extra capacity; queries that must see current truth stay on the primary. The discipline is the assignment, not the plumbing.

When can a read be sent to a replica?

When it tolerates lag: catalogs, listings, feeds, analytics, most reads in most systems. Reads that must show the user their own recent write need stickiness, and strictly current reads: balances, authorization, checkout; stay on the primary always, under the same rule that keeps strictly fresh data out of caches.

What is replica lag, and how do you manage it?

The distance between the primary’s log position and the position a replica has applied; spiky, tail-heavy, and the budget every separation decision spends. Manage it with a declared staleness tolerance per read class, per-replica lag monitoring alerted against those budgets, and honest degradation (back to the primary, staleness named, or a short wait) when a budget breaks.

How do you keep a user from seeing stale data after their own write?

Sticky reads: pin the session to the primary for a short window after a write, route only to replicas under a lag threshold, or use version tokens that block the read until the replica has applied the write’s position. The mechanism varies with the workload; the requirement (a user always sees their own change) does not.

When does read/write separation stop scaling?

When the write path or the working set is the constraint: replicas copy the commit rate and the dataset, so neither is relieved by more copies. That is sharding’s trigger (splitting writes across machines) and separation’s design carries into it unchanged, applied per shard.

How many read replicas do you need?

Enough for three constraints at once: the tolerant read volume each replica can serve, the lag budget those reads may spend, and the loss of one replica without routing everything back to the primary. In practice, behind indexing and caching, the count is small, and it is the loss constraint, not raw volume, that usually sets the minimum.

Related articles

• Next read: database sharding; the last rung: when the write path or the dataset, not the read volume, is the ceiling this article cannot move.
• database replication, the mechanics underneath: topologies, snapshot-and-catch-up, and the failover this routing layer survives.
• sharding vs replication; the ladder this article is the third rung of, and the decision at its end.
• caching in system design, the rung below: reduce the read volume before spreading it.
• database indexing; the rung below that: the cheapest reads are the ones that never happen.
• search engines in system design, the lag-budget cousin on the derived-copy side of the stack.
• latency vs throughput; the tail-latency discipline that reads a spiky lag number honestly.