Read Replicas: Architecture, Replication Lag, and Consistency Trade-offs

Read Replicas matter because a single primary cannot serve every read once traffic grows. A replica copies the data and answers reads so the primary can spend its budget on writes. The copy is not instant. If you send the wrong read to a replica, the user sees a stale row and files a bug you cannot reproduce on the primary.

What a Read Replica Is

A read replica is a database copy that receives changes from the primary and serves select traffic. It is not a second writer. Writes still go to the primary. The replica applies those changes as they arrive, and it can lag when apply cannot keep up.

PostgreSQL does this with streaming replication and hot standby. The primary ships WAL. The replica replays it and allows queries while it replays.

The PostgreSQL hot standby docs describe the query limits during replay. The warm standby chapter covers how the copy is built and kept current.

Managed systems use the same idea. The Amazon RDS read replica guide explains promotion, lag, and the cases where a replica cannot be created. The product names differ. The lag problem does not.

A replica can also be your failover target. That second job changes how you treat it. A replica that is busy with heavy reports may fall behind, and then it is a poor candidate to promote. Split report load and failover capacity when you can.

Why Replicas Fail in Production

The failure users notice is a stale read after a write. Someone saves a profile and the next page loads from a replica that has not applied the change. The screen shows the old name. A refresh fixes it, which trains people not to trust the product.

I have seen this on a checkout flow that wrote an order on the primary and then read it back from a replica to render the receipt. Under load, lag was a few hundred milliseconds. The receipt page raced the apply and sometimes returned not found.

The row existed. The replica had not seen it yet.

Lag also spikes when the primary does heavy work. A bulk load, a long index build, or a vacuum storm ships a lot of WAL. Replicas fall behind together.

Reads then return data that is minutes old, not milliseconds. Database migrations at scale should plan for that lag, or you pause replica reads during the job.

A replica can fall so far behind that it is no longer useful, or it can disconnect and serve an old snapshot if you allow it. You need a lag cap. Past that cap, stop sending user traffic to that host. Serve an error or send the read to the primary.

Replicas do not fix a bad query. If the statement scans a huge table, every replica runs the same scan. You moved the pain and multiplied the disk use. Tune the statement with query optimization before you add copies.

How Routing Should Work

Split traffic by what the read must see. A read that must include the user’s last write goes to the primary. A read that can be a little old can go to a replica.

Lists, search, and feeds often fit the second group. Receipts, balances, and permission checks often fit the first.

A practical rule is read-your-writes for a short window. After this user writes, pin their reads to the primary for a few seconds. Then allow replicas again.

Store the pin in the session or in a cookie with a deadline. Do not pin every user forever or the primary stays hot.

Route by lag as well as by role. If the replica is past your cap, skip it. A load balancer that only checks TCP health will keep sending traffic to a lagged host.

Export replica lag and use it in the choice. A host that is up but minutes behind is not healthy for users.

Keep pools separate. One pool for the primary and one for replicas, as covered in connection pooling. A shared pool behind a balancer can run a write and the follow-up read on different hosts by accident. That is the stale read, even when lag is small.

Apply lag and visibility

Two clocks matter. Send lag is how far the replica is from receiving WAL. Apply lag is how far it is from replaying it.

Users see apply lag. Alert on apply lag, not only on the network delay.

Hot standby can also pause a query that conflicts with replay. A long report on the replica can block apply, which grows lag for everyone else. Set a cap on statement time for replica traffic. Kill the report rather than stall the copy.

Replica reads are a form of eventual consistency. The system becomes current after apply catches up. Design the screen so a short delay is honest. If the screen cannot tolerate that, do not use the replica for that screen.

Promotion and failover

When the primary dies, you promote a replica. That host starts to accept writes. Other replicas must follow the new primary.

Practice this. A replica you have never promoted is a hope, not a plan.

During failover, lag becomes data loss if the last WAL never arrived. Synchronous replication can shrink that window, at the cost of slower commits. Pick sync for a small set of critical writes if you need it. Do not turn every replica synchronous or the primary waits on the slowest one.

Trade-offs You Should Weigh

Replicas buy read capacity and a failover option. They cost lag, extra machines, and more routing rules. Use the table when you decide which reads may leave the primary.

Async replicas are the default for read scale. Sync replicas are for durability, and they are a poor place to dump heavy reads. If you mix both jobs on one host, you will stall commits when a report runs. Keep those roles apart.

More replicas add cost in a straight line. Each one stores a full copy and replays every write. If the write rate is the bottleneck, another replica does not help.

It only adds apply work on the side. Scale writes another way, or reduce them.

Pitfalls and Failure Modes

Most replica incidents are routing bugs or lag bugs. The data is not corrupt. It is late, or you asked the wrong host. Check these before you call the design done.

1. You read after write from a replica and sometimes miss the new row.
2. You ignore apply lag and keep a lagged host in the pool.
3. A long query on the replica blocks replay and lag grows for all readers.
4. You run a migration that ships a WAL flood and do not shed replica traffic.
5. Failover promotes a replica that was minutes behind on purpose for reports.
6. Health checks open new database sessions and add to the connection storm.

A common mistake I have seen is a cache key that does not include the freshness rule. The app pins the user to the primary, then stores the page in a shared cache filled from a replica. The next user gets the stale page. If you cache, fill the cache from the same host class you promise.

Schema changes must be replica safe. A migration that locks the primary will also delay apply. Test the migration against a replica before you run it at noon.

A Realistic Routing Example

The helper below sends a read to the primary when the user wrote recently or when replica lag is too high. Otherwise it uses the replica pool. Keep the lag check cheap and cached for a short time so you do not query lag on every request.

def connection_for_read(user, lag_ms):
    wrote_at = user.last_write_at
    if wrote_at and time.time() - wrote_at < 3:
        return primary_pool.checkout()
    if lag_ms > 500:
        return primary_pool.checkout()
    return replica_pool.checkout()

Three seconds and 500 milliseconds are starting points, not laws. Pick them from your lag chart and from how stale a screen can be. A stock ticker can wait longer than a payment receipt. Say the bound in the product note so the next change does not guess.

Always send writes to the primary pool. Do not use this helper for inserts or updates. If a code path is mixed, split it.

Read, return the session, then write on the primary. Holding one session for both roles is how pins leak.

What to test

When you review routing, add a test that writes a row and reads it at once on the replica path. It should miss or it should have been pinned. Either result is fine if it matches the rule. A test that only uses one database will hide the bug.

Also test the lag cap. Force a fake lag above the limit and confirm reads leave that replica. If the cap is only a graph and not a routing input, it will not save you during an incident.

Performance, Scale, and Cost

Each replica can take a share of read QPS, up to the point where its disk and CPU match the primary. In an illustrative production range, two or three replicas often double or triple read headroom if the queries are already cheap. They do nothing for a scan that saturates one host. Fix the plan first.

Apply uses CPU and disk on every replica. A write-heavy primary forces every replica to replay the same writes. At some write rate, replicas cannot catch up no matter how many you add.

Then lag grows without bound. You must reduce WAL or make apply faster, not add readers.

Cost is a full extra database for each copy, plus backup and monitoring. A replica that only exists for failover still needs to stay near the primary, so you pay for a hot spare. A replica that serves reads can earn that cost back by keeping the primary smaller. Measure read share before you buy the third copy.

Watch apply lag, replica CPU, and the share of reads that still hit the primary. If the pin rule sends most traffic back to the primary, the replicas are idle and the primary is still hot. Loosen the pin before you add another replica.

Key Takeaways

• Send writes and read-your-writes to the primary, and stale-ok reads to replicas.
• Pin a user to the primary for a short time after they write.
• Route around replicas whose apply lag is past a hard cap.
• Use separate pools so a write and a follow-up read cannot switch hosts by accident.
• Do not run heavy reports on the replica you plan to promote.
• Tune slow queries before you scale out copies of a bad plan.
• Treat replica reads as eventual, and only use them where the screen allows it.

FAQ

Can you write to a read replica?

Not in the normal setup. The replica accepts the replicated stream and rejects local writes, or it diverges and breaks. Send inserts and updates to the primary. If you need a writable copy, that is a different topology, such as logical replication with conflict handling, and it needs its own design.

How much lag is too much?

Set the cap from the screen, not from a default. A feed can often tolerate a second or two. A balance or a new order often cannot.

When lag passes the cap, stop using that replica for user traffic. Also page a human when lag stays high, because the cap only protects reads. It does not fix apply.

Does a replica replace a cache?

No. A replica still runs SQL and still lags. A cache is faster and usually staler, and it needs invalidation.

Use a replica to scale queries you must answer from current-enough rows. Use a cache for hot keys that change slowly. Many systems use both, with different freshness rules.

What happens to replicas when the primary fails?

You promote one replica and point writers at it. Other replicas should follow the new primary. Reads may fail or go stale during the cut.

Practice the cut so the steps are short. If you promote a host that was far behind, you lose the writes it never received.

List the reads that must see the latest write, and keep those on the primary. Send the rest to replicas with a lag cap and a short pin after each write. Then load-test with injected lag so the stale path fails in a way users can understand.

Last updated on 22 September 2026.