Eventual Consistency in Practice: Models, Conflict Resolution, and User Experience

Eventual Consistency in Practice matters because many systems accept a write on one node and show it on another node later. That delay keeps the service up when the network is slow. It also means a user can read stale data, or two writes can clash. You should pick this model on purpose, and you should show the delay in the product instead of hiding it.

What Eventual Consistency Means

Eventual consistency means that if writes stop, every copy will agree after some time. It does not mean the copies agree now. It does not give you a bound unless you add one. Werner Vogels described the idea in Eventually Consistent, and the gaps he named still show up in production.

A single primary with async copies is the common case. The write commits on the primary. Replicas apply it later.

Until they do, a read can miss the write. That is still eventual consistency, even if you do not use a multi-writer store. Read replicas are this pattern with SQL.

Multi-writer systems go further. Two nodes can accept updates to the same key while they cannot talk. When they reconnect, they must merge.

If you keep only the last write, you can drop an update. If you keep both, the user must resolve the clash, or your code must.

Stronger reads exist when you need them. DynamoDB, for example, offers eventually consistent reads and strongly consistent reads. The DynamoDB read consistency page states the trade.

The strong read costs more and may fail when the system is partitioned. Use it for the few keys that cannot be stale.

Why It Fails in Production

The failure is usually a product bug, not a crashed node. A user saves a setting and the next screen reads a stale copy. They save again.

Now you have two writes and a support ticket. The system did what you asked. The screen promised something you did not build.

I have seen a cart that wrote to one region and read from another. Most of the time lag was short. During a partition, the read region served an empty cart and the client deleted local state.

When the partition healed, the items came back and the user had already checked out a second time. The merge had no rule, so both orders shipped.

Another failure is a job that assumes a read is complete. A worker lists rows and acts on them. A row written a second ago is missing, so the worker never sees it.

If the job does not run again, that row is stuck. Eventual does not mean the next read will be fresh. It means a later read will be, if you try again.

Conflicts get worse when every field is last-write-wins and the clients have skewed clocks. A delayed packet with an old timestamp can overwrite a newer edit. If you use wall clocks to order writes, you will lose updates. Use a version or a counter the store assigns.

How to Build the Read Path

Start by naming the promise for each screen. Some screens can be a few seconds old. Some must show this user’s last write.

Some must show a single global order, such as a balance. Write the promise down. The storage choice follows from it.

For read-your-writes, pin that user to the node that took the write for a short time. You can also read a version token from the write response and retry the read until the copy has that version. Give up after a small number of tries and show a pending state. Do not spin forever.

For global freshness, read from the primary or use a strong read. Accept the extra latency and the lower availability. If the strong read fails because of a partition, fail the request. A silent fall back to a stale read breaks the promise you just made.

Logical replication can feed a read model that is shaped for the screen. PostgreSQL documents this in the logical replication chapter. The read model lags.

Build the UI so a missing row looks pending, not deleted. A tombstone and a not-yet-copied row must not look the same.

Conflict rules that hold up

Pick one merge rule per key and test it. Last-write-wins is fine for a display name if you can tolerate a lost edit. It is not fine for a counter. Increments must be merges, or you must send them through one writer.

A version column rejects a stale write instead of merging it. That is optimistic locking. The client reloads and tries again.

Use it when a clash is rare and the user can resolve it. When clashes are constant, the retry storm is the outage.

A single writer removes the clash by design. All updates for that key go through one process or one row lock. Pessimistic locking is the database form of that choice.

It costs wait time. It is the right cost when a lost update moves money.

What the user should see

Say when data can be late. A label such as updated a moment ago is more honest than a spinner that pretends the write is global. After a save, show the value the user just sent, even if a background read is still stale. That local echo is read-your-writes without a round trip.

If a conflict needs a human, show both versions. Do not drop one in silence. A support tool that hides the loser will ship the wrong order again. Keep the discarded write in a log so you can explain it.

Trade-offs You Should Weigh

Consistency, latency, and availability pull apart when the network fails. You do not get all three for every key. Choose per use case. The table is the menu I use in design reviews.

Do not pick one model for the whole product. A feed can be eventual. A payment cannot.

Mixing them is normal. The bug is mixing them inside one screen without a label, so the user cannot tell which number is fresh.

Also weigh operator load. A multi-writer store needs conflict metrics. A single primary needs a failover plan. Deadlocks are the page you already know if you stay with row locks.

Pitfalls and Failure Modes

These are the ways an eventual design surprises a team that tested on one node. Add a test that delays a copy before you ship.

1. You read your write from a lagged copy and clear local state.
2. You use client clocks to order writes and lose the newer edit.
3. A worker runs once, misses a late row, and never scans again.
4. Last-write-wins hits a counter and drops increments.
5. You fall back from a strong read to a stale read and hide the error.
6. A retry runs a handler twice and applies a charge two times.

A common mistake I have seen is a unique job id that is checked on a replica. Both workers miss the row, and both send the email. Check the idempotency key on the primary, or in a store that is linear for that key. The rest of the data can lag.

Deletes need a tombstone. If you remove the row at once, a lagged copy can copy the old row back after the delete. Keep a delete marker until every copy has seen it.

Then garbage-collect the marker. Without that wait, the row returns.

Bounds matter. Eventual with no deadline is not an SLO. Set a target for how late a copy may be, and alert when you miss it. If you cannot meet the target, stop offering that read or move it to a stronger path.

A Realistic Merge Example

The function below merges a profile with a version check. If the stored version does not match, it refuses the write. The caller reloads and decides what to keep. Counters are not in this object, because a version check is the wrong tool for increments.

def save_profile(store, user_id, new_name, seen_version):
    current = store.get_primary(user_id)
    if current.version != seen_version:
        return {"ok": False, "conflict": current}
    current.name = new_name
    current.version = seen_version + 1
    store.put_primary(current)
    return {"ok": True, "version": current.version}

Read the fresh profile from the primary inside this path. A replica read can hand you an old version and then fail the next save for no user-visible reason. After a success, the client can show the new name at once. Other users may see it later, and that is acceptable for a profile.

If this key were a balance, do not use this function. Send the debit through one writer so two spends cannot both pass. The profile rule and the balance rule can live in the same app. They should not share one helper that pretends they are the same.

How to test delay

When you review a feature, add a test that writes on one fake node and reads on another before sync. The UI or the API should show pending, not empty, and not a false delete. Then deliver the sync and confirm the read converges.

Also test a clash. Two writers save different names with the same version. One succeeds.

The other gets the conflict payload. If the second save silently wins, the merge rule is not the one you think you shipped.

Performance, Scale, and Cost

Eventual reads scale because you can add copies. The write path still has a limit. In an illustrative production range, async replicas absorb large read QPS while the primary stays near its write budget.

Your ceiling is the writer, not the number of readers. Measure both before you add a region.

Strong reads cost extra round trips and they fail closed during a partition. That is cheaper than a wrong balance, and more expensive than a stale feed. Put the strong read on the small set of keys that need it. If every read is strong, you paid for a distributed system and got a slower single node.

Watch lag, conflict rate, and how often clients retry a version check. If retries climb, you have a hot key, not a slow network. Shard that key or send it through a single writer. Adding regions will not calm a key that everyone updates.

Key Takeaways

• Eventual means copies agree later, not that the next read is fresh.
• Name the freshness promise per screen before you pick a store.
• Pin read-your-writes, and do not fall back from a strong read to a stale one.
• Use versions or a single writer for data you cannot merge with last-write-wins.
• Make handlers safe to run twice, because delivery will duplicate.
• Keep tombstones until every copy has seen the delete.
• Alert on a lag bound so eventual has a deadline you can operate.

FAQ

Is a primary with replicas eventually consistent?

Yes, if replica reads are allowed before apply catches up. The primary itself can still be linear for writers. The product becomes eventual at the moment you read from the copy. If every read hits the primary, you are not serving an eventual view, and you also are not scaling reads.

When is last-write-wins enough?

Use it when losing one edit is cheap and easy to redo, such as a nickname or a theme. Do not use it for counters, stock, or money. Those need a merge that adds the operations, or a single writer. If you cannot explain the lost edit to a user, the rule is too weak.

How do you give the user read-your-writes?

Show the value they just submitted from the client, and pin their next reads to the node that accepted the write. You can also pass a version token and retry until the copy catches up. Stop after a few tries and show pending. A tight retry loop will stampede the store.

Can you bound the word eventual?

You can set an operational target, such as most copies catch up within a second. That is a lag SLO, not a proof. Alert when you miss it, and route critical reads around a late copy. If you need a hard cap for every read, use a stronger model for that path.

Write down which screens can be stale and which cannot. Pin or strongly read the strict ones, and put a visible pending state on the rest. Then add a test that delays a replica so the clash and the stale read show up before users do.