Race Conditions in Backend Systems: Detection, Prevention, and Testing

Race Conditions in Backend Systems matter because two correct requests can still corrupt a row. Each request reads a value, decides, and writes. The second write wipes the first. You only see it when load overlaps.

What a Race Condition Is

A race is a bug that depends on timing. The code is fine when calls run one by one. It breaks when two calls interleave. The result then depends on which write lands last.

The classic shape is check then act. You read a balance, see that it covers the charge, and then subtract. A second request does the same read before the first write commits.

Both charges pass. The balance goes negative.

Races also hide in caches, queues, and file moves. Two workers can both claim one job. Two hosts can both create the same unique key. The database is the usual place you notice first, because the bad row stays.

This is not the same as a slow query. A slow query is stable. A race is not.

Retries, deploys, and extra hosts make it more likely. If your tests use one thread, they will stay green.

Why It Fails in Production

Production has overlap that a laptop does not. A sale, a retry storm, or a cron plus an API call is enough. I have seen a nightly job and a user request double spend a credit. The logs looked clean, because each request was valid alone.

Lost updates are the costly form. The last writer wins, and the earlier change vanishes. Support sees a user who paid and still has the old status.

There is no error to page on. You need a metric that counts clashes, not only errors.

Read skew across services is another form. Service A reads a row, calls service B, and then writes. Service B changed the row during the call.

Your write now stores a decision that used stale facts. The user sees a state that no single service would have chosen.

Deploys make races worse for a short window. Old and new code can use different write paths. One path checks a version.

The other path writes the whole row. While both run, the check is optional. Ship one write path, or you will chase a bug that fades after the rollout.

How to Stop the Overlap

Push the decision into one atomic write. Instead of read, decide, and write, use one update that checks the predicate in the same statement. The database applies it under its own row lock. Two callers cannot both pass the same check.

A version column is the other common fix. You read the row and its version. You write only if the version is unchanged.

If the update touches zero rows, someone else won. You reload and retry, or you fail the request. That pattern is optimistic locking.

When clashes are common, take the lock before you read. Pessimistic locking makes the second writer wait. The wait is the price of a strict order. Keep the locked section short, or the queue will dominate your latency.

Idempotency keys stop a different race. A client retry can apply the same charge twice if the first response was lost. Store the key with the result.

If the key exists, return the saved result. Do not run the charge again.

Where the check must live

The check has to sit in the same store as the write. A check in the app, then a blind write, is still a race. A check on a replica is also weak, because the replica can lag. Run the conditional write on the primary.

Cross-row rules need a single work unit or a single owner. If you decrement stock and insert an order, both steps must commit together. If they live in two databases, you need a saga with a clear undo. Otherwise one side can commit and the other can fail.

The PostgreSQL isolation docs explain why a plain read does not block a later write. Snapshot reads are fast. They do not freeze the row for you. If you need a freeze, ask for a lock or use a conditional update.

Trade-offs You Should Weigh

An atomic update is simple and fast when the rule fits in one statement. It fails when the rule needs data from another service. You then lock, call out, and hold the lock too long. Or you accept a retry loop.

A version check scales well when clashes are rare. Writers do not block each other. The loser repeats the read.

On a hot row, the losers retry in a tight loop and the database melts. Add a cap and a delay.

A queue with one consumer removes the race by doing the work in order. Throughput is the speed of that consumer. Lag grows under a burst. Use it when the order of events matters more than parallel speed.

DynamoDB offers a similar choice. A condition expression rejects a write that lost the race. TransactWriteItems groups several checks. The DynamoDB transaction API documents the limits, including item count and when the call cancels.

Pitfalls and Failure Modes

First, a unique index does not fix a lost update on an existing row. It only stops two inserts with the same key. Two updates can still clobber a balance.

Use the index for create races. Use a version or a predicate for updates.

Second, read your own write through a cache and you can reload stale data. After a clash, the retry must read the primary, not the cache. I have seen a retry loop that kept reading a cached version and failed until the TTL expired.

Third, partial failure looks like success. You charge the card, then the database write loses the race. The user is billed and the order is missing.

Before you call a partner, write a pending row with an idempotency key. After the partner returns, finish that same row.

Fourth, clocks are a weak lock. Two hosts can both believe they own a lease if the lease check is not atomic. Compare and set the lease token in one write.

If you use wall time, add skew margin. Still, prefer a token over a timestamp alone.

Fifth, deadlocks in databases and distributed systems show up once you lock more than one row. A race fix that locks in random order will trade lost updates for cycles. Sort the keys. Retry the whole unit when the engine aborts you.

A Practical Guard Example

The update below subtracts a charge only when the version matches and the balance is high enough. Zero rows means you lost the race or the funds were short. Do not treat zero rows as success. Reload, or return a conflict to the client.

UPDATE accounts
SET balance = balance - 100,
    version_no = version_no + 1
WHERE account_id = 10
  AND version_no = 7
  AND balance >= 100;

Check the row count in the app. If it is one, commit. If it is zero, roll back and decide.

A common mistake I have seen is to log the zero and still publish an event that says the charge worked. The event then lies.

How to test the race

A single-threaded test will not catch this. Run two workers against one row. Each worker tries the same charge many times.

The final balance must match the successful writes only. If it drops by more than that, you lost an update.

1. Seed one account with a known balance and version.
2. Start two workers that issue the same charge.
3. Wait until both workers finish.
4. Assert the balance and the version.
5. Fail the test if either value drifted.

Also run the test with a pause injected between read and write, if you still have a read path. The pause makes the overlap reliable. When the conditional update is the only path, the pause test should be impossible to write. That is a good sign.

Performance, Scale, and Cost

A conditional update costs about the same as a normal update when nobody clashes. The extra predicate is an index lookup you already pay for. The cost appears when many writers hit one row. Then you pay in retries, lock waits, or both.

In an illustrative production range, a hot balance can absorb only a few hundred contended writes per second before retries dominate. The exact cap depends on how long you hold the row. Measure clash rate, retry rate, and the slow tail. Do not guess from a quiet staging box.

Scale out by splitting the hot key. Give a counter several rows and sum them on read. Each writer hashes to one shard.

Clashes fall. The read gets a bit more work. That trade is cheap next to a retry storm.

Idempotency storage has its own cost. You keep a row per key for a retention window. A short window saves space and can double charge a late retry.

A long window costs disk. Pick the window from how late your clients retry, then add margin.

When a bad deploy starts double writes, you need a fast way back. Rollback strategies for backend systems should include the write path, not only the binary. If the new code drops the version check, rolling back the binary is the fix. A feature flag on the check is safer than a long bake with the check off.

The PostgreSQL locking guide is worth a pass when you choose between a predicate and an explicit lock. Use the predicate when the rule is local. Use the lock when you must read several rows and then decide.

Key Takeaways

• A race is a timing bug, so single-thread tests will not show it.
• Put the check in the same statement as the write.
• Treat a zero-row update as a clash, not as success.
• Use an idempotency key so retries do not double charge.
• Lock in a stable order when one request touches many rows.
• Split hot keys when retries or waits dominate latency.
• Watch clash rate in production, because lost updates stay quiet.

FAQ

How do you spot a race in logs?

Look for writes that succeed with no error while state goes backward. A version that jumps, or a balance that skips, is a clue. If you count zero-row updates, a spike means overlap.

Also compare request rate with distinct keys. A high rate on one key is where to start.

Is a database transaction enough?

A transaction groups your statements. It does not stop another transaction from writing the same row between your read and your write, unless you lock or use a stricter mode. Snapshot reads are the default in many setups. Add a predicate or a lock when the write depends on the read.

Should every table use a version column?

No. Add it where a lost update costs money or trust. A settings row that one admin edits can live without it.

A ledger cannot. Extra versions on cold tables add noise and little safety. Start with the hot, high-cost rows.

Can a queue remove every race?

A single consumer serializes the work it owns. It does not stop a second writer that bypasses the queue. Any direct update path can still race.

If you choose a queue, close the other doors. Then size the consumer so lag stays inside your product limit.

List every flow that reads a value and later writes it back. Turn the hot ones into a conditional update or a short lock, and add a two-worker test that fails if the total drifts. Then alert on clash rate before the next sale. If the rate stays high on one key, split that key instead of adding more retries.

Last updated on 09 September 2026.