By Imran M ·
· 11 min read
Clock Skew in Distributed Systems: Causes, Impact, and Clock Synchronization
Clock Skew in Distributed Systems breaks leases, caches, and event order. Learn the causes, the impact on reads, and how to sync clocks before bugs ship.
Clock Skew in Distributed Systems is the gap between two clocks at the same moment. It matters because leases, caches, tokens, and logs all pretend those clocks match. If they do not, a live owner looks expired, or a dead owner looks current. You should bound that gap before you trust wall time in a protocol.
What skew is
Skew is an offset. At one instant, host A reads 12:00:01 and host B reads 12:00:04. The skew is three seconds.
Drift is the rate that offset grows when nobody corrects it. A cheap crystal can drift more under heat or load. Then a small offset becomes a large one.
Jitter is the noise in the offset from one sample to the next. A single NTP reply can be early or late because the path is uneven. If you step the clock on every sample, you will thrash. Therefore a good daemon slews the clock in small steps, and it steps only when the gap is too large to slew.
Also, separate wall time from a monotonic clock. Wall time can jump back after a bad step or a leap second smear. A monotonic clock on one host only moves forward.
Use it for durations on that host. Do not send it to another host and compare it there. Since it has no shared epoch, the compare is meaningless.
Why this fails in production
In my experience, the bug report says the cache is wrong. The cause is often a clock. A lease that should last ten seconds lasts thirty on a host that is behind.
The old owner keeps acting after the new owner starts. That is a fence with a hole. When the same gap hits a token, the token is valid on one service and expired on the next.
Order breaks in the same way. If you sort events by wall time from different hosts, a later event can look earlier. Then a replay job applies the wrong last write.
We once hit a bottleneck when log lines from two zones were merged by timestamp. Because the clocks disagreed by more than the true gap between events, the story lied. Store UTC, and treat a laptop zone offset as a setting, not as skew.
Where the gap comes from
Virtual machines pause. Live migration, steal time, and a long garbage collection stall freeze the guest clock, or they let it jump when the guest wakes. Containers usually share the host clock.
If the host is wrong, every container is wrong together. Then your checks inside the pod all agree, and they are still late relative to the rest of the fleet.
NTP itself can be wrong when the path is uneven. The protocol assumes the delay out and back is similar. If it is not, the offset estimate is biased.
A firewall that slows one direction will do that. Also, a host with no sync at all will drift until a human notices. After a suspend and resume, assume the clock is dirty until the daemon reports a fresh sample.
Architecture and synchronization
Pick a source, a bound, and a failure action. The source is a time service you trust. The bound is the largest error you will tolerate.
The action is what you do when the error is larger. If you have no action, the bound is a comment, not a control.
NTP as the default
RFC 5905 defines NTP version 4. It estimates offset and delay, and it disciplines the clock. Run a daemon such as chrony or ntpd, not a cron job that sets the clock once an hour.
A step once an hour leaves a long window of drift, and the step itself reorders events. Slew while the host is up. Step only at boot, when no leases are held.
On cloud hosts, use the provider source. Amazon Time Sync is reached at a link local address, so it does not depend on your VPC route to the public internet. That matters during a partial network fault.
If time sync shares the same broken path as your data plane, both fail together. Point the daemon at the link local source, and keep a second source for when you run outside that cloud.
Bounds, not identical clocks
You will not make two clocks identical. You can keep the error inside a bound you publish. Spanner TrueTime treats each timestamp as an interval.
A commit waits until that interval has passed, so a later reader can trust the order. Therefore a bad time source shows up as slower commits, not as silent wrong order.
You can copy the idea at smaller scale without copying the hardware. Give each lease an uncertainty margin. If your sync stack claims an error under 50 ms in an illustrative range, do not expire a peer on a 50 ms gap.
Add the margin to the lease, or refuse to grant the lease when the daemon reports a worse error. When the margin is unknown, do not use wall time as the only fence.
When wall time is the wrong tool
Lamport clocks and hybrid logical clocks order events without trusting a global wall clock. A hybrid clock keeps a wall component for humans and a counter for ties. If host B is behind, the counter still moves past events it has seen.
Then a merge has a total order that does not reverse a cause. Use that for logs and for last write wins. Keep wall time for display and for TTL only when the TTL can absorb the skew.
Do not build a total order from NTP alone during network partitions. A cut can freeze sync and still leave both sides writing. The clocks will drift while they are split.
After the heal, a naive timestamp merge will pick a winner that was not last in real time. Fence writers with a quorum and an epoch first. Then let time order the events inside one epoch.
Trade-offs
Tighter clocks cost money, gear, or latency. NTP on a normal host is enough for leases of many seconds. It is a weak tool for a commit order at millisecond scale.
PTP with hardware timestamps can get much tighter, and it needs NICs, switches, and a team that can debug them. A logical clock is cheap, and it will not tell you the civil time of an event.
| Approach | What you get | What it costs | Use when. |
|---|---|---|---|
| NTP slew | A bounded offset | Daemon care | Leases of seconds are enough. |
| PTP hardware | A tight stamp | Special gear | You measure delay in microseconds. |
| Commit wait | A safe order | Extra latency | A wrong order is a data bug. |
| Hybrid logical clock | Causal order | No civil time | You merge logs or keys. |
If you skip the commit wait, a reader on a faster clock can miss the write. That bug shows up as a lost update you cannot replay from one host. Prefer the wait on keys that move money or inventory.
Pitfalls and failure modes
Stepping the clock backward is the failure I trust least. Processes that already slept until a deadline wake late, or they wake twice. Certificates look not yet valid.
A lease that had expired becomes valid again. If a daemon must step, do it before the process accepts traffic. After that, slew only.
- Two NTP sources disagree, and the daemon hops between them all day.
- The guest syncs to the host, and the host itself is free running.
- A leap smear on one fleet and a leap step on another meet in one log.
- TTL uses wall time, so a clock jump keeps a cache entry forever.
- Monitors graph offset and never page when the offset exceeds the lease margin.
- A job compares timestamps across regions with no published error bound.
Timeouts hide the same bug. A short deadline plus a late clock looks like a slow dependency. You then add retries, and the tail gets worse.
Read timeouts in distributed systems with the clock bound in mind. If the clock can be off by a second, a 200 ms timeout is not a measure of the network. It is a coin flip.
Checks before you trust a host
- Confirm the daemon is running and the last sample is fresh.
- Read the reported offset and the root delay, not only the year.
- Refuse traffic if the offset exceeds the lease margin.
- Slew instead of step while the process holds a lease.
- Alert on a source change, not only on a large offset.
- Record UTC in every event, and keep a logical tie break.
A guard you can ship
Put the margin in the lease code, next to the TTL. The snippet below rejects a grant when the sync error is too wide, and it extends the lease by that error so a peer does not expire the owner early. The numbers are an illustrative budget, not a lab result. Set them from the offset you actually see on the fleet.
max_sync_error_ms: 100
lease_ttl_ms: 10000
def grant_lease(sync):
if sync.error_ms > max_sync_error_ms:
return None
ttl = lease_ttl_ms + sync.error_ms
return Lease(ttl_ms=ttl, epoch=next_epoch())
I used a text escape for the compare so the page stays plain HTML. Read it as a greater than check against the max error. If sync error is unknown, return no lease. A missing sample is not a zero error.
Also, store the epoch from the leader election. Time is a backstop. The epoch is the fence during a split.
When you run multi-region deployments, do not assume one region’s time service matches another’s within your lease. Measure the offset between regions. If the gap is larger than the margin, stretch the lease or stop using it across that path. A lease that is shorter than the skew is a split brain timer.
Performance, scale, and cost
NTP traffic is small next to the data plane. The cost is operational: a source, a dashboard of offset, and a page when the bound breaks. At fleet scale, do not let every host poll one public server. Use the provider source or your own stratum servers.
Commit wait spends latency, not bandwidth. If uncertainty is a few milliseconds, the wait is in the noise of a cross zone write. If uncertainty spikes to hundreds of milliseconds, every commit slows, and tail latency moves with it.
That is a useful signal. Page on uncertainty, because the user pain will follow. You can also read how that pain lands on P99 latency explained for the commit path.
Buy PTP or GPS only when a wrong microsecond order loses money. Do not buy them to fix a lease of many seconds. A logical clock is the cheap fix for merge order, and wall time still serves the auditor.
Label which clock is authoritative for which decision. Then step a staging host backward and confirm the service stops granting leases.
Key Takeaways
- Skew is an offset between hosts, and drift is how fast that offset grows.
- Slew a running host, and step only when no lease is in force.
- Publish an error bound, and refuse leases when the bound is too wide.
- Use a monotonic clock for local durations, not for order across hosts.
- Use a hybrid logical clock when you merge events from many writers.
- Point sync at a source that survives the same faults as your data plane.
- Drill a forward step and a backward step before you trust token expiry.
FAQ
Is skew the same thing as drift?
No. Skew is the gap right now. Drift is the speed of the gap when you do not correct it.
A host can have low drift and still sit seconds off if nobody synced it after boot. Measure both. Alert on the offset the protocol cannot absorb.
Can NTP force two clocks to match?
It can keep them close. It cannot make them equal at every instant. Path delay, jitter, and a pause in the guest all leave a residual error.
Design for that residual. If your protocol needs a total order tighter than the residual, add a commit wait or a logical clock.
Should I sort cluster events by wall time?
Only inside one host, or after you have a bound that is smaller than the gaps you care about. Across hosts, wall time can reverse cause and effect. Prefer an epoch plus a logical counter for the order you will replay. Keep wall time as a hint for humans.
What should happen if a clock steps backward?
Stop granting leases and stop trusting new tokens until the daemon reports a stable offset. A backward step can revive an expired lease. Slew forward later if you must correct a small gap.
Do not let application code set the clock. That path should belong to one daemon.
Write down the maximum clock error your leases and tokens can absorb. Then alert when the sync daemon reports a worse error, and refuse new grants until it recovers. Next, run a staging step forward and backward, and confirm the service fails closed. If it keeps serving, the bound is only a comment.
Last updated on 10 September 2026.
