Clocks and Ordering Events
In a distributed system there is no single clock everyone can read. Each machine has its own quartz clock that drifts, gets corrected by NTP, and can even jump backwards. So “which happened first?” is a hard question: comparing timestamps from two machines can give the wrong answer. Distributed systems use physical clocks when approximate real time is enough, and logical clocks when they need to get causality (what could have influenced what) exactly right.
Why wall-clock timestamps lie
- Drift: a typical server clock drifts tens of parts per million, which adds up to seconds per day without correction.
- Sync error: NTP over the internet keeps clocks within a few to tens of milliseconds; inside a good datacenter, around a millisecond or better. PTP with hardware timestamping (for example AWS Time Sync’s PTP hardware clock on supported instances) can reach microseconds.
- Jumps: when NTP corrects a clock, or a leap second is handled badly, time-of-day can move backwards.
If node A stamps an order update at 10:00:00.120 and node B, whose clock runs 100 ms slow, stamps a later update at 10:00:00.050, a last-write-wins database keeps the older one. That is real, silent data loss.
Logical clocks: count events, not seconds
- Three nodes, three clocks that disagree by up to 130 ms. Sorting events by these timestamps would be guesswork.
- Lamport clock: every node keeps a counter and adds 1 per event. Node A has events a = 1 and b = 2, and sends its counter with the message.
- Node B receives it and sets its counter to max(own, received) + 1 = 3. A receive is always numbered after its send.
- Node C's first local event d gets 1. The same rule gives e = 4 and f = 5. If x caused y, then L(x) < L(y), so sorting by Lamport time never puts an effect before its cause.
- The reverse is not true: d = 1 is smaller than c = 3, yet neither caused the other. Vector clocks keep one counter per node; [0,0,1] and [2,1,0] are not comparable, which proves d and c are concurrent.
Clock types compared
| Clock | What it gives you | Size | Used in |
|---|---|---|---|
| Wall clock + NTP | Approximate real time, can jump | 8 bytes | Logs, TTLs, event time in payloads |
| Monotonic clock | Never goes backwards, only valid on one machine | 8 bytes | Timeouts, measuring durations |
| Lamport timestamp | A total order consistent with causality | 1 counter | Raft/Paxos terms and log indexes are close cousins |
| Vector clock / version vector | Detects whether two events are causal or concurrent | 1 counter per node | Dynamo, Riak, CRDTs, conflict detection |
| Hybrid logical clock (HLC) | Close to wall time, still respects causality | wall time + counter | CockroachDB, YugabyteDB, MongoDB cluster time |
| TrueTime | A time interval with a guaranteed error bound | interval | Google Spanner (commit wait); AWS ClockBound offers a similar API |
Spanner’s trick: TrueTime says “now is between t − ε and t + ε” with ε of at most a few milliseconds, using GPS and atomic clocks. Before committing, Spanner waits out that uncertainty, so commit timestamps match real-time order across the planet. See Distributed Transactions.
Why data engineers care
- Event time vs processing time: streaming engines order and window events by the timestamp inside the event, not by arrival. See Batch and Stream Processing.
- CDC ordering: order database changes by the log position (PostgreSQL LSN, MySQL binlog position) or Kafka offset, not by
updated_at. - Deduplication and upserts: “keep the latest row” logic based on timestamps from different services can keep the wrong row. Prefer versions or sequence numbers from the source.
- Conflict resolution: last-write-wins in Cassandra or DynamoDB global tables depends on clocks. See Consistency vs Eventual Consistency.
Common mistakes
- Using
now()from different machines to order events. Use a single sequencer, a log offset, or a logical clock. - Measuring timeouts with wall-clock time. Use the monotonic clock (
time.monotonic(),System.nanoTime()). - Storing local time without a zone. Store UTC (
timestamptz), convert at the edge. - Assuming NTP means “in sync”. Monitor clock offset; some databases (CockroachDB) refuse to run when skew exceeds a limit, for good reason.
Learning Resources
Papers and articles
- Time, Clocks, and the Ordering of Events in a Distributed System: Leslie Lamport, 1978
- Logical Physical Clocks (HLC): Kulkarni et al., 2014
- Spanner: Google’s Globally-Distributed Database: TrueTime and commit wait
- There is No Now: Justin Sheehy, ACM Queue
Books
- Designing Data-Intensive Applications (2nd ed.), Martin Kleppmann and Chris Riccomini: the chapter on the trouble with distributed systems