Deep dive
Now depends on who you ask
UTC fixes how time is written down. It does not settle whose clock won, when a transaction sampled it, or which event happened first.
Share
Every service is set to UTC, and the trace still says the callback arrived before the request that caused it.
A perfectly ordinary trace can look like this:
| Record | Timestamp | Written by |
|---|---|---|
| request sent | 14:02:11.418Z | application node A |
| callback received | 14:02:11.401Z | application node B |
| row created | 14:02:11.392Z | database |
All three values use UTC. None of them came from the same clock, and they do not describe the same boundary.
The first reflex is usually to check time zones. That is worth doing, but it only checks the representation. UTC can make timestamps comparable on paper. It cannot make several machines share one clock, turn created_at into commit order, or make wall time safe for measuring a timeout.
The word “timestamp” hides three questions:
- Whose clock produced it? The application host, database, browser, device, upstream service, or telemetry collector.
- When does that clock get sampled? At the function call, statement start, transaction start, event creation, receipt, or commit.
- What fact must the value prove? Human time, elapsed duration, expiry, or order.
Most timestamp bugs start when code answers one question and a reader quietly asks another.
One server has more than one clock
An operating system exposes clocks with different contracts because “what time is it?” and “how long did this take?” are different jobs.
Linux calls its calendar clock CLOCK_REALTIME. An administrator can change it, and time synchronization adjusts it. CLOCK_MONOTONIC has an arbitrary origin and does not go backwards, which makes it useful for elapsed time on one machine.1
Ruby exposes both:
started_at = Time.now.utc
started_tick = Process.clock_gettime(Process::CLOCK_MONOTONIC)
result = call_upstream
finished_at = Time.now.utc
elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started_tick
started_at and finished_at belong in a log or user-facing audit trail. elapsed belongs in latency measurements and timeout budgets.
Subtracting two wall-clock timestamps often appears to work. Then the system clock is corrected during the operation and a request acquires a negative duration or an extra second it never spent running. Go avoids that trap in common cases by carrying a monotonic reading inside values returned by time.Now() and using it for comparisons and subtraction.2
A monotonic value is not a better wall timestamp. Its zero point is local and arbitrary. A reading is generally not portable through serialization, a machine reboot, or comparison with another host. It is excellent evidence for one duration and useless as “Tuesday at 14:02 UTC.”
PostgreSQL freezes now() on purpose
PostgreSQL has three useful answers to “now”:
SELECT
now(), -- start of this transaction
statement_timestamp(), -- start of this statement
clock_timestamp(); -- time at this function call
now() is an alias for transaction_timestamp(). It stays fixed from BEGIN through COMMIT. statement_timestamp() advances with each statement. clock_timestamp() advances even within one statement.3
The frozen value is intentional. Every modification in a transaction can receive one consistent timestamp. The surprise appears when someone later reads it as insertion time or commit time.
Run the two transactions below. PostgreSQL gives A the earlier created_at, even though its row becomes visible last.
created_at lost the race.
Commit order ≠ timestamp order.
created_at 10:00 waits on lock commit10:05 created_at 10:02 commit10:03 Visible
- Bcommit 10:03
- Acommit 10:05
ORDER BY created_at
- A
created_at10:00 - B
created_at10:02
A starts at 10:00.
A begins at 10:00 and waits on a lock. B begins later but commits first. When A finally commits, its row still carries 10:00. ORDER BY created_at returns A before B—the reverse of the order in which the rows became visible.
That is not clock skew. One database is behaving exactly as documented.
Other databases choose different sampling boundaries. MySQL evaluates NOW() once at the start of a query, including the stored programs called by it. SQLite keeps 'now' stable within one sqlite3_step() call.4 Porting SQL can preserve the function name and change its meaning.
created_at has an author
A column name does not reveal which clock filled it.
Rails normally writes model timestamps from Time.now in the application process. A PostgreSQL default such as DEFAULT now() uses the database transaction clock. An ingest job may preserve a timestamp supplied by a phone. A backfill may copy the time from an old file.5
All four rows can land in the same created_at column.
That makes created_at useful for many product questions and a poor universal cursor. It might mean when code constructed a record, when a database transaction began, or when an event claims it occurred. It usually does not mean when the transaction committed.
Precision does not repair the ambiguity. Six decimal places tell us how finely a value is represented. They do not guarantee the underlying clock was that accurate, that two calls cannot tie, or that another node agrees.
This is why a stable pagination cursor commonly needs (created_at, id) rather than created_at alone. The second value breaks ties. It still does not create causal or commit order. When order changes correctness, use a value whose contract actually contains order: a per-entity version, an offset in an ordered log, a database change position, or a sequence allocated by the authority that owns the write.
Synchronized clocks still disagree
NTP does not replace all clocks with one central clock. It measures offsets and network delay, estimates error, and disciplines each machine's local clock. Its model includes jitter and dispersion because synchronization always carries uncertainty.6
Usually the disagreement is small enough that nobody notices. Small is not the same as ordered.
If node A reports 10:00:00.120 and node B reports 10:00:00.115, either event might have happened first. The five-millisecond difference may describe real order, clock offset, measurement error, or some combination of them.
Lamport's old result still bites modern systems: causality gives events a partial order, and physical timestamps alone do not recover that order.7 If B handles a message from A, the message itself proves a relationship. Two unrelated wall-clock readings do not.
Some databases make clock discipline part of their correctness model. Cassandra's last-write-wins reconciliation compares mutation timestamps supplied by the client or coordinator; its current documentation says accurate clocks are required for correctness.8 A node whose clock is ahead can make an older logical write appear newer.
Spanner takes the opposite lesson seriously. TrueTime returns an interval of possible real time, and Spanner uses that bounded uncertainty to assign transaction timestamps that follow its serial order. The commit timestamp is trustworthy because the database built a protocol around uncertainty, not because a TIMESTAMP column happened to exist.9
Unless a system documents a guarantee like that, two server timestamps should be treated as observations from two clocks.
Event time and observation time are different facts
A mobile device records an action at 09:00 while offline. The event reaches the API at 12:00. A consumer processes it at 12:03.
Which timestamp is correct?
All of them, if the field names keep their meaning:
{
"event_id": "evt_481",
"occurred_at": "2026-09-11T09:00:00Z",
"received_at": "2026-09-11T12:00:04Z",
"processed_at": "2026-09-11T12:03:17Z"
}
OpenTelemetry's log data model makes the same distinction. Timestamp is when the event occurred according to the origin clock. ObservedTimestamp is when the collection system saw it, according to the collector's clock.10
Keeping both is useful during incidents. A sudden gap between them can expose offline clients, queue delay, a slow collector, or a clock problem. Overwriting source time at ingestion destroys that evidence. Trusting source time as arrival order invents evidence that never existed.
For a durable event, I want at least an event ID, the source timestamp when it matters, the server receipt timestamp, and an ordering value for every stream whose order matters. One field should not perform all four jobs.
Expiry forces wall time back into the design
Monotonic clocks solve duration measurement inside one clock domain. They cannot answer “expire this record at 5 PM tomorrow.” A stored monotonic reading is also a poor cross-host or post-reboot deadline because the reader may not share its clock origin.
Persistent deadlines need wall time and an authority.
Redis stores expiry as an absolute Unix timestamp in milliseconds. Time keeps passing while Redis is stopped, and moving the host clock forward can expire keys immediately.11 A TTL is therefore part data structure and part clock policy.
Security protocols expose the same seam. JWT's exp and nbf checks compare against the verifier's current time; the specification permits a small leeway to account for clock skew.12 That leeway is a product and security decision. Too little rejects valid traffic. Too much extends the period in which a token is accepted.
For a retry loop inside one process, use a monotonic deadline. For a lease or token that must survive processes and machines, store a wall-clock deadline, decide which system owns it, define tolerated skew, and make the consequence of a bad clock explicit.
Time-ordered IDs need an ordering contract too
UUIDv7 puts a Unix millisecond timestamp in the high bits, which makes IDs broadly time-sortable. The standard also spends a long section on counters, extra precision, rollback detection, and monotonic error handling.13
That detail matters. “Contains a timestamp” and “globally ordered” are not equivalent statements.
Two IDs created in one millisecond need an implementation rule if their generation order must be retained. IDs from separate nodes still reflect separate clocks. A library may handle rollback by carrying the previous timestamp forward, waiting, adding a counter, or reporting an error. The UUID version alone does not choose the application's required semantics.
The same caution applies to raw millisecond timestamps. Java documents that currentTimeMillis() can have a granularity coarser than one millisecond even though the value is expressed in milliseconds.14 More digits in storage do not manufacture more clock ticks.
Ask what time is being hired to do
The right value follows from the job:
| Question | Useful value | Boundary to state |
|---|---|---|
| When should a human see this happened? | UTC wall timestamp | source clock and uncertainty |
| How long did this call take? | monotonic clock difference | one clock domain and its suspend behavior |
| When did this database transaction begin? | transaction timestamp | database transaction |
| Which update follows the previous update? | version, sequence, or ordered-log position | the authority allocating order |
| When did a collector first see the event? | observation or receipt timestamp | collector clock |
| When should durable state expire? | wall-clock deadline | clock owner and skew policy |
I still store UTC. It is the least surprising representation for an instant and it keeps presentation rules out of persistence. I just do not ask UTC to prove more than it can.
Server time is relative, though not in the physics sense. It is relative to the server that read a clock, the kind of clock it read, and the boundary at which the software sampled it.
The fix is rarely a better date format. Name the fact first. Then choose a clock that can honestly supply it.
Footnotes
-
Linux man-pages,
clock_gettime(3), and Ruby 3.3,Process.clock_gettime.CLOCK_REALTIMEis adjustable wall time;CLOCK_MONOTONICcannot jump backwards, though successive readings may tie and its relationship to suspend depends on the clock variant. ↩ -
The Go standard library,
timepackage. Go combines wall and monotonic readings for local comparison and strips the monotonic part when values are serialized. ↩ -
PostgreSQL 18 documentation, Current Date/Time. It defines transaction, statement, and call-time clocks and explains why transaction time remains fixed. ↩
-
MySQL 8.4, Date and Time Functions, and SQLite, Date and Time Functions. Their current-time functions use query and
sqlite3_step()boundaries respectively. ↩ -
Rails source, Active Record timestamping. Normal model writes obtain
created_atandupdated_atfromTime.nowin the application process. ↩ -
IETF RFC 5905, Network Time Protocol Version 4. NTP estimates offset, delay, dispersion, and jitter while disciplining a local system clock. ↩
-
Leslie Lamport, Time, Clocks, and the Ordering of Events in a Distributed System. The paper defines causal ordering and the logical clocks that preserve it. ↩
-
Apache Cassandra documentation, Data Versioning. Its last-write-wins reconciliation uses mutation timestamps and explicitly depends on synchronized clocks. ↩
-
Google Cloud documentation, Spanner: TrueTime and external consistency. Spanner ties transaction timestamps to serial order using bounded clock uncertainty and commit rules. ↩
-
OpenTelemetry specification, Logs data model. It separately defines event-origin
Timestampand collector-sideObservedTimestamp. ↩ -
Redis documentation,
EXPIRE. Expiry is stored as absolute Unix time and is affected by changes to the host clock. ↩ -
IETF RFC 7519, JSON Web Token. The
expandnbfrules permit limited leeway for clock skew. ↩ -
IETF RFC 9562, Universally Unique IDentifiers. UUIDv7 is millisecond-based; stronger monotonicity depends on implementation choices for precision, counters, and rollback. ↩
-
Java SE 25,
System.currentTimeMillis(). The unit is milliseconds, while the operating system's measurement granularity may be coarser. ↩
Comments