Trace IDs Are Opaque. Storage Still Needs a Time Signal.

Why timestamps still cannot answer the question#

The logs table orders by (FeatureId, Date, TimestampMs) for feature-scoped time ranges. traces_aggregate orders by (FeatureId, Date, TraceId) for known-trace lookups. TraceId is stored as a String: it identifies but does not tell time.

The obvious response is to use the span timestamp. The schema article and the Redis coordination article cover how the pipeline materializes Date and coordinates root timestamps.

Those timestamps solve a different problem. Even after the backend knows the root timestamp, a user may not provide it. The user copies a trace ID from a log line, a browser error, or a message from a teammate. They do not copy the exact root start time, the feature ID, and the likely partition with it.

That is the practical argument for putting a weak time signal into the ID: the ID is what most often crosses the boundary between a user and a lookup API.

OpenTelemetry leaves the layout open#

The OpenTelemetry Tracing API defines a valid TraceId as a 16-byte array with at least one non-zero byte, written canonically as 32 lowercase hexadecimal characters. It assigns no timestamp bits. A trace context must travel between SDKs, collectors, vendors, and services without every participant agreeing on one storage format.

The SDK specification currently says that SDKs must by default randomly generate IDs, and it requires a mechanism for customizing ID generation. Vendors get room to choose a different generator; backends get no permission to reject every other shape.

OpenTelemetry defines an identifier contract, not an identifier sort order.

Issue #1947 proposes changing the default generator toward an AWS X-Ray-like timestamp prefix, and it remains open. Even if accepted, backends would still have to accept opaque IDs from older SDKs, other vendors, and systems that generate their own context.

Three ways to fill 128 bits#

Neither time-prefixed format gives storage a causal truth. Two machines can have skewed clocks, multiple traces can begin in the same millisecond, and a retry can arrive much later than its root was created.

A custom generator must also preserve the trace-ID randomness behavior that sampling relies on. “Sortable” is not the only property that matters.

What locality buys a backend#

The ClickHouse layout already chooses the dependable baseline: feature first, date second, then the value that table’s read path needs. A time-prefixed trace ID would not replace those keys. It could add a useful correlation when a query starts with an ID, or when a different store organizes objects by identifier.

Consider a hypothetical object-store backend with one sorted chunk per hour, each recording its smallest and largest trace ID. With random IDs, a chunk from 10<00> can overlap the lexical range of nearly every other hour, so a pasted ID may need many chunks inspected.

With controlled, time-prefixed root IDs, a query can turn a rough time window into a narrower ID range. Some chunks still survive pruning without containing the trace, but the candidate set can shrink.

That is a plausible optimization, not a measured result. The gain depends on what fraction of IDs come from the controlled generator, how objects are sorted, and how much clock skew and late delivery widen the ranges.

The counterexample matters as much. A service receives a trace context from a mobile client or another vendor, and its root may have started yesterday on a device with a bad clock. A backend that trusts the first bits as a timestamp will misplace the query, reject valid data, or quietly miss the trace.

The useful boundary sits at the generator#

The OTLP path carries client-generated trace context into the pipeline, so the backend cannot choose those IDs after the fact.

The older REST ingestion path is different. Its parsing service generates a trace ID with a ULID generator when it creates a trace from the API format, so it can experiment with a time-correlated value without pretending that every OTLP trace has the same property. Both paths converge on one string TraceID downstream.

It would be a mistake to “normalize” every incoming ID into a new sortable value. A child service would search for the ID it received while the backend stored a different one, and logs, links, support tickets, and downstream systems would all disagree about the trace’s identity.

The backend can add a derived field or an auxiliary index, but it cannot rewrite the identifier that defines the trace.

A compatibility-first storage strategy#

1. Accept every valid incoming trace ID. Validate the 16-byte contract. Do not require ULID bits, X-Ray bits, a particular clock, or a particular vendor.
2. Keep time and tenant dimensions explicit. Partition and order data by the constraints the backend can trust.
3. Use sortable generators only for controlled roots, and treat the hint as optional. Do not use a decoded timestamp to decide whether a trace is valid, which partition is authoritative, or whether a late span belongs to a trace.
4. Add statistics beside the identifier. Min/max values, bloom filters, external indexes, or a derived creation-time field can improve lookup without changing trace identity.
5. Keep the wire representation honest. A ULID-like 128-bit layout can be hex-encoded into the 32 lowercase characters W3C Trace Context requires. A 26-character Crockford Base32 ULID string cannot go directly in a W3C trace ID field.

The experiment I would run first#

I have not measured any of this. Before changing a default generator or a table key, I would compare random, ULID-like, and timestamp-prefixed IDs on one identical workload, changing only the ID population and the ordering or statistics under test.

The measurements that matter: objects inspected to find a pasted ID, bytes read for a time window, write cost, and lookup accuracy under clock skew, retries, and child-first delivery. The benchmark may show that the current feature/date layout already wins.

Use explicit timestamps for correctness and a sortable generator only where you control the root and can prove that the locality helps a real workload. Then keep accepting the random, inherited, skewed, and unfamiliar IDs that make distributed tracing interoperable in the first place.