№57
Trace IDs Are Opaque. Storage Still Needs a Time Signal.
The first useful clue in our ClickHouse schema was not a query. It was this line:
Devlog Rebuilding an Ingestion Pipeline · Part 7 11 min
Devlog
Real production work debriefed after the fact — what broke, what it cost, and the lessons carried forward.
8 published entries
Begin with a path
These series give this subject an intentional reading order.
-
Rebuilding an Ingestion Pipeline
A reading order through the rebuilt tracing pipeline: Kafka delivery, Go processing, Redis coordination, ClickHouse analytics, and Quickwit search.
See the reading order →
All writing
№55
Out-of-Order Traces: Redis Timestamp Coordination Model
A child span can reach the aggregate processor before its root. If the root’s timestamp falls in the previous month, publishing both rows under their own timestamps splits the trace across monthly partitions.
Devlog Rebuilding an Ingestion Pipeline · Part 6 5 min
№53
Five ClickHouse Decisions for LLM Trace Analytics
I had moved trace ingestion away from PostgreSQL, split large payloads before Kafka rejected them, and separated text search from analytical reads. ClickHouse accepted the rows without drama. Then I tried to show one trace, list a feature’s recent runs, and chart latency by day. Each read wanted the data arranged differently.
Devlog Rebuilding an Ingestion Pipeline · Part 5 8 min
№51
When One Go Binary Beats Three Separate Kubernetes Services
I needed three ingestion services, but I did not need three Go binaries.
Devlog Rebuilding an Ingestion Pipeline · Part 8 5 min
№50
Streaming OTLP JSON in Go Without Loading the Payload
The collector had CPU headroom and responsive Kafka producers, yet one OTLP batch made the hot path queue. The parser retained the whole payload while four downstream writers finished.
Devlog Rebuilding an Ingestion Pipeline · Part 4 4 min
№48
Kafka Rejects Records Over 1 MB. LLM Spans Do Not Care.
A record-size test exposed an accounting error: an 870 KiB text field fit, while an 880 KiB field did not.
Devlog Rebuilding an Ingestion Pipeline · Part 3 7 min
№46
We Put Full-Text Search in PostgreSQL, Then ClickHouse. Both Were the Wrong Place.
The logs table stored trace IDs, users, token counts, latency, prompt input, model output, metadata, and variables. We used the same table to show one trace, graph cost by day, filter a feature’s recent spans, and find text a user remembered typing three weeks ago.
Devlog Rebuilding an Ingestion Pipeline · Part 2 6 min
№44
PostgreSQL Was Down Again: Rebuilding LLM Trace Ingestion
At 6 p.m. on December 19, 2025, the product was down again.
Devlog Rebuilding an Ingestion Pipeline · Part 1 12 min