Rebuilding an Ingestion Pipeline

A practical reading order. Start with Part 1; each published part builds on the previous one.

5 published parts

Reading order

Part 1 of 5

13 Aug №44

PostgreSQL Was Down Again: Rebuilding LLM Trace Ingestion

At 6 p.m. on December 19, 2025, the product was down again.

Devlog 12 min

Part 2 of 5

20 Aug №46

We Put Full-Text Search in PostgreSQL, Then ClickHouse. Both Were the Wrong Place.

The logs table stored trace IDs, users, token counts, latency, prompt input, model output, metadata, and variables. We used the same table to show one trace, graph cost by day, filter a feature’s recent spans, and find text a user remembered typing three weeks ago.

Devlog 6 min

Part 3 of 5

27 Aug №48

Kafka Rejects Records Over 1 MB. LLM Spans Do Not Care.

A record-size test exposed an accounting error: an 870 KiB text field fit, while an 880 KiB field did not.

Devlog 7 min

Part 4 of 5

3 Sept №50

Streaming OTLP JSON in Go Without Loading the Payload

The collector had CPU headroom and responsive Kafka producers, yet one OTLP batch made the hot path queue. The parser retained the whole payload while four downstream writers finished.

Devlog 4 min

Part 5 of 5

10 Sept №51

When One Go Binary Beats Three Separate Kubernetes Services

I needed three ingestion services, but I did not need three Go binaries.

Devlog 5 min