Skip to content
Corentin GS

One Terabyte a Day: The Full Ingestion Architecture

№61 · · ·1302 words ·7 min read
In this piece

data/{workspaceId}/{year}/{month}/{day}/{timestamp}-{traceId}.json

That is a span’s address in the rebuilt pipeline: an object in a bucket, with a key that a human can read. Everything this series built is organized around making that address cheap to create and never the reason the product falls over.

Nine articles covered the pieces. This one assembles them into the machine you would actually operate, then reports the bills still unpaid. The rule the series kept relearning: choose boundaries by failure mode, not by technology.

One honesty note on scale. A terabyte of telemetry a day is the design target: it was the peak volume of a single customer, and we built the pipeline expecting larger volumes over time.

The query work that motivated the redesign went from tens of seconds to under a second; that is the reported outcome, not a benchmark this series measured.

The path one trace takes

A trace enters as OTLP from a customer’s SDK. The collector, a custom OpenTelemetry Collector extension, authenticates the workspace and transforms the payload at the edge. It writes the raw payload to S3 first, then publishes a small pointer to Kafka. Payload in objects and notifications on the broker: the whole design rests on that inversion.

  1. INGRESSOTLP SDKs
    1. OTLP / HTTP
  2. EDGECollector
    1. durable write
    2. pointerto Kafka raw pointers
  3. OBJECTSS3 raw payloads
    1. read payloadto Transformer
  4. EVENTSKafka raw pointers
    1. consume
  5. GOTransformer
    1. NDJSON batch
    2. batch pointerto Kafka batch topic
  6. OBJECTSS3 NDJSON batches
    1. read batchto Splitter
  7. EVENTSKafka batch topic
    1. batch event
  8. GOSplitter
    1. search documents + jobs
    2. structured rowsto ClickHouse
    3. per-span JSONto S3 span objects
  9. EVENTSKafka search and job topics
    1. search documentsto Quickwit
    2. aggregation jobto Aggregator
    3. evaluation jobto Evaluation jobs
  10. OBJECTSS3 span objects
    1. presigned payloadto Product UI
    2. read spansto Evaluation jobs
  11. TEXT SEARCHQuickwit
    1. candidate IDsto Product UI
  12. GOAggregator
    1. root time + pendingto Redis timestamps
    2. aggregate rowsto ClickHouse
  13. JOBSEvaluation jobs
  14. COORDINATIONRedis timestamps
  15. ANALYTICSClickHouse
    1. analytics + previews
  16. READSProduct UI
The full pipeline, one trace end to end

A trace enters through the collector, lands durable in S3, is announced on Kafka, processed by three modes of one Go binary, and served from ClickHouse, Quickwit, and presigned S3 objects. Splitter publishes search documents and jobs back to Kafka.

Three Go services do the processing:

  • Transformer parses OTLP JSON a span at a time (streaming parser), then writes normalized NDJSON batches back to S3 and announces them.
  • Splitter fans each span out three ways: structured rows to ClickHouse, full-text documents to Quickwit through Kafka, and durable per-span JSON to S3. It also publishes the aggregation and evaluation jobs.
  • Aggregator handles trace-level rollups, coordinating out-of-order children against root timestamps in Redis.

All three are the same Go binary, run in three Kubernetes Deployments with a mode flag: the one-binary trade. The interesting decision was where the process boundaries sit, not what runs inside them.

Durability before notification, replay as the consequence

S3 first and Kafka second is what makes downstream failures recoverable. The object is the copy, and a failed consumer retries against a key that is still where it landed. Within the retention window, every projection can be rebuilt from the objects and batches in S3.

The consequence is at-least-once everywhere. A reprocessed batch re-writes rows, and a consumer restart can re-deliver a job. Duplicates are a normal input: Quickwit’s candidate IDs are deduplicated against ClickHouse at read time, as the search design requires.

Every boundary trades a failure mode

Each boundary exists because of a specific mechanism that took the product down on December 19, 2025. Each one trades that failure for a smaller, named exposure rather than deleting risk outright.

BoundaryDecember failure it removesMechanism
S3 before processing15 MB payload inside one ACID transactionObject write is atomic and lock-free
Kafka pointers81 SQS messages invisible inside 300-second LambdasOffset replay; no visibility-timeout purgatory
Containers, not Lambdasp99 reached 300,000 ms, the function timeoutNo Lambda execution limit on one unit of work
One store per query shapeAuth queries queued behind trace inserts; 403 connections demanded against 400Failure domains stop sharing an instance
Previews in, blobs outTrigram indexes updating inside the insert transactionDatabases store references, not megabytes

The record-size article shows how the 900 KiB batch budget keeps search documents under Kafka’s limit.

What each store owns

Store choice followed query shape, not preference.

StoreWritten byRead by
PostgreSQLProduct services (auth, metadata)Product API and UI
ClickHouseSplitter, AggregatorAnalytics UI, trace screens, filters
QuickwitSplitter (via Kafka)Text search, candidate IDs
S3Collector, HTTP API, Transformer, SplitterUI payloads, evaluation, replay

The schema article covers the ClickHouse table layouts, and the delivery article covers how payloads reach the browser.

The bill this design pays on purpose is storage. Every span is materialized about five times before anyone reads it: raw payload, NDJSON batch, per-span object, ClickHouse row, Quickwit document. At the terabyte-a-day design target, that multiplication is real money. Retention is the control, and it is set per customer contract rather than once for the whole platform.

What we still owe

These debts are all still true.

Data can still be lost or duplicated:

  • The entry point has an orphan window. An S3 write whose Kafka publish fails leaves a durable, unreferenced object, and no inventory sweep exists to republish those pointers.
  • The pending-list flush has races. The read-publish-delete sequence around Redis can duplicate or lose aggregate rows under partial failure.
  • Duplicates are tolerated, not made impossible. Idempotent writes keyed by span identity would remove that reliance.
  • Dead-letter topics are still a lesson, not a fixture. A poison message today still depends on an operator.

Limits are not enforced everywhere:

  • The aggregation-job path has no chunker. Splitter publishes those jobs to otel.evaluation.jobs without record-size validation.
  • The splitter loads a full batch into memory. It is the next place a batch-size regression will bite.
  • The daily analytics view mixes aggregate states with scalar columns. It works until a cross-partition total silently lies. The replacement schema is designed, not shipped.

Decisions remain open:

  • ParsedSpan is defined twice. On March 12, a span-kind fix crossed eleven files. Generated schemas remain the right answer.
  • The trace-ID experiment has not run. Until it does, explicit timestamps stay the source of truth, and a derived creation-time column is still only a proposal.
  • Signed cookies have only been considered. Nobody has designed or deployed them yet.

The decision

S3, Kafka, Go, ClickHouse, Quickwit, Redis, and PostgreSQL are all boring. What made the rebuild work was asking, for every December failure mechanism, which boundary contains it, then paying the operational price of that boundary and no more.

The old system was one database that could do everything and therefore did. The new one splits the work across stores and services that each do one thing and refuse the rest. On this workload, that trade wins, and the debts above plus the on-call surface of Kafka, Redis, and Quickwit are its running cost.

A lost store now degrades the queries it serves. A lost broker delays processing: payloads already in S3 stay durable, but a write whose pointer never publishes falls into the orphan window above. A lost Redis can still corrupt aggregate correctness, which is why its races sit on the debt list.

December’s failure was customers who could not log in. A set of boundaries buys you a smaller failure than that.

Explore this subject

More on Devlog