Streaming OTLP JSON in Go Without Loading the Payload

What a full-batch unmarshal costs#

A naive OTLP JSON parser is short:

var req otlpcollectortrace.ExportTraceServiceRequest
if err := json.Unmarshal(body, &req); err != nil {
    return err
}
return fanOut(ctx, req)

req is a deeply nested object graph. A full unmarshal retains every resource, scope, span, attribute, and value until fanOut finishes. The downstream transformer handles one span at a time, so retaining the complete tree adds memory pressure without helping that stage.

The envelope is the unit of work#

OTLP JSON looks like this at the top level:

{
	"resourceSpans": [
		{
			"resource": { "...": "..." },
			"scopeSpans": [
				{ "scope": { "name": "..." }, "spans": [/* spans here */] }
			]
		}
	]
}

The unit of work is one span. The parser needs to find every span, hand each one to the rest of the pipeline, and forget about it. The envelope above and the metadata beside each span are mostly not interesting for the splitter, but they are required for routing and enrichment, so the parser still has to read them.

The streaming shape is a tree descent:

The parser reads a field token, compares tok.String() with the fields it needs, and calls SkipValue for the rest:

for s.dec.PeekKind() != '}' {
    fieldTok, _ := s.dec.ReadToken()
    if !s.matchesField(fieldTok, keyScopeSpans) {
        _ = s.dec.SkipValue()
        continue
    }
    // descend into scopeSpans...
}

At the span boundary, json.UnmarshalDecode deserializes one otelJSONSpan. The parser reads the enclosing resource and scope objects field by field or skips them.

Field order is not guaranteed#

The OTLP protobuf and JSON wire formats do not promise a field order. The protobuf spec lets producers serialize fields in any order, and consumers must tolerate that. The parser cannot assume scope comes before spans, or that name comes before attributes.

Inside a scopeSpan, the scope name is needed for routing but may arrive before or after the spans. The parser handles this with bounded temporary buffering:

var pendingSpans []otelJSONSpan
var scopeName string

for s.dec.PeekKind() != '}' {
    fieldTok, _ := s.dec.ReadToken()
    switch {
    case s.matchesField(fieldTok, "scope"):
        // read scope name into scopeName, skip the rest
    case s.matchesField(fieldTok, "spans"):
        // buffer spans into pendingSpans
    default:
        _ = s.dec.SkipValue()
    }
}

// after the closing brace, emit every buffered span with the resolved scope name
for _, span := range pendingSpans {
    s.emit(span, scopeName)
}
pendingSpans = pendingSpans[:0] // reuse the backing array

Stream into the next representation#

The parser uses a 256 KiB bufio.Reader and emits ParsedSpan values through a channel buffered to 100. The Transformer marshals each value into one NDJSON line and sends it to its input channel. A completed batch then moves through io.Pipe into the S3 upload.

Backpressure exists at each bounded channel. The parser blocks when its output channel is full; the Transformer can drop an item when its own input channel is full. Monitor those channels and the upload path under representative load rather than inferring capacity from the parser alone.

Measure the boundary#

The parser benchmarks live in parser_bench_test.go. Run them with -benchmem, compare the old and new implementations on the same machine, then capture pprof -alloc_objects under representative load:

go tool pprof -alloc_objects -seconds=30 http://localhost:6060/debug/pprof/heap

The parser still allocates per span and may buffer all spans in one scopeSpan while it resolves field order. Measure those bounds; do not describe the path as zero-allocation.

The Splitter still loads a batch#

The streaming boundary ends at the Transformer output. Downstream, the Splitter reads the NDJSON object from S3 into []ParsedSpan:

spans, err := parseNDJSON(reader)

The Splitter therefore retains a full batch while it fans out to ClickHouse, Quickwit, aggregation, and per-span S3 uploads. A 20 MB input can create significant memory pressure even though the JSON parser processes spans incrementally.

The next improvement needs a different Splitter architecture: page spans from S3 or make its fan-out incremental. Both change its ordering and cross-destination behavior. This article’s narrower claim is that the JSON parser no longer constructs a complete OTLP object graph before it emits work.