Skip to content
Corentin GS

Kafka Rejects Records Over 1 MB. LLM Spans Do Not Care.

№48 · · ·1004 words ·5 min read
In this piece

A record-size test exposed an accounting error: an 870 KiB text field fit, while an 880 KiB field did not.

The test used a 900,000-byte batch target and reserved 4 KiB for each record, which leaves a budget of 895,904 bytes. The 880 KiB field is already 901,120 raw bytes, and its JSON document adds about 141 more. It exceeded the budget before JSON escaping mattered.

LLM spans accumulate prompt input, model output, tool arguments, metadata, and variables. The producer must account for the serialized record: value, key, headers, protocol framing, and JSON envelope. An oversized record fails under load and enters the retry path.

The test that made the problem visible

The full-pipeline size test builds a realistic OTEL batch, runs it through the splitter, and inspects the records that would be sent to the Quickwit topic:

go
assert.LessOrEqualf(t, rec.recordSize, maxRecordBytes,
    "record %d (key=%s) recordSize=%d exceeds budget=%d (valueSize=%d, overhead=%d)",
    i, rec.key, rec.recordSize, maxRecordBytes, rec.valueSize, kafkaRecordOverhead,
)

Define an application budget

Kafka’s message.max.bytes broker setting and max.message.bytes topic setting constrain record size. An application still needs a smaller budget for one field, because it has to reserve the parts it does not own:

text
text budget = configured batch target
            - record safety margin
            - Kafka framing allowance
            - serialized document envelope

The processor keeps these reservations as named constants so that configuration changes have one sizing path to audit:

go
const (
    defaultBatchMaxBytes       = 900 * kibibyte
    quickwitBatchSafetyBytes   = 8 * kibibyte
    quickwitRecordSafetyBytes  = 4 * kibibyte
    kafkaRecordFramingBytes    = 256
)

The 4 KiB reserve protects an individual Quickwit document. The 8 KiB reserve protects the later batch packing step. They solve different problems, so they should not be collapsed into one unexplained “buffer.”

The configured target matters too. The Go default is 900 * 1024 bytes; the full-flow sizing test pins a production configuration value of 900_000 bytes. Both sit below the broker ceiling, but they are not interchangeable. A size calculation must use the configuration that actually publishes the record.

JSON changes the size of text

A normal byte needs one byte in JSON. Quotes, backslashes, and several control characters need an escape sequence. Go’s default encoder also escapes <, >, and & with a six-byte \uXXXX sequence. LLM output regularly includes code, HTML fragments, quoted strings, and newlines, so raw length is unreliable.

Marshaling each candidate chunk to learn whether it fits would allocate on a hot ingestion path. The implementation calculates the encoded length instead:

go
func jsonEscapedStringLen(s string) int {
    n := 2 // surrounding quotes
    for i := 0; i < len(s); i++ {
        switch s[i] {
        case '"', '\\', '\b', '\f', '\n', '\r', '\t':
            n += 2
        case '<', '>', '&':
            n += 6
        default:
            if s[i] < 0x20 {
                n += 6
            } else {
                n++
            }
        }
    }
    return n
}

A test compares its result with json.Marshal for plain text, quotes, backslashes, newlines, tabs, HTML characters, control characters, UTF-8 text, and representative LLM output. The calculation is an optimization only if it stays identical to the serializer it replaces.

The document envelope receives the same treatment. The empty JSON shape has a fixed 83-byte skeleton:

json
{"span_id":"","trace_id":"","feature_id":"","text_type":"","text":"","created_at":}

The processor adds the escaped lengths of the identifier fields and the decimal length of the timestamp. A separate test fails if that accounting drifts after a field is renamed, added, or removed.

Split at a real boundary

Once the system knows the available text budget, the ordering matters:

  1. Reserve the record and envelope bytes before looking at text.
  2. Take a candidate slice no larger than the remaining raw-byte budget.
  3. Move the endpoint off any partial UTF-8 rune.
  4. Check the JSON-escaped length, then shrink if necessary.
  5. Prefer a space boundary only after the candidate is known to fit.
  6. Validate every fully serialized outbound message before publishing.

A single rune larger than the remaining budget still moves forward, so malformed or pathological input cannot create an infinite loop.

The final validation earns its place: analytical sizing protects the hot path; validation protects the delivery contract. The serializer, headers, or producer adapter can change independently of the splitter. A strict final check turns a bad assumption into an explicit error instead of an opaque broker rejection.

After the splitter proves each record safe, it packs the pre-marshaled messages into batches under MaxBatchBytes - 8 KiB:

  1. INPUTlarge text field
    1. text
  2. PROCESSINGsplitter
    1. chunk + escape
  3. RECORD BUDGETUTF-8-safe records
    1. pack
  4. BATCH BUDGETpacked batches
    1. produce
  5. BROKERKafka
One limit, two levels of packing

The splitter reserves record and envelope bytes, chunks UTF-8-safe text within a record budget, then packs those records into batches within a batch budget before publishing to Kafka.

Treating those as one operation is how a correctly sized document ends up inside an oversized batch.

The missing chunker is part of the lesson

The Quickwit path now has a serialized-size calculation, UTF-8-safe splitting, and final record validation. That does not mean every Kafka path is safe.

  1. PROCESSINGsplitter
    1. chunk + validate
    2. raw batchto aggregation path
  2. VALIDATEDQuickwit path
    1. safeto otel.search.documents
  3. NO CHUNKERaggregation path
    1. at riskto otel.evaluation.jobs
  4. TOPICotel.search.documents
  5. TOPICotel.evaluation.jobs
One producer has a chunker, one does not

The splitter fans the same parsed spans out to multiple Kafka topics. Only the Quickwit path reserves record bytes, splits text, and validates the serialized message. The aggregation path publishes the raw batch.

The aggregation-job path still has no equivalent chunking or record-size validation. A sufficiently large aggregation batch can be rejected and retried as a whole. The correct response is to reuse the decision rule: identify the serialized unit, reserve its overhead, impose a bound before publishing, and test the boundary with realistic data.

A size limit is a contract at every producer boundary. Each producer needs to honor its own contract.

Next in Rebuilding an Ingestion Pipeline

Streaming OTLP JSON in Go Without Loading the Payload

Part 4 continues the series.