Kafka Rejects Records Over 1 MB. LLM Spans Do Not Care.

The test that made the problem visible#

The full-pipeline size test builds a realistic OTEL batch, runs it through the splitter, and inspects the records that would be sent to the Quickwit topic. Its central assertion is plain:

assert.LessOrEqualf(t, rec.recordSize, maxRecordBytes,
    "record %d (key=%s) recordSize=%d exceeds budget=%d (valueSize=%d, overhead=%d)",
    i, rec.key, rec.recordSize, maxRecordBytes, rec.valueSize, kafkaRecordOverhead,
)

One fixture uses 880 KiB of plain text. That is 901,120 raw bytes. Under the test configuration, the record budget is 895,904 bytes after a 4 KiB safety reserve. Even before considering escaping, the field is too large. The document wrapper adds roughly another 141 bytes.

The 870 KiB fixture fits because its raw field plus wrapper remain below that same budget.

Define an application budget#

Kafka’s message.max.bytes broker setting and max.message.bytes topic setting constrain record size. An application still needs a smaller value budget for one field.

See Kafka’s broker configuration reference and topic configuration reference for the effective limits in a deployment.

The application has to reserve the parts it does not own.

The processor’s default batch target is 900 KiB, deliberately below Kafka’s approximate 1 MiB default. The Quickwit path takes another 4 KiB from that number for record safety and reserves 256 bytes for Kafka framing and the key. It then subtracts the JSON envelope that identifies the span, trace, feature, text type, and timestamp.

Conceptually, the calculation is:

text budget = configured batch target
            - record safety margin
            - Kafka framing allowance
            - serialized document envelope

The exact configured target matters. The Go default is 900 * 1024 bytes; the full-flow sizing test pins a production configuration value of 900_000 bytes. Both sit below the broker ceiling, but they are not interchangeable. A size calculation must use the configuration that actually publishes the record.

The limit must be expressed in the same unit as the publisher configuration. Account for the key, headers, framing, and JSON envelope before accepting a text chunk.

The processor keeps these reservations as named constants so that configuration changes have one sizing path to audit:

const (
    defaultBatchMaxBytes       = 900 * kibibyte
    quickwitBatchSafetyBytes   = 8 * kibibyte
    quickwitRecordSafetyBytes  = 4 * kibibyte
    kafkaRecordFramingBytes    = 256
)

The 4 KiB reserve protects an individual Quickwit document. The 8 KiB reserve protects the later batch packing step. They solve different problems, so they should not be collapsed into one unexplained “buffer.”

JSON changes the size of text#

A text field does not retain its raw length when it becomes JSON.

A normal byte needs one byte. Quotes, backslashes, and several control characters need an escape sequence. Go’s default JSON encoder also escapes <, >, and & with a six-byte \uXXXX sequence. LLM output regularly includes code, JSON, HTML fragments, quoted strings, and newlines, the characters that make raw length unreliable.

The implementation measures this expansion without marshaling each candidate chunk. Marshaling only to learn whether a chunk fits would allocate on a hot ingestion path and would force the system to build a value it may immediately discard or split again.

Instead, it calculates the encoded length:

func jsonEscapedStringLen(s string) int {
    n := 2 // surrounding quotes
    for i := 0; i < len(s); i++ {
        switch s[i] {
        case '"', '\\', '\b', '\f', '\n', '\r', '\t':
            n += 2
        case '<', '>', '&':
            n += 6
        default:
            if s[i] < 0x20 {
                n += 6
            } else {
                n++
            }
        }
    }
    return n
}

That function has a less glamorous companion: a test compares its result with json.Marshal for plain text, quotes, backslashes, newlines, tabs, HTML characters, control characters, UTF-8 text, and representative LLM output. The calculation is an optimization only if it stays identical to the serializer it replaces.

The document envelope receives the same treatment. The empty JSON shape has a fixed 83-byte skeleton:

{"span_id":"","trace_id":"","feature_id":"","text_type":"","text":"","created_at":}

The processor adds the escaped lengths of the identifier fields and the decimal length of the timestamp. A separate test marshals representative documents and fails if that envelope accounting drifts after a field is renamed, added, or removed.

The 83-byte envelope skeleton must remain visible because it determines whether a near-limit document fits.

Split at a real boundary#

Once the system knows the available text budget, splitting is uncomplicated in principle and full of sharp edges in practice.

The splitter starts with a raw-byte estimate, backs up to a UTF-8 rune boundary, computes the escaped JSON length, and shrinks until the text fits. When there is a suitable space before the boundary, it prefers that split. When there is not, it keeps the hard boundary. A single rune larger than the remaining budget still moves forward so malformed or pathological input cannot create an infinite loop.

The ordering matters:

1. Reserve the record and envelope bytes before looking at text.
2. Take a candidate slice no larger than the remaining raw-byte budget.
3. Move the endpoint off any partial UTF-8 rune.
4. Check the JSON-escaped length, then shrink if necessary.
5. Prefer a space boundary only after the candidate is known to fit.
6. Validate every fully serialized outbound message before publishing.

The final validation earns its place: analytical sizing protects the hot path; validation protects the delivery contract. The serializer, headers, or producer adapter can change independently of the splitter. A strict final check turns a bad assumption into an explicit error instead of an opaque broker rejection.

The code emits each Quickwit document as a pre-marshaled json.RawMessage, so the broker adapter does not serialize it a second time. After the splitter proves each record safe, it packs the messages into batches under MaxBatchBytes - 8 KiB.

One limit, two levels of packing:

Treating those as one operation is how a correctly sized document ends up inside an oversized batch.

The missing chunker is part of the lesson#

The Quickwit path now has a serialized-size calculation, UTF-8-safe splitting, and final record validation. That does not mean every Kafka path is safe.

The aggregation-job path still has no equivalent chunking or record-size validation. A sufficiently large aggregation batch can be rejected and retried as a whole. The correct response is to reuse the decision rule: identify the serialized unit, reserve its overhead, impose a bound before publishing, and test the boundary with realistic data.

A size limit is a contract at every producer boundary. Each producer needs to honor its own contract.