Building a Custom OpenTelemetry Collector in Go

Why route through a Collector?#

The three-options post ended with this: send both [automatic and manual telemetry] through the same OpenTelemetry Collector. The reason is operational leverage.

Without a Collector, your service talks directly to every backend. Swapping Jaeger for Tempo, adding a Prometheus sidecar, or changing the sampling rate means redeploying every service. With a Collector in between, you change one config file.

The Collector also gives you transforms you cannot do inside the service:

• Attribute redaction: strip user.email, user.id, http.request.header.authorization before telemetry leaves the host.
• Sampling: tail-based sampling that keeps errors and slow traces, drops the rest.
• Routing: send traces to Jaeger, metrics to Prometheus, and logs to Loki — from the same OTLP stream.
• ** cardinality control**: drop url.path (high cardinality, useless for alerting) and keep http.route (normalized, stable).

This is the layer where platform teams enforce telemetry policy without touching application code.

The minimal Collector config#

The Collector is configured in YAML. The binary is otelcol (the OpenTelemetry Collector Core) or otelcol-contrib (the distribution that includes hundreds of community receivers and exporters). For everything in this post, otelcol-contrib covers the base.

Install it:

# Linux amd64 — other architectures at https://github.com/open-telemetry/opentelemetry-collector-releases/releases
curl -LO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.115.0/otelcol-contrib_0.115.0_linux_amd64.tar.gz
tar -xzf otelcol-contrib_0.115.0_linux_amd64.tar.gz
sudo mv otelcol-contrib /usr/local/bin/

The minimal config that receives OTLP/gRPC and prints traces to stdout:

# collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]

Run it:

otelcol --config collector-config.yaml

Point otelc-instrumented service at it:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
./myapp

Expected output (representative, trimmed):

2026/08/05 10:23:41 Received span{
    TraceID: 7d3f2a1b4c5e6f...
    SpanID: a1b2c3d4e5f6
    Name: "GET"
    Kind: Server
    Status: Unset
    Attributes:
        http.request.method: GET
        url.path: /users/42
        http.response.status_code: 200
        user.id: usr_7f3a2b1c
        user.email: [email protected]
        db.query.text: SELECT name, email FROM users WHERE id = $1
}

The user.id and user.email are a problem. That is PII leaving your infrastructure. The next section fixes it.

A business-relevant transform: PII redaction#

The filter processor drops spans and attributes by signal, severity, or attribute pattern. The transform processor rewrites attributes using an expression language. Combine them.

Drop all spans from the /health and /metrics routes — they add noise to traces without diagnostic value:

processors:
  filter/health_spans:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/health"'
        - 'attributes["http.route"] == "/metrics"'

Redact PII from span attributes. The transform processor replaces email addresses, user IDs, and authorization headers with [REDACTED]:

transform/pii_redact:
  error_mode: ignore
  traces:
    queries:
      - replace_pattern(attributes["user.email"], ".*", "[REDACTED]")
      - replace_pattern(attributes["user.id"], ".*", "[REDACTED]")
      - replace_pattern(attributes["http.request.header.authorization"], ".*", "[REDACTED]")
      - replace_pattern(attributes["db.query.text"], "SELECT .* FROM .*", "[SQL REDACTED]")

The replace_pattern function uses RE2 regex. The SQL redaction above replaces any SELECT ... FROM ... clause with a fixed token — it removes the query text while keeping the operation name and parameters if they appear separately.

Apply both processors to the traces pipeline:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/health_spans, transform/pii_redact]
      exporters: [debug]

Re-run the collector and hit the service again. Expected output:

2026/08/05 10:24:03 Received span{
    Name: "GET"
    Kind: Server
    Attributes:
        http.request.method: GET
        url.path: /users/42
        http.response.status_code: 200
        user.email: [REDACTED]
        user.id: [REDACTED]
        db.query.text: [SQL REDACTED]
}

/health and /metrics spans are gone entirely. PII attributes are scrubbed. Your backend receives telemetry that is still useful for latency and error analysis without exposing user data.

Routing by service name#

A fleet often has multiple services sharing one Collector endpoint. Add a groupbyattrs processor to stamp telemetry with a service.name resource attribute, then route to different exporters based on that label.

processors:
  groupbyattrs/service_stamp:
    actions:
      - key: service.name
        action: upsert
        value: "my-service"

exporters:
  debug:
    verbosity: normal

  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors:
        [groupbyattrs/service_stamp, filter/health_spans, transform/pii_redact]
      exporters: [debug, jaeger]

The debug exporter prints everything. In production, drop debug and use only jaeger (or otlphttp to a Grafana Tempo endpoint). The routing is in the exporter list order — both receive every span after processors run.

To route conditionally (e.g., send only errors to a dedicated backend), use the ottl condition in a filter processor:

filter/errors_only:
  error_mode: ignore
  traces:
    span:
      - 'attributes["error"] == true'

Building a custom Collector binary#

The YAML config covers most transforms. When you need logic that YAML cannot express, build a custom Collector binary with Go.

The Collector is a go.opentelemetry.io/collector/collector module. You add your custom logic as a processor, then wire it into a new main.go:

// main.go
package main

import (
	"go.opentelemetry.io/collector/cmd/builder"
)

func main() {
	// builder.New creates a Collector from a config struct.
	// It replaces the YAML config file for embedded deployments.
	_, err := builder.New(builder.Config{
		Receivers: builder.ReceiveConfigs{
			"otlp": struct{}{},
		},
		Processors: builder.ReceiveConfigs{
			"otlp": struct{}{},
		},
		Exporters: builder.ReceiveConfigs{
			"debug":    struct{}{},
			"jaeger":   struct{}{},
		},
	})
	if err != nil {
		panic(err)
	}
}

This is the boilerplate the otelcol and otelcol-contrib distributions already provide. The real value is writing a custom processor:

// myprocessor/factory.go
package myprocessor

import (
	"context"

	"go.opentelemetry.io/collector/component"
	"go.opentelemetry.io/collector/processor"
	"go.opentelemetry.io/collector/processor/processorcapability"
)

func NewFactory() processor.Factory {
	return processor.NewFactory(
		component.Type("myprocessor"),
		createDefaultConfig,
		processor.WithTraces(createTracesProcessor, processorcapability.Traces{...}),
		processor.WithMetrics(...),
		processor.WithLogs(...),
	)
}

Then register it:

builder.New(builder.Config{
    Processors: builder.ReceiveConfigs{
        "myprocessor": struct{}{},
    },
    // ...
})

The Collector builder documentation covers the full factory API, config struct, and test harness. For a custom processor that does not exist in collector-contrib — a proprietary sampling algorithm, a company-specific attribute normalization, a gateway that routes to an internal backend — this is the path.

Running the Collector#

The Collector runs as a sidecar, a daemon per host, or a central service. The right topology depends on your throughput and reliability requirements.

Sidecar per pod (Kubernetes):

# otelcol-sidecar.yaml — Collector as a sidecar container in the same pod.
containers:
  - name: myapp
    image: myapp:latest
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: "localhost:4317"
  - name: otelcol
    image: otelcol-contrib:0.115.0
    args: ["--config", "/etc/otelcol/collector-config.yaml"]
    volumeMounts:
      - name: otelcol-config
        mountPath: /etc/otelcol

DaemonSet per host:

# otelcol-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otelcol
spec:
  selector:
    matchLabels:
      app: otelcol
  template:
    metadata:
      labels:
        app: otelcol
    spec:
      hostNetwork: true # access host-level metrics
      containers:
        - name: otelcol
          image: otelcol-contrib:0.115.0
          args: ["--config", "/etc/otelcol/collector-config.yaml"]
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "localhost:4317"
          volumeMounts:
            - name: config
              mountPath: /etc/otelcol
              readOnly: true

Central (for low-to-medium traffic):

One Collector instance receives telemetry from all services, applies transforms, and forwards to the backend. Scale it horizontally behind a load balancer for high throughput.

For the local prototype from this post, run it directly:

otelcol --config collector-config.yaml

Then run your otelc-instrumented service in another terminal:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
./myapp

What collector-contrib adds#

The base otelcol binary ships with OTLP, debug, and a handful of exporters. otelcol-contrib adds hundreds of community-maintained receivers and exporters. The components most teams reach for first:

• Receivers: kafka, awscloudwatch, awsxray, datadog, fluentbit, prometheus, statsd, zipkin
• Exporters: awsprometheusremotewrite, azuremonitor, clickhouse, datadog, dynatrace, elasticsearch, honeycomb, influxdb, jaeger, kafka, loadbalancing, loki, prometheus, prometheusremotewrite, sentry, splunk, tempo, tencentcloud
• Processors: batch, filter, memorylimiter, probabilistic_sampler, resourcedetection, tail_sampling, transform

The tail_sampling processor is worth calling out. It makes decisions after seeing a full trace — keeping traces that contain errors, have high latency, or match a custom policy — rather than sampling at the client. This is the processor you want when head-based sampling loses too many interesting traces.

Next steps#

The series started with compile-time instrumentation — zero-code spans and metrics from your service. The three-options post put that in context against OBI and the manual API. This post connected those telemetry signals to a Collector that redacts PII, filters noise, and routes to your backend.

Where you go next depends on your stack:

• Jaeger or Tempo for traces? Use the jaeger or otlphttp exporter. Tempo is otlphttp pointing at your Grafana Tempo endpoint.
• Prometheus for metrics? The Collector has a Prometheus exporter, and otelcol-contrib has a Prometheus receiver that scrapes targets and converts metrics to the OTel format.
• Sampling? Start with tail_sampling and keep 100 % of errors and traces above a latency threshold. Tune the ratio from there.
• Cardinality? The transform processor normalizes url.path to http.route using the ottl replace_pattern` function. Drop raw paths before they reach your backend.

The Collector is the telemetry routing layer that keeps your services clean and your backends honest. It is the piece between the signal and the storage.

References#

• OpenTelemetry Collector documentation
• Collector builder
• Collector contrib components
• OTL (OpenTelemetry Language) reference
• Tail sampling processor: GitHub — open-telemetry/opentelemetry-collector-contrib
• Related on this blog: compile-time instrumentation with otelc, three ways to instrument Go with OTel, context propagation and cancellation in Go