Three Ways to Instrument Go with OpenTelemetry

The decision table#

otelc avoids a runtime agent, but the code it injects still creates spans, records metrics, propagates context, and exports data through the SDK. OBI moves instrumentation outside the application process, where eBPF probes and the OBI process consume resources. Manual instrumentation runs the same SDK operations under your control.

Measure request latency, CPU, allocation rate, telemetry volume, and build time on a representative service.

Pick compile-time instrumentation when you control the build#

otelc wraps the Go toolchain through -toolexec. It matches functions against instrumentation rules and injects hooks into the binary during compilation.

Replace the standard build command:

otelc go build -o service .

This works well when:

• your team owns the build pipeline;
• the service uses supported packages such as net/http, database/sql, or gRPC;
• you want consistent instrumentation without wrappers in application code;
• a longer build does not threaten your deployment time.

Compile-time instrumentation shifts operational cost into the build pipeline. You must rebuild every binary, keep otelc compatible with the Go toolchain, and verify that its rules cover the libraries you use. Unsupported frameworks and clients remain invisible unless you add manual instrumentation or write a rule.

I tested this path in OpenTelemetry Go Compile-Time with otelc. On a one-package net/http and database/sql prototype with a warm Go cache, measured across three runs, it produced HTTP and SQL spans without source changes. Median build time increased from 0.39 s to 12.17 s, and binary size grew from 14.1 MB to 31.2 MB.

Start with one non-critical service. Inspect the generated spans, check for duplicate instrumentation, then measure the build and request paths.

Pick OBI when rebuilding is not an option#

OpenTelemetry eBPF Instrumentation observes executables and network activity from outside the application. It can capture traces and RED metrics for supported protocols and libraries without modifying code, installing an SDK in the target service, or rebuilding its binary.

OBI fits when:

• you run vendor-supplied or legacy binaries;
• you manage services written in several languages;
• you want fleet-level coverage from the platform layer;
• your applications run on compatible Linux hosts or Kubernetes nodes;
• your platform team can grant and audit the required eBPF privileges.

Fleet-wide deployment makes OBI useful during migrations and incident response. A platform team can obtain an HTTP and database baseline across Go, Java, Python, Node.js, Rust, and other workloads without waiting for every application team to change its build.

Those requirements exclude macOS, Windows, older kernels, and hosts where the platform team cannot grant eBPF capabilities. Some context propagation features also depend on the runtime and protocol.

OBI can identify an HTTP request or PostgreSQL operation. Only your application code knows that the request represents a failed subscription renewal or that a database transaction belongs to an inventory reservation.

Pick manual instrumentation for domain telemetry#

The OpenTelemetry Go API gives you control over span boundaries, names, attributes, events, errors, and status.

A manual span can describe the operation your users care about:

func reserveStock(ctx context.Context, orderID string) error {
	ctx, span := tracer.Start(ctx, "inventory.reserve")
	defer span.End()

	span.SetAttributes(attribute.String("order.id", orderID))

	if err := reserve(ctx, orderID); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "stock reservation failed")
		return err
	}

	return nil
}

Use manual instrumentation when:

• you need spans for business operations;
• you need stable domain attributes for querying traces;
• automatic instrumentation places span boundaries at the wrong level;
• you must record errors or events that do not cross an observable protocol;
• you maintain a library that should expose instrumentation through the OTel API.

Manual instrumentation costs engineering time. You must propagate context.Context, choose useful span boundaries, control attribute cardinality, and maintain the code as the domain changes.

Avoid tracing every function. A span should represent an operation that helps you explain latency, failure, or a user-visible transition. Function-level traces add volume without improving diagnosis.

The three options compose#

Combine one automatic baseline with manual domain spans.

A common setup uses automatic instrumentation for infrastructure spans and the Go API for domain spans:

HTTP request                         automatic: otelc or OBI
└── checkout.confirm                manual
    ├── inventory.reserve           manual
    │   └── PostgreSQL query        automatic: otelc or OBI
    └── payment.authorize           manual
        └── HTTP request            automatic: otelc or OBI

The automatic layer covers HTTP, RPC, database, messaging, and runtime activity. Manual spans explain why those operations happened.

Pass the active context.Context into your domain functions so manual spans inherit the automatic parent span.

Overlapping instrumentation can create duplicate spans. If otelc instruments net/http while your service also wraps the same handler with otelhttp, one request may produce two server spans. Keep the domain spans, then disable or remove one of the overlapping library-level instrumentations.

OBI has an additional Go-specific constraint. Its documentation warns against setting a global tracer provider when combining eBPF zero-code instrumentation with manual spans. Use the OpenTelemetry Auto SDK integration documented for that setup rather than initializing a competing global provider.

A practical selection order#

Start with the deployment constraint.

You control the build and run Go services: pilot otelc. Confirm that its supported rules match your dependencies and that the build-time cost fits your pipeline.

You cannot rebuild, or the fleet uses several languages: deploy OBI on a compatible Linux environment. Treat its protocol-level visibility as a baseline.

You need to explain business work: add manual spans. Neither compiler rules nor eBPF probes know your domain model.

Most Go services need two layers:

1. otelc or OBI for broad automatic coverage.
2. Manual spans for the operations that matter to users and operators.

Send both through the same OpenTelemetry Collector. Collector configuration is the next operational decision: sampling, attribute filtering, and telemetry-volume controls determine what reaches the backend.

FAQ#