14 min left
2726 words
14 minutes

OpenTelemetry Go Compile-Time with otelc

OpenTelemetry Go Compile-Time Instrumentation v1 lets you swap go build for otelc go build and get traces, metrics, and log correlation with zero source changes. No wrappers, no SDK init, no agent at startup. A SIG formed by Alibaba and Datadog, with maintainers from Cabify. Vendor-neutral, stable.

TLDR: otelc go build injects OpenTelemetry into your binary at compile time: stdlib, dependencies, and all. It works, it costs build time, and it has gaps. This post covers what ships, what doesn’t, and where it breaks.

What is otelc?#

otelc is the Go toolchain wrapper for OpenTelemetry compile-time instrumentation. It sits between go build and the compiler via -toolexec, matches packages and functions against an instrumentation rule set, injects trampolines and hooks, and ships traces, metrics, and log correlation in the resulting binary. No source changes, no runtime agent, no shared library at startup.

Install and build#

Terminal window
go install go.opentelemetry.io/otelc/tool/cmd/[email protected]
otelc go build -o myapp .

That is it. No annotations, no wrappers, no SDK init in your main.go source. The tool finds what it can instrument by scanning your module graph and rewrites the compiled code to emit spans and metrics on its own.

Now point the binary at a collector. Set the standard OTel env var so the auto-initialized SDK knows where to export, then run the collector with a minimal config:

Terminal window
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc # be explicit; default differs across SDKs
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
otelcol --config otelcol-config.yaml
otelcol-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
exporters: [debug]

Now a service that is nothing but a net/http handler and a database/sql query. Before, a curl localhost:8080/users/42 produces no telemetry. After the one-line build swap, the same curl produces spans shaped like this (representative, trimmed):

{
"resourceSpans": [{
"scopeSpans": [{
"spans": [{
"name": "GET",
"kind": "SPAN_KIND_SERVER",
"attributes": {
"http.request.method": "GET",
"url.path": "/users/42",
"http.response.status_code": 200,
"network.protocol.version": "1.1"
}
}, {
"name": "DB Query",
"parentSpanId": "0000000000000000",
"attributes": {
"db.system.name": "postgresql",
"db.query.text": "SELECT name, email FROM users WHERE id = $1",
"db.operation.name": "SELECT",
"server.address": "db.local"
}
}]
}]
}]
}

The HTTP span carries the route, method, status, protocol: all semantic conventions, no ad-hoc fields.

The SQL span sits alongside the HTTP span because QueryRowContext carries context through the database/sql stack. Whether it becomes a real parent/child link depends on the OTel rules propagating the span context. The runtime hooks are present in the matched set below; I did not decode the JSON excerpt to confirm parent/child in this prototype.

The diff between “before” and “after” for application source is zero; the build-command diff is one line: go buildotelc go build.

To verify, I ran it on a minimal net/http + database/sql (modernc.org/sqlite, in-memory) service with no otel.instrumentation.go file. After the build swap, .otelc-build/matched.json listed the rules that fired:

  • database/sql — Open, BeginTx, Exec/QueryContext on DB/Conn/Tx/Stmt, Commit/Rollback
  • net/http — client RoundTrip, server ServeHTTP
  • log.Logger.Output — for slog/logrus correlation
  • runtime — goroutine-local-storage via newproc1
  • init_sdk — file rule injected into main

Three curls against /users/42 produced three HTTP spans and three SELECT spans with real OTel semantic conventions attribute names (http.request.method, http.response.status_code, url.path, db.query.text, db.system.name, db.operation.name) and a schema URL of https://opentelemetry.io/schemas/1.41.0 on the resource. Numbers from this run appear in the performance section below.

One caveat before you copy-paste: v1.0.0 was retracted the day it shipped because otelc pin wrote broken module paths into go.mod. v1.0.1 fixed it the same day. Pin v1.0.1 explicitly in your Dockerfile and CI; @latest will resolve cleanly today but the retraction is recent enough to be worth dodging on purpose.

How otelc works#

The Go toolchain’s go build accepts a -toolexec flag that interposes a wrapper command around every compile and link invocation. otelc is that wrapper: for every package being built, it loads its rule set, finds matching functions in the AST, and rewrites the code before the real compiler runs. The rewritten code is what ships.

Much of the built-in instrumentation uses injected hook points and trampolines (tiny functions injected at the top of instrumented calls), invoking hooks via auto-generated //go:linkname directives (a compiler mechanism that lets the injected code call into unexported OTel internals without a public import). The trampoline isolates panics: your app keeps running if a hook blows up. It also gives every hook a uniform HookContext API regardless of the target function’s signature. The rule engine can also rewrite call sites, append arguments, inject struct fields, and expand annotated declarations; see docs/rules.md for the full set of transforms. At runtime, the only cost is whatever the rule injected plus whatever the hook itself does. No agent attaches at startup, no shared library gets loaded.

otelc commands#

Terminal window
otelc go build # instrument + build (default)
otelc go install # instrument + install
otelc go test # instrument + test
otelc setup # generate .otelc-build/ artifacts once
otelc pin # write otel.instrumentation.go (local workflow)
otelc cleanup # remove the .otelc-build/ directory

Invocation modes#

Three invocation modes exist: otelc go build, a GOFLAGS export after otelc setup, and a Dockerfile swap. otelc go build is the one to reach for by default. The Dockerfile version is the same line inside a build stage:

FROM golang:1.25 AS build
RUN go install go.opentelemetry.io/otelc/tool/cmd/[email protected]
COPY . /src
WORKDIR /src
RUN otelc go build -o /out/myapp .

What otelc instruments in v1#

Coverage is the point. v1 ships instrumentation for the libraries you use, split across four clusters.

HTTP and RPC#

net/http (client and server) and google.golang.org/grpc (client and server) are pre-instrumented. Add github.com/gin-gonic/gin and you get the same server-side spans without touching your handlers.

Data stores#

database/sql covers the bulk of Go services. github.com/redis/go-redis/v9 and go.mongodb.org/mongo-driver/mongo extend that to the common NoSQL pair.

Messaging and AI#

github.com/segmentio/kafka-go handles the Kafka client most new Go services reach for. github.com/openai/openai-go is a signal of where the SIG thinks the audience is going.

Kubernetes and logs#

k8s.io/client-go informers cover controller workloads. log/slog and sirupsen/logrus are instrumented for log records (per the supported-libraries page). otelc can inject trace context into those records, making trace–log correlation possible without manually wiring every logger. Whether logs reach an OTel backend is a separate question and depends on the logger, handler, SDK configuration, and the export pipeline you choose.

What otelc does not instrument in v1#

Echo, Fiber, gorilla/mux, Hertz. pgx, sqlx, gorm. Sarama, Franz-Kafka. The Anthropic SDK. This list is partial; anything outside the supported-libraries page needs an answer of its own. If you depend on one of these, you either write your own rule or look sideways at the vendor-backed predecessor and parallel implementations. Alibaba still ships Loongsuite-go with dozens of supported libraries including echo, fiber, gorm, sqlx, sarama, kratos, and kitex. Datadog still ships Orchestrion. The OTel SIG was formed by contributors from Alibaba, Datadog, and Quesma to unify approaches; those projects are not simple forks of this repository. They keep shipping in parallel. Read that however you want; my read is “coverage gaps are real, and the vendors kept their escape hatches.”

Performance: what the benchmarks don’t show#

The official claim is “no added runtime overhead.” That phrasing specifically means no runtime agent and no attach step at startup (both true, and worth having). There is still a per-request cost: the injected hooks, span creation, metrics recording, context propagation, and attribute extraction all consume CPU and may allocate. The project has not yet published request-path benchmarks that characterize this, so for a high-traffic hot path, measure on your own code before trusting any “zero overhead” framing. From a single observed run, treat the SDK init time of sub-five-millisecond as a floor, not a guarantee (trace/meter/logger providers log their init lines on startup, all synchronous, none blocking on the exporter in my test setup).

Build-time overhead is where otelc shows its cost. -toolexec places otelc in the compiler path for package builds, so even packages without matching instrumentation can contribute per-package overhead. The project’s own Makefile exposes BENCH_MAX_OVERHEAD_PCT=150 to a benchmark/threshold Go test that fails the build if overhead exceeds 150 %, a guardrail ceiling the project enforces on itself (not a measurement of typical overhead). A clean per-package floor is harder to find. Here is what I measured on the one-package prototype (net/http + database/sql + modernc.org/sqlite, warm Go cache, three runs each):

Plain go buildotelc go buildDelta
Median0.39 s12.17 s+11.78 s (+3051 %)
Min / Max0.38 / 0.44 s11.99 / 12.39 s
Binary size14.1 MB31.2 MB+17.1 MB (+121 %)

Plain go build versus otelc go build on a one-package prototype (net/http + database/sql + modernc.org/sqlite, warm cache, three runs each). The +3051 % delta is dominated by first-time SDK download cost; the per-package toolexec floor is what amortizes on a real codebase.

The 3051 % number is what to expect when the package count is small: you are paying the SDK + OTel exporter dependency load plus per-package toolexec process spawn, almost none of which amortizes. On a real codebase with hundreds of packages, the SDK download amortizes to zero and the per-package spawn cost dominates, which is where the project’s 150 % CI ceiling lives. Issue #570 is still open and there is no public runtime SLO. For a build-critical CI path, measure on a representative commit before wiring otelc in.

Two issues the announcement did not address.

Metric cardinality. Auto-instrumentation emits more telemetry and more attributes than you may expect. Audit metric labels separately from span attributes:

  • Normalize routes so unparameterized paths don’t blow up label series
  • Drop high-cardinality labels in the SDK or Collector
  • Configure aggregation or filtering before exporting
  • Use OTEL_GO_DISABLED_INSTRUMENTATIONS to turn off specific auto-instrumented sources

Trace sampling will reduce trace volume. It does not solve metric-series cardinality.

SDK init at startup. In my run the init was fast, but otelc does inject runtime-metric collection and an OTel pipeline initialization. Nothing in the documented behavior says slow DNS resolution would block your main() from returning; OTLP exporters batch and connect asynchronously. Do not pre-emptively redesign for this; measure on your own startup path first. Expect benign processor export timeout warnings on graceful shutdown if the collector connection drops before the final flush. The prior batch has already left the process by then.

A v1 semver release is not the same as production-proven. The official docs page still carries a NOTE that the project “is under active development and is not yet ready for production use.” The table below uses the docs wording, because that note is operationally honest.

Limitations you will hit#

  1. Which go subcommands are wrapped. The documented and tested entry points are otelc go build, otelc go install, and otelc go test. Any subcommand that drives the compiler through -toolexec can be instrumented in principle, but the tested surface is those three. For a fast dev loop where you do not want instrumentation, the GOFLAGS mode after otelc setup keeps plain go build working.

  2. Generics may expose rule limitations. Generic functions can hit rule and hook gaps, especially when instrumentation needs to inspect or replace typed parameters and return values. Type parameters are not always available in the injected code, and the public docs do not enumerate every edge case. If your database layer or hot paths are generic, test those paths explicitly before adopting. Auto-instrumentation there may be partial.

  3. Restricted hook imports (for packaged rules). A packaged hook for a target like github.com/foo/bar is generally limited to importing the target library, the OTel packages, and the standard library. Any other third-party import breaks the build graph. This constraint applies to packaged instrumentation rules; if you write your own rules, the rule engine can declare additional imports as long as the target module resolves them (see docs/instrument-guide.md). If you need custom marshalling or a proprietary logger inside a packaged hook, inline it or use a side-channel.

  4. Migration and coexistence. If you already instrument manually with otelhttp or the OTel SDK, you can compose manual and zero-code instrumentation (the v1 announcement explicitly supports this). The risk is double-counting when your manual code and a compile-time rule both wrap the same call. Audit overlap before enabling a built-in rule: keep manual domain spans, and either disable the overlapping otelc instrumentation for that surface (OTEL_GO_DISABLED_INSTRUMENTATIONS covers runtime metrics and adjacent controls; per-rule disable exists for built-ins) or remove duplicate library wrappers. This is the first question teams with existing instrumentation ask, and it is worth a dedicated migration note in your own docs.

  5. Go version and toolchain. The otelc module requires Go 1.25.0+ (per its go.mod), so go install go.opentelemetry.io/otelc/tool/cmd/otelc@latest needs a recent toolchain. Two platform questions get confused: where otelc itself builds (the Makefile PLATFORMS variable lists darwin/linux amd64/arm64 plus windows amd64 for the tool binary), and what GOOS/GOARCH targets you can instrument (which is a separate question). Using a host otelc binary to instrument arbitrary cross-compile targets should be tested rather than inferred from the tool’s release matrix. go mod vendor is supported since v1 (it broke setup previously).

  6. First-build troubleshooting. Private VCS, GOTOOLCHAIN=auto mismatches, and a stale module cache are the usual suspects when the first otelc go build fails. On a fresh cache otelc also pulls in a long tail of OTel exporter modules (seventeen direct deps on a small app), so a low-quota sandbox can run out of disk on the first build. otelc’s state snapshotting (track backup files) writes to the module directory. Run otelc setup again after any go.mod change, and clear your GOCACHE if the build drifts.

  7. -toolexec is brittle. Romain Muller, a Datadog engineer who worked on the SIG, opened golang/go#69887 listing the mechanism’s limits: influence on object build IDs, help-output parsing, past bugs with packages like nats.go. v1 works around these with a job server and packages.Load, but the upstream complaint that -toolexec is brittle remains valid. The proposal is still open.

otelc vs otelhttp vs OBI#

The three Go OpenTelemetry paths cover different constraints. The decision table below is the short version.

Criterionotelc (compile-time)OBI (eBPF)Manual SDK (otelhttp)
Zero-codeYesYesNo
Runtime costIn-process hooks and SDK workOut-of-process eBPF plus in-process SDK where applicableIn-process SDK under developer control
Build-time overheadSignificantNoneNone
Stdlib coverageYesPartialNo
Third-party coverageLimited (see v1 list)Broad (network protocols)No
Custom domain spansVia rulesLimitedYes, easy
No rebuild possibleN/AYes (runtime attach)No
Polyglot fleetNoYes (multi-language)No
Production-readyv1.0.1 — docs say not production-readyOBI v0.10.0 (Development)Stable

Compile-time wins for teams that can rebuild and want zero-code coverage of stdlib and dependencies. If you cannot rebuild, or you run a polyglot fleet, eBPF is the right call on principle. Many internal services can be rebuilt more easily than teams initially assume, but signed, vendor-supplied, legacy, or tightly regulated binaries are real exceptions.

Verdict#

I would pilot v1.0.1 on a non-critical service this week, with build-time and request-path benchmarks before broader rollout. The one-line build swap, the auto-context-propagation, and the slog correlation are worth the build-time tax for anything that is not on your build-critical CI path. Hold until there is a public runtime SLO and the build overhead is documented per-codebase-size rather than per-CI-ceiling. If a 90-second build becoming four minutes means a deploy SLA breach, this is not the tool for that path today.

Run otelc go build on one service. Measure the build delta, the binary size, and whether the spans carry the attributes you expect. Email me what you measure; I’m collecting data points ahead of a future comparison, and real numbers beat vendor benchmarks. On a small Go service my numbers were +3051 % build time and +121 % binary size; the spread on a real monorepo is what I want to see.

FAQ#

What is otelc?#

otelc is the Go toolchain wrapper for OpenTelemetry compile-time instrumentation. It scans your module graph, injects hooks and trampolines into matches via -toolexec, and ships traces, metrics, and log correlation in the resulting binary — no source changes, no runtime agent.

Is otelc production-ready?#

I would pilot, not ship. v1.0.1 is a stable semver release, but the official docs page still carries a NOTE that the project is “not yet ready for production use.” The comparison table above reflects that wording.

How much does otelc slow down the Go build?#

On my one-package prototype (net/http + database/sql + modernc.org/sqlite, warm cache, three runs), median build time went from 0.39 s to 12.17 s — a +3051 % delta. On a real codebase the SDK download amortizes to zero and the per-package toolexec cost dominates; the project’s own threshold ceiling is BENCH_MAX_OVERHEAD_PCT=150 in the Makefile.

otelc vs otelhttp vs OBI — when to choose which?#

otelc wins when you can rebuild and want zero-code coverage of stdlib and dependencies. OBI (eBPF) wins when you cannot rebuild or you run a polyglot fleet. Manual SDK with otelhttp wins when you need full control over domain spans and are willing to maintain the wrapper code. The comparison table above covers the trade-offs in detail.

References#

OpenTelemetry Go Compile-Time with otelc
https://corentings.dev/blog/go-otel-compile-time-v1/
Author
Corentin Giaufer Saubert
Published at
2026-07-18
License
CC BY-NC-SA 4.0
Share this post