OpenTelemetry

Overview

OpenTelemetry (OTel) is the CNCF standard that ended the era of vendor-proprietary instrumentation: one API to instrument against, one wire protocol (OTLP) to export over, and any backend — Jaeger, Prometheus/Mimir, Tempo, Datadog, Honeycomb — on the receiving end. Its central design decision is the API / SDK / Collector split: libraries and application code depend only on the no-op-by-default API; the SDK, configured once at the composition root, decides sampling, processing, and export; and the Collector moves pipeline concerns (fan-out, filtering, vendor credentials) out of the process entirely. Instrument once, choose backends forever after.

Key points

• API vs SDK: the API (Tracer, Meter, Logger providers) is a lightweight, dependency-safe facade — a library that instruments against it costs nothing when no SDK is installed. The SDK is the implementation the application wires: resource, samplers, processors, exporters. Never let library code touch SDK classes.
• Collector = receiver → processor → exporter pipelines: receives OTLP (and Prometheus, Zipkin, …), processes (batching, memory limits, filtering, attribute redaction, tail sampling), exports to one or many backends. Run it as sidecar/daemonset agent or a central gateway; it is the fan-out point that keeps vendor choice out of app config.
• Traces: a trace is a tree of spans — each with name, kind (server/client/internal/producer/consumer), attributes, timed events, links to other traces, and a status. Head sampling is configured in the SDK (parentbased_traceidratio is the sane default); tail sampling lives in the Collector.
• Metrics: instruments on a Meter — Counter, UpDownCounter, Histogram, Gauge, plus async/observable variants for pull-style readings. Views rename, re-bucket, or drop attributes at SDK level; temporality (cumulative vs delta) is the exporter-dependent detail that bites when bridging to Prometheus (cumulative) vs some vendors (delta).
• Logs: deliberately not a new logging API — a bridge from existing frameworks (Logback/Log4j appenders, a slog handler, a tracing layer, a pino transport) into the OTel log data model, so logs share Resource and trace context with the other signals.
• Resource + semantic conventions: the Resource describes the emitter (service.name, service.version, deployment.environment) and is attached to all signals; semantic conventions standardize attribute names (http.request.method, http.response.status_code, db.system) so dashboards and alerts port across services and vendors.
• OTLP: one protocol for all three signals — gRPC on 4317, HTTP/protobuf on 4318. Standard env vars configure any SDK without code: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, OTEL_PROPAGATORS, and OTEL_SDK_DISABLED=true as the kill-switch (12-factor telemetry).
• Auto vs manual: auto-instrumentation (Java agent, Node registrations) gives HTTP/ DB/queue spans for free via bytecode or module patching; manual instrumentation adds the domain-meaningful spans and metrics. Frameworks increasingly ship OTel natively (build-time in Quarkus), replacing the generic agent.
• The correlation story: propagators carry traceparent/baggage across services; trace_id/span_id stamped into log MDC joins logs to traces; exemplars attach sample trace-ids to histogram buckets, joining metrics to traces. That triangle is the payoff of adopting the standard end-to-end.

Details

Instrumentation per platform

Go and Rust have no auto-instrumentation worth the name — middleware wrappers at the edges plus manual spans is the idiom, which fits the hexagonal placement anyway: telemetry wiring belongs at the assembly point (parent note), and the domain never sees the SDK.

Minimal SDK wiring shape

Every language's setup reduces to the same four steps, done once in the composition root:

1. Build a Resource (service.name at minimum — anonymous services are unfindable).
2. Create provider(s) — TracerProvider with sampler + BatchSpanProcessor, MeterProvider with a periodic reader — pointing at an OTLP exporter.
3. Register the propagator (W3C tracecontext + baggage is the default).
4. Register globally, and flush/shutdown on exit — the batch processor holds data; skipping shutdown silently drops the tail of every process's telemetry.

Choosing what to emit

Semantic-convention attributes for the edges (HTTP, DB, messaging) come free with instrumentation libraries; the manual layer should add domain telemetry: a span per use case (command/query name as span name), business-meaningful attributes, and a small number of domain metrics (histograms cost cardinality — bound your label sets). One custom metric plus one custom span is enough to prove the pipeline end-to-end in a walking skeleton.

Related

• Observability & SRE practice — the strategic map; OTel is its "now-standard instrumentation layer" made concrete.
• Correlation IDs & context propagation — the propagator half of OTel: traceparent, tracestate, baggage.
• Structured logging & MDC — the log bridge and the trace_id-in-MDC join.
• Health checks: liveness, readiness, startup — sibling operational contract; keep probe traffic out of traces (sampler or filter).
• Time-series stores from Java — Micrometer as the JVM metrics facade OTel meets via the tracing bridge and OTLP registry.
• Time-series stores from Go — Prometheus-native metrics and OTel as the converging facade.
• Time-series stores from Rust — the metrics facade and exposition landscape OTel's Rust SDK sits beside.
• Frameworks × the stdlib contracts — otelhttp ships as func(http.Handler) http.Handler; contract fidelity is what makes it universal.

Citations

[1] OpenTelemetry documentation [2] OTLP specification [3] OTel semantic conventions [4] SDK environment variables