rgoussu.dev
rgoussu@goussu: ~/library/go/toolchain
❯ ~/library/go/toolchain cat execution-tracer.md

Execution tracer — go tool trace

# The runtime execution tracer — scheduler-level event recording, go tool trace analysis, flight-recorder capture, and when tracing beats profiling.

Conceptsaved 2026-08-09 #go#tooling#performance#runtime#concurrency

Overview

The execution tracer records what the runtime did, event by event: every goroutine state change, GC phase, syscall, and network block, with nanosecond timestamps. Where pprof answers "where is CPU spent", the trace answers "why did this request take 300 ms while using 2 ms of CPU" — the latency mysteries sampling profilers are blind to.

Key points

  • Capture: runtime/trace.Start(w) / Stop() in code, go test -trace=trace.out, or /debug/pprof/trace?seconds=5 via net/http/pprof; analyse with go tool trace trace.out.
  • What it shows: per-P and per-goroutine timelines — running, runnable (waiting for a CPU), blocked on channel/mutex/network/syscall — plus GC assists, STW pauses, and heap-size evolution.
  • Runnable-but-not-running gaps are the signature find: work ready to go but starved of a P — scheduler contention that no CPU profile can reveal.
  • Overhead collapsed in 1.21: from a "too expensive for production" 10–20% to ~1–2%, making short traces of live services routine.
  • 1.22 rework: the trace format became streamable and partition-based — traces can be consumed while being written and a crash no longer loses the whole recording; the golang.org/x/exp/trace reader parses them programmatically.
  • Flight recorder: keep tracing into a bounded in-memory ring and snapshot only when something interesting happens (SLO breach, timeout) — capturing the moments before the anomaly; introduced experimentally in x/exp/trace, now in runtime/trace.
  • Tasks and regions: trace.NewTask / trace.WithRegion annotate application-level work (one request, one job) so the timeline maps to your domain, not just goroutine IDs.
  • When trace beats pprof: tail latency, goroutines stuck runnable, GC pause impact, syscall/network stalls, lock convoys. When pprof beats trace: pure CPU hotspots and allocation sites — the trace is far denser data than you need for those.

Examples

# Trace 5 seconds of a live service, then explore in the browser
curl -o trace.out "http://localhost:6060/debug/pprof/trace?seconds=5"
go tool trace trace.out

ctx, task := trace.NewTask(ctx, "handleOrder")
defer task.End()
trace.WithRegion(ctx, "chargeCard", func() { charge(ctx, card) })

Related

  • The go command & tool catalog — parent catalog of the toolchain.
  • pprof — the sampling complement: hotspots there, causality here.
  • go tool cover — the third go tool analysis surface, aimed at tests rather than runtime behaviour.
  • Concurrency & parallelism — the concepts (scheduling, contention, starvation) the timeline makes visible.
  • JFR & JMC — the JVM's flight recorder; Go's tracer converged on the same always-on, dump-on-anomaly model.