rgoussu.dev
rgoussu@goussu: ~/library/system-design
❯ ~/library/system-design cat performance-engineering.md

Performance engineering

# Measuring before guessing — profiling, benchmarking methodology, mechanical sympathy, and a shared vocabulary from CPU cache to p99.

Conceptsaved 2026-08-08updated 2026-08-20 #performance#profiling#benchmarking#mechanical-sympathy

Overview

Performance engineering is the discipline of making systems fast on evidence rather than folklore: measure first, profile to find where time actually goes, change one thing, and measure again. It spans every layer already in this KB — CPU caches under Rust and Java, the Linux scheduler and page cache, GC pauses, network round trips — and its core skill is knowing which layer the bottleneck lives in. Gregg's Systems Performance and the USE method anchor the systems side; Hyde's "mechanical sympathy" names the hardware side.

Key points

  • Methodology beats tools: define the question (throughput? p99? cost?), form a hypothesis, measure under realistic load, change one variable. USE (utilization, saturation, errors) for resources; latency percentiles — never averages — for user experience; beware coordinated omission in load tests.
  • Profiling: CPU flame graphs (perf, async-profiler, pprof) for where time goes; off-CPU analysis for where time waits; allocation profiling for GC pressure; continuous profiling in production as the modern default.
  • Microbenchmark honestly or not at all: JIT warmup, dead-code elimination, and cache effects invalidate naive loops — use JMH (JVM), go test -bench + benchstat, Criterion (Rust); still distrust results that lack variance.
  • Mechanical sympathy: cache lines and false sharing, branch prediction, NUMA, sequential vs. random access (why LSM-trees and Kafka are fast), SIMD — the "latency numbers every programmer should know" ladder as intuition.
  • The usual suspects, in order: unnecessary work (N+1 queries, chatty APIs, missing cache) → serialization and allocation churn → contention (locks, connection pools) → actual compute. Algorithmic wins dwarf micro-tuning.
  • Little's law: concurrency = throughput × latency — the one formula that sizes pools, queues, and explains why saturated systems' latency explodes.
  • To explore: tail-latency amplification in fan-out systems, GC tuning as a last resort, eBPF-based production tracing, capacity planning.

Practice

  • Flame-graph a real service (source) — profile something you actually run (perf, async-profiler, pprof), read the widest towers, and explain them before changing anything.
  • Microbenchmark pitfalls with JMH (source) — work through the official samples: watch dead-code elimination and missing warmup destroy a naive loop, then measure the same thing honestly.
  • Load-test without lying (source) — put a service under constant-throughput load and compare wrk2's percentiles with a naive request loop's: coordinated omission, demonstrated on your own numbers.
  • One Billion Row Challenge (exercise) — the full loop at scale: baseline, profile, fix the biggest cost, repeat — I/O, parsing, allocation churn, parallelism, and mechanical sympathy each get their turn.

Related