Performance & load testing — criterion, divan & friends

Overview

Rust's built-in #[bench] never stabilized, so benchmarking lives in the ecosystem — and matured there. criterion is the statistical standard: warmup, outlier detection, regression comparison against saved baselines, HTML reports. divan is the ergonomic newcomer for quick, readable suites; iai-callgrind measures instruction counts for noise-free CI comparisons. Load testing above the process is goose (Rust's Locust) or the polyglot standards (k6). The cross-stack contrast: this is JMH's role without a JIT to outwit — no warmup-to-compile phase, but the same discipline against the optimizer.

Key points

• criterion: cargo bench with [[bench]] harness = false targets; measures wall-time distributions, flags statistically significant changes vs --save-baseline runs — the answer to "did this PR make it slower?".
• Defeat the optimizer or measure nothing: std::hint::black_box around inputs and outputs — dead-code elimination happily deletes an unused computation and reports femtoseconds; same trap JMH's blackholes guard against.
• divan: #[divan::bench] attributes, generic/const-parameterized benches, allocation counting — much less ceremony than criterion; good default for library-internal suites.
• iai-callgrind counts instructions/cache events under Valgrind: deterministic, so CI can fail on a 2% regression without a quiet machine — the complement to criterion's wall time (which needs stable hardware to compare honestly).
• Benchmarks live in benches/, get the library like integration tests do, and should benchmark the release profile semantics (cargo bench does) — debug-build numbers are fiction.
• Explaining beats measuring: when a bench regresses, the whys live in profiling & debugging — flamegraphs, samply, dhat; criterion's --profile-time hooks profilers directly.
• Load testing: goose scripts scenarios in Rust (users, transactions, ramp plans) and scales well per node; k6/vegeta/hey remain fine polyglot choices — the target service being Rust doesn't oblige the load generator to be.
• CI reality: wall-time benches on shared runners are noise machines — either iai-callgrind, dedicated hardware (bencher/codspeed services), or trend-only reporting; a red build on ±5% wall time is a flake generator.

Examples

// benches/parse.rs — criterion
use criterion::{criterion_group, criterion_main, Criterion};
use std::hint::black_box;

fn bench_parse(c: &mut Criterion) {
    c.bench_function("parse_duration 1h30m", |b| {
        b.iter(|| mycrate::parse_duration(black_box("1h30m")))
    });
}
criterion_group!(benches, bench_parse);
criterion_main!(benches);

cargo bench -- --save-baseline main     # before
cargo bench -- --baseline main          # after: significant changes flagged

Related

• Testing in Rust — strategies & tooling map — parent map.
• Profiling & debugging — explaining what these benchmarks measure.
• Cargo in depth — profiles; why bench builds are release-shaped.
• Performance & load testing in Java — JMH and the JIT-warmup problem Rust doesn't have.
• Performance & load testing in Go — testing.B/benchstat: the built-in counterpart.
• Performance engineering — the system-level discipline above the microbenchmark.