rgoussu.dev
rgoussu@goussu: ~/library/java/testing
❯ ~/library/java/testing cat mutation-testing.md

Mutation testing — PIT

# PIT mutates production bytecode to measure whether tests actually assert behaviour, giving a mutation score stronger than line coverage.

Conceptsaved 2026-08-09 #java#testing#pit#quality#build

Overview

Mutation testing judges the test suite instead of the code: it seeds small faults (mutants) into production code and reruns the tests — a good suite kills the mutant (some test fails), a weak one lets it survive. PIT (pitest) is the Java standard, operating on bytecode with per-mutant test selection, which makes it practical where earlier academic tools were not. Its mutation score exposes what line coverage hides: code that is executed by tests but never actually asserted.

Key points

  • Mutant: a single small change to the code (invert a conditional, drop a call, change a constant). Killed = at least one test fails; survived = tests pass anyway; also no coverage, and timed out (usually killed infinite loops — counts as killed).
  • Mutation score = killed / total mutants. 100% line coverage with a mediocre mutation score is common — and is precisely the smell the technique exists to catch.
  • PIT is fast by design: it mutates bytecode (no recompilation) and runs only the tests that cover the mutated line, using per-test coverage data.
  • Default mutators are a curated, low-noise set: conditionals boundary (< → <=), negate conditionals, math operator replacement, return-value mutators (empty/null/ primitive returns), void method call removal, increments. STRONGER/ALL groups exist.
  • Equivalent mutants — mutants that don't change observable behaviour — can never be killed; they are the irreducible noise floor, so chase surviving mutants, not 100%.
  • Incremental analysis (withHistory) persists results and re-examines only mutants affected by changed code — the key to keeping PIT in regular use on a real codebase.
  • Surviving mutants are actionable: each one names a line and a fault your suite would ship; either add the missing assertion or delete untestable/dead code.

Details

Build integration

  • Maven: pitest-maven plugin, mvn org.pitest:pitest-maven:mutationCoverage; add pitest-junit5-plugin for JUnit 5 suites. Configure targetClasses/targetTests, mutationThreshold to fail the build below a score.
  • Gradle: the gradle-pitest-plugin (info.solidsoft.pitest); same knobs in the pitest { } block.
  • Reports: HTML per-class views marking each mutant killed/survived, plus XML for CI.
  • CI pattern: nightly full run + threshold; PR runs incremental or scoped to changed modules.

Cost management on big codebases

  • Scope by value: run PIT on domain/money/algorithmic modules; skip generated code, DTOs and glue (excludedClasses, avoidCallsTo for logging).
  • Incremental history across runs; features=+GIT(from[...])-style change scoping or simply running per changed Maven module in PRs.
  • Parallelism: threads for local cores; shard by module in CI.
  • Keep the covering tests fast — PIT reruns them thousands of times; a slow integration test in the covering set dominates the bill (prefer unit-level coverage for mutated code).

Examples

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>${pitest.version}</version>
  <dependencies>
    <dependency>
      <groupId>org.pitest</groupId>
      <artifactId>pitest-junit5-plugin</artifactId>
      <version>${pitest.junit5.version}</version>
    </dependency>
  </dependencies>
  <configuration>
    <targetClasses><param>com.example.pricing.*</param></targetClasses>
    <mutationThreshold>80</mutationThreshold>
    <withHistory>true</withHistory>
  </configuration>
</plugin>

A survived "conditionals boundary" mutant on if (amount > limit) means no test pins the amount == limit case — exactly the off-by-one class of bug example-based suites miss.

Related