Test-suite scheduling & job partitioning

Overview

Long test suites get split. The split is usually done by intuition — "this job takes twenty minutes, let's run it as four jobs" — and often buys much less than expected, sometimes nothing at all. The reasons are mechanical: parallel runners have a scheduling granularity that is coarser than a single test, expensive setup is frequently duplicated by exactly the split meant to help, and total wall clock is set by the slowest indivisible unit rather than by the total amount of work.

This note is about deciding whether and how to divide, and about the fact that both questions want measurements rather than instinct.

Key points

• Find your runner's unit of scheduling — it varies, and everything below depends on it. Vitest and Jest parallelise across files and run one file's tests sequentially; go test parallelises across packages (and within one only where t.Parallel() says so); JUnit 5 parallelises by class or method once configured; pytest-xdist is configurable and can distribute individual tests. Whatever that unit is, it is the atom: no amount of concurrency splits it, and the slowest single unit sets a floor under the whole job.
• The floor is the number to look at first. Adding workers to a job whose longest unit is most of its wall clock changes almost nothing. Measure the longest unit and the total before deciding anything — if they are close, the problem is that unit, not the parallelism.
• Where the unit is coarser than a test, a long case deserves its own unit — precisely because the unit is what gets scheduled. With a file-level runner, appending one more slow case to an already-slow file makes the floor worse in a way no scheduling change can recover. With a test-level distributor this pressure largely disappears, which is itself a reason to know which you have.
• Where a job is slow because each unit does real work — builds, containers, databases — setup is usually the bigger half, and splitting duplicates it. Dependency resolution, toolchain provisioning, image pulls, database seeding are all paid per unit. (For a pure unit-test suite the opposite holds: setup is trivial and the tests are the cost.) Two cases sharing one expensive setup can be dramatically cheaper than the same two cases in separate units, each paying for it cold.
• So fix the setup before changing the layout. Amortize what is genuinely reusable across cases (a dependency cache, a downloaded toolchain, a warmed image) and isolate only what actually carries state between cases (the working tree, the database contents). The instinct to split two slow cases into two units is often exactly backwards — it hands the second one its cold setup back.
• Divide along an axis that means something. If a partition is a real dimension of the thing under test, a failure names the cause. If it is an arbitrary split by count, a failure is a log to go read. That difference is most of the value of partitioning.
• Extra jobs are cheap in wall clock only while your runner concurrency can absorb them, and never free in machine time. Given spare capacity, jobs finishing faster than the pipeline's slowest job cost nothing perceptible. Against a concurrency cap they queue, and the wall clock you were protecting is exactly what you lose — so check the cap before adding cells, because this is the assumption that silently flips the trade-off. Either way every job pays its own setup and its own minutes: spend them where attribution is worth it.
• Never create a cell nothing populates. A partition that runs no tests reports green, and its name then asserts coverage that does not exist — worse than an absence, because it looks like a presence.
• A suite in no partition never runs, and looks exactly like a suite that passed. Once the partitioning is an explicit list, that list and the actual set of suites can drift silently. Check the correspondence mechanically in the fast gate; convention will not hold it.
• Verify the assumptions that motivated the split. Which half is slower, whether a given tool is the expensive one, whether a case is really independent — all cheap to measure, and frequently inverted in practice.

Details

Deciding whether to split

Work down this list; most slow jobs are fixed before the last step.

1. Measure the longest unit against the total. A job whose longest unit is a small fraction of its total is genuinely under-parallelised — add workers or units. A job whose longest unit dominates cannot be helped by either.
2. Find the repeated setup. Anything paid once per unit that could be paid once per job is a candidate for amortization, and amortizing it usually beats splitting.
3. Check the isolation you actually need. Cases often share more safely than assumed — a downloaded artifact cache is shared state carrying no test-visible information, while a database or a working tree is. Conflating the two is what makes setup expensive.
4. Only then split, and split the long unit, not the job.

Deciding how to divide

Two partitions of identical cost can differ enormously in value:

If the suite exists to prove something works across a matrix of combinations, the matrix is the partition — and the job name becomes the diagnosis. If the suite is merely long, an arbitrary split is honest and cheaper to maintain.

Two follow-on rules:

• Keep the body shared. Cells that duplicate their assertions drift into subtly different tests, and the matrix stops meaning what its name says. One parameterised body, many cells.
• Assert per-axis facts once. A property belonging to one dimension of the matrix should be checked in one cell of that dimension, not in every cell of the whole grid. Repeating it multiplies cost and proves the same thing.

The enumeration hazard

Any scheme where jobs name their work explicitly has a silent failure mode: work that no job names. It produces no error, no skipped-test count, and no red — the pipeline simply does less than it claims.

The first answer is to not declare it: derive the enumeration from the filesystem or from the runner's own discovery, so the question cannot arise. Where the partitioning must be declared — a static matrix, a hand-tuned split, per-cell configuration a generated list cannot express — the fallback is a check comparing the declaration against the actual set and failing on disagreement. Treat that check as part of the pipeline rather than a tooling nicety; where the list is hand-maintained it is the one thing standing between a new suite and never running.

Related

• CI/CD & delivery engineering — the wider pipeline discipline; fast feedback first is the principle this serves.
• Testing strategies — deciding what the slow suites should contain in the first place.
• Performance engineering — measure-first and bottleneck reasoning, generalised.
• Developer environments — toolchain provisioning is the setup cost most often duplicated per job.