Mutation Testing False Kills: Baselines, File Access, Kill Reasons

Published 2026-10-19

Mutation testing changes your code on purpose (a < becomes <=, a return x becomes return None) and checks whether your tests notice. A "killed" mutant means a test failed. But a test can fail for reasons that have nothing to do with the mutation — and then your mutation score looks better than your tests really are. This guide shows three checks that catch these false kills.

Why false kills happen

Cause	What you see
The suite was already failing or flaky before mutation	Many kills with the same unrelated error
Tests import the original file, not the mutated copy	Kills only from timeouts or environment errors
Timeouts from slow machines	Kills clustered on long-running tests
Shared state between runs (temp files, caches, ports)	Kill result changes when you rerun the same mutant

Timeouts matter for the score: in Stryker, detected = killed + timeout, and mutation score = detected / valid × 100. A machine that times out a lot inflates the score. Source: Stryker mutant states and metrics, checked 2026-10-19.

Check 1 — Baseline on a pristine copy

Copy the project to a fresh directory, exactly as the mutation tool will (same sandbox, same env vars).
Run the full test suite there with no mutation, three times.
It must pass all three times. Any failure here is a pre-existing failure that will show up as fake kills later — fix or quarantine it first.

cp -r project /tmp/mt-baseline && cd /tmp/mt-baseline
for i in 1 2 3; do pytest -q -p no:cacheprovider || echo "BASELINE FAIL $i"; done

Check 2 — Prove the tests read the mutated file

If a test never loads the mutated file, it cannot have killed that mutant. Record file access during each mutant run. In Python, an audit hook is enough:

# conftest.py
import sys, os
TARGET = os.environ.get("MUT_TARGET")  # path of the mutated file
def hook(event, args):
    if event == "open" and TARGET and str(args[0]).endswith(TARGET):
        open("/tmp/mut_touched", "a").write("1")
sys.addaudithook(hook)

Run each mutant with stale bytecode removed (delete __pycache__ in the copy) so the source file is actually read. Rule: a kill where the target file was never opened is marked suspect and rerun. On other stacks, strace -e trace=openat (Linux) or the test runner's coverage data serve the same purpose.

Check 3 — Look at the kill-reason distribution

For every killed mutant, store why the run failed: assertion error, other exception, timeout, crash, collection/import error.
Make a simple count table.

Reason	Healthy pattern	Red flag
Assertion failure	Most kills	—
Other exception	Some	One identical message on many mutants
Timeout	Few (infinite-loop mutants)	Large share; rises on slower machines
Import / collection error	Almost none	Any meaningful share — the harness, not the tests, is failing

Putting it together

Baseline green 3/3 on a pristine copy
Every kill has a "target file opened" record
Assertion failures are the largest kill reason
Rerun a random 5% of kills; results should not change

FAQ

Should I raise the timeout?: Set it from the baseline: measure the normal suite time, then allow a multiple of it. Too low creates fake kills; too high slows the run.
Do compile errors count?: In Stryker, compile and runtime errors are "invalid" mutants and are not part of the score.

Sources

Stryker: mutant states and metrics — checked 2026-10-19

Related (Korean): 개발 지식 모음 (한국어)

📤 Share