4.1 KiB
Context
@suaveplan/testing already enforces a 100% Istanbul line/branch/function/statement gate (openspec/specs/testing/spec.md "Coverage and docs gates" — updated from the legacy 95% in the 2026-04-25 audit) and a 10% mitata benchmark regression gate. Neither detects assertion-quality gaps: a test that exercises a branch without asserting on its observable effect is invisible to coverage tooling but kills no mutants. The 2026-04-25 audit (§2 row "No mutation testing") flags this as a high-severity gap and recommends Stryker as the remediation, citing crypto, error, validation, and serialize as the obvious foundation targets.
Goals / Non-Goals
Goals:
- Add a normative mutation-score requirement layered on top of 100% line coverage. Both gates run; both gates must pass.
- Tier the score thresholds by package risk: foundation packages (substrate of the monorepo) at ≥ 85%; all others at ≥ 70%.
- Keep CI wall-clock predictable: scoped PR runs, full nightly runs.
- Wire surviving-mutant triage into the change-landing workflow so silent gaps are not left for "later".
Non-Goals:
- Replacing or relaxing the 100% line-coverage gate.
- Mutation-testing infrastructure files (barrels, schemas, benchmarks, stories) where there is no behavioural assertion to score.
- Cross-runner mutation testing — Stryker's
vitest-runneris the only supported configuration on genesis. Bun-test mutation runs are out of scope until the upstream runner exists.
Decisions
D1: Tooling
@stryker-mutator/core + @stryker-mutator/vitest-runner from catalogs.testing. Per-package stryker.conf.json at the package root. The vitest runner reuses the package's existing vitest.config.ts so test environment, polyfills, and pool settings carry through unchanged.
D2: Layering, not replacement
Mutation score is a separate gate from line coverage. The pre-publish sequence becomes: lint → typecheck → test → test:coverage (100%) → test:mutation (≥ threshold). None of these substitutes for any other.
D3: Tiered thresholds
Foundation packages (substrate everything else builds on) at ≥ 85%. All other packages at ≥ 70%. Foundation set is enumerated explicitly in the spec to prevent threshold-shopping. Tightening either threshold over time is a separate spec change; loosening is forbidden by the 100% audit-rule precedent.
D4: Exclusions
Barrel index.ts, *.schemas.ts, *.bench.ts, __benchmarks__/, and *.stories.tsx carry no behaviour to mutate. They are excluded via Stryker's mutate glob — not via test-skip — so coverage of their existence remains visible.
D5: CI cadence
Nightly: full per-package mutation run; HTML + JSON reports archived to artefact storage; surviving mutants enumerated in the run summary. PR: scoped via --mutate to changed files only; threshold check applies to the scoped score so a PR that touches only well-covered code is not blocked by historical drift elsewhere.
D6: Surviving-mutant triage
Three terminal states per surviving mutant: (a) killed by a new assertion in the same change; (b) marked as a Stryker-equivalent mutant with a code-comment justification (// stryker-disable-next-line <mutator>: <reason>); (c) explicitly deferred via an issue link. No silent surviving mutants land.
D7: Custom-tooling self-coverage
If Suaveplan ships a custom Stryker mutator or runner adapter (e.g. for the bun-test runner once it exists), the adapter itself MUST meet 100% Istanbul coverage. Mutation tooling does not exempt itself from the project's own coverage rules.
Risks / Trade-offs
- Wall-clock cost. Full runs are expensive on large packages. Mitigated by scoped PR runs and nightly full runs.
- Equivalent-mutant noise. Some surviving mutants are semantically equivalent to the original (e.g.
i++vs++iin an isolated context). Mitigated by D6's explicit triage state with comment-level justification and reviewer sign-off. - Threshold drift. Tiered thresholds invite "downgrade this package out of foundation". Mitigated by enumerating the foundation set explicitly in the spec and by the audit-rule precedent that thresholds tighten over time, never loosen.