Initial Commit

openspec/changes/enterprise-testing-mutation/specs/testing/spec.md

@@ -0,0 +1,104 @@
+## ADDED Requirements
+
+### Requirement: Mutation testing layered on 100% line coverage
+
+Every `@suaveplan/*` package SHALL run a Stryker mutation analysis as a quality gate layered on top of the existing 100% Istanbul statement / branch / function / line coverage requirement. 100% line coverage is a precondition for mutation testing; mutation score is an additional, independent gate. Neither requirement substitutes for the other. A package SHALL NOT be published while either gate is red.
+
+#### Scenario: Mutation gate runs only after coverage gate passes
+
+- **WHEN** the pre-publish task graph executes for any `@suaveplan/<pkg>`
+- **THEN** `test:mutation` MUST run only after `test:coverage` has reported 100% statement / branch / function / line coverage AND the publish step MUST be blocked unless both gates are green
+
+#### Scenario: Coverage at 100% does not exempt mutation gate
+
+- **WHEN** a package reports 100% Istanbul coverage but its mutation score is below its tier threshold
+- **THEN** the publish step MUST fail AND the failure message MUST cite the mutation-score tier threshold and the actual score
+
+### Requirement: Stryker tooling and configuration
+
+The mutation-testing tier SHALL be implemented with `@stryker-mutator/core` and `@stryker-mutator/vitest-runner`, both referenced via `catalog:testing` in package `devDependencies` (no bare version strings). Each package SHALL co-locate a `stryker.conf.json` (or `stryker.conf.cjs` extending a shared base config) at the package root that reuses the package's existing `vitest.config.ts` for the test environment, polyfills, and pool settings.
+
+#### Scenario: Per-package config present
+
+- **WHEN** the CI runs `bunx turbo test:mutation --filter=@suaveplan/<pkg>`
+- **THEN** Stryker MUST resolve a `stryker.conf.json` (or `stryker.conf.cjs`) at the package root AND the resolved config MUST set `testRunner: "vitest"` AND MUST point `vitest.configFile` at the package's existing `vitest.config.ts`
+
+#### Scenario: Catalog-pinned tooling
+
+- **WHEN** a package's `devDependencies` are inspected
+- **THEN** `@stryker-mutator/core` and `@stryker-mutator/vitest-runner` MUST resolve to `catalog:testing` (or another catalog key) AND MUST NOT contain a bare version string
+
+### Requirement: Tiered mutation-score thresholds
+
+Mutation scores SHALL be enforced at two tiers. Foundation packages — `crypto`, `crypto-utils`, `crypto-hash`, `crypto-curve`, `crypto-cipher`, `crypto-keys`, `crypto-vault`, `error`, `validation`, `serialize`, `datetime`, `timing`, `retry`, `pagination`, `env`, `config`, `mime`, `templates`, `i18n`, `idempotency`, `slsa`, `diff` — SHALL achieve a mutation score of at least 85% killed. Every other `@suaveplan/*` package SHALL achieve at least 70% killed. Threshold loosening SHALL NOT occur on a per-package basis; tightening over time is permitted via a follow-up spec change.
+
+#### Scenario: Foundation package below 85% fails
+
+- **WHEN** `@suaveplan/crypto` reports a mutation score of 80% killed
+- **THEN** the CI mutation step MUST fail the build AND the publish step MUST be blocked
+
+#### Scenario: Non-foundation package below 70% fails
+
+- **WHEN** `@suaveplan/sitemap` reports a mutation score of 65% killed
+- **THEN** the CI mutation step MUST fail the build AND the publish step MUST be blocked
+
+#### Scenario: Foundation package at 85% or above passes
+
+- **WHEN** `@suaveplan/error` reports a mutation score of 86% killed
+- **THEN** the CI mutation step MUST pass
+
+### Requirement: Mutation-analysis exclusions
+
+The Stryker `mutate` glob SHALL exclude paths that carry no behavioural assertions to mutate: barrel `index.ts` files, `*.schemas.ts` files, `*.bench.ts` files, every file under `__benchmarks__/` directories, and `*.stories.tsx` files. Exclusions SHALL be expressed via Stryker's `mutate` glob (not via test-skip), so the coverage of those files via the existing 100% line gate remains visible.
+
+#### Scenario: Schemas file is not mutated
+
+- **WHEN** Stryker runs against `universal/<domain>/<pkg>` and the package contains `src/foo/foo.schemas.ts`
+- **THEN** that file MUST NOT appear in the Stryker `files` set for mutation AND MUST still appear in the Istanbul coverage report at 100%
+
+#### Scenario: Benchmark file is not mutated
+
+- **WHEN** Stryker runs and the package contains `src/__benchmarks__/foo.bench.ts`
+- **THEN** that file MUST NOT be mutated
+
+### Requirement: CI integration — nightly and per-PR runs
+
+A nightly CI job SHALL run the full mutation analysis for every `@suaveplan/*` package and publish HTML + JSON reports to artefact storage. Per-PR CI runs SHALL invoke Stryker with the `--mutate` filter scoped to files changed against the PR base branch, so the wall-clock cost stays within the PR CI envelope. The PR-scoped score SHALL be enforced against the same tier threshold as the full run; a PR that introduces no mutated files passes the gate trivially.
+
+#### Scenario: Nightly run publishes artefacts
+
+- **WHEN** the nightly mutation job completes for `@suaveplan/<pkg>`
+- **THEN** an HTML report and a JSON report MUST be archived as CI artefacts AND the JSON report MUST be machine-readable for downstream tooling
+
+#### Scenario: PR-scoped run uses --mutate filter
+
+- **WHEN** a PR modifies `packages/crypto/crypto/src/aead/aead.ts` and `packages/crypto/crypto/src/sign/sign.ts`
+- **THEN** the PR mutation job MUST invoke Stryker with `--mutate` scoped to those two files AND MUST evaluate the resulting score against the package's tier threshold
+
+### Requirement: Surviving-mutant reporting and triage
+
+Every CI mutation run SHALL emit a structured PR comment enumerating surviving mutants with `path`, `line`, `mutator`, and `status` fields. Surviving mutants in a landing change MUST resolve to one of three terminal states before the change merges: (a) killed by a new assertion added in the same change; (b) explicitly marked equivalent via a `stryker-disable-next-line <mutator>: <reason>` comment in source with reviewer sign-off; (c) deferred via a tracked issue linked from the PR description. No silent surviving mutants SHALL land.
+
+#### Scenario: PR comment lists surviving mutants
+
+- **WHEN** a PR mutation run completes with 3 surviving mutants
+- **THEN** a PR comment MUST be posted listing all 3 with `path`, `line`, `mutator`, and `status` columns
+
+#### Scenario: Surviving mutant blocks merge without triage
+
+- **WHEN** a PR mutation run reports 1 surviving mutant AND the PR description does not link a tracked issue AND the source contains no `stryker-disable-next-line` for that location AND the change adds no assertion that kills the mutant
+- **THEN** the merge gate MUST block the PR
+
+### Requirement: Custom mutator and adapter conformance
+
+Any custom Stryker mutator or runner adapter Suaveplan ships (for example, a future bun-test runner adapter) SHALL throw `TestingError` subclasses with stable codes prefixed `TESTING.MUTATION.*` (e.g. `TESTING.MUTATION.RUNNER_FAILED`, `TESTING.MUTATION.CONFIG_INVALID`). The adapter package itself SHALL meet 100% Istanbul coverage AND SHALL meet the same mutation-score tier threshold it advertises to consumers.
+
+#### Scenario: Adapter throws coded error on misconfiguration
+
+- **WHEN** a custom Suaveplan Stryker adapter receives a config missing a required field
+- **THEN** it MUST throw `TestingError` with `code === "TESTING.MUTATION.CONFIG_INVALID"` AND `context` MUST include the missing field name
+
+#### Scenario: Adapter self-coverage
+
+- **WHEN** the adapter package's `test:coverage` task runs
+- **THEN** statement, branch, function, and line coverage MUST each be 100% AND the adapter's own `test:mutation` task MUST meet at least the 70% non-foundation threshold (or 85% if the adapter is itself foundation)