Initial Commit

openspec/changes/enterprise-testing-mutation/design.md

@@ -0,0 +1,54 @@
+## Context
+
+`@suaveplan/testing` already enforces a 100% Istanbul line/branch/function/statement gate (`openspec/specs/testing/spec.md` "Coverage and docs gates" — updated from the legacy 95% in the 2026-04-25 audit) and a 10% mitata benchmark regression gate. Neither detects assertion-quality gaps: a test that exercises a branch without asserting on its observable effect is invisible to coverage tooling but kills no mutants. The 2026-04-25 audit (§2 row "No mutation testing") flags this as a high-severity gap and recommends Stryker as the remediation, citing crypto, error, validation, and serialize as the obvious foundation targets.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Add a normative mutation-score requirement layered on top of 100% line coverage. Both gates run; both gates must pass.
+- Tier the score thresholds by package risk: foundation packages (substrate of the monorepo) at ≥ 85%; all others at ≥ 70%.
+- Keep CI wall-clock predictable: scoped PR runs, full nightly runs.
+- Wire surviving-mutant triage into the change-landing workflow so silent gaps are not left for "later".
+
+**Non-Goals:**
+
+- Replacing or relaxing the 100% line-coverage gate.
+- Mutation-testing infrastructure files (barrels, schemas, benchmarks, stories) where there is no behavioural assertion to score.
+- Cross-runner mutation testing — Stryker's `vitest-runner` is the only supported configuration on genesis. Bun-test mutation runs are out of scope until the upstream runner exists.
+
+## Decisions
+
+### D1: Tooling
+
+`@stryker-mutator/core` + `@stryker-mutator/vitest-runner` from `catalogs.testing`. Per-package `stryker.conf.json` at the package root. The vitest runner reuses the package's existing `vitest.config.ts` so test environment, polyfills, and pool settings carry through unchanged.
+
+### D2: Layering, not replacement
+
+Mutation score is a separate gate from line coverage. The pre-publish sequence becomes: `lint → typecheck → test → test:coverage (100%) → test:mutation (≥ threshold)`. None of these substitutes for any other.
+
+### D3: Tiered thresholds
+
+Foundation packages (substrate everything else builds on) at ≥ 85%. All other packages at ≥ 70%. Foundation set is enumerated explicitly in the spec to prevent threshold-shopping. Tightening either threshold over time is a separate spec change; loosening is forbidden by the 100% audit-rule precedent.
+
+### D4: Exclusions
+
+Barrel `index.ts`, `*.schemas.ts`, `*.bench.ts`, `__benchmarks__/`, and `*.stories.tsx` carry no behaviour to mutate. They are excluded via Stryker's `mutate` glob — not via test-skip — so coverage of their existence remains visible.
+
+### D5: CI cadence
+
+Nightly: full per-package mutation run; HTML + JSON reports archived to artefact storage; surviving mutants enumerated in the run summary. PR: scoped via `--mutate` to changed files only; threshold check applies to the scoped score so a PR that touches only well-covered code is not blocked by historical drift elsewhere.
+
+### D6: Surviving-mutant triage
+
+Three terminal states per surviving mutant: (a) killed by a new assertion in the same change; (b) marked as a Stryker-equivalent mutant with a code-comment justification (`// stryker-disable-next-line <mutator>: <reason>`); (c) explicitly deferred via an issue link. No silent surviving mutants land.
+
+### D7: Custom-tooling self-coverage
+
+If Suaveplan ships a custom Stryker mutator or runner adapter (e.g. for the bun-test runner once it exists), the adapter itself MUST meet 100% Istanbul coverage. Mutation tooling does not exempt itself from the project's own coverage rules.
+
+## Risks / Trade-offs
+
+- **Wall-clock cost.** Full runs are expensive on large packages. Mitigated by scoped PR runs and nightly full runs.
+- **Equivalent-mutant noise.** Some surviving mutants are semantically equivalent to the original (e.g. `i++` vs `++i` in an isolated context). Mitigated by D6's explicit triage state with comment-level justification and reviewer sign-off.
+- **Threshold drift.** Tiered thresholds invite "downgrade this package out of foundation". Mitigated by enumerating the foundation set explicitly in the spec and by the audit-rule precedent that thresholds tighten over time, never loosen.

openspec/changes/enterprise-testing-mutation/proposal.md

@@ -0,0 +1,32 @@
+## Why
+
+The 2026-04-25 test-suite audit (`/mnt/s/web/genesis/docs/superpowers/specs/2026-04-25-test-suite-audit.md` §2) identifies mutation testing as a critical missing tier of the genesis test pyramid. The canonical `openspec/specs/testing/spec.md` enforces 100% Istanbul line/branch/function/statement coverage and a mitata 10% benchmark regression gate, but neither catches a whole class of defects: tests that execute a code path without actually asserting its behaviour. A mutated implementation that still passes the test suite is a silent gap in the contract.
+
+CLAUDE.md (`/mnt/s/web/genesis/CLAUDE.md` §Critical and §Quality) mandates enterprise-grade quality on every package and explicitly forbids skipping or disabling tests. Foundation packages (`crypto-*`, `error`, `validation`, `serialize`, `datetime`, `timing`, `retry`, `pagination`, `env`, `config`, `mime`, `templates`, `i18n`, `idempotency`, `slsa`, `diff`) are the substrate for every downstream consumer; a surviving mutant in one of those packages is a latent defect propagated everywhere. Layering a mutation-score gate on top of the existing 100% line-coverage gate closes the assertion-quality hole without diluting either signal.
+
+## What Changes
+
+This change adds **mutation testing** as a quality gate on top of (not in place of) 100% Istanbul line coverage. 100% line coverage remains a precondition; mutation score is an additional, independent gate. Neither substitutes for the other.
+
+### New requirements
+
+- Stryker (`@stryker-mutator/core` + `@stryker-mutator/vitest-runner`) configured per-package via `stryker.conf.json` co-located at the package root, with external deps routed through the catalog (no bare versions per `openspec/AGENTS.md` §1).
+- **Mutation score gate** — foundation packages SHALL achieve ≥ 85% mutants killed; all other `@suaveplan/*` packages SHALL achieve ≥ 70% mutants killed. Surviving mutants MUST be triaged (killed, equivalent-marked with justification, or covered by a follow-up issue) before the change lands.
+- **Excluded paths** — barrel `index.ts`, `*.schemas.ts`, `*.bench.ts`, `__benchmarks__/`, and `*.stories.tsx` are excluded from mutation analysis (no behavioural assertions to mutate).
+- **CI integration** — nightly per-package mutation runs publish HTML + JSON reports as artefacts; on-PR runs use Stryker's `--mutate` filter to score only changed files, keeping wall-clock under budget.
+- **Reporter contract** — surviving mutants are enumerated in a structured PR comment (path, line, mutator, status) so reviewers can act without opening artefact links.
+- **Error-code prefix** — any custom Stryker mutator or runner adapter Suaveplan ships throws `TESTING.MUTATION.*` codes via the local `TestingError` class, matching the existing `TESTING.<DOMAIN>.<REASON>` convention from `openspec/specs/testing/spec.md`.
+- **Self-coverage** — any custom Suaveplan mutator or adapter SHALL itself meet 100% Istanbul coverage; mutation tooling does not exempt itself.
+
+## Capabilities
+
+### Modified Capabilities
+
+- `testing` — adds normative requirements for the mutation-score gate, excluded paths, CI cadence, reporter contract, and `TESTING.MUTATION.*` error codes. Does not modify or relax the existing 100% line-coverage requirement or the mitata benchmark gate.
+
+## Impact
+
+- **Affected code:** every `@suaveplan/*` package gains a `stryker.conf.json` and a CI hook. No source code changes mandated; surviving mutants drive follow-up fixes per package.
+- **Affected catalogs:** root `package.json` gains `@stryker-mutator/core` and `@stryker-mutator/vitest-runner` under `catalogs.testing`.
+- **Pre-publish gate:** the existing pre-publish gate is unchanged. Mutation score is enforced by a separate `bunx turbo test:mutation --filter=@suaveplan/<pkg>` step that MUST also pass before publish.
+- **Wall-clock budget:** PR runs scoped to changed files keep mutation analysis within the existing PR CI envelope; full runs move to nightly to absorb the higher cost.

openspec/changes/enterprise-testing-mutation/specs/testing/spec.md

@@ -0,0 +1,104 @@
+## ADDED Requirements
+
+### Requirement: Mutation testing layered on 100% line coverage
+
+Every `@suaveplan/*` package SHALL run a Stryker mutation analysis as a quality gate layered on top of the existing 100% Istanbul statement / branch / function / line coverage requirement. 100% line coverage is a precondition for mutation testing; mutation score is an additional, independent gate. Neither requirement substitutes for the other. A package SHALL NOT be published while either gate is red.
+
+#### Scenario: Mutation gate runs only after coverage gate passes
+
+- **WHEN** the pre-publish task graph executes for any `@suaveplan/<pkg>`
+- **THEN** `test:mutation` MUST run only after `test:coverage` has reported 100% statement / branch / function / line coverage AND the publish step MUST be blocked unless both gates are green
+
+#### Scenario: Coverage at 100% does not exempt mutation gate
+
+- **WHEN** a package reports 100% Istanbul coverage but its mutation score is below its tier threshold
+- **THEN** the publish step MUST fail AND the failure message MUST cite the mutation-score tier threshold and the actual score
+
+### Requirement: Stryker tooling and configuration
+
+The mutation-testing tier SHALL be implemented with `@stryker-mutator/core` and `@stryker-mutator/vitest-runner`, both referenced via `catalog:testing` in package `devDependencies` (no bare version strings). Each package SHALL co-locate a `stryker.conf.json` (or `stryker.conf.cjs` extending a shared base config) at the package root that reuses the package's existing `vitest.config.ts` for the test environment, polyfills, and pool settings.
+
+#### Scenario: Per-package config present
+
+- **WHEN** the CI runs `bunx turbo test:mutation --filter=@suaveplan/<pkg>`
+- **THEN** Stryker MUST resolve a `stryker.conf.json` (or `stryker.conf.cjs`) at the package root AND the resolved config MUST set `testRunner: "vitest"` AND MUST point `vitest.configFile` at the package's existing `vitest.config.ts`
+
+#### Scenario: Catalog-pinned tooling
+
+- **WHEN** a package's `devDependencies` are inspected
+- **THEN** `@stryker-mutator/core` and `@stryker-mutator/vitest-runner` MUST resolve to `catalog:testing` (or another catalog key) AND MUST NOT contain a bare version string
+
+### Requirement: Tiered mutation-score thresholds
+
+Mutation scores SHALL be enforced at two tiers. Foundation packages — `crypto`, `crypto-utils`, `crypto-hash`, `crypto-curve`, `crypto-cipher`, `crypto-keys`, `crypto-vault`, `error`, `validation`, `serialize`, `datetime`, `timing`, `retry`, `pagination`, `env`, `config`, `mime`, `templates`, `i18n`, `idempotency`, `slsa`, `diff` — SHALL achieve a mutation score of at least 85% killed. Every other `@suaveplan/*` package SHALL achieve at least 70% killed. Threshold loosening SHALL NOT occur on a per-package basis; tightening over time is permitted via a follow-up spec change.
+
+#### Scenario: Foundation package below 85% fails
+
+- **WHEN** `@suaveplan/crypto` reports a mutation score of 80% killed
+- **THEN** the CI mutation step MUST fail the build AND the publish step MUST be blocked
+
+#### Scenario: Non-foundation package below 70% fails
+
+- **WHEN** `@suaveplan/sitemap` reports a mutation score of 65% killed
+- **THEN** the CI mutation step MUST fail the build AND the publish step MUST be blocked
+
+#### Scenario: Foundation package at 85% or above passes
+
+- **WHEN** `@suaveplan/error` reports a mutation score of 86% killed
+- **THEN** the CI mutation step MUST pass
+
+### Requirement: Mutation-analysis exclusions
+
+The Stryker `mutate` glob SHALL exclude paths that carry no behavioural assertions to mutate: barrel `index.ts` files, `*.schemas.ts` files, `*.bench.ts` files, every file under `__benchmarks__/` directories, and `*.stories.tsx` files. Exclusions SHALL be expressed via Stryker's `mutate` glob (not via test-skip), so the coverage of those files via the existing 100% line gate remains visible.
+
+#### Scenario: Schemas file is not mutated
+
+- **WHEN** Stryker runs against `universal/<domain>/<pkg>` and the package contains `src/foo/foo.schemas.ts`
+- **THEN** that file MUST NOT appear in the Stryker `files` set for mutation AND MUST still appear in the Istanbul coverage report at 100%
+
+#### Scenario: Benchmark file is not mutated
+
+- **WHEN** Stryker runs and the package contains `src/__benchmarks__/foo.bench.ts`
+- **THEN** that file MUST NOT be mutated
+
+### Requirement: CI integration — nightly and per-PR runs
+
+A nightly CI job SHALL run the full mutation analysis for every `@suaveplan/*` package and publish HTML + JSON reports to artefact storage. Per-PR CI runs SHALL invoke Stryker with the `--mutate` filter scoped to files changed against the PR base branch, so the wall-clock cost stays within the PR CI envelope. The PR-scoped score SHALL be enforced against the same tier threshold as the full run; a PR that introduces no mutated files passes the gate trivially.
+
+#### Scenario: Nightly run publishes artefacts
+
+- **WHEN** the nightly mutation job completes for `@suaveplan/<pkg>`
+- **THEN** an HTML report and a JSON report MUST be archived as CI artefacts AND the JSON report MUST be machine-readable for downstream tooling
+
+#### Scenario: PR-scoped run uses --mutate filter
+
+- **WHEN** a PR modifies `packages/crypto/crypto/src/aead/aead.ts` and `packages/crypto/crypto/src/sign/sign.ts`
+- **THEN** the PR mutation job MUST invoke Stryker with `--mutate` scoped to those two files AND MUST evaluate the resulting score against the package's tier threshold
+
+### Requirement: Surviving-mutant reporting and triage
+
+Every CI mutation run SHALL emit a structured PR comment enumerating surviving mutants with `path`, `line`, `mutator`, and `status` fields. Surviving mutants in a landing change MUST resolve to one of three terminal states before the change merges: (a) killed by a new assertion added in the same change; (b) explicitly marked equivalent via a `stryker-disable-next-line <mutator>: <reason>` comment in source with reviewer sign-off; (c) deferred via a tracked issue linked from the PR description. No silent surviving mutants SHALL land.
+
+#### Scenario: PR comment lists surviving mutants
+
+- **WHEN** a PR mutation run completes with 3 surviving mutants
+- **THEN** a PR comment MUST be posted listing all 3 with `path`, `line`, `mutator`, and `status` columns
+
+#### Scenario: Surviving mutant blocks merge without triage
+
+- **WHEN** a PR mutation run reports 1 surviving mutant AND the PR description does not link a tracked issue AND the source contains no `stryker-disable-next-line` for that location AND the change adds no assertion that kills the mutant
+- **THEN** the merge gate MUST block the PR
+
+### Requirement: Custom mutator and adapter conformance
+
+Any custom Stryker mutator or runner adapter Suaveplan ships (for example, a future bun-test runner adapter) SHALL throw `TestingError` subclasses with stable codes prefixed `TESTING.MUTATION.*` (e.g. `TESTING.MUTATION.RUNNER_FAILED`, `TESTING.MUTATION.CONFIG_INVALID`). The adapter package itself SHALL meet 100% Istanbul coverage AND SHALL meet the same mutation-score tier threshold it advertises to consumers.
+
+#### Scenario: Adapter throws coded error on misconfiguration
+
+- **WHEN** a custom Suaveplan Stryker adapter receives a config missing a required field
+- **THEN** it MUST throw `TestingError` with `code === "TESTING.MUTATION.CONFIG_INVALID"` AND `context` MUST include the missing field name
+
+#### Scenario: Adapter self-coverage
+
+- **WHEN** the adapter package's `test:coverage` task runs
+- **THEN** statement, branch, function, and line coverage MUST each be 100% AND the adapter's own `test:mutation` task MUST meet at least the 70% non-foundation threshold (or 85% if the adapter is itself foundation)

openspec/changes/enterprise-testing-mutation/tasks.md

@@ -0,0 +1,51 @@
+
+
+## 0. Pre-flight (non-negotiable)
+
+- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
+- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
+- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
+
+## 1. Catalog and tooling setup
+
+- [ ] 1.1 Add `@stryker-mutator/core` and `@stryker-mutator/vitest-runner` to root `package.json` `catalogs.testing` (no bare versions per `openspec/AGENTS.md` §1)
+- [ ] 1.2 Run `bun install`; confirm lockfile updates cleanly
+- [ ] 1.3 Add `test:mutation` task to root `turbo.json` with `dependsOn: ["test:coverage"]` so 100% line coverage is a precondition
+- [ ] 1.4 Document the foundation-package set in `openspec/specs/testing/spec.md` once this change is folded; the same enumeration appears in the spec delta in this folder
+
+## 2. Per-package Stryker configuration
+
+- [ ] 2.1 Author a reusable `stryker.base.conf.cjs` template under `tooling/stryker/` (path TBD) that every package extends — testRunner: `vitest`, reporters: `html, json, clear-text`, mutate exclusions per design D4
+- [ ] 2.2 For every `@suaveplan/*` package, create `stryker.conf.json` at the package root extending the base template, declaring `mutate: ["src/**/*.ts"]` and the package-specific exclusions
+- [ ] 2.3 For every package, add a `test:mutation` script in `package.json` invoking Stryker against the package's `vitest.config.ts`
+
+## 3. Threshold enforcement
+
+- [ ] 3.1 Encode the foundation-package list in the base config via a tiered `thresholds` block (≥ 85% for foundation, ≥ 70% otherwise)
+- [ ] 3.2 Wire a CI step that fails the build on `mutationScore` below the package's tier threshold
+- [ ] 3.3 Wire the PR-scoped mutation run via `--mutate` filtered to files changed in the PR (resolved through `git diff --name-only origin/main...HEAD`)
+
+## 4. Reporter and artefacts
+
+- [ ] 4.1 Configure HTML + JSON reporters; archive both as CI artefacts per nightly run
+- [ ] 4.2 Author a `scripts/report-mutants.ts` that reads the JSON report and emits a Markdown table (path, line, mutator, status) for the PR comment
+- [ ] 4.3 Wire the PR-comment step to post the table via `gh pr comment` (or workflow equivalent)
+
+## 5. Custom-mutator hook (extension point)
+
+- [ ] 5.1 If a custom Stryker mutator or runner adapter is required (e.g. once a bun-test runner exists), package it as `@suaveplan/testing-mutation` under `packages/testing/`
+- [ ] 5.2 Errors thrown by the adapter MUST extend `TestingError` with codes prefixed `TESTING.MUTATION.*` (e.g. `TESTING.MUTATION.RUNNER_FAILED`, `TESTING.MUTATION.CONFIG_INVALID`)
+- [ ] 5.3 The adapter package itself MUST meet 100% Istanbul coverage and pass the same mutation-score tier it advertises
+
+## 6. Spec delta and validation
+
+- [ ] 6.1 `openspec validate enterprise-testing-mutation --type change` passes
+- [ ] 6.2 Fold `specs/testing/spec.md` ADDED Requirements into `openspec/specs/testing/spec.md`
+- [ ] 6.3 Move this change folder to `openspec/archive/<yyyy-mm-dd>-enterprise-testing-mutation/`
+
+## 7. Rollout
+
+- [ ] 7.1 Run baseline mutation analysis across every existing genesis package; record current scores
+- [ ] 7.2 For each package below its tier threshold, file a triage issue listing surviving mutants by mutator class
+- [ ] 7.3 Land assertion-strengthening fixes per package until every existing package meets its tier threshold
+- [ ] 7.4 Flip the CI gate from advisory to enforcing once every package is green