Initial Commit
This commit is contained in:
135
docs/superpowers/specs/2026-04-25-test-suite-audit.md
Normal file
135
docs/superpowers/specs/2026-04-25-test-suite-audit.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Test-Suite Documentation Audit — Genesis Monorepo (2026-04-24 wave)
|
||||
|
||||
**Date:** 2026-04-25
|
||||
**Scope:** review of `openspec/specs/testing*/spec.md` (canonical, 2026-04-19) plus the 2026-04-24 testbed design + plans in `docs/superpowers/`.
|
||||
**Goal:** identify gaps and improvements to elevate the test-suite documentation to enterprise-class quality.
|
||||
|
||||
## Executive summary — top critical gaps
|
||||
|
||||
1. **Spec drift is severe.** The canonical `openspec/specs/testing*/spec.md` files (2026-04-19) say nothing about `testing-integration*`, `console-monitor`, `network-monitor`, `multi-session`, or `extension-fixture`. All recent 2026-04-24 work lives only in `docs/superpowers/plans/` and has not been folded into the contract — there is no normative spec for the integration substrate that the entire testbed and CI nightly job depends on.
|
||||
2. **Test-pyramid is missing several mandatory enterprise tiers.** Zero accessibility (axe-core), zero visual-regression, zero contract testing, zero mutation testing, zero load/perf, zero fuzzing for non-pure code, zero chaos/fault injection beyond the single "disable signalling route" test.
|
||||
3. **Service-fixture coverage is too narrow for the package list.** `oauth-client`, `mcp`, `server-emailer`, `search`, future Kafka/NATS-style event pipes, and any object-store/secret/identity work have no fixtures. The minimum missing set: MongoDB, MySQL/MariaDB, OIDC issuer (Keycloak/dex/Hydra), MinIO (broader-S3 parity vs Garage), an OTel collector, and a NATS/Kafka broker.
|
||||
4. **`apps/testbed` only partially enforces CLAUDE.md's six "Web App & Page Verification" bullets.** Console + network monitors and Docker staging are wired; rendered-DOM content checks per page, mixed-content, asset-404 detection, and the "zero-defect-to-live" gate are not formalised in the plan.
|
||||
5. **Test-suite observability is absent.** No JUnit/Allure output, no OTel export of test execution despite OTel being a first-class concern in `testing/spec.md`, no flake-rate metric, no coverage trend, no trace-on-failure shipped to a sink. CI matrix is also single-arch (no arm64, no Bun version matrix, no Node fallback parity check).
|
||||
|
||||
---
|
||||
|
||||
## 1. Coverage gaps in service fixtures
|
||||
|
||||
| Severity | Gap | Where it belongs | Concrete fix |
|
||||
|---|---|---|---|
|
||||
| Critical | No OIDC issuer fixture, but `@suaveplan/oauth-client` already exists | new plugin `testing-integration-oidc` (Keycloak or `oauth2-mock-server`) under `packages/testing/` | Add to `enterprise-testbed-design.md §4.1` and create plan; required to integration-test `oauth-client` against a real Authorization Code + PKCE flow. |
|
||||
| Critical | No Mongo / MySQL fixture | `enterprise-testbed-design.md §4.1` | Add `testing-integration-mongodb` (mongo:7) and `testing-integration-mysql` (mysql:8) — many real apps use these and there is no parity coverage for `db-*` siblings of `db-sqlite`. |
|
||||
| High | No object-store parity beyond Garage | new `testing-integration-minio` | Garage has quirks (admin-port health, layout assignment) that mask AWS-S3 differences; MinIO is the closest free AWS-equivalent and exposes a different surface. |
|
||||
| High | No Kafka/NATS/RabbitMQ broker fixture | `§4.1` | Add at least one (NATS is lightest) — events, schema-registry, idempotency, and any future stream-processing package needs it. |
|
||||
| High | No OpenTelemetry collector fixture | `§4.1` | Required to actually verify the OTel instrumentation that every package ships; today you can only assert "no SDK registered = noop", not end-to-end export. |
|
||||
| Medium | No Elasticsearch/OpenSearch fixture | `§4.1` | Typesense covers search but Elastic-API consumers are common — flag for later wave. |
|
||||
| Medium | No Vault / secrets fixture, no LocalStack | `§4.1` | Mark as deferred; document the gap so consumers aren't surprised. |
|
||||
| Low | No Caddy/nginx reverse-proxy test fixture | `§4.1` | Caddy already runs in `testbed` stack; expose a minimal fixture for routing tests. |
|
||||
|
||||
## 2. Test-pyramid completeness
|
||||
|
||||
| Severity | Gap | Where it belongs | Concrete fix |
|
||||
|---|---|---|---|
|
||||
| Critical | No accessibility testing | `testing-e2e/spec.md` + `testbed-pages-and-e2e.md §7.4` | Add `attachAxe(page)` helper (using `@axe-core/playwright`) and require every spec's invariant list to include `await axe.assertNoViolations()` alongside console/network monitors. |
|
||||
| Critical | No visual regression | `testing-e2e/spec.md` + `testbed-foundation.md` Storybook section | Add `expect(page).toHaveScreenshot()` baseline strategy plus a Storybook test-runner job; or pin a `pixelmatch`-based helper into `testing-e2e`. Decide and write the spec — "stories exist" is not visual coverage. |
|
||||
| High | No mutation testing | `testing/spec.md` quality gate | Add a Stryker job (Bun-compatible via `@stryker-mutator/core` + node) for `crypto*`, `error`, `validation`, `serialize`, and other foundation packages; gate at e.g. ≥ 70% mutation score. |
|
||||
| High | No load/perf gate beyond `mitata` micro-bench | `testing/spec.md` benchmark requirement | Extend with a k6/autocannon "API smoke" gate against `apps/testbed`'s Hono routes, run nightly; current 10% mitata gate doesn't cover end-to-end latency or concurrency. |
|
||||
| High | No contract tests (consumer/provider) | new `testing-contract` package or `testing-integration` extension | `oauth-client`, `webhook-client`, `schema-registry`, `api-schemas` are textbook Pact candidates; spec a `recordPactInteraction()` helper. |
|
||||
| Medium | Property tests (fast-check) only mandated for `@suaveplan/testing` itself | `testing/spec.md` "Property-based coverage" requirement | Generalise into a project-wide rule: every pure-function package SHALL include fast-check round-trip tests; cite `crypto-utils`, `diff`, `pagination`, `validation`. |
|
||||
| Medium | No fuzzing for parsers | per-package specs (`mime`, `dmarc`, `password-formats`, `metadata`, `exif`, `serialize`) | Require a `*.fuzz.test.ts` using `fast-check` arbitraries seeded from real-world corpora; these are all attack-surface parsers. |
|
||||
| Medium | No chaos/fault injection beyond single "disable route" | `testing-e2e/spec.md` + multi-session specs | Add network-throttle, latency, and abort fixtures (`page.route` with random failure injection); already half-built in `network-monitor`. |
|
||||
| Low | No snapshot strategy doc | `testing-dom/spec.md` | One-paragraph stance on inline snapshots vs file snapshots; right now nothing forbids snapshot drift. |
|
||||
|
||||
## 3. Cross-cutting concerns
|
||||
|
||||
| Severity | Gap | Where it belongs | Concrete fix |
|
||||
|---|---|---|---|
|
||||
| Critical | Multi-fixture lifecycle order is parallel-start / reverse-stop only — no declared dependencies between fixtures | `testing-integration-core.md` `TestEnvironment` section + `testing` spec delta | Add `dependsOn?: string[]` to `ServiceFixture`; topo-sort start order; document for cases like "OIDC issuer must be ready before app under test seeds clients". |
|
||||
| Critical | No deterministic time/RNG injection contract for integration tests | new requirement in `testing-integration` spec | The `testing` spec mandates `Clock`/`Rng` injection but `testing-integration` fixtures don't propagate; add `clock`/`rng` to `TestEnvironmentOptions` and require fixture configs to pass them through where applicable. |
|
||||
| High | No flake quarantine / retry policy for integration tests | `testing-integration` spec | Add `quarantine.json` + a `markFlaky()` helper that retries N times and reports — without it, transient Docker failures will block PRs. |
|
||||
| High | No leak/handle-tracking on teardown | `TestEnvironment` requirement | After `teardown()`, assert via `process._getActiveHandles()` (or Bun equivalent) that no handles attributable to fixtures remain; fail the run if so. |
|
||||
| High | No CI matrix on Bun versions, OS, or Node-fallback parity | `testbed-pages-and-e2e.md §7.6` | Add matrix: `bun-1.x` + `bun-canary`, `ubuntu-latest` + `macos-14` + `windows-latest` (Playwright at minimum); add a Node-runner parity job for browser-runnable packages. |
|
||||
| High | No secret management policy for CI | `testing-integration` spec + `docker/README.md` | Document where `GARAGE_S3_SECRET_ACCESS_KEY` and any future OIDC client secrets live (GitHub OIDC + Vault/SOPS); current `.env.example` leaves them blank with no guidance. |
|
||||
| Medium | Port collision policy is static range only | `enterprise-testbed-design.md §5.2` | testcontainers already uses random ports for the integration path; spec the dev/test-stack collision behaviour (fail-loud vs auto-shift) and document in `docker/README.md`. |
|
||||
| Medium | No OOM/timeout budget per spec | `testing/spec.md` and Playwright config | Per-spec memory cap and global-suite wall-clock budget; current 30 s test timeout is per-test, no suite ceiling. |
|
||||
| Medium | No network sandboxing rule | `testing-e2e/spec.md` | Default `page.route('**/*')` to block external origins unless explicitly allow-listed; today specs could hit live internet. |
|
||||
|
||||
## 4. Observability of the tests themselves
|
||||
|
||||
| Severity | Gap | Where it belongs | Concrete fix |
|
||||
|---|---|---|---|
|
||||
| Critical | No JUnit / Allure / structured test-result output | `testing/spec.md` + Playwright config | Mandate `junit` + `playwright html` reporters and an Allure aggregation step in CI; currently only `["html","github"]` is set. |
|
||||
| High | OTel test-execution export missing despite first-class OTel in spec | new requirement in `testing/spec.md` | Wire test runner to the `testing-integration-otel-collector` fixture (see §1) and export every test's span; failed-test traces become triage gold. |
|
||||
| High | No flake-rate / coverage trend tracking | `testing/spec.md` quality gate | Persist Istanbul + JUnit artefacts to a static bucket (Garage S3 itself!) and compute per-package flake rate over 30 days; gate "new flaky test" on PR. |
|
||||
| Medium | Trace-on-failure exists locally (`trace: on-first-retry`) but is not exported anywhere | Playwright config | Upload `trace.zip` artefacts to the same bucket on red runs. |
|
||||
|
||||
## 5. Operational rigor
|
||||
|
||||
| Severity | Gap | Where it belongs | Concrete fix |
|
||||
|---|---|---|---|
|
||||
| Critical | Images pinned by tag, not digest | `docker-infrastructure.md` + plugin specs | Replace every `image: foo:tag` with `image: foo:tag@sha256:…`; tag-based pinning is non-reproducible (especially `axllent/mailpit:latest`, which is wrong for any "enterprise" claim). |
|
||||
| Critical | `axllent/mailpit:latest` in particular | `docker-infrastructure.md` Chunk 1 + `testing-integration-mailpit` plugin | Pin to a specific minor (`v1.x`) and digest. |
|
||||
| High | No arm64 ↔ amd64 matrix for testcontainers | `testbed-pages-and-e2e.md §7.6` | Garage and Typesense images don't always ship arm64; CI must run both arches or the spec must document amd64-only. |
|
||||
| High | No airgap / offline-cache test | new requirement | Verify the test suite runs after `bun install --offline` with all images pre-pulled; many enterprises require this. |
|
||||
| High | No Verdaccio integration in test path | `testing-integration` spec | The repo uses Verdaccio at `npmjs.fq.io` — there should be a `testing-integration-verdaccio` fixture (or compose service) so publish flows are testable, not just lint+typecheck+test. |
|
||||
| Medium | No container resource limits | `docker-compose.*.yml` | Add `deploy.resources.limits` (cpu, memory) — without them, a runaway test can starve CI. |
|
||||
| Medium | Healthcheck timeouts not coherent with WSL2 fork-pool timing | `testing-integration-core.md` HealthPoller defaults + CLAUDE.md "Vitest / WSL2" | 120 s default is fine, but document interaction with `fileParallelism: false` for browser tests; a parallel sweep can serialise health polls and blow the budget. |
|
||||
|
||||
## 6. Documentation / spec coherence
|
||||
|
||||
| Severity | Gap | Where it belongs | Concrete fix |
|
||||
|---|---|---|---|
|
||||
| Critical | None of the 2026-04-24 work is in `openspec/specs/` | `openspec/specs/testing-integration/spec.md` (new), and ADDED requirements in `openspec/specs/testing-e2e/spec.md` | Create an OpenSpec change folder, write deltas covering: `ServiceFixture<T>`, `TestEnvironment`, `createIntegrationSetup`, `skipIfUnavailable`, `HealthPoller`, env-bridge, plus the new e2e modules (`console-monitor`, `network-monitor`, `multi-session`, `extension-fixture`); fold on ship. |
|
||||
| High | Six service-fixture plugins have no spec at all | new `openspec/specs/testing-integration-<svc>/spec.md` per plugin (or one umbrella) | Each needs at minimum: image policy, env-bridge contract, healthcheck contract, config shape, error code prefix. |
|
||||
| Critical | `testing` spec mandates ≥ 95 % coverage; project rule is **100 % for every package, no exceptions** (statements / branches / functions / lines, Istanbul provider) | `openspec/specs/testing/spec.md` | Replace the 95 % threshold with a flat 100 % gate. No "foundation-only" carve-out. Pre-publish gate (`bunx turbo test:coverage --filter=@suaveplan/<pkg>`) must hard-fail under 100 %. Remove every reference to 95 % in plans and specs. |
|
||||
| Medium | The `testing` spec's `provide("__integration_failed__", …)` interface is invented in plans without being mirrored in any spec | `testing-integration` spec | Document it as a stable inject-key contract or rename to `__suaveplan_integration_failed__` to avoid name collisions. |
|
||||
|
||||
## 7. apps/testbed verification rigor
|
||||
|
||||
CLAUDE.md mandates six bullets; the plan delivers ~3.5 of them.
|
||||
|
||||
| Severity | Bullet | Status | Concrete fix |
|
||||
|---|---|---|---|
|
||||
| Met | Docker staging environment | `docker-compose.testbed.yml` is fine | — |
|
||||
| Partial | Full Playwright sweep of every page + every component | Pages enumerated via `routes.ts`; components only via Storybook stories — there is no spec that says "every component on every page is interacted with" | `testbed-pages-and-e2e.md §7.4` + spec | Add a per-page `interaction.spec.ts` template that asserts every `data-testid="demo-card"` triggers `data-testid="output"` to populate; baseline only navigates. |
|
||||
| Critical | Never trust HTTP 200 alone — verify rendered DOM | Plans require `data-testid="page-title"` non-empty but not other rendered content | `enterprise-testbed-design.md §7.4` | Strengthen invariant 5 to "`output` AND at least one demo-specific assertion per page", and forbid baseline-only specs as the sole coverage for a route. |
|
||||
| Met | Inspect browser console | `attachConsoleMonitor` covers it | — |
|
||||
| Partial | Inspect network activity | `attachNetworkMonitor` exists; no rules for missing-asset / mixed-content / CORS | `testing-e2e/spec.md` | Extend `NetworkMonitor` contract: `assertNoMissingAssets()`, `assertNoMixedContent()`, `assertNoCorsErrors()` as separate methods, all called by invariant. |
|
||||
| Critical | Zero-defect push to live | No gate exists | CI workflow + `enterprise-testbed-design.md` | Add a "release readiness" job that fails on any of: console error, network failure, axe violation, visual diff, broken-link crawler hit. Without this, the bullet is aspirational. |
|
||||
|
||||
## 8. Test-suite onboarding
|
||||
|
||||
| Severity | Gap | Where it belongs | Concrete fix |
|
||||
|---|---|---|---|
|
||||
| High | No single "how do I write a test for X" entry point | new `docs/testing/README.md` (or `openspec/specs/testing/AGENTS.md`) | Decision tree: pure fn → vitest+fast-check; React → `testing-dom`; needs Docker → `testing-integration` + plugin; cross-browser → `testing-e2e`; performance → mitata + k6. Link from root README and CLAUDE.md. |
|
||||
| Medium | No "first test in 5 minutes" tutorial against `apps/testbed` | `apps/testbed/README.md` | The plan's README requirements list operational topics but not a tutorial; add one. |
|
||||
| Medium | No discoverability index of fixtures | `docker/README.md` or new `docs/testing/fixtures.md` | Table mapping "I need to test X" → "use fixture Y" for every plugin. |
|
||||
|
||||
---
|
||||
|
||||
## Universal coverage rule
|
||||
|
||||
**100 % Istanbul coverage (statements / branches / functions / lines) is mandatory for every `@suaveplan/*` package — no exceptions, no "foundation only", no "95 % is fine for X".** The pre-publish gate (`bunx turbo test:coverage --filter=@suaveplan/<pkg>`) MUST hard-fail below 100 %. Every spec, plan, and tooling default in this repo SHALL state 100 % and only 100 %. Any 95 % reference is a defect to fix on sight.
|
||||
|
||||
Mutation testing, contract testing, and property/fuzz coverage are layered ON TOP of 100 % line coverage — they do not substitute for it, and 100 % line coverage does not substitute for them.
|
||||
|
||||
## Recommended sequencing
|
||||
|
||||
1. **Lift 2026-04-24 plans into normative specs.** Open an OpenSpec change adding `openspec/specs/testing-integration/spec.md` + per-plugin specs + ADDED requirements in `testing-e2e/spec.md` for the new modules. Without this, every future plugin compounds drift.
|
||||
2. **Replace every 95 % coverage reference with 100 %** in the same change — `testing/spec.md`, the archived `2026-04-19-enterprise-testing` design notes if still authoritative, and any plan defaults. Add an explicit "100 %, no exceptions" line to `openspec/specs/testing/spec.md` quality requirements.
|
||||
3. **Add the missing pyramid tiers as separate OpenSpec changes**, in this order: a11y (cheap, high-signal) → visual regression → OTel-collector fixture + test-execution export → contract testing → mutation testing → load/perf gate.
|
||||
4. **Tighten testbed verification** to fully meet CLAUDE.md's six bullets — strengthen invariants, add missing-asset/mixed-content/CORS assertions, add the release-readiness gate.
|
||||
5. **Fix operational hygiene last** — digest-pin all images, add resource limits, add CI matrix, add airgap rebuild test.
|
||||
|
||||
## Files referenced
|
||||
|
||||
- `/mnt/s/web/genesis/openspec/specs/testing/spec.md`
|
||||
- `/mnt/s/web/genesis/openspec/specs/testing-dom/spec.md`
|
||||
- `/mnt/s/web/genesis/openspec/specs/testing-e2e/spec.md`
|
||||
- `/mnt/s/web/genesis/docs/superpowers/specs/2026-04-24-enterprise-testbed-design.md`
|
||||
- `/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-docker-infrastructure.md`
|
||||
- `/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testing-integration-core.md`
|
||||
- `/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testing-integration-plugins.md`
|
||||
- `/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testbed-foundation.md`
|
||||
- `/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testbed-pages-and-e2e.md`
|
||||
Reference in New Issue
Block a user