18 KiB
18 KiB
Test-Suite Documentation Audit — Genesis Monorepo (2026-04-24 wave)
Date: 2026-04-25
Scope: review of openspec/specs/testing*/spec.md (canonical, 2026-04-19) plus the 2026-04-24 testbed design + plans in docs/superpowers/.
Goal: identify gaps and improvements to elevate the test-suite documentation to enterprise-class quality.
Executive summary — top critical gaps
- Spec drift is severe. The canonical
openspec/specs/testing*/spec.mdfiles (2026-04-19) say nothing abouttesting-integration*,console-monitor,network-monitor,multi-session, orextension-fixture. All recent 2026-04-24 work lives only indocs/superpowers/plans/and has not been folded into the contract — there is no normative spec for the integration substrate that the entire testbed and CI nightly job depends on. - Test-pyramid is missing several mandatory enterprise tiers. Zero accessibility (axe-core), zero visual-regression, zero contract testing, zero mutation testing, zero load/perf, zero fuzzing for non-pure code, zero chaos/fault injection beyond the single "disable signalling route" test.
- Service-fixture coverage is too narrow for the package list.
oauth-client,mcp,server-emailer,search, future Kafka/NATS-style event pipes, and any object-store/secret/identity work have no fixtures. The minimum missing set: MongoDB, MySQL/MariaDB, OIDC issuer (Keycloak/dex/Hydra), MinIO (broader-S3 parity vs Garage), an OTel collector, and a NATS/Kafka broker. apps/testbedonly partially enforces CLAUDE.md's six "Web App & Page Verification" bullets. Console + network monitors and Docker staging are wired; rendered-DOM content checks per page, mixed-content, asset-404 detection, and the "zero-defect-to-live" gate are not formalised in the plan.- Test-suite observability is absent. No JUnit/Allure output, no OTel export of test execution despite OTel being a first-class concern in
testing/spec.md, no flake-rate metric, no coverage trend, no trace-on-failure shipped to a sink. CI matrix is also single-arch (no arm64, no Bun version matrix, no Node fallback parity check).
1. Coverage gaps in service fixtures
| Severity | Gap | Where it belongs | Concrete fix |
|---|---|---|---|
| Critical | No OIDC issuer fixture, but @suaveplan/oauth-client already exists |
new plugin testing-integration-oidc (Keycloak or oauth2-mock-server) under packages/testing/ |
Add to enterprise-testbed-design.md §4.1 and create plan; required to integration-test oauth-client against a real Authorization Code + PKCE flow. |
| Critical | No Mongo / MySQL fixture | enterprise-testbed-design.md §4.1 |
Add testing-integration-mongodb (mongo:7) and testing-integration-mysql (mysql:8) — many real apps use these and there is no parity coverage for db-* siblings of db-sqlite. |
| High | No object-store parity beyond Garage | new testing-integration-minio |
Garage has quirks (admin-port health, layout assignment) that mask AWS-S3 differences; MinIO is the closest free AWS-equivalent and exposes a different surface. |
| High | No Kafka/NATS/RabbitMQ broker fixture | §4.1 |
Add at least one (NATS is lightest) — events, schema-registry, idempotency, and any future stream-processing package needs it. |
| High | No OpenTelemetry collector fixture | §4.1 |
Required to actually verify the OTel instrumentation that every package ships; today you can only assert "no SDK registered = noop", not end-to-end export. |
| Medium | No Elasticsearch/OpenSearch fixture | §4.1 |
Typesense covers search but Elastic-API consumers are common — flag for later wave. |
| Medium | No Vault / secrets fixture, no LocalStack | §4.1 |
Mark as deferred; document the gap so consumers aren't surprised. |
| Low | No Caddy/nginx reverse-proxy test fixture | §4.1 |
Caddy already runs in testbed stack; expose a minimal fixture for routing tests. |
2. Test-pyramid completeness
| Severity | Gap | Where it belongs | Concrete fix |
|---|---|---|---|
| Critical | No accessibility testing | testing-e2e/spec.md + testbed-pages-and-e2e.md §7.4 |
Add attachAxe(page) helper (using @axe-core/playwright) and require every spec's invariant list to include await axe.assertNoViolations() alongside console/network monitors. |
| Critical | No visual regression | testing-e2e/spec.md + testbed-foundation.md Storybook section |
Add expect(page).toHaveScreenshot() baseline strategy plus a Storybook test-runner job; or pin a pixelmatch-based helper into testing-e2e. Decide and write the spec — "stories exist" is not visual coverage. |
| High | No mutation testing | testing/spec.md quality gate |
Add a Stryker job (Bun-compatible via @stryker-mutator/core + node) for crypto*, error, validation, serialize, and other foundation packages; gate at e.g. ≥ 70% mutation score. |
| High | No load/perf gate beyond mitata micro-bench |
testing/spec.md benchmark requirement |
Extend with a k6/autocannon "API smoke" gate against apps/testbed's Hono routes, run nightly; current 10% mitata gate doesn't cover end-to-end latency or concurrency. |
| High | No contract tests (consumer/provider) | new testing-contract package or testing-integration extension |
oauth-client, webhook-client, schema-registry, api-schemas are textbook Pact candidates; spec a recordPactInteraction() helper. |
| Medium | Property tests (fast-check) only mandated for @suaveplan/testing itself |
testing/spec.md "Property-based coverage" requirement |
Generalise into a project-wide rule: every pure-function package SHALL include fast-check round-trip tests; cite crypto-utils, diff, pagination, validation. |
| Medium | No fuzzing for parsers | per-package specs (mime, dmarc, password-formats, metadata, exif, serialize) |
Require a *.fuzz.test.ts using fast-check arbitraries seeded from real-world corpora; these are all attack-surface parsers. |
| Medium | No chaos/fault injection beyond single "disable route" | testing-e2e/spec.md + multi-session specs |
Add network-throttle, latency, and abort fixtures (page.route with random failure injection); already half-built in network-monitor. |
| Low | No snapshot strategy doc | testing-dom/spec.md |
One-paragraph stance on inline snapshots vs file snapshots; right now nothing forbids snapshot drift. |
3. Cross-cutting concerns
| Severity | Gap | Where it belongs | Concrete fix |
|---|---|---|---|
| Critical | Multi-fixture lifecycle order is parallel-start / reverse-stop only — no declared dependencies between fixtures | testing-integration-core.md TestEnvironment section + testing spec delta |
Add dependsOn?: string[] to ServiceFixture; topo-sort start order; document for cases like "OIDC issuer must be ready before app under test seeds clients". |
| Critical | No deterministic time/RNG injection contract for integration tests | new requirement in testing-integration spec |
The testing spec mandates Clock/Rng injection but testing-integration fixtures don't propagate; add clock/rng to TestEnvironmentOptions and require fixture configs to pass them through where applicable. |
| High | No flake quarantine / retry policy for integration tests | testing-integration spec |
Add quarantine.json + a markFlaky() helper that retries N times and reports — without it, transient Docker failures will block PRs. |
| High | No leak/handle-tracking on teardown | TestEnvironment requirement |
After teardown(), assert via process._getActiveHandles() (or Bun equivalent) that no handles attributable to fixtures remain; fail the run if so. |
| High | No CI matrix on Bun versions, OS, or Node-fallback parity | testbed-pages-and-e2e.md §7.6 |
Add matrix: bun-1.x + bun-canary, ubuntu-latest + macos-14 + windows-latest (Playwright at minimum); add a Node-runner parity job for browser-runnable packages. |
| High | No secret management policy for CI | testing-integration spec + docker/README.md |
Document where GARAGE_S3_SECRET_ACCESS_KEY and any future OIDC client secrets live (GitHub OIDC + Vault/SOPS); current .env.example leaves them blank with no guidance. |
| Medium | Port collision policy is static range only | enterprise-testbed-design.md §5.2 |
testcontainers already uses random ports for the integration path; spec the dev/test-stack collision behaviour (fail-loud vs auto-shift) and document in docker/README.md. |
| Medium | No OOM/timeout budget per spec | testing/spec.md and Playwright config |
Per-spec memory cap and global-suite wall-clock budget; current 30 s test timeout is per-test, no suite ceiling. |
| Medium | No network sandboxing rule | testing-e2e/spec.md |
Default page.route('**/*') to block external origins unless explicitly allow-listed; today specs could hit live internet. |
4. Observability of the tests themselves
| Severity | Gap | Where it belongs | Concrete fix |
|---|---|---|---|
| Critical | No JUnit / Allure / structured test-result output | testing/spec.md + Playwright config |
Mandate junit + playwright html reporters and an Allure aggregation step in CI; currently only ["html","github"] is set. |
| High | OTel test-execution export missing despite first-class OTel in spec | new requirement in testing/spec.md |
Wire test runner to the testing-integration-otel-collector fixture (see §1) and export every test's span; failed-test traces become triage gold. |
| High | No flake-rate / coverage trend tracking | testing/spec.md quality gate |
Persist Istanbul + JUnit artefacts to a static bucket (Garage S3 itself!) and compute per-package flake rate over 30 days; gate "new flaky test" on PR. |
| Medium | Trace-on-failure exists locally (trace: on-first-retry) but is not exported anywhere |
Playwright config | Upload trace.zip artefacts to the same bucket on red runs. |
5. Operational rigor
| Severity | Gap | Where it belongs | Concrete fix |
|---|---|---|---|
| Critical | Images pinned by tag, not digest | docker-infrastructure.md + plugin specs |
Replace every image: foo:tag with image: foo:tag@sha256:…; tag-based pinning is non-reproducible (especially axllent/mailpit:latest, which is wrong for any "enterprise" claim). |
| Critical | axllent/mailpit:latest in particular |
docker-infrastructure.md Chunk 1 + testing-integration-mailpit plugin |
Pin to a specific minor (v1.x) and digest. |
| High | No arm64 ↔ amd64 matrix for testcontainers | testbed-pages-and-e2e.md §7.6 |
Garage and Typesense images don't always ship arm64; CI must run both arches or the spec must document amd64-only. |
| High | No airgap / offline-cache test | new requirement | Verify the test suite runs after bun install --offline with all images pre-pulled; many enterprises require this. |
| High | No Verdaccio integration in test path | testing-integration spec |
The repo uses Verdaccio at npmjs.fq.io — there should be a testing-integration-verdaccio fixture (or compose service) so publish flows are testable, not just lint+typecheck+test. |
| Medium | No container resource limits | docker-compose.*.yml |
Add deploy.resources.limits (cpu, memory) — without them, a runaway test can starve CI. |
| Medium | Healthcheck timeouts not coherent with WSL2 fork-pool timing | testing-integration-core.md HealthPoller defaults + CLAUDE.md "Vitest / WSL2" |
120 s default is fine, but document interaction with fileParallelism: false for browser tests; a parallel sweep can serialise health polls and blow the budget. |
6. Documentation / spec coherence
| Severity | Gap | Where it belongs | Concrete fix |
|---|---|---|---|
| Critical | None of the 2026-04-24 work is in openspec/specs/ |
openspec/specs/testing-integration/spec.md (new), and ADDED requirements in openspec/specs/testing-e2e/spec.md |
Create an OpenSpec change folder, write deltas covering: ServiceFixture<T>, TestEnvironment, createIntegrationSetup, skipIfUnavailable, HealthPoller, env-bridge, plus the new e2e modules (console-monitor, network-monitor, multi-session, extension-fixture); fold on ship. |
| High | Six service-fixture plugins have no spec at all | new openspec/specs/testing-integration-<svc>/spec.md per plugin (or one umbrella) |
Each needs at minimum: image policy, env-bridge contract, healthcheck contract, config shape, error code prefix. |
| Critical | testing spec mandates ≥ 95 % coverage; project rule is 100 % for every package, no exceptions (statements / branches / functions / lines, Istanbul provider) |
openspec/specs/testing/spec.md |
Replace the 95 % threshold with a flat 100 % gate. No "foundation-only" carve-out. Pre-publish gate (bunx turbo test:coverage --filter=@suaveplan/<pkg>) must hard-fail under 100 %. Remove every reference to 95 % in plans and specs. |
| Medium | The testing spec's provide("__integration_failed__", …) interface is invented in plans without being mirrored in any spec |
testing-integration spec |
Document it as a stable inject-key contract or rename to __suaveplan_integration_failed__ to avoid name collisions. |
7. apps/testbed verification rigor
CLAUDE.md mandates six bullets; the plan delivers ~3.5 of them.
| Severity | Bullet | Status | Concrete fix |
|---|---|---|---|
| Met | Docker staging environment | docker-compose.testbed.yml is fine |
— |
| Partial | Full Playwright sweep of every page + every component | Pages enumerated via routes.ts; components only via Storybook stories — there is no spec that says "every component on every page is interacted with" |
testbed-pages-and-e2e.md §7.4 + spec |
| Critical | Never trust HTTP 200 alone — verify rendered DOM | Plans require data-testid="page-title" non-empty but not other rendered content |
enterprise-testbed-design.md §7.4 |
| Met | Inspect browser console | attachConsoleMonitor covers it |
— |
| Partial | Inspect network activity | attachNetworkMonitor exists; no rules for missing-asset / mixed-content / CORS |
testing-e2e/spec.md |
| Critical | Zero-defect push to live | No gate exists | CI workflow + enterprise-testbed-design.md |
8. Test-suite onboarding
| Severity | Gap | Where it belongs | Concrete fix |
|---|---|---|---|
| High | No single "how do I write a test for X" entry point | new docs/testing/README.md (or openspec/specs/testing/AGENTS.md) |
Decision tree: pure fn → vitest+fast-check; React → testing-dom; needs Docker → testing-integration + plugin; cross-browser → testing-e2e; performance → mitata + k6. Link from root README and CLAUDE.md. |
| Medium | No "first test in 5 minutes" tutorial against apps/testbed |
apps/testbed/README.md |
The plan's README requirements list operational topics but not a tutorial; add one. |
| Medium | No discoverability index of fixtures | docker/README.md or new docs/testing/fixtures.md |
Table mapping "I need to test X" → "use fixture Y" for every plugin. |
Universal coverage rule
100 % Istanbul coverage (statements / branches / functions / lines) is mandatory for every @suaveplan/* package — no exceptions, no "foundation only", no "95 % is fine for X". The pre-publish gate (bunx turbo test:coverage --filter=@suaveplan/<pkg>) MUST hard-fail below 100 %. Every spec, plan, and tooling default in this repo SHALL state 100 % and only 100 %. Any 95 % reference is a defect to fix on sight.
Mutation testing, contract testing, and property/fuzz coverage are layered ON TOP of 100 % line coverage — they do not substitute for it, and 100 % line coverage does not substitute for them.
Recommended sequencing
- Lift 2026-04-24 plans into normative specs. Open an OpenSpec change adding
openspec/specs/testing-integration/spec.md+ per-plugin specs + ADDED requirements intesting-e2e/spec.mdfor the new modules. Without this, every future plugin compounds drift. - Replace every 95 % coverage reference with 100 % in the same change —
testing/spec.md, the archived2026-04-19-enterprise-testingdesign notes if still authoritative, and any plan defaults. Add an explicit "100 %, no exceptions" line toopenspec/specs/testing/spec.mdquality requirements. - Add the missing pyramid tiers as separate OpenSpec changes, in this order: a11y (cheap, high-signal) → visual regression → OTel-collector fixture + test-execution export → contract testing → mutation testing → load/perf gate.
- Tighten testbed verification to fully meet CLAUDE.md's six bullets — strengthen invariants, add missing-asset/mixed-content/CORS assertions, add the release-readiness gate.
- Fix operational hygiene last — digest-pin all images, add resource limits, add CI matrix, add airgap rebuild test.
Files referenced
/mnt/s/web/genesis/openspec/specs/testing/spec.md/mnt/s/web/genesis/openspec/specs/testing-dom/spec.md/mnt/s/web/genesis/openspec/specs/testing-e2e/spec.md/mnt/s/web/genesis/docs/superpowers/specs/2026-04-24-enterprise-testbed-design.md/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-docker-infrastructure.md/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testing-integration-core.md/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testing-integration-plugins.md/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testbed-foundation.md/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testbed-pages-and-e2e.md