Files
suaveplan/openspec/changes/enterprise-testing-load-perf/proposal.md
2026-05-27 23:29:53 +01:00

4.5 KiB

Why

The 2026-04-25 test-suite audit (/mnt/s/web/genesis/docs/superpowers/specs/2026-04-25-test-suite-audit.md §2 row "No load/perf gate beyond mitata micro-bench") identifies an end-to-end performance gap. The canonical openspec/specs/testing/spec.md already enforces a mitata 10% benchmark regression gate, but mitata measures function-level micro-throughput in isolation. It does not exercise the request-response cycle, websocket fan-out, or concurrent-user behaviour of apps/testbed's Hono routes or the websocket-gateway. A regression in middleware ordering, async-context propagation, or Drizzle query plans can pass every micro-benchmark while doubling p95 latency under realistic load.

CLAUDE.md (/mnt/s/web/genesis/CLAUDE.md §Web App & Page Verification) requires Docker-staging verification, full Playwright sweeps, and zero-defect-to-live before pushing client-side work. Audit §7 confirms apps/testbed only partially meets this bar and that "release readiness" gating is missing. End-to-end load and perf gates close the third missing piece (after audit §2's a11y and visual regression, scheduled separately): proving the system holds up under load, not just that it renders.

What Changes

This change introduces a new @suaveplan/testing-load package and extends openspec/specs/testing/spec.md with an end-to-end performance gate that complements (does not replace) the existing mitata micro-benchmark gate.

New @suaveplan/testing-load package

  • Layout: packages/testing/testing-load. Zero @suaveplan/* runtime deps except @suaveplan/types (consistent with the rule-#9 carve-out per CLAUDE.md §Critical and the existing testing-integration* precedent).
  • Public API: defineLoadScenario({ name, target, vus, duration, thresholds, stages? }), runLoadScenario(scenario, opts) returning { summary, raw }, assertThresholds(summary) throwing on violation.
  • Tooling: k6 (binary, run via testcontainers) is the primary driver. autocannon (npm, catalog ref) is supported as a lighter Node-native alternative for environments without Docker.
  • Default thresholds: http_req_duration{p(95)} < 500ms, http_req_failed < 1%, iteration_duration{p(99)} < 2000ms. Per-scenario overrides supported.
  • Run modes: smoke (1 vu, 30s), load (10 vu, 5m), stress (50 vu, 10m), spike (0→100 vu in 30s).
  • Errors: TestingLoadError extending Error with { code, cause?, context? } and stable TESTING_LOAD.* codes. Shape-compatible with @suaveplan/error for a future adapter, matching the testing-package precedent in openspec/specs/testing/spec.md.
  • Quality: 100% Istanbul coverage on the wrapper, sibling .md per implementation file, Zod schemas in sibling *.schemas.ts, mitata bench file, fast-check property tests where applicable.

Extension to openspec/specs/testing/spec.md

A new requirement layered on the existing Benchmark gate: every app under apps/ SHALL ship a *.smoke.load.ts exercising the smoke scenario nightly, failing on threshold violation. apps/testbed SHALL define a load scenario per route declared in its routes.ts. The websocket-gateway SHALL define a connection-fan-out scenario. The release-readiness gate (cross-referenced from audit §7) consumes the smoke result as one of its inputs.

Capabilities

New Capabilities

  • testing-load — end-to-end load and performance scenarios with k6/autocannon drivers, threshold assertions, structured errors, and Zod-validated configs. Layer 0 utility (no @suaveplan/* runtime deps except @suaveplan/types).

Modified Capabilities

  • testing — adds an end-to-end performance gate requirement for every app under apps/, layered on top of the existing mitata 10% micro-benchmark gate. Does not modify or relax the mitata gate.

Impact

  • Affected code: new packages/testing/testing-load/ package; per-app *.smoke.load.ts files added under each apps/<name>/. apps/testbed gains one scenario per route in routes.ts. The websocket-gateway app gains a connection-fan-out scenario.
  • Affected catalogs: root package.json gains autocannon under catalogs.testing. k6 is run via the grafana/k6 testcontainers image (no npm catalog entry needed).
  • CI cadence: smoke scenarios run nightly per app. Load/stress/spike scenarios run weekly or on-demand against staging.
  • Coverage: wrapper meets 100% Istanbul; integration with k6 binary mocked via injected exec/fetch shims to keep the package self-testable without Docker.
  • No replacement: the existing mitata 10% regression gate stays in place. Both gates run; both gates must pass.