Initial Commit

openspec/changes/enterprise-testing-load-perf/design.md

@@ -0,0 +1,83 @@
+## Context
+
+`mitata` benchmarks under `src/__benchmarks__/` already gate per-package micro-performance at the 10% regression floor (`openspec/specs/testing/spec.md` "Benchmark gate in CI"). This protects pure-function throughput but says nothing about request-response latency, concurrent-user behaviour, websocket fan-out, or the full Hono middleware stack of `apps/testbed`. The 2026-04-25 audit (§2) flags this as a high-severity gap and recommends k6 or autocannon as the remediation.
+
+`apps/testbed` already ships a Docker staging environment and a Playwright sweep. Adding a load gate against the same staging environment closes the third side of the verification triangle: render correctness (Playwright), behavioural assertions (vitest), and performance under load (k6/autocannon).
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- A single, runner-agnostic public API (`defineLoadScenario`, `runLoadScenario`, `assertThresholds`) that callers use without knowing which driver runs underneath.
+- k6 as primary driver via the `grafana/k6` testcontainers image; autocannon as a lighter Node-native alternative for environments without Docker (CI matrix on macOS/Windows runners that lack Docker support).
+- Default thresholds tuned for typical Hono REST workloads: `http_req_duration{p(95)} < 500ms`, `http_req_failed < 1%`, `iteration_duration{p(99)} < 2000ms`. All overridable per scenario.
+- Four run modes: `smoke` (gating, 1 vu / 30s), `load` (10 vu / 5m), `stress` (50 vu / 10m), `spike` (0→100 vu in 30s). Smoke runs nightly in CI; the others run weekly or on demand.
+- 100% Istanbul coverage on the wrapper itself; the k6 binary invocation is mocked via injected spawn/fetch shims so the package tests run without Docker.
+- Stable error contract: `TestingLoadError` with codes prefixed `TESTING_LOAD.*` (e.g. `TESTING_LOAD.THRESHOLD_VIOLATION`, `TESTING_LOAD.DRIVER_FAILED`, `TESTING_LOAD.CONFIG_INVALID`).
+
+**Non-Goals:**
+
+- Replacing or relaxing the mitata 10% micro-benchmark gate.
+- Distributed load generation. k6 cloud, k6 operator on Kubernetes, and Locust are out of scope. Single-runner load is sufficient for the smoke gate.
+- Custom k6 extensions (xk6). The wrapper consumes k6's stock features only.
+- Load testing against production. All scenarios target the Docker staging environment defined by `apps/testbed/docker-compose.testbed.yml` (or equivalent per-app stack).
+
+## Decisions
+
+### D1: Runner-agnostic API with pluggable drivers
+
+```ts
+export interface LoadDriver {
+    name: "k6" | "autocannon";
+    run(scenario: LoadScenario, opts: LoadRunOptions): Promise<LoadSummary>;
+}
+
+export function defineLoadScenario(spec: LoadScenarioSpec): LoadScenario;
+export function runLoadScenario(scenario: LoadScenario, opts?: LoadRunOptions): Promise<{ summary: LoadSummary; raw: unknown }>;
+export function assertThresholds(summary: LoadSummary): void;
+```
+
+`LoadScenario` carries `{ name, target, vus, duration, thresholds, stages? }`. `LoadRunOptions` carries `{ driver?: LoadDriver; signal?: AbortSignal; onMetric?: (m) => void }`. The driver is injected; k6 is the default when present, autocannon otherwise.
+
+### D2: k6 via testcontainers, autocannon via npm
+
+k6 runs in the `grafana/k6` container, with the script generated from the `LoadScenario` and mounted into the container. Stdout is parsed via k6's structured-summary JSON output (`--summary-export`). Autocannon runs in-process via the npm package, with options mapped from the same `LoadScenario`. Both drivers populate the same `LoadSummary` shape so `assertThresholds` is driver-agnostic.
+
+### D3: Threshold contract
+
+Thresholds are RFC-style key→predicate maps:
+
+```ts
+export type Thresholds = Record<string, string[]>;
+// e.g. { "http_req_duration": ["p(95)<500", "p(99)<1000"], "http_req_failed": ["rate<0.01"] }
+```
+
+Defaults applied if the caller omits the field; overrides merge on top. `assertThresholds(summary)` throws `TestingLoadError("TESTING_LOAD.THRESHOLD_VIOLATION", …, { context: { failed: [...] } })` listing every failed predicate.
+
+### D4: Run modes as named presets
+
+`smoke`, `load`, `stress`, `spike` are exported constants composing `vus` + `stages` + `duration`. Callers spread them into `defineLoadScenario`:
+
+```ts
+defineLoadScenario({ ...presets.smoke, name: "GET /healthz", target: "http://testbed:3000/healthz" });
+```
+
+### D5: Self-testability without Docker
+
+The k6 driver accepts an injectable spawn function and an injectable `fetch`. Tests stub both to feed canned k6 JSON summaries; no Docker required for the wrapper's own test suite. Same pattern as `testing-integration-postgres`'s injectable runner function (per `CLAUDE.md` package listing).
+
+### D6: Error shape parallels `@suaveplan/testing`
+
+`TestingLoadError extends Error` with `{ code, cause?, context? }` and codes prefixed `TESTING_LOAD.*`. Shape-compatible with the existing `TestingError` so a future `@suaveplan/error` adapter bridges both at once. No `@suaveplan/error` dep on day one (keeps Layer-0 carve-out per CLAUDE.md rule 8).
+
+### D7: App integration via `*.smoke.load.ts`
+
+Each app under `apps/` ships at least one `*.smoke.load.ts` file. CI runs them nightly against the app's staging stack. `apps/testbed` enumerates one smoke scenario per route in `routes.ts`. The websocket-gateway app's smoke scenario opens N websocket connections and asserts message round-trip latency.
+
+## Risks / Trade-offs
+
+- **k6 binary availability.** Mitigated by the testcontainers image plus the autocannon fallback path.
+- **CI wall-clock cost.** Smoke scenarios are 30s × N apps; load/stress/spike are out-of-band (weekly or on demand).
+- **Threshold tuning.** Defaults may be too tight for some routes. Mitigated by per-scenario overrides; the smoke run is intended to catch regressions, not enforce SLOs.
+- **Driver-summary impedance.** k6 and autocannon report different metric names. Mitigated by the wrapper normalising both into `LoadSummary` before threshold evaluation.
+- **Layer-0 dep purity.** The package depends on `@suaveplan/types` only (parallel to `testing-integration*`). External catalog deps cover everything else.