Initial Commit

openspec/changes/enterprise-testing-load-perf/specs/testing-load/spec.md

@@ -0,0 +1,109 @@
+## ADDED Requirements
+
+### Requirement: Package layout and dependency policy
+
+The `@suaveplan/testing-load` package SHALL live at `packages/testing/testing-load/`. Its `dependencies`, `peerDependencies`, and `devDependencies` MUST NOT list any `@suaveplan/*` workspace package other than `@suaveplan/types`. External deps SHALL be referenced via catalog entries (no bare version strings). The package SHALL declare `"sideEffects": false` and SHALL provide a subpath export per feature module per `openspec/AGENTS.md` §6.
+
+#### Scenario: Workspace deps are limited to @suaveplan/types
+
+- **WHEN** `bunx turbo ls --filter=@suaveplan/testing-load...` is run
+- **THEN** the only `@suaveplan/*` package in the forward closure MUST be `@suaveplan/types`
+
+#### Scenario: No bare versions
+
+- **WHEN** the package's `package.json` `dependencies` and `devDependencies` are scanned
+- **THEN** every external dep MUST resolve to a `catalog:<key>` reference AND no bare semver string MUST appear
+
+### Requirement: Public API — defineLoadScenario, runLoadScenario, assertThresholds
+
+The package SHALL export `defineLoadScenario(spec)` returning a frozen `LoadScenario`, `runLoadScenario(scenario, opts?)` returning `Promise<{ summary: LoadSummary; raw: unknown }>`, and `assertThresholds(summary)` returning `void` (throws on violation). Inputs SHALL be Zod-validated at the boundary; outputs SHALL conform to the `LoadSummary` shape regardless of which driver produced them. Every public function SHALL accept an optional `{ signal?: AbortSignal }` and SHALL respect cancellation at bounded checkpoints.
+
+#### Scenario: defineLoadScenario validates input
+
+- **WHEN** `defineLoadScenario({ name: "x", target: "not-a-url", vus: -1, duration: "30s" })` is called
+- **THEN** it MUST throw `TestingLoadError` with `code === "TESTING_LOAD.CONFIG_INVALID"` AND `context.issues` MUST list the offending fields
+
+#### Scenario: runLoadScenario honours AbortSignal
+
+- **WHEN** `runLoadScenario(scenario, { signal })` is awaited and `signal.abort()` is called mid-run
+- **THEN** the promise MUST reject with `TestingLoadError` where `code === "TESTING_LOAD.ABORTED"` AND `cause === signal.reason`
+
+#### Scenario: assertThresholds enumerates failed predicates
+
+- **WHEN** a `LoadSummary` reports `http_req_duration{p(95)} = 750` against threshold `p(95)<500`
+- **THEN** `assertThresholds(summary)` MUST throw `TestingLoadError` with `code === "TESTING_LOAD.THRESHOLD_VIOLATION"` AND `context.failed` MUST list the predicate `"http_req_duration: p(95)<500 (observed 750)"`
+
+### Requirement: Driver pluggability — k6 primary, autocannon fallback
+
+The package SHALL expose a `LoadDriver` interface and SHALL ship two implementations: a k6 driver invoking `grafana/k6` via testcontainers, and an autocannon driver running in-process via the `autocannon` npm package. `runLoadScenario` SHALL default to the k6 driver when reachable and SHALL fall back to autocannon when k6 is unavailable. Both drivers SHALL normalise their output into the same `LoadSummary` shape so callers and `assertThresholds` are driver-agnostic.
+
+#### Scenario: k6 selected when available
+
+- **WHEN** `runLoadScenario(scenario)` is called without an explicit driver AND a reachable Docker daemon is detected AND the `grafana/k6` image can be pulled
+- **THEN** the k6 driver MUST be selected AND the resulting `summary.driverName` MUST equal `"k6"`
+
+#### Scenario: autocannon fallback when k6 unavailable
+
+- **WHEN** `runLoadScenario(scenario)` is called without an explicit driver AND no Docker daemon is reachable
+- **THEN** the autocannon driver MUST be selected AND the resulting `summary.driverName` MUST equal `"autocannon"`
+
+#### Scenario: Both drivers produce LoadSummary
+
+- **WHEN** the same `LoadScenario` is executed once via the k6 driver and once via the autocannon driver
+- **THEN** both runs MUST return a `LoadSummary` with the same set of metric keys (e.g. `http_req_duration`, `http_req_failed`, `iteration_duration`) AND `assertThresholds` MUST evaluate identically against either summary
+
+### Requirement: Default thresholds and per-scenario overrides
+
+The package SHALL apply default thresholds when the caller omits the `thresholds` field on a `LoadScenario`: `http_req_duration{p(95)} < 500ms`, `http_req_failed < 1%` (rate < 0.01), `iteration_duration{p(99)} < 2000ms`. Caller-supplied thresholds SHALL merge on top of (not replace) the defaults; an explicit `thresholds: {}` SHALL retain the defaults; callers SHALL be able to override a single key without restating the rest.
+
+#### Scenario: Defaults applied when omitted
+
+- **WHEN** `defineLoadScenario({ name: "x", target: "http://t/", vus: 1, duration: "30s" })` is called and the resulting scenario is inspected
+- **THEN** the resolved `thresholds` MUST contain `http_req_duration: ["p(95)<500"]` AND `http_req_failed: ["rate<0.01"]` AND `iteration_duration: ["p(99)<2000"]`
+
+#### Scenario: Single-key override merges with defaults
+
+- **WHEN** `defineLoadScenario({ ..., thresholds: { http_req_duration: ["p(95)<200"] } })` is called
+- **THEN** the resolved scenario's `http_req_duration` MUST equal `["p(95)<200"]` AND the other default keys MUST remain present unchanged
+
+### Requirement: Run-mode presets — smoke, load, stress, spike
+
+The package SHALL export named run-mode presets composable with `defineLoadScenario`: `smoke` (1 vu, 30s), `load` (10 vu, 5m), `stress` (50 vu, 10m), `spike` (stages from 0 to 100 vu over 30s). Presets SHALL be plain objects spreadable into `defineLoadScenario`'s spec argument; callers SHALL be able to override any preset field on a per-scenario basis.
+
+#### Scenario: smoke preset values
+
+- **WHEN** the `smoke` preset is inspected
+- **THEN** `vus === 1` AND `duration === "30s"` AND `stages` MUST be undefined
+
+#### Scenario: spike preset values
+
+- **WHEN** the `spike` preset is inspected
+- **THEN** `stages` MUST describe a ramp from `0` to `100` virtual users over `30s` AND the total scenario duration MUST equal the sum of stage durations
+
+### Requirement: Structured errors with TESTING_LOAD.* codes
+
+The package SHALL throw only subclasses of `TestingLoadError extends Error` with stable codes prefixed `TESTING_LOAD.*`. Errors MUST carry `{ code, cause?, context? }` fields. The package MUST NOT throw plain `Error` or `TypeError` from any public code path. Defined codes include `TESTING_LOAD.CONFIG_INVALID`, `TESTING_LOAD.DRIVER_FAILED`, `TESTING_LOAD.DRIVER_UNAVAILABLE`, `TESTING_LOAD.THRESHOLD_VIOLATION`, and `TESTING_LOAD.ABORTED`.
+
+#### Scenario: Driver failure produces coded error
+
+- **WHEN** the k6 driver's container exits non-zero
+- **THEN** the driver MUST throw `TestingLoadError` with `code === "TESTING_LOAD.DRIVER_FAILED"` AND `context.driverName === "k6"` AND `context.exitCode` MUST equal the observed exit code
+
+#### Scenario: Driver unavailable surfaces a typed error
+
+- **WHEN** `runLoadScenario` is called with `driver: "k6"` explicitly AND no Docker daemon is reachable
+- **THEN** the call MUST throw `TestingLoadError` with `code === "TESTING_LOAD.DRIVER_UNAVAILABLE"` AND `context.driverName === "k6"`
+
+### Requirement: 100% Istanbul coverage and self-testability
+
+The package SHALL meet 100% Istanbul statement / branch / function / line coverage. The k6 driver SHALL accept injectable `exec` and `fetch` shims so the package's own test suite runs without a Docker daemon. The autocannon driver SHALL accept an injectable autocannon factory so the package's own test suite runs without performing real network requests.
+
+#### Scenario: Coverage gate at 100%
+
+- **WHEN** `bunx turbo test:coverage --filter=@suaveplan/testing-load` runs
+- **THEN** statement, branch, function, and line coverage MUST each be 100%
+
+#### Scenario: Tests pass without Docker
+
+- **WHEN** the package's test suite is invoked on a host with no Docker daemon and no network access
+- **THEN** every test MUST pass AND no test MUST attempt a real `docker` invocation or a real outbound network request

openspec/changes/enterprise-testing-load-perf/specs/testing/spec.md

@@ -0,0 +1,52 @@
+## ADDED Requirements
+
+### Requirement: End-to-end performance gate
+
+Every application under `apps/` SHALL ship at least one `*.smoke.load.ts` file exercising the `smoke` load preset (1 virtual user, 30 seconds) against the app's Docker staging stack. CI SHALL run every `*.smoke.load.ts` nightly. A threshold violation in any smoke scenario SHALL fail the build. This gate layers on top of — and does not replace — the existing mitata 10% micro-benchmark regression gate; both gates run independently and both must pass.
+
+#### Scenario: Every app has at least one smoke scenario
+
+- **WHEN** a script enumerates directories under `apps/` and looks for `tests/load/*.smoke.load.ts` (or equivalent path) in each
+- **THEN** every app directory MUST contain at least one matching file
+
+#### Scenario: Smoke threshold violation fails CI
+
+- **WHEN** the nightly CI job runs `apps/testbed/tests/load/healthz.smoke.load.ts` and the resulting summary reports `http_req_duration{p(95)} >= 500ms`
+- **THEN** the CI job MUST exit non-zero AND the failure summary MUST cite the failed threshold predicate and observed value
+
+#### Scenario: Mitata gate is unaffected
+
+- **WHEN** the smoke gate fails for `apps/testbed`
+- **THEN** the existing mitata 10%-regression gate for every `@suaveplan/*` package MUST still run independently AND its result MUST NOT be masked by the smoke-gate failure
+
+### Requirement: apps/testbed scenario coverage
+
+`apps/testbed` SHALL define one smoke load scenario per route declared in its `routes.ts` (or equivalent route manifest). The scenario SHALL target the route's HTTP method, path, and minimum-viable request body, and SHALL use the default thresholds from `@suaveplan/testing-load` unless the route documents a per-route override.
+
+#### Scenario: Every testbed route has a smoke scenario
+
+- **WHEN** the routes manifest in `apps/testbed/src/routes.ts` declares routes `[A, B, C]`
+- **THEN** `apps/testbed/tests/load/` MUST contain a `*.smoke.load.ts` covering each of A, B, and C AND each scenario MUST set `target` to the corresponding route URL
+
+#### Scenario: Per-route threshold override is documented
+
+- **WHEN** a testbed route requires a threshold looser than the `@suaveplan/testing-load` default
+- **THEN** the scenario file MUST set the override explicitly via `defineLoadScenario({ thresholds: { … } })` AND the route's documentation MUST justify the override
+
+### Requirement: Websocket-gateway connection-fan-out scenario
+
+The websocket-gateway app SHALL define a smoke load scenario that opens at least 10 concurrent websocket connections, sends a ping per connection, and asserts message round-trip latency below the smoke threshold. The scenario SHALL run nightly alongside the HTTP smoke scenarios.
+
+#### Scenario: Connection-fan-out scenario exists and runs nightly
+
+- **WHEN** CI's nightly load step runs against the websocket-gateway staging stack
+- **THEN** a `*.smoke.load.ts` MUST exercise at least 10 concurrent websocket connections AND the run MUST report message round-trip latency AND a violation of the configured round-trip threshold MUST fail the build
+
+### Requirement: Cross-link to release-readiness gate
+
+The smoke load-gate result SHALL be exposed as one of the inputs to the release-readiness gate (referenced in the 2026-04-25 audit §7). A red smoke gate SHALL block any "release ready" CI signal. The release-readiness gate definition itself is out of scope for this change but SHALL be authored in a follow-up change that consumes this signal.
+
+#### Scenario: Red smoke blocks release-readiness signal
+
+- **WHEN** any `*.smoke.load.ts` reports a threshold violation in the nightly run
+- **THEN** the release-readiness CI job (when present) MUST not emit a green signal AND the failure surface MUST cite the load gate as the cause