Initial Commit
This commit is contained in:
83
openspec/changes/enterprise-testing-load-perf/design.md
Normal file
83
openspec/changes/enterprise-testing-load-perf/design.md
Normal file
@@ -0,0 +1,83 @@
|
||||
## Context
|
||||
|
||||
`mitata` benchmarks under `src/__benchmarks__/` already gate per-package micro-performance at the 10% regression floor (`openspec/specs/testing/spec.md` "Benchmark gate in CI"). This protects pure-function throughput but says nothing about request-response latency, concurrent-user behaviour, websocket fan-out, or the full Hono middleware stack of `apps/testbed`. The 2026-04-25 audit (§2) flags this as a high-severity gap and recommends k6 or autocannon as the remediation.
|
||||
|
||||
`apps/testbed` already ships a Docker staging environment and a Playwright sweep. Adding a load gate against the same staging environment closes the third side of the verification triangle: render correctness (Playwright), behavioural assertions (vitest), and performance under load (k6/autocannon).
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- A single, runner-agnostic public API (`defineLoadScenario`, `runLoadScenario`, `assertThresholds`) that callers use without knowing which driver runs underneath.
|
||||
- k6 as primary driver via the `grafana/k6` testcontainers image; autocannon as a lighter Node-native alternative for environments without Docker (CI matrix on macOS/Windows runners that lack Docker support).
|
||||
- Default thresholds tuned for typical Hono REST workloads: `http_req_duration{p(95)} < 500ms`, `http_req_failed < 1%`, `iteration_duration{p(99)} < 2000ms`. All overridable per scenario.
|
||||
- Four run modes: `smoke` (gating, 1 vu / 30s), `load` (10 vu / 5m), `stress` (50 vu / 10m), `spike` (0→100 vu in 30s). Smoke runs nightly in CI; the others run weekly or on demand.
|
||||
- 100% Istanbul coverage on the wrapper itself; the k6 binary invocation is mocked via injected spawn/fetch shims so the package tests run without Docker.
|
||||
- Stable error contract: `TestingLoadError` with codes prefixed `TESTING_LOAD.*` (e.g. `TESTING_LOAD.THRESHOLD_VIOLATION`, `TESTING_LOAD.DRIVER_FAILED`, `TESTING_LOAD.CONFIG_INVALID`).
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- Replacing or relaxing the mitata 10% micro-benchmark gate.
|
||||
- Distributed load generation. k6 cloud, k6 operator on Kubernetes, and Locust are out of scope. Single-runner load is sufficient for the smoke gate.
|
||||
- Custom k6 extensions (xk6). The wrapper consumes k6's stock features only.
|
||||
- Load testing against production. All scenarios target the Docker staging environment defined by `apps/testbed/docker-compose.testbed.yml` (or equivalent per-app stack).
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Runner-agnostic API with pluggable drivers
|
||||
|
||||
```ts
|
||||
export interface LoadDriver {
|
||||
name: "k6" | "autocannon";
|
||||
run(scenario: LoadScenario, opts: LoadRunOptions): Promise<LoadSummary>;
|
||||
}
|
||||
|
||||
export function defineLoadScenario(spec: LoadScenarioSpec): LoadScenario;
|
||||
export function runLoadScenario(scenario: LoadScenario, opts?: LoadRunOptions): Promise<{ summary: LoadSummary; raw: unknown }>;
|
||||
export function assertThresholds(summary: LoadSummary): void;
|
||||
```
|
||||
|
||||
`LoadScenario` carries `{ name, target, vus, duration, thresholds, stages? }`. `LoadRunOptions` carries `{ driver?: LoadDriver; signal?: AbortSignal; onMetric?: (m) => void }`. The driver is injected; k6 is the default when present, autocannon otherwise.
|
||||
|
||||
### D2: k6 via testcontainers, autocannon via npm
|
||||
|
||||
k6 runs in the `grafana/k6` container, with the script generated from the `LoadScenario` and mounted into the container. Stdout is parsed via k6's structured-summary JSON output (`--summary-export`). Autocannon runs in-process via the npm package, with options mapped from the same `LoadScenario`. Both drivers populate the same `LoadSummary` shape so `assertThresholds` is driver-agnostic.
|
||||
|
||||
### D3: Threshold contract
|
||||
|
||||
Thresholds are RFC-style key→predicate maps:
|
||||
|
||||
```ts
|
||||
export type Thresholds = Record<string, string[]>;
|
||||
// e.g. { "http_req_duration": ["p(95)<500", "p(99)<1000"], "http_req_failed": ["rate<0.01"] }
|
||||
```
|
||||
|
||||
Defaults applied if the caller omits the field; overrides merge on top. `assertThresholds(summary)` throws `TestingLoadError("TESTING_LOAD.THRESHOLD_VIOLATION", …, { context: { failed: [...] } })` listing every failed predicate.
|
||||
|
||||
### D4: Run modes as named presets
|
||||
|
||||
`smoke`, `load`, `stress`, `spike` are exported constants composing `vus` + `stages` + `duration`. Callers spread them into `defineLoadScenario`:
|
||||
|
||||
```ts
|
||||
defineLoadScenario({ ...presets.smoke, name: "GET /healthz", target: "http://testbed:3000/healthz" });
|
||||
```
|
||||
|
||||
### D5: Self-testability without Docker
|
||||
|
||||
The k6 driver accepts an injectable spawn function and an injectable `fetch`. Tests stub both to feed canned k6 JSON summaries; no Docker required for the wrapper's own test suite. Same pattern as `testing-integration-postgres`'s injectable runner function (per `CLAUDE.md` package listing).
|
||||
|
||||
### D6: Error shape parallels `@suaveplan/testing`
|
||||
|
||||
`TestingLoadError extends Error` with `{ code, cause?, context? }` and codes prefixed `TESTING_LOAD.*`. Shape-compatible with the existing `TestingError` so a future `@suaveplan/error` adapter bridges both at once. No `@suaveplan/error` dep on day one (keeps Layer-0 carve-out per CLAUDE.md rule 8).
|
||||
|
||||
### D7: App integration via `*.smoke.load.ts`
|
||||
|
||||
Each app under `apps/` ships at least one `*.smoke.load.ts` file. CI runs them nightly against the app's staging stack. `apps/testbed` enumerates one smoke scenario per route in `routes.ts`. The websocket-gateway app's smoke scenario opens N websocket connections and asserts message round-trip latency.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **k6 binary availability.** Mitigated by the testcontainers image plus the autocannon fallback path.
|
||||
- **CI wall-clock cost.** Smoke scenarios are 30s × N apps; load/stress/spike are out-of-band (weekly or on demand).
|
||||
- **Threshold tuning.** Defaults may be too tight for some routes. Mitigated by per-scenario overrides; the smoke run is intended to catch regressions, not enforce SLOs.
|
||||
- **Driver-summary impedance.** k6 and autocannon report different metric names. Mitigated by the wrapper normalising both into `LoadSummary` before threshold evaluation.
|
||||
- **Layer-0 dep purity.** The package depends on `@suaveplan/types` only (parallel to `testing-integration*`). External catalog deps cover everything else.
|
||||
41
openspec/changes/enterprise-testing-load-perf/proposal.md
Normal file
41
openspec/changes/enterprise-testing-load-perf/proposal.md
Normal file
@@ -0,0 +1,41 @@
|
||||
## Why
|
||||
|
||||
The 2026-04-25 test-suite audit (`/mnt/s/web/genesis/docs/superpowers/specs/2026-04-25-test-suite-audit.md` §2 row "No load/perf gate beyond `mitata` micro-bench") identifies an end-to-end performance gap. The canonical `openspec/specs/testing/spec.md` already enforces a mitata 10% benchmark regression gate, but mitata measures function-level micro-throughput in isolation. It does not exercise the request-response cycle, websocket fan-out, or concurrent-user behaviour of `apps/testbed`'s Hono routes or the websocket-gateway. A regression in middleware ordering, async-context propagation, or Drizzle query plans can pass every micro-benchmark while doubling p95 latency under realistic load.
|
||||
|
||||
CLAUDE.md (`/mnt/s/web/genesis/CLAUDE.md` §Web App & Page Verification) requires Docker-staging verification, full Playwright sweeps, and zero-defect-to-live before pushing client-side work. Audit §7 confirms `apps/testbed` only partially meets this bar and that "release readiness" gating is missing. End-to-end load and perf gates close the third missing piece (after audit §2's a11y and visual regression, scheduled separately): proving the system holds up under load, not just that it renders.
|
||||
|
||||
## What Changes
|
||||
|
||||
This change introduces a new `@suaveplan/testing-load` package and extends `openspec/specs/testing/spec.md` with an end-to-end performance gate that complements (does not replace) the existing mitata micro-benchmark gate.
|
||||
|
||||
### New `@suaveplan/testing-load` package
|
||||
|
||||
- Layout: `packages/testing/testing-load`. Zero `@suaveplan/*` runtime deps except `@suaveplan/types` (consistent with the rule-#9 carve-out per `CLAUDE.md` §Critical and the existing `testing-integration*` precedent).
|
||||
- Public API: `defineLoadScenario({ name, target, vus, duration, thresholds, stages? })`, `runLoadScenario(scenario, opts)` returning `{ summary, raw }`, `assertThresholds(summary)` throwing on violation.
|
||||
- Tooling: `k6` (binary, run via testcontainers) is the primary driver. `autocannon` (npm, catalog ref) is supported as a lighter Node-native alternative for environments without Docker.
|
||||
- Default thresholds: `http_req_duration{p(95)} < 500ms`, `http_req_failed < 1%`, `iteration_duration{p(99)} < 2000ms`. Per-scenario overrides supported.
|
||||
- Run modes: `smoke` (1 vu, 30s), `load` (10 vu, 5m), `stress` (50 vu, 10m), `spike` (0→100 vu in 30s).
|
||||
- Errors: `TestingLoadError` extending `Error` with `{ code, cause?, context? }` and stable `TESTING_LOAD.*` codes. Shape-compatible with `@suaveplan/error` for a future adapter, matching the testing-package precedent in `openspec/specs/testing/spec.md`.
|
||||
- Quality: 100% Istanbul coverage on the wrapper, sibling `.md` per implementation file, Zod schemas in sibling `*.schemas.ts`, mitata bench file, fast-check property tests where applicable.
|
||||
|
||||
### Extension to `openspec/specs/testing/spec.md`
|
||||
|
||||
A new requirement layered on the existing Benchmark gate: every app under `apps/` SHALL ship a `*.smoke.load.ts` exercising the smoke scenario nightly, failing on threshold violation. `apps/testbed` SHALL define a load scenario per route declared in its `routes.ts`. The websocket-gateway SHALL define a connection-fan-out scenario. The release-readiness gate (cross-referenced from audit §7) consumes the smoke result as one of its inputs.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `testing-load` — end-to-end load and performance scenarios with k6/autocannon drivers, threshold assertions, structured errors, and Zod-validated configs. Layer 0 utility (no `@suaveplan/*` runtime deps except `@suaveplan/types`).
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `testing` — adds an end-to-end performance gate requirement for every app under `apps/`, layered on top of the existing mitata 10% micro-benchmark gate. Does not modify or relax the mitata gate.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Affected code:** new `packages/testing/testing-load/` package; per-app `*.smoke.load.ts` files added under each `apps/<name>/`. `apps/testbed` gains one scenario per route in `routes.ts`. The `websocket-gateway` app gains a connection-fan-out scenario.
|
||||
- **Affected catalogs:** root `package.json` gains `autocannon` under `catalogs.testing`. `k6` is run via the `grafana/k6` testcontainers image (no npm catalog entry needed).
|
||||
- **CI cadence:** smoke scenarios run nightly per app. Load/stress/spike scenarios run weekly or on-demand against staging.
|
||||
- **Coverage:** wrapper meets 100% Istanbul; integration with k6 binary mocked via injected exec/fetch shims to keep the package self-testable without Docker.
|
||||
- **No replacement:** the existing mitata 10% regression gate stays in place. Both gates run; both gates must pass.
|
||||
@@ -0,0 +1,109 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Package layout and dependency policy
|
||||
|
||||
The `@suaveplan/testing-load` package SHALL live at `packages/testing/testing-load/`. Its `dependencies`, `peerDependencies`, and `devDependencies` MUST NOT list any `@suaveplan/*` workspace package other than `@suaveplan/types`. External deps SHALL be referenced via catalog entries (no bare version strings). The package SHALL declare `"sideEffects": false` and SHALL provide a subpath export per feature module per `openspec/AGENTS.md` §6.
|
||||
|
||||
#### Scenario: Workspace deps are limited to @suaveplan/types
|
||||
|
||||
- **WHEN** `bunx turbo ls --filter=@suaveplan/testing-load...` is run
|
||||
- **THEN** the only `@suaveplan/*` package in the forward closure MUST be `@suaveplan/types`
|
||||
|
||||
#### Scenario: No bare versions
|
||||
|
||||
- **WHEN** the package's `package.json` `dependencies` and `devDependencies` are scanned
|
||||
- **THEN** every external dep MUST resolve to a `catalog:<key>` reference AND no bare semver string MUST appear
|
||||
|
||||
### Requirement: Public API — defineLoadScenario, runLoadScenario, assertThresholds
|
||||
|
||||
The package SHALL export `defineLoadScenario(spec)` returning a frozen `LoadScenario`, `runLoadScenario(scenario, opts?)` returning `Promise<{ summary: LoadSummary; raw: unknown }>`, and `assertThresholds(summary)` returning `void` (throws on violation). Inputs SHALL be Zod-validated at the boundary; outputs SHALL conform to the `LoadSummary` shape regardless of which driver produced them. Every public function SHALL accept an optional `{ signal?: AbortSignal }` and SHALL respect cancellation at bounded checkpoints.
|
||||
|
||||
#### Scenario: defineLoadScenario validates input
|
||||
|
||||
- **WHEN** `defineLoadScenario({ name: "x", target: "not-a-url", vus: -1, duration: "30s" })` is called
|
||||
- **THEN** it MUST throw `TestingLoadError` with `code === "TESTING_LOAD.CONFIG_INVALID"` AND `context.issues` MUST list the offending fields
|
||||
|
||||
#### Scenario: runLoadScenario honours AbortSignal
|
||||
|
||||
- **WHEN** `runLoadScenario(scenario, { signal })` is awaited and `signal.abort()` is called mid-run
|
||||
- **THEN** the promise MUST reject with `TestingLoadError` where `code === "TESTING_LOAD.ABORTED"` AND `cause === signal.reason`
|
||||
|
||||
#### Scenario: assertThresholds enumerates failed predicates
|
||||
|
||||
- **WHEN** a `LoadSummary` reports `http_req_duration{p(95)} = 750` against threshold `p(95)<500`
|
||||
- **THEN** `assertThresholds(summary)` MUST throw `TestingLoadError` with `code === "TESTING_LOAD.THRESHOLD_VIOLATION"` AND `context.failed` MUST list the predicate `"http_req_duration: p(95)<500 (observed 750)"`
|
||||
|
||||
### Requirement: Driver pluggability — k6 primary, autocannon fallback
|
||||
|
||||
The package SHALL expose a `LoadDriver` interface and SHALL ship two implementations: a k6 driver invoking `grafana/k6` via testcontainers, and an autocannon driver running in-process via the `autocannon` npm package. `runLoadScenario` SHALL default to the k6 driver when reachable and SHALL fall back to autocannon when k6 is unavailable. Both drivers SHALL normalise their output into the same `LoadSummary` shape so callers and `assertThresholds` are driver-agnostic.
|
||||
|
||||
#### Scenario: k6 selected when available
|
||||
|
||||
- **WHEN** `runLoadScenario(scenario)` is called without an explicit driver AND a reachable Docker daemon is detected AND the `grafana/k6` image can be pulled
|
||||
- **THEN** the k6 driver MUST be selected AND the resulting `summary.driverName` MUST equal `"k6"`
|
||||
|
||||
#### Scenario: autocannon fallback when k6 unavailable
|
||||
|
||||
- **WHEN** `runLoadScenario(scenario)` is called without an explicit driver AND no Docker daemon is reachable
|
||||
- **THEN** the autocannon driver MUST be selected AND the resulting `summary.driverName` MUST equal `"autocannon"`
|
||||
|
||||
#### Scenario: Both drivers produce LoadSummary
|
||||
|
||||
- **WHEN** the same `LoadScenario` is executed once via the k6 driver and once via the autocannon driver
|
||||
- **THEN** both runs MUST return a `LoadSummary` with the same set of metric keys (e.g. `http_req_duration`, `http_req_failed`, `iteration_duration`) AND `assertThresholds` MUST evaluate identically against either summary
|
||||
|
||||
### Requirement: Default thresholds and per-scenario overrides
|
||||
|
||||
The package SHALL apply default thresholds when the caller omits the `thresholds` field on a `LoadScenario`: `http_req_duration{p(95)} < 500ms`, `http_req_failed < 1%` (rate < 0.01), `iteration_duration{p(99)} < 2000ms`. Caller-supplied thresholds SHALL merge on top of (not replace) the defaults; an explicit `thresholds: {}` SHALL retain the defaults; callers SHALL be able to override a single key without restating the rest.
|
||||
|
||||
#### Scenario: Defaults applied when omitted
|
||||
|
||||
- **WHEN** `defineLoadScenario({ name: "x", target: "http://t/", vus: 1, duration: "30s" })` is called and the resulting scenario is inspected
|
||||
- **THEN** the resolved `thresholds` MUST contain `http_req_duration: ["p(95)<500"]` AND `http_req_failed: ["rate<0.01"]` AND `iteration_duration: ["p(99)<2000"]`
|
||||
|
||||
#### Scenario: Single-key override merges with defaults
|
||||
|
||||
- **WHEN** `defineLoadScenario({ ..., thresholds: { http_req_duration: ["p(95)<200"] } })` is called
|
||||
- **THEN** the resolved scenario's `http_req_duration` MUST equal `["p(95)<200"]` AND the other default keys MUST remain present unchanged
|
||||
|
||||
### Requirement: Run-mode presets — smoke, load, stress, spike
|
||||
|
||||
The package SHALL export named run-mode presets composable with `defineLoadScenario`: `smoke` (1 vu, 30s), `load` (10 vu, 5m), `stress` (50 vu, 10m), `spike` (stages from 0 to 100 vu over 30s). Presets SHALL be plain objects spreadable into `defineLoadScenario`'s spec argument; callers SHALL be able to override any preset field on a per-scenario basis.
|
||||
|
||||
#### Scenario: smoke preset values
|
||||
|
||||
- **WHEN** the `smoke` preset is inspected
|
||||
- **THEN** `vus === 1` AND `duration === "30s"` AND `stages` MUST be undefined
|
||||
|
||||
#### Scenario: spike preset values
|
||||
|
||||
- **WHEN** the `spike` preset is inspected
|
||||
- **THEN** `stages` MUST describe a ramp from `0` to `100` virtual users over `30s` AND the total scenario duration MUST equal the sum of stage durations
|
||||
|
||||
### Requirement: Structured errors with TESTING_LOAD.* codes
|
||||
|
||||
The package SHALL throw only subclasses of `TestingLoadError extends Error` with stable codes prefixed `TESTING_LOAD.*`. Errors MUST carry `{ code, cause?, context? }` fields. The package MUST NOT throw plain `Error` or `TypeError` from any public code path. Defined codes include `TESTING_LOAD.CONFIG_INVALID`, `TESTING_LOAD.DRIVER_FAILED`, `TESTING_LOAD.DRIVER_UNAVAILABLE`, `TESTING_LOAD.THRESHOLD_VIOLATION`, and `TESTING_LOAD.ABORTED`.
|
||||
|
||||
#### Scenario: Driver failure produces coded error
|
||||
|
||||
- **WHEN** the k6 driver's container exits non-zero
|
||||
- **THEN** the driver MUST throw `TestingLoadError` with `code === "TESTING_LOAD.DRIVER_FAILED"` AND `context.driverName === "k6"` AND `context.exitCode` MUST equal the observed exit code
|
||||
|
||||
#### Scenario: Driver unavailable surfaces a typed error
|
||||
|
||||
- **WHEN** `runLoadScenario` is called with `driver: "k6"` explicitly AND no Docker daemon is reachable
|
||||
- **THEN** the call MUST throw `TestingLoadError` with `code === "TESTING_LOAD.DRIVER_UNAVAILABLE"` AND `context.driverName === "k6"`
|
||||
|
||||
### Requirement: 100% Istanbul coverage and self-testability
|
||||
|
||||
The package SHALL meet 100% Istanbul statement / branch / function / line coverage. The k6 driver SHALL accept injectable `exec` and `fetch` shims so the package's own test suite runs without a Docker daemon. The autocannon driver SHALL accept an injectable autocannon factory so the package's own test suite runs without performing real network requests.
|
||||
|
||||
#### Scenario: Coverage gate at 100%
|
||||
|
||||
- **WHEN** `bunx turbo test:coverage --filter=@suaveplan/testing-load` runs
|
||||
- **THEN** statement, branch, function, and line coverage MUST each be 100%
|
||||
|
||||
#### Scenario: Tests pass without Docker
|
||||
|
||||
- **WHEN** the package's test suite is invoked on a host with no Docker daemon and no network access
|
||||
- **THEN** every test MUST pass AND no test MUST attempt a real `docker` invocation or a real outbound network request
|
||||
@@ -0,0 +1,52 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: End-to-end performance gate
|
||||
|
||||
Every application under `apps/` SHALL ship at least one `*.smoke.load.ts` file exercising the `smoke` load preset (1 virtual user, 30 seconds) against the app's Docker staging stack. CI SHALL run every `*.smoke.load.ts` nightly. A threshold violation in any smoke scenario SHALL fail the build. This gate layers on top of — and does not replace — the existing mitata 10% micro-benchmark regression gate; both gates run independently and both must pass.
|
||||
|
||||
#### Scenario: Every app has at least one smoke scenario
|
||||
|
||||
- **WHEN** a script enumerates directories under `apps/` and looks for `tests/load/*.smoke.load.ts` (or equivalent path) in each
|
||||
- **THEN** every app directory MUST contain at least one matching file
|
||||
|
||||
#### Scenario: Smoke threshold violation fails CI
|
||||
|
||||
- **WHEN** the nightly CI job runs `apps/testbed/tests/load/healthz.smoke.load.ts` and the resulting summary reports `http_req_duration{p(95)} >= 500ms`
|
||||
- **THEN** the CI job MUST exit non-zero AND the failure summary MUST cite the failed threshold predicate and observed value
|
||||
|
||||
#### Scenario: Mitata gate is unaffected
|
||||
|
||||
- **WHEN** the smoke gate fails for `apps/testbed`
|
||||
- **THEN** the existing mitata 10%-regression gate for every `@suaveplan/*` package MUST still run independently AND its result MUST NOT be masked by the smoke-gate failure
|
||||
|
||||
### Requirement: apps/testbed scenario coverage
|
||||
|
||||
`apps/testbed` SHALL define one smoke load scenario per route declared in its `routes.ts` (or equivalent route manifest). The scenario SHALL target the route's HTTP method, path, and minimum-viable request body, and SHALL use the default thresholds from `@suaveplan/testing-load` unless the route documents a per-route override.
|
||||
|
||||
#### Scenario: Every testbed route has a smoke scenario
|
||||
|
||||
- **WHEN** the routes manifest in `apps/testbed/src/routes.ts` declares routes `[A, B, C]`
|
||||
- **THEN** `apps/testbed/tests/load/` MUST contain a `*.smoke.load.ts` covering each of A, B, and C AND each scenario MUST set `target` to the corresponding route URL
|
||||
|
||||
#### Scenario: Per-route threshold override is documented
|
||||
|
||||
- **WHEN** a testbed route requires a threshold looser than the `@suaveplan/testing-load` default
|
||||
- **THEN** the scenario file MUST set the override explicitly via `defineLoadScenario({ thresholds: { … } })` AND the route's documentation MUST justify the override
|
||||
|
||||
### Requirement: Websocket-gateway connection-fan-out scenario
|
||||
|
||||
The websocket-gateway app SHALL define a smoke load scenario that opens at least 10 concurrent websocket connections, sends a ping per connection, and asserts message round-trip latency below the smoke threshold. The scenario SHALL run nightly alongside the HTTP smoke scenarios.
|
||||
|
||||
#### Scenario: Connection-fan-out scenario exists and runs nightly
|
||||
|
||||
- **WHEN** CI's nightly load step runs against the websocket-gateway staging stack
|
||||
- **THEN** a `*.smoke.load.ts` MUST exercise at least 10 concurrent websocket connections AND the run MUST report message round-trip latency AND a violation of the configured round-trip threshold MUST fail the build
|
||||
|
||||
### Requirement: Cross-link to release-readiness gate
|
||||
|
||||
The smoke load-gate result SHALL be exposed as one of the inputs to the release-readiness gate (referenced in the 2026-04-25 audit §7). A red smoke gate SHALL block any "release ready" CI signal. The release-readiness gate definition itself is out of scope for this change but SHALL be authored in a follow-up change that consumes this signal.
|
||||
|
||||
#### Scenario: Red smoke blocks release-readiness signal
|
||||
|
||||
- **WHEN** any `*.smoke.load.ts` reports a threshold violation in the nightly run
|
||||
- **THEN** the release-readiness CI job (when present) MUST not emit a green signal AND the failure surface MUST cite the load gate as the cause
|
||||
73
openspec/changes/enterprise-testing-load-perf/tasks.md
Normal file
73
openspec/changes/enterprise-testing-load-perf/tasks.md
Normal file
@@ -0,0 +1,73 @@
|
||||
|
||||
|
||||
## 0. Pre-flight (non-negotiable)
|
||||
|
||||
- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
|
||||
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
|
||||
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
|
||||
|
||||
## 1. Catalog setup
|
||||
|
||||
- [ ] 1.1 Add `autocannon` to root `package.json` `catalogs.testing` (k6 runs via the `grafana/k6` container; no npm entry needed)
|
||||
- [ ] 1.2 Run `bun install`; confirm lockfile updates cleanly
|
||||
|
||||
## 2. Package skeleton — `@suaveplan/testing-load`
|
||||
|
||||
- [ ] 2.1 Create `packages/testing/testing-load/` with `package.json` (version `0.1.0`, `type: "module"`, `sideEffects: false`, `@suaveplan/types` workspace dep, external catalog deps only)
|
||||
- [ ] 2.2 Create `tsconfig.lib.json` extending `@suaveplan/typescript-config/library.json`
|
||||
- [ ] 2.3 Create `vite.config.ts` (preserveModules, ESM only)
|
||||
- [ ] 2.4 Create `vitest.config.ts` consuming `@suaveplan/testing/config` once available; thresholds at 100/100/100/100 (istanbul)
|
||||
- [ ] 2.5 Create `README.md` with purpose, install, quick-start, error-code index, links to module `.md` files
|
||||
- [ ] 2.6 Subpath exports in `package.json` for every feature module per `openspec/AGENTS.md` §6
|
||||
|
||||
## 3. Public API — scenario, driver, runner
|
||||
|
||||
- [ ] 3.1 `src/scenario/scenario.ts` — `defineLoadScenario`, `LoadScenario`, `LoadScenarioSpec`; Zod schemas in `scenario.schemas.ts`
|
||||
- [ ] 3.2 `src/scenario/presets.ts` — `smoke`, `load`, `stress`, `spike` constants composing vus/stages/duration
|
||||
- [ ] 3.3 `src/driver/driver.ts` — `LoadDriver` interface, `LoadSummary` normalised shape
|
||||
- [ ] 3.4 `src/driver/k6-driver.ts` — k6 implementation; injectable `exec` (testcontainers) + `fetch` for self-testability
|
||||
- [ ] 3.5 `src/driver/autocannon-driver.ts` — autocannon implementation; in-process Node fallback
|
||||
- [ ] 3.6 `src/runner/run-load-scenario.ts` — `runLoadScenario(scenario, opts)`; resolves driver default to k6 if reachable, autocannon otherwise; honours `AbortSignal`
|
||||
- [ ] 3.7 `src/thresholds/assert-thresholds.ts` — `assertThresholds(summary)`; throws `TestingLoadError("TESTING_LOAD.THRESHOLD_VIOLATION", …)` listing every failed predicate
|
||||
|
||||
## 4. Errors and observability
|
||||
|
||||
- [ ] 4.1 `src/errors/testing-load-error.ts` — `TestingLoadError extends Error` with `{ code, cause?, context? }`
|
||||
- [ ] 4.2 `src/errors/codes.ts` — `TESTING_LOAD.THRESHOLD_VIOLATION`, `TESTING_LOAD.DRIVER_FAILED`, `TESTING_LOAD.CONFIG_INVALID`, `TESTING_LOAD.DRIVER_UNAVAILABLE`, `TESTING_LOAD.ABORTED`
|
||||
- [ ] 4.3 OTel counters/histograms via `@opentelemetry/api` per existing `testing/spec.md` pattern; spans for `runLoadScenario` only
|
||||
|
||||
## 5. Tests, docs, schemas
|
||||
|
||||
- [ ] 5.1 Co-locate `*.test.ts` per implementation file; 100% Istanbul coverage (statements / branches / functions / lines)
|
||||
- [ ] 5.2 Co-locate `*.md` per implementation file (≥ 200 words, simple + advanced examples, API reference, implementation notes) per `openspec/AGENTS.md` §4
|
||||
- [ ] 5.3 Co-locate `*.schemas.ts` per implementation file; tree-shake test proving schemas don't bloat functional bundle
|
||||
- [ ] 5.4 Property tests (fast-check) for `assertThresholds` predicate evaluation, scenario merge associativity
|
||||
- [ ] 5.5 Mitata benchmarks under `src/__benchmarks__/` for scenario-build and threshold-evaluation hot paths
|
||||
|
||||
## 6. App integration — `*.smoke.load.ts`
|
||||
|
||||
- [ ] 6.1 `apps/testbed/tests/load/healthz.smoke.load.ts` — smoke scenario for `GET /healthz`
|
||||
- [ ] 6.2 One `*.smoke.load.ts` per route declared in `apps/testbed/src/routes.ts`
|
||||
- [ ] 6.3 `apps/websocket-gateway/tests/load/connection.smoke.load.ts` — websocket-fan-out smoke scenario
|
||||
- [ ] 6.4 Each app's `package.json` exposes `test:load:smoke` invoking its smoke files via the package's own runner
|
||||
|
||||
## 7. CI integration
|
||||
|
||||
- [ ] 7.1 Nightly job: run every `*.smoke.load.ts` against the app's Docker staging stack
|
||||
- [ ] 7.2 Failure on any threshold violation breaks the build
|
||||
- [ ] 7.3 Weekly job (or manual workflow_dispatch): run `load`, `stress`, `spike` modes against staging
|
||||
- [ ] 7.4 Cross-link smoke result into the release-readiness gate (audit §7) once that gate lands
|
||||
|
||||
## 8. Spec deltas and validation
|
||||
|
||||
- [ ] 8.1 `openspec validate enterprise-testing-load-perf --type change` passes
|
||||
- [ ] 8.2 Fold `specs/testing-load/spec.md` ADDED Requirements into a new `openspec/specs/testing-load/spec.md`
|
||||
- [ ] 8.3 Fold `specs/testing/spec.md` ADDED Requirement into the existing `openspec/specs/testing/spec.md`
|
||||
- [ ] 8.4 Move this change folder to `openspec/archive/<yyyy-mm-dd>-enterprise-testing-load-perf/`
|
||||
|
||||
## 9. Pre-publish gate and publish
|
||||
|
||||
- [ ] 9.1 `bunx turbo lint typecheck test --filter=@suaveplan/testing-load` green
|
||||
- [ ] 9.2 `bunx turbo test:coverage --filter=@suaveplan/testing-load` reports 100% statements / branches / functions / lines
|
||||
- [ ] 9.3 `bun scripts/publish.ts --package @suaveplan/testing-load --dry-run` produces a sane diff
|
||||
- [ ] 9.4 `bun scripts/publish.ts --package @suaveplan/testing-load` — publish `0.1.0` to Verdaccio
|
||||
Reference in New Issue
Block a user