84 lines
6.0 KiB
Markdown
84 lines
6.0 KiB
Markdown
## Context
|
||
|
||
`mitata` benchmarks under `src/__benchmarks__/` already gate per-package micro-performance at the 10% regression floor (`openspec/specs/testing/spec.md` "Benchmark gate in CI"). This protects pure-function throughput but says nothing about request-response latency, concurrent-user behaviour, websocket fan-out, or the full Hono middleware stack of `apps/testbed`. The 2026-04-25 audit (§2) flags this as a high-severity gap and recommends k6 or autocannon as the remediation.
|
||
|
||
`apps/testbed` already ships a Docker staging environment and a Playwright sweep. Adding a load gate against the same staging environment closes the third side of the verification triangle: render correctness (Playwright), behavioural assertions (vitest), and performance under load (k6/autocannon).
|
||
|
||
## Goals / Non-Goals
|
||
|
||
**Goals:**
|
||
|
||
- A single, runner-agnostic public API (`defineLoadScenario`, `runLoadScenario`, `assertThresholds`) that callers use without knowing which driver runs underneath.
|
||
- k6 as primary driver via the `grafana/k6` testcontainers image; autocannon as a lighter Node-native alternative for environments without Docker (CI matrix on macOS/Windows runners that lack Docker support).
|
||
- Default thresholds tuned for typical Hono REST workloads: `http_req_duration{p(95)} < 500ms`, `http_req_failed < 1%`, `iteration_duration{p(99)} < 2000ms`. All overridable per scenario.
|
||
- Four run modes: `smoke` (gating, 1 vu / 30s), `load` (10 vu / 5m), `stress` (50 vu / 10m), `spike` (0→100 vu in 30s). Smoke runs nightly in CI; the others run weekly or on demand.
|
||
- 100% Istanbul coverage on the wrapper itself; the k6 binary invocation is mocked via injected spawn/fetch shims so the package tests run without Docker.
|
||
- Stable error contract: `TestingLoadError` with codes prefixed `TESTING_LOAD.*` (e.g. `TESTING_LOAD.THRESHOLD_VIOLATION`, `TESTING_LOAD.DRIVER_FAILED`, `TESTING_LOAD.CONFIG_INVALID`).
|
||
|
||
**Non-Goals:**
|
||
|
||
- Replacing or relaxing the mitata 10% micro-benchmark gate.
|
||
- Distributed load generation. k6 cloud, k6 operator on Kubernetes, and Locust are out of scope. Single-runner load is sufficient for the smoke gate.
|
||
- Custom k6 extensions (xk6). The wrapper consumes k6's stock features only.
|
||
- Load testing against production. All scenarios target the Docker staging environment defined by `apps/testbed/docker-compose.testbed.yml` (or equivalent per-app stack).
|
||
|
||
## Decisions
|
||
|
||
### D1: Runner-agnostic API with pluggable drivers
|
||
|
||
```ts
|
||
export interface LoadDriver {
|
||
name: "k6" | "autocannon";
|
||
run(scenario: LoadScenario, opts: LoadRunOptions): Promise<LoadSummary>;
|
||
}
|
||
|
||
export function defineLoadScenario(spec: LoadScenarioSpec): LoadScenario;
|
||
export function runLoadScenario(scenario: LoadScenario, opts?: LoadRunOptions): Promise<{ summary: LoadSummary; raw: unknown }>;
|
||
export function assertThresholds(summary: LoadSummary): void;
|
||
```
|
||
|
||
`LoadScenario` carries `{ name, target, vus, duration, thresholds, stages? }`. `LoadRunOptions` carries `{ driver?: LoadDriver; signal?: AbortSignal; onMetric?: (m) => void }`. The driver is injected; k6 is the default when present, autocannon otherwise.
|
||
|
||
### D2: k6 via testcontainers, autocannon via npm
|
||
|
||
k6 runs in the `grafana/k6` container, with the script generated from the `LoadScenario` and mounted into the container. Stdout is parsed via k6's structured-summary JSON output (`--summary-export`). Autocannon runs in-process via the npm package, with options mapped from the same `LoadScenario`. Both drivers populate the same `LoadSummary` shape so `assertThresholds` is driver-agnostic.
|
||
|
||
### D3: Threshold contract
|
||
|
||
Thresholds are RFC-style key→predicate maps:
|
||
|
||
```ts
|
||
export type Thresholds = Record<string, string[]>;
|
||
// e.g. { "http_req_duration": ["p(95)<500", "p(99)<1000"], "http_req_failed": ["rate<0.01"] }
|
||
```
|
||
|
||
Defaults applied if the caller omits the field; overrides merge on top. `assertThresholds(summary)` throws `TestingLoadError("TESTING_LOAD.THRESHOLD_VIOLATION", …, { context: { failed: [...] } })` listing every failed predicate.
|
||
|
||
### D4: Run modes as named presets
|
||
|
||
`smoke`, `load`, `stress`, `spike` are exported constants composing `vus` + `stages` + `duration`. Callers spread them into `defineLoadScenario`:
|
||
|
||
```ts
|
||
defineLoadScenario({ ...presets.smoke, name: "GET /healthz", target: "http://testbed:3000/healthz" });
|
||
```
|
||
|
||
### D5: Self-testability without Docker
|
||
|
||
The k6 driver accepts an injectable spawn function and an injectable `fetch`. Tests stub both to feed canned k6 JSON summaries; no Docker required for the wrapper's own test suite. Same pattern as `testing-integration-postgres`'s injectable runner function (per `CLAUDE.md` package listing).
|
||
|
||
### D6: Error shape parallels `@suaveplan/testing`
|
||
|
||
`TestingLoadError extends Error` with `{ code, cause?, context? }` and codes prefixed `TESTING_LOAD.*`. Shape-compatible with the existing `TestingError` so a future `@suaveplan/error` adapter bridges both at once. No `@suaveplan/error` dep on day one (keeps Layer-0 carve-out per CLAUDE.md rule 8).
|
||
|
||
### D7: App integration via `*.smoke.load.ts`
|
||
|
||
Each app under `apps/` ships at least one `*.smoke.load.ts` file. CI runs them nightly against the app's staging stack. `apps/testbed` enumerates one smoke scenario per route in `routes.ts`. The websocket-gateway app's smoke scenario opens N websocket connections and asserts message round-trip latency.
|
||
|
||
## Risks / Trade-offs
|
||
|
||
- **k6 binary availability.** Mitigated by the testcontainers image plus the autocannon fallback path.
|
||
- **CI wall-clock cost.** Smoke scenarios are 30s × N apps; load/stress/spike are out-of-band (weekly or on demand).
|
||
- **Threshold tuning.** Defaults may be too tight for some routes. Mitigated by per-scenario overrides; the smoke run is intended to catch regressions, not enforce SLOs.
|
||
- **Driver-summary impedance.** k6 and autocannon report different metric names. Mitigated by the wrapper normalising both into `LoadSummary` before threshold evaluation.
|
||
- **Layer-0 dep purity.** The package depends on `@suaveplan/types` only (parallel to `testing-integration*`). External catalog deps cover everything else.
|