Initial Commit

openspec/changes/enterprise-testing-operational/design.md

@@ -0,0 +1,42 @@
+## Design notes
+
+### Image pinning enforcement
+
+The verifier (`scripts/verify-image-pinning.ts`) walks every `docker-compose.*.yml`, every plugin's image-config TS source, and every reference inside testcontainers config helpers. It uses a strict regex `^[a-z0-9./-]+:[A-Za-z0-9._-]+@sha256:[a-f0-9]{64}$` and fails on any reference that does not match. Tag-only references are a hard error, not a warning, because partial enforcement is what allowed `axllent/mailpit:latest` to ship in the first place.
+
+Digest refresh follows a dependabot-style monthly cadence: a scheduled job opens a PR with the updated digests pulled from `docker manifest inspect`. CVE advisories from upstream (Mailpit, Redis, Postgres, SurrealDB, Garage, Typesense, Verdaccio) trigger immediate refresh outside the monthly window. The reviewer SHALL verify the upstream changelog before merge.
+
+### Resource limits — compose vs testcontainers
+
+Compose stacks declare limits under `deploy.resources.limits`. testcontainers does not honour the compose v3 schema, so plugin fixture configs accept `{ memoryMiB?: number; cpuCount?: number }` and translate to `HostConfig.Memory` (bytes) and `HostConfig.NanoCpus` (cpu × 1e9). Defaults are identical between the two paths so behaviour is consistent regardless of which entry point CI uses.
+
+Database services get the larger budget (2.0 cpu, 2Gi) because a parallel test sweep can drive Surreal/Postgres to its working-set limit. Garage and MinIO sit in the middle (1.5 cpu, 1.5Gi) because both have a memory-resident metadata index that grows with bucket size during long-running E2E runs.
+
+### CI matrix shape
+
+Three axes — runtime, OS, arch — multiplied conservatively because Playwright on Windows triples job time and arm64 is unsupported by some images we depend on. The plugin-side `archSupport` declaration is required so the matrix-generator script can prune unsupported cells before scheduling. `bun-canary` is allowed to fail-open because canary breakage is an upstream signal, not a merge blocker — but the warning is loud and surfaces in PR checks.
+
+The Node 22 LTS parity job runs only the universal-purity suite (no integration, no E2E). It catches accidental Bun-only API leakage in browser-runnable packages (`testing`, `testing-dom`, `crypto-utils`, etc.) which is otherwise invisible until a downstream consumer hits it.
+
+### Airgap rebuild
+
+The nightly `airgap-rebuild` job exists because enterprise consumers regularly run in environments without internet egress at build/test time. The job pre-warms a Bun cache via a normal `bun install` in a separate step, then runs `bun install --offline`, `bunx turbo build`, and a network-isolated test phase. Failure indicates a missing offline-cache entry — typically a transitive dep that pulled at runtime — and blocks the next release. We test the smallest changed-since-last-week filter (`...[HEAD~7]`) to keep the job under the 30-minute wall-clock budget.
+
+### Verdaccio fixture
+
+Verdaccio is the only "test-the-test-of-the-test" case where a `@suaveplan/*` package's primary use case (publishing) requires a fixture. We accept the meta-circularity: `testing-integration-verdaccio` does not depend on any `@suaveplan/*` package by Layer-0 rule, but `bun scripts/publish.ts` (a repo-level script, not a package) is exercised against it in CI.
+
+Memory storage is the default for speed; `tmpfs` is available for tests that need filesystem persistence semantics. Users and packages config follow Verdaccio's native YAML shape, translated from the TS config object at fixture startup.
+
+### Network sandboxing
+
+The default block is implemented via `page.route('**/*')` registered before each test. The allow-list is checked by URL origin (not full path) using the `URL` constructor; the auto-allow set is `localhost`, `127.0.0.1`, and the configured testbed origin from env. The `test.use({ allowExternalNetwork: true })` escape hatch logs a warning at suite start so the reviewer can challenge it during code review.
+
+### Leak detection
+
+After every `TestEnvironment.teardown()` the runner snapshots active handles. On Bun we use `Bun.peek` and the runtime's internal handle table; on Node fallback we use `process._getActiveHandles()`. The diff between pre-setup and post-teardown is the leak set. Categories: `HANDLE` (file descriptors, dns lookups), `TIMER` (setTimeout/setInterval), `SOCKET` (TCP/UDP). The error code is granular so triage can route to the responsible fixture.
+
+### Open questions
+
+- Whether to extend the airgap test to E2E (Playwright requires browser binaries that may not be in the offline cache by default).
+- Whether `quarantine.json` should live at repo root or under `openspec/`. Preference is repo root so CI tooling does not need OpenSpec parsing.

openspec/changes/enterprise-testing-operational/proposal.md

@@ -0,0 +1,79 @@
+## Why
+
+The 2026-04-25 test-suite audit (`/mnt/s/web/genesis/docs/superpowers/specs/2026-04-25-test-suite-audit.md`, sections 3, 4, 5) catalogued a class of gaps that the in-flight `enterprise-testing-integration` lift does not cover: the **operational hygiene** of the test infrastructure itself. Image pinning is by tag only (`axllent/mailpit:latest` is the worst offender); compose stacks declare no resource limits; CI runs on a single Bun version and a single architecture; there is no airgap rebuild gate; secrets are referenced inline in workflow YAML; testcontainers and compose stacks share no port-collision contract; flaky tests have no quarantine path; integration teardown does not assert handle/timer/socket leak freedom; E2E specs may reach the live internet by default; and the publish flow against Verdaccio (`https://npmjs.fq.io/`) has no fixture, so `bun scripts/publish.ts` cannot be exercised end-to-end before a real publish.
+
+Each gap is a near-miss for a CI outage or a non-reproducible release. Pinning by digest, resource caps, a Bun/OS/arch matrix, an airgap rebuild test, a Verdaccio fixture, secret-management policy, port pre-flight, OOM/timeout budgets, network sandboxing, and flake/leak detection are all standard at the enterprise tier but absent from the genesis testing substrate today. This change closes them in one coherent set of normative requirements.
+
+This change is **scoped narrowly** to operational hygiene. The pyramid-tier additions called out in audit §2 (a11y, visual regression, mutation, contract, load/perf) are out of scope and will land as separate OpenSpec changes per audit §"Recommended sequencing".
+
+## What Changes
+
+### Image policy and supply chain
+
+- Every image referenced in any `docker-compose.*.yml`, plugin spec, or testcontainers config SHALL be pinned by tag **and** sha256 digest (`image: foo/bar:v1.2.3@sha256:…`).
+- A repo-level script `scripts/verify-image-pinning.ts` SHALL fail CI on any tag-only reference.
+- `axllent/mailpit:latest` SHALL be replaced with a pinned `v1.x@sha256:…` form immediately.
+- Digest refresh policy is monthly dependabot-style bump with reviewer approval; CVE advisories trigger immediate refresh.
+
+### Container resource limits
+
+- Every service in `docker-compose.{dev,test,testbed}.yml` SHALL declare `deploy.resources.limits.{cpus,memory}`. Defaults: 1.0 cpu / 1Gi (services), 2.0 cpu / 2Gi (databases), 1.5 cpu / 1.5Gi (Garage/MinIO).
+- testcontainers-managed containers SHALL set `HostConfig.Memory` and `HostConfig.NanoCpus` via the existing config shape with the same defaults.
+
+### CI matrix and airgap
+
+- CI matrix: `bun-latest` × `bun-canary`, `ubuntu-22.04` × `ubuntu-24.04` × `macos-14` × `windows-latest` (Playwright-only on Windows), `amd64` × `arm64` (Linux only), with per-plugin `archSupport: ['amd64','arm64']` declaration to skip unsupported cells.
+- Node-fallback parity: a separate Node 22 LTS job runs the universal-purity suite for browser-runnable packages.
+- Bun version pin lives in root `package.json` `engines.bun`; `bun-canary` is fail-open with a warning.
+- Nightly `airgap-rebuild` job: `bun install --offline` from a pre-warmed cache, `bunx turbo build`, `bunx turbo test --filter=...[HEAD~7]` with images pre-pulled and `--network=none` during the test phase.
+
+### testing-integration-verdaccio fixture (new)
+
+- Image: `verdaccio/verdaccio:6@sha256:…`.
+- Env-bridge: `NPM_REGISTRY_URL`, `NPM_TOKEN`.
+- Healthcheck: HTTP `GET /-/ping`.
+- Config: `{ storage?: 'memory'|'tmpfs', users?, packages?, uplinks? }`.
+- Error prefix `INTEGRATION.VERDACCIO.*`.
+- Enables end-to-end testing of `bun scripts/publish.ts --package @suaveplan/<pkg>` in CI before real publish.
+
+### Secret management
+
+- CI secrets sourced from GitHub OIDC + a Vault/SOPS bridge. `GARAGE_S3_SECRET_ACCESS_KEY`, `OIDC_CLIENT_SECRET`, `NPM_TOKEN`, `PACT_BROKER_TOKEN` MUST be referenced via the bridge, never inline in workflow YAML.
+- `.env.example` documents each secret's CI vs local-dev source.
+- `scripts/scan-secrets.ts` (using `@suaveplan/security-codes` detectors) fails CI on inline secret patterns.
+
+### Port collision and budgets
+
+- Static port ranges (dev 8001-8902, test 7004-7011, testbed 3200) reserved for compose stacks; testcontainers integration tests use random ephemeral ports.
+- `docker/scripts/preflight.ts` fails loudly on bound reserved ports with a remediation hint and `PORTS_OFFSET` escape hatch.
+- Per-spec memory cap: 1Gi default, configurable via `vitest.config.ts` `pool.forks.maxMemory`.
+- Suite wall-clock ceiling: 30 min per `bunx turbo test`; soft warning at 20 min, abort at 30, no retries.
+
+### Network sandboxing (testing-e2e)
+
+- `createE2ETestConfig()` defaults to blocking external origins via `page.route`. `localhost`, `127.0.0.1`, and the configured testbed origin are auto-allowed. `allowedOrigins: string[]` and `test.use({ allowExternalNetwork: true })` are explicit opt-outs (the latter emits a warning).
+
+### Flake quarantine and leak detection (testing)
+
+- `markFlaky({ retries, reason, ticket })` wraps a test with bounded retries (max 3), emits `testing.flake.count` and `testing.flake.retry_used` metrics, requires a non-empty issue URL.
+- `quarantine.json` at repo root enumerates flaky tests with `{ path, name, ticket, addedAt, expiresAt }`; expired entries fail CI.
+- After every fixture teardown the runner asserts no leaked timers, sockets, or file descriptors. Leaks fail with `TESTING.LEAK.{HANDLE,TIMER,SOCKET}`.
+
+## Capabilities
+
+### New Capabilities
+
+- `testing-integration-verdaccio` — Verdaccio v6 ServiceFixture for testing the genesis publish path end-to-end.
+
+### Modified Capabilities
+
+- `testing-integration` — operational additions: image pinning, resource limits, CI matrix, airgap rebuild, secret policy, port pre-flight, OOM/wall-clock budgets, leak detection on teardown.
+- `testing` — flake quarantine + handle/timer/socket leak assertions. (Wave-1 lift may also touch this spec; this change layers on the operational specifics without conflict.)
+- `testing-e2e` — network sandboxing default plus `allowedOrigins` + `allowExternalNetwork` opt-outs.
+
+## Impact
+
+- **Affected code:** `docker-compose.{dev,test,testbed}.yml`, every `testing-integration-*` plugin's image config, `scripts/verify-image-pinning.ts` (new), `scripts/scan-secrets.ts` (new), `docker/scripts/preflight.ts` (new), `packages/testing/testing-integration-verdaccio/**` (new package), `packages/testing/testing-e2e/src/config/**`, `packages/testing/testing/src/quarantine/**` and `src/leak/**`, `quarantine.json` (new at repo root), `.env.example`, GitHub Actions workflow YAML.
+- **Affected consumers:** `bun scripts/publish.ts` gains an integration-test path; every existing testing-integration plugin gains digest pins and resource caps; every E2E spec inherits the network block (no source change required for the common case).
+- **Spec deltas:** ADDED requirements in `testing-integration` and `testing-integration-verdaccio` specs; ADDED + MODIFIED requirements in `testing` and `testing-e2e` specs.
+- **Pre-publish gate:** unchanged thresholds; coverage stays at 100% per `openspec/project.md`.

openspec/changes/enterprise-testing-operational/specs/testing-e2e/spec.md

@@ -0,0 +1,63 @@
+## ADDED Requirements
+
+### Requirement: Default network sandboxing
+
+`createE2ETestConfig()` SHALL default to blocking external origins via a per-test `page.route('**/*')` handler that aborts requests with `'blockedbyclient'` unless their origin is allow-listed. The auto-allowed set SHALL include `localhost`, `127.0.0.1`, and the configured testbed origin (read from the `TESTBED_ORIGIN` env var, falling back to the factory's `baseURL`). The factory SHALL accept an `allowedOrigins: string[]` option (default `[]`) which lifts the block per-origin. Origin matching SHALL be performed via `URL` constructor parsing on full origin (`<scheme>://<host>:<port>`), never substring or regex matching.
+
+#### Scenario: External origin blocked by default
+
+- **WHEN** a spec configured by `createE2ETestConfig({ baseURL: 'http://localhost:3200' })` issues a request to `https://example.com/data`
+- **THEN** the request is aborted with reason `'blockedbyclient'` AND the assertion library surfaces the abort as a network failure
+
+#### Scenario: localhost auto-allowed
+
+- **WHEN** a spec under the same config issues a request to `http://localhost:3200/api/users` or `http://127.0.0.1:9000/`
+- **THEN** the request proceeds normally AND no abort is raised
+
+#### Scenario: Testbed origin auto-allowed
+
+- **WHEN** `TESTBED_ORIGIN=https://testbed.internal:8443` is set in the environment
+- **THEN** requests to that exact origin proceed normally
+
+#### Scenario: allowedOrigins lifts the block
+
+- **WHEN** `createE2ETestConfig({ allowedOrigins: ['https://api.partner.test'] })` is used and a spec issues `GET https://api.partner.test/v1/foo`
+- **THEN** the request proceeds normally AND requests to any other external origin remain blocked
+
+#### Scenario: Origin matching is exact
+
+- **WHEN** `allowedOrigins: ['https://api.partner.test']` is set and a spec issues `GET https://evil.api.partner.test.attacker.example/`
+- **THEN** the request is blocked AND the substring overlap does NOT lift the block
+
+### Requirement: Spec-level escape hatch with warning
+
+The package SHALL expose a `test.use({ allowExternalNetwork: true })` Playwright fixture extension that disables the default block for the enclosing spec only. Use of this escape hatch SHALL emit a warning at suite start naming the offending spec file path, encouraging migration to explicit `allowedOrigins`. The escape hatch MUST be implemented as a Playwright `test.extend` fixture and MUST NOT mutate other specs' route handlers.
+
+#### Scenario: Escape hatch lifts the block locally
+
+- **WHEN** a spec declares `test.use({ allowExternalNetwork: true })` and issues a request to `https://example.com/`
+- **THEN** the request proceeds normally AND no abort is raised within that spec
+
+#### Scenario: Escape-hatch warning emitted
+
+- **WHEN** the suite starts and at least one spec opts into `allowExternalNetwork: true`
+- **THEN** a warning line is written to stderr naming each offending spec file path AND the warning persists in the test report
+
+#### Scenario: Escape hatch is spec-local
+
+- **WHEN** spec A declares `test.use({ allowExternalNetwork: true })` and spec B does not
+- **THEN** spec B's external requests are still blocked AND spec A's opt-in does not leak across
+
+### Requirement: Origin-matching algorithm is documented and tested
+
+The origin-matching algorithm used by the network sandbox SHALL be exported as a pure function (e.g. `isOriginAllowed(url, allowedOrigins, autoAllowed)`) so it is testable in isolation. The function MUST: parse `url` via the `URL` constructor and reject inputs that fail to parse with `TESTING_E2E.NETWORK.URL_INVALID`; compare on the `<scheme>://<host>:<port>` triple, treating the default port for the scheme as equivalent to an explicit port; return a boolean. fast-check property tests SHALL cover round-trip parsing and the substring-attack scenario above.
+
+#### Scenario: Default-port equivalence
+
+- **WHEN** `isOriginAllowed('https://api.partner.test/', ['https://api.partner.test:443'], [])` is called
+- **THEN** the function returns `true`
+
+#### Scenario: Invalid URL rejected
+
+- **WHEN** `isOriginAllowed('not a url', [], [])` is called
+- **THEN** the function throws `TESTING_E2E.NETWORK.URL_INVALID`

openspec/changes/enterprise-testing-operational/specs/testing-integration-verdaccio/spec.md

@@ -0,0 +1,75 @@
+## ADDED Requirements
+
+### Requirement: Verdaccio image and pinning
+
+`@suaveplan/testing-integration-verdaccio` SHALL ship a `VerdaccioFixture` ServiceFixture backed by `verdaccio/verdaccio:6@sha256:<digest>`. The image MUST be pinned by both tag and sha256 digest and MUST be tracked by `scripts/verify-image-pinning.ts`.
+
+#### Scenario: Image is pinned
+
+- **WHEN** the fixture's image config is read
+- **THEN** the value matches `^verdaccio/verdaccio:6@sha256:[a-f0-9]{64}$`
+
+### Requirement: Environment-variable bridge
+
+The fixture SHALL expose its endpoint and credentials to the test environment via the variables `NPM_REGISTRY_URL` and `NPM_TOKEN`. `NPM_REGISTRY_URL` MUST be the fully-qualified `http://<host>:<port>/` form. `NPM_TOKEN` MUST be a Verdaccio-compatible bearer token generated at fixture startup.
+
+#### Scenario: Env-bridge populated after start
+
+- **WHEN** `VerdaccioFixture.setup()` resolves
+- **THEN** `process.env.NPM_REGISTRY_URL` is a parseable URL with `protocol === 'http:'` AND `process.env.NPM_TOKEN` has length ≥ 32
+
+### Requirement: Healthcheck via /-/ping
+
+Readiness SHALL be determined by a polling HTTP `GET /-/ping` against the fixture's endpoint. The poller MUST respect `HealthPoller` defaults from `@suaveplan/testing-integration` and SHALL accept an injectable `fetch` implementation for unit tests.
+
+#### Scenario: Ping returns 200 → ready
+
+- **WHEN** `GET /-/ping` returns HTTP 200
+- **THEN** `HealthPoller` resolves and the fixture is considered ready
+
+#### Scenario: Ping never succeeds → timeout
+
+- **WHEN** `/-/ping` never returns 200 within the configured `healthTimeoutMs`
+- **THEN** `setup()` rejects with `INTEGRATION.VERDACCIO.HEALTH_TIMEOUT`
+
+### Requirement: Configuration shape
+
+The fixture SHALL accept a config object of shape `{ storage?: 'memory'|'tmpfs'; users?: Array<{ username: string; password: string }>; packages?: Array<{ name: string; access?: string; publish?: string }>; uplinks?: Record<string, string> }`. `storage` defaults to `'memory'`. The fixture SHALL translate this object into Verdaccio's native YAML config at startup. Invalid config MUST be rejected via Zod with a `INTEGRATION.VERDACCIO.CONFIG_INVALID` error.
+
+#### Scenario: Memory storage default
+
+- **WHEN** the fixture is started without a `storage` field
+- **THEN** Verdaccio runs with in-memory storage AND no host filesystem path is mounted
+
+#### Scenario: tmpfs storage
+
+- **WHEN** the fixture is started with `storage: 'tmpfs'`
+- **THEN** the container's storage path is backed by a tmpfs mount AND publish state survives within the fixture lifetime but not across restarts
+
+#### Scenario: User and package config translated
+
+- **WHEN** `users: [{ username: 'ci', password: 'p' }]` and `packages: [{ name: '@suaveplan/*', access: '$authenticated', publish: 'ci' }]` are supplied
+- **THEN** the rendered Verdaccio YAML contains an htpasswd entry for `ci` AND a `packages` block matching the input
+
+### Requirement: Error code prefix
+
+Every error thrown by `@suaveplan/testing-integration-verdaccio` SHALL extend the package's structured error class and carry a `code` of the form `INTEGRATION.VERDACCIO.<REASON>` (uppercase ASCII, dots as separators).
+
+#### Scenario: Coded error on health timeout
+
+- **WHEN** the fixture's healthcheck never succeeds
+- **THEN** the thrown error has `code === 'INTEGRATION.VERDACCIO.HEALTH_TIMEOUT'`
+
+#### Scenario: Coded error on invalid config
+
+- **WHEN** an invalid `users` array is supplied
+- **THEN** the thrown error has `code === 'INTEGRATION.VERDACCIO.CONFIG_INVALID'` AND `cause` is the underlying `ZodError`
+
+### Requirement: Publish flow integration test
+
+The fixture SHALL be exercisable end-to-end against `bun scripts/publish.ts --package @suaveplan/<pkg>` so the genesis publish path is testable in CI before any real publish to the production Verdaccio at `https://npmjs.fq.io/`. A reference integration test SHALL ship in the package demonstrating: fixture startup, env-bridge wiring, `bun scripts/publish.ts` invocation against the fixture, and verification of the published tarball via `npm view --registry $NPM_REGISTRY_URL`.
+
+#### Scenario: Publish round-trip
+
+- **WHEN** the integration test runs `bun scripts/publish.ts --package <probe>` against the fixture
+- **THEN** `npm view <probe>@0.0.0 --registry $NPM_REGISTRY_URL` returns the package metadata AND the tarball digest matches the locally built artefact

openspec/changes/enterprise-testing-operational/specs/testing-integration/spec.md

@@ -0,0 +1,128 @@
+## ADDED Requirements
+
+### Requirement: Image pinning by tag and digest
+
+Every Docker image referenced by any `testing-integration-*` plugin, by `docker-compose.{dev,test,testbed}.yml`, or by any testcontainers config helper SHALL be pinned by both tag AND sha256 digest in the canonical form `image: <registry>/<name>:<tag>@sha256:<64-hex>`. Tag-only pinning is a defect. The pin SHALL be verified by `scripts/verify-image-pinning.ts`, which fails CI on any reference that does not match `^[a-z0-9./-]+:[A-Za-z0-9._-]+@sha256:[a-f0-9]{64}$`. Digest refresh follows a monthly dependabot-style cadence with reviewer approval; security CVE advisories trigger immediate refresh outside the monthly window.
+
+#### Scenario: Tag-only image fails verification
+
+- **WHEN** a plugin or compose file declares `image: foo/bar:v1.2.3` without a `@sha256:` digest
+- **THEN** `scripts/verify-image-pinning.ts` exits non-zero AND CI fails the lint stage
+
+#### Scenario: Mailpit is pinned to v1.x with a digest
+
+- **WHEN** the Mailpit plugin or testbed compose stack references the Mailpit image
+- **THEN** the reference is `axllent/mailpit:v1.<minor>@sha256:<digest>` (never `:latest`)
+
+#### Scenario: CVE refresh bypasses monthly cadence
+
+- **WHEN** an upstream CVE advisory is filed against a pinned image
+- **THEN** a digest-bump PR SHALL be opened immediately AND the monthly cadence does not block it
+
+### Requirement: Container resource limits
+
+Every service in `docker-compose.{dev,test,testbed}.yml` SHALL declare `deploy.resources.limits.{cpus,memory}`. Defaults: 1.0 cpu / 1Gi for application services, 2.0 cpu / 2Gi for database services (Postgres, SurrealDB, Redis, Typesense), 1.5 cpu / 1.5Gi for object stores (Garage, MinIO). testcontainers-managed containers SHALL set `HostConfig.Memory` (bytes) and `HostConfig.NanoCpus` (cpu × 1e9) via the plugin config shape `{ memoryMiB?: number; cpuCount?: number }`, with the same defaults applied when fields are omitted.
+
+#### Scenario: Compose service lacking limits fails verification
+
+- **WHEN** a service in any tracked `docker-compose.*.yml` does not declare `deploy.resources.limits.cpus` AND `deploy.resources.limits.memory`
+- **THEN** the compose-lint step fails
+
+#### Scenario: testcontainer translates plugin budget to docker HostConfig
+
+- **WHEN** a plugin starts with `{ memoryMiB: 1024, cpuCount: 1.0 }`
+- **THEN** the resulting docker `HostConfig.Memory === 1073741824` AND `HostConfig.NanoCpus === 1_000_000_000`
+
+### Requirement: CI matrix on Bun, OS, and architecture
+
+CI SHALL execute on a matrix of `bun-latest` × `bun-canary`, `ubuntu-22.04` × `ubuntu-24.04` × `macos-14` × `windows-latest` (Playwright-only on Windows), and `amd64` × `arm64` on Linux. Each `testing-integration-*` plugin SHALL declare `archSupport: ('amd64'|'arm64')[]` and the matrix-generator SHALL skip unsupported cells. The canonical Bun version is pinned via root `package.json` `engines.bun`; the `bun-canary` job is allowed to fail-open with a warning annotation. A separate Node 22 LTS job SHALL run the universal-purity suite for browser-runnable packages (`testing`, `testing-dom`); failures here block merge.
+
+#### Scenario: Plugin without arm64 support skips arm64 cell
+
+- **WHEN** a plugin declares `archSupport: ['amd64']`
+- **THEN** the matrix-generator emits no `arm64` job for that plugin AND CI does not flag the absence as a failure
+
+#### Scenario: Bun canary fails
+
+- **WHEN** the `bun-canary` job exits non-zero
+- **THEN** the PR check surfaces a warning annotation AND merge is not blocked
+
+#### Scenario: Node parity fails
+
+- **WHEN** the Node 22 LTS parity job for browser-runnable packages exits non-zero
+- **THEN** merge is blocked
+
+### Requirement: Airgap rebuild gate
+
+A nightly workflow `airgap-rebuild` SHALL: pre-warm the Bun cache via a normal install, run `bun install --offline` from that cache, run `bunx turbo build`, then run `bunx turbo test --filter=...[HEAD~7]` with all required images pre-pulled and the test phase executed inside `--network=none`. Failure indicates a missing offline-cache entry and SHALL block the next release.
+
+#### Scenario: Offline install missing dep fails the job
+
+- **WHEN** any package pulls a dependency that is not present in the pre-warmed cache during `bun install --offline`
+- **THEN** the airgap job fails AND release is blocked until the offline cache is fixed
+
+#### Scenario: Test phase isolated from network
+
+- **WHEN** the airgap job enters the test phase
+- **THEN** the Docker network for the runner is `--network=none` AND any test that attempts external network egress fails fast
+
+### Requirement: Secret management policy
+
+CI secrets SHALL be sourced from GitHub OIDC plus a Vault or SOPS bridge. The variables `GARAGE_S3_SECRET_ACCESS_KEY`, `OIDC_CLIENT_SECRET`, `NPM_TOKEN`, and `PACT_BROKER_TOKEN` MUST be referenced via the bridge — never inline in workflow YAML. `.env.example` SHALL document each secret's CI source and its local-dev source. `scripts/scan-secrets.ts` SHALL fail CI on any inline secret pattern using `@suaveplan/security-codes` detectors.
+
+#### Scenario: Inline secret in workflow YAML fails CI
+
+- **WHEN** a workflow declares `env: NPM_TOKEN: <literal-token>` (any pattern detected by `@suaveplan/security-codes`)
+- **THEN** `scripts/scan-secrets.ts` exits non-zero AND CI fails
+
+#### Scenario: Bridge resolution
+
+- **WHEN** a workflow references `${{ secrets.NPM_TOKEN }}` via the OIDC + Vault/SOPS bridge step
+- **THEN** the secret is fetched at job start AND no inline value appears in the YAML source
+
+### Requirement: Port collision policy
+
+Static port ranges (dev 8001-8902, test 7004-7011, testbed 3200) are reserved for compose stacks. testcontainers-managed integration tests SHALL use random ephemeral ports (the testcontainers default). `docker/scripts/preflight.ts` SHALL fail loudly if any reserved port is already bound, with a remediation hint pointing the user at either killing the conflicting process or remapping via `PORTS_OFFSET`.
+
+#### Scenario: Bound reserved port aborts startup
+
+- **WHEN** `docker compose -f docker-compose.dev.yml up` runs and port 8001 is already bound by another process
+- **THEN** preflight exits non-zero AND prints the offending PID + a `PORTS_OFFSET` remediation hint
+
+#### Scenario: PORTS_OFFSET shift
+
+- **WHEN** `PORTS_OFFSET=100` is set in the environment
+- **THEN** every reserved port is shifted by 100 AND preflight passes if the shifted range is free
+
+### Requirement: OOM and suite wall-clock budgets
+
+Per-spec memory cap SHALL default to 1Gi via the shared `createTestConfig()`'s `pool.forks.maxMemory`; packages MAY override per-package. Per-`bunx turbo test` invocation wall-clock ceiling SHALL be 30 minutes; a soft warning SHALL be emitted at 20 minutes; the run SHALL be aborted at 30 minutes. No automatic retries on timeout — timeout indicates a pathological condition that must be diagnosed.
+
+#### Scenario: Single-spec OOM aborts that fork only
+
+- **WHEN** a single test spec exceeds 1Gi resident
+- **THEN** the forked vitest worker is killed with a `TESTING.MEMORY.LIMIT_EXCEEDED` error AND sibling specs continue
+
+#### Scenario: Wall-clock 30-minute abort
+
+- **WHEN** `bunx turbo test` exceeds 30 minutes wall-clock
+- **THEN** the run is aborted with a non-zero exit AND no automatic retry is attempted
+
+### Requirement: Leak detection on fixture teardown
+
+After every `TestEnvironment.teardown()` the runner SHALL assert no leaked timers, sockets, or file descriptors via `Bun.peek` / runtime handle table on Bun or `process._getActiveHandles()` on Node fallback. Leaks SHALL fail the suite with a categorised error: `TESTING.LEAK.HANDLE`, `TESTING.LEAK.TIMER`, or `TESTING.LEAK.SOCKET`, each carrying `{ count, sample }` context.
+
+#### Scenario: Unreleased timer leaks
+
+- **WHEN** a fixture sets `setInterval` and does not clear it before teardown
+- **THEN** teardown throws `TESTING.LEAK.TIMER` AND `context.count >= 1`
+
+#### Scenario: Unclosed socket leaks
+
+- **WHEN** a fixture opens a TCP socket and does not close it before teardown
+- **THEN** teardown throws `TESTING.LEAK.SOCKET` AND `context.sample` contains the offending socket descriptor
+
+#### Scenario: Clean teardown passes
+
+- **WHEN** all timers are cleared and all sockets are closed before teardown returns
+- **THEN** the leak assertion passes silently

openspec/changes/enterprise-testing-operational/specs/testing/spec.md

@@ -0,0 +1,81 @@
+## ADDED Requirements
+
+### Requirement: Flake quarantine via `markFlaky`
+
+The package SHALL export `markFlaky({ retries, reason, ticket })` that wraps a Vitest `it`/`test` callback with bounded retries (max 3). Each retry SHALL emit `testing.flake.retry_used` as an OTel counter with attributes `{ test, attempt }`; the wrapper SHALL emit `testing.flake.count` as a counter on initial registration. The `ticket` field MUST be a non-empty issue URL validated by Zod (`url()` + `.min(1)`); a missing or invalid ticket MUST throw `TESTING.FLAKE.TICKET_REQUIRED` at registration time, not at first run.
+
+#### Scenario: Bounded retries succeed
+
+- **WHEN** `markFlaky({ retries: 2, reason: 'docker DNS jitter', ticket: 'https://example.com/issues/42' })` wraps a test that fails once then passes
+- **THEN** the test reports green AND `testing.flake.retry_used` records exactly one increment
+
+#### Scenario: Retries above the cap rejected
+
+- **WHEN** `markFlaky({ retries: 5, ... })` is called
+- **THEN** `markFlaky` throws `TESTING.FLAKE.RETRIES_OUT_OF_RANGE` with `context.max === 3`
+
+#### Scenario: Missing ticket rejected at registration
+
+- **WHEN** `markFlaky({ retries: 1, reason: 'x', ticket: '' })` is called
+- **THEN** `markFlaky` throws `TESTING.FLAKE.TICKET_REQUIRED` immediately AND the test is not registered
+
+### Requirement: Quarantine ledger
+
+A `quarantine.json` file SHALL live at the repository root and enumerate every flaky test as `{ path: string; name: string; ticket: string; addedAt: ISO8601; expiresAt: ISO8601 }`. CI SHALL fail when any entry's `expiresAt` is in the past, forcing triage. The ledger schema MUST be exported as a Zod schema from the testing package so external tooling can validate it without re-implementing the shape.
+
+#### Scenario: Expired entry fails CI
+
+- **WHEN** `quarantine.json` contains an entry with `expiresAt` earlier than the current time
+- **THEN** the quarantine-lint step exits non-zero AND CI fails with a message naming the expired entries
+
+#### Scenario: Schema mismatch rejected
+
+- **WHEN** `quarantine.json` is missing the `ticket` field for any entry
+- **THEN** the loader throws `TESTING.QUARANTINE.SCHEMA_INVALID` AND the suite refuses to start
+
+#### Scenario: Active entry passes
+
+- **WHEN** every entry's `expiresAt` is in the future AND each entry validates against the Zod schema
+- **THEN** the quarantine-lint step passes silently
+
+### Requirement: Handle, timer, and socket leak detection on teardown
+
+After every fixture or `TestEnvironment.teardown()` call, the runner SHALL snapshot active timers, sockets, and file descriptors via `Bun.peek` / runtime handle table on Bun or `process._getActiveHandles()` on Node fallback. Any leak SHALL fail the suite with a categorised error: `TESTING.LEAK.HANDLE`, `TESTING.LEAK.TIMER`, or `TESTING.LEAK.SOCKET`, each carrying `{ count: number; sample: unknown[] }` context.
+
+#### Scenario: Timer leaked
+
+- **WHEN** a teardown returns while at least one `setInterval` is still active
+- **THEN** the runner throws `TESTING.LEAK.TIMER` with `context.count >= 1`
+
+#### Scenario: Socket leaked
+
+- **WHEN** a teardown returns while at least one TCP socket remains open
+- **THEN** the runner throws `TESTING.LEAK.SOCKET` AND `context.sample[0]` describes the offending descriptor
+
+#### Scenario: Clean teardown
+
+- **WHEN** all timers, sockets, and handles attributable to fixtures are released before teardown returns
+- **THEN** the leak assertion passes silently AND no error is thrown
+
+## MODIFIED Requirements
+
+### Requirement: Coverage and docs gates
+
+Istanbul coverage across `packages/universal/testing/src/` (excluding `*.bench.ts`, `*.schemas.ts`, `*.test.ts`, inner-dir `*.types.ts`, and barrel `index.ts`) SHALL be 100% on statements, branches, functions, and lines — no exceptions, no "foundation only" carve-out, no degraded-situation floor (per `openspec/project.md`). The pre-publish gate (`bunx turbo test:coverage --filter=@suaveplan/testing`) MUST hard-fail under 100%. Every implementation file MUST have a sibling `.md` of ≥ 200 non-code words containing both a simple AND an advanced usage example. Every test wrapped by `markFlaky` SHALL be exempted from coverage attribution to avoid retry-based double counting; the exemption MUST be recorded in the istanbul ignore-set automatically by the wrapper.
+
+This requirement replaces the previous "≥ 95%" formulation verbatim quoted here for traceability: "Istanbul statement + branch coverage across `packages/universal/testing/src/` (excluding `*.bench.ts` and `*.schemas.ts`) MUST be ≥ 95%."
+
+#### Scenario: Coverage gate in CI
+
+- **WHEN** `bunx turbo test:coverage --filter=@suaveplan/testing` runs
+- **THEN** reported statement, branch, function, and line coverage are each 100% AND the gate exits non-zero on anything less
+
+#### Scenario: Docs coverage enforced
+
+- **WHEN** a script enumerates implementation files under `src/` (excluding `*.test.ts`, `*.bench.ts`, `*.schemas.ts`, inner-dir `*.types.ts`, and barrel `index.ts`) and checks for sibling `.md` files
+- **THEN** every implementation file has a sibling `.md` AND every `.md` is ≥ 200 non-code words AND each `.md` contains both a simple and an advanced usage example
+
+#### Scenario: Flaky retries do not inflate coverage
+
+- **WHEN** a `markFlaky`-wrapped test executes its body twice (initial + one retry)
+- **THEN** the second execution does NOT contribute additional counts to istanbul AND the coverage report is identical to the single-execution case

openspec/changes/enterprise-testing-operational/tasks.md

@@ -0,0 +1,83 @@
+# Tasks
+
+## 0. Pre-flight (non-negotiable)
+
+- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
+- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
+- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
+
+## 1. Image pinning
+
+- [ ] 1.1 Author `scripts/verify-image-pinning.ts` (regex-based scan of compose files + plugin TS sources)
+- [ ] 1.2 Wire `verify-image-pinning` into the root `lint` turbo task so CI hard-fails on violation
+- [ ] 1.3 Replace `axllent/mailpit:latest` with `axllent/mailpit:v1.x@sha256:…` in the Mailpit plugin and `docker-compose.testbed.yml`
+- [ ] 1.4 Pin every other image referenced in `docker-compose.{dev,test,testbed}.yml` and in every `testing-integration-*` plugin
+- [ ] 1.5 Document monthly digest-refresh cadence + CVE-trigger policy in `docker/README.md`
+- [ ] 1.6 Add a scheduled GitHub Actions workflow that opens a digest-bump PR monthly
+
+## 2. Container resource limits
+
+- [ ] 2.1 Add `deploy.resources.limits.{cpus,memory}` to every service in `docker-compose.{dev,test,testbed}.yml` using the documented defaults
+- [ ] 2.2 Extend each `testing-integration-*` plugin's config shape with `memoryMiB?` + `cpuCount?` and translate to `HostConfig.Memory` / `HostConfig.NanoCpus`
+- [ ] 2.3 Set defaults: 1.0 cpu / 1Gi (services), 2.0 cpu / 2Gi (databases — Postgres, SurrealDB, Redis, Typesense), 1.5 cpu / 1.5Gi (Garage, MinIO)
+- [ ] 2.4 Add a unit test per plugin verifying the translated docker `HostConfig` matches the configured budget
+
+## 3. CI matrix and airgap
+
+- [ ] 3.1 Pin `engines.bun` in root `package.json` to the canonical genesis Bun version
+- [ ] 3.2 Define matrix axes in `.github/workflows/ci.yml`: bun (latest, canary), os (ubuntu-22.04, ubuntu-24.04, macos-14, windows-latest), arch (amd64, arm64 — linux only)
+- [ ] 3.3 Add `archSupport: ['amd64','arm64']` declaration to every `testing-integration-*` plugin and skip unsupported matrix cells
+- [ ] 3.4 Add a Node-22-LTS parity job running the universal-purity suite for browser-runnable packages
+- [ ] 3.5 Mark `bun-canary` job as `continue-on-error: true` with a warning annotation
+- [ ] 3.6 Author the nightly `airgap-rebuild` workflow (offline install → build → network-isolated test phase)
+- [ ] 3.7 Document the matrix and airgap policy in `docker/README.md`
+
+## 4. testing-integration-verdaccio
+
+- [ ] 4.1 Scaffold `packages/testing/testing-integration-verdaccio` package per Layer-0 rules (no `@suaveplan/*` runtime deps; testcontainers + zod + `@opentelemetry/api`)
+- [ ] 4.2 Implement `VerdaccioFixture` (image, env-bridge, healthcheck, config types)
+- [ ] 4.3 Pin `verdaccio/verdaccio:6@sha256:…` and add to image-pinning verifier
+- [ ] 4.4 Implement memory + tmpfs storage modes; users/packages/uplinks config translation
+- [ ] 4.5 Add `INTEGRATION.VERDACCIO.*` error codes
+- [ ] 4.6 Author integration test exercising `bun scripts/publish.ts --package @suaveplan/<probe>` against the fixture
+- [ ] 4.7 Add the fixture to the testbed compose stack as an optional service for manual repro
+- [ ] 4.8 100% Istanbul coverage gate
+
+## 5. Secret management
+
+- [ ] 5.1 Wire GitHub OIDC + Vault/SOPS bridge in `.github/workflows/_secrets.yml` (reusable workflow)
+- [ ] 5.2 Refactor every workflow that references `GARAGE_S3_SECRET_ACCESS_KEY`, `OIDC_CLIENT_SECRET`, `NPM_TOKEN`, `PACT_BROKER_TOKEN` to source from the bridge
+- [ ] 5.3 Author `scripts/scan-secrets.ts` using `@suaveplan/security-codes` detectors; fail CI on inline secret patterns
+- [ ] 5.4 Update `.env.example` to document each secret's CI source vs local-dev source
+
+## 6. Port collision and budgets
+
+- [ ] 6.1 Author `docker/scripts/preflight.ts` that probes reserved port ranges (dev 8001-8902, test 7004-7011, testbed 3200) and fails loudly on collision
+- [ ] 6.2 Implement `PORTS_OFFSET` env shift escape hatch
+- [ ] 6.3 Add per-spec memory cap (`pool.forks.maxMemory: 1Gi` default) to the shared `createTestConfig`
+- [ ] 6.4 Add suite wall-clock budget enforcement (20 min warn, 30 min abort) to the root turbo `test` task
+
+## 7. Network sandboxing (testing-e2e)
+
+- [ ] 7.1 Default-block external origins in `createE2ETestConfig` via `page.route('**/*')`
+- [ ] 7.2 Auto-allow `localhost`, `127.0.0.1`, and configured testbed origin
+- [ ] 7.3 Add `allowedOrigins: string[]` option (default `[]`)
+- [ ] 7.4 Add `test.use({ allowExternalNetwork: true })` escape hatch with a startup warning
+- [ ] 7.5 Tests covering all four code paths
+
+## 8. Flake quarantine and leak detection (testing)
+
+- [ ] 8.1 Implement `markFlaky({ retries, reason, ticket })` with bounded retries (max 3)
+- [ ] 8.2 Emit `testing.flake.count` and `testing.flake.retry_used` OTel metrics
+- [ ] 8.3 Validate ticket field is a non-empty issue URL (zod schema)
+- [ ] 8.4 Define `quarantine.json` schema (`{ path, name, ticket, addedAt, expiresAt }`) and load at suite start
+- [ ] 8.5 Fail CI on expired quarantine entries
+- [ ] 8.6 Implement post-teardown leak assertion: timers / sockets / handles
+- [ ] 8.7 Throw `TESTING.LEAK.{HANDLE,TIMER,SOCKET}` with the offending stack
+- [ ] 8.8 Tests covering each leak category
+
+## 9. Spec archival
+
+- [ ] 9.1 Run `openspec validate enterprise-testing-operational --strict`
+- [ ] 9.2 Fold deltas into `openspec/specs/{testing-integration,testing-integration-verdaccio,testing,testing-e2e}/spec.md`
+- [ ] 9.3 Move `openspec/changes/enterprise-testing-operational/` to `openspec/archive/<yyyy-mm-dd>-enterprise-testing-operational/`