suaveplan/openspec/changes/enterprise-testing-operational/proposal.md

Why

The 2026-04-25 test-suite audit (/mnt/s/web/genesis/docs/superpowers/specs/2026-04-25-test-suite-audit.md, sections 3, 4, 5) catalogued a class of gaps that the in-flight enterprise-testing-integration lift does not cover: the operational hygiene of the test infrastructure itself. Image pinning is by tag only (axllent/mailpit:latest is the worst offender); compose stacks declare no resource limits; CI runs on a single Bun version and a single architecture; there is no airgap rebuild gate; secrets are referenced inline in workflow YAML; testcontainers and compose stacks share no port-collision contract; flaky tests have no quarantine path; integration teardown does not assert handle/timer/socket leak freedom; E2E specs may reach the live internet by default; and the publish flow against Verdaccio (https://npmjs.fq.io/) has no fixture, so bun scripts/publish.ts cannot be exercised end-to-end before a real publish.

Each gap is a near-miss for a CI outage or a non-reproducible release. Pinning by digest, resource caps, a Bun/OS/arch matrix, an airgap rebuild test, a Verdaccio fixture, secret-management policy, port pre-flight, OOM/timeout budgets, network sandboxing, and flake/leak detection are all standard at the enterprise tier but absent from the genesis testing substrate today. This change closes them in one coherent set of normative requirements.

This change is scoped narrowly to operational hygiene. The pyramid-tier additions called out in audit §2 (a11y, visual regression, mutation, contract, load/perf) are out of scope and will land as separate OpenSpec changes per audit §"Recommended sequencing".

What Changes

Image policy and supply chain

• Every image referenced in any docker-compose.*.yml, plugin spec, or testcontainers config SHALL be pinned by tag and sha256 digest (image: foo/bar:v1.2.3@sha256:…).
• A repo-level script scripts/verify-image-pinning.ts SHALL fail CI on any tag-only reference.
• axllent/mailpit:latest SHALL be replaced with a pinned v1.x@sha256:… form immediately.
• Digest refresh policy is monthly dependabot-style bump with reviewer approval; CVE advisories trigger immediate refresh.

Container resource limits

• Every service in docker-compose.{dev,test,testbed}.yml SHALL declare deploy.resources.limits.{cpus,memory}. Defaults: 1.0 cpu / 1Gi (services), 2.0 cpu / 2Gi (databases), 1.5 cpu / 1.5Gi (Garage/MinIO).
• testcontainers-managed containers SHALL set HostConfig.Memory and HostConfig.NanoCpus via the existing config shape with the same defaults.

CI matrix and airgap

• CI matrix: bun-latest × bun-canary, ubuntu-22.04 × ubuntu-24.04 × macos-14 × windows-latest (Playwright-only on Windows), amd64 × arm64 (Linux only), with per-plugin archSupport: ['amd64','arm64'] declaration to skip unsupported cells.
• Node-fallback parity: a separate Node 22 LTS job runs the universal-purity suite for browser-runnable packages.
• Bun version pin lives in root package.json engines.bun; bun-canary is fail-open with a warning.
• Nightly airgap-rebuild job: bun install --offline from a pre-warmed cache, bunx turbo build, bunx turbo test --filter=...[HEAD~7] with images pre-pulled and --network=none during the test phase.

testing-integration-verdaccio fixture (new)

• Image: verdaccio/verdaccio:6@sha256:….
• Env-bridge: NPM_REGISTRY_URL, NPM_TOKEN.
• Healthcheck: HTTP GET /-/ping.
• Config: { storage?: 'memory'|'tmpfs', users?, packages?, uplinks? }.
• Error prefix INTEGRATION.VERDACCIO.*.
• Enables end-to-end testing of bun scripts/publish.ts --package @suaveplan/<pkg> in CI before real publish.

Secret management

• CI secrets sourced from GitHub OIDC + a Vault/SOPS bridge. GARAGE_S3_SECRET_ACCESS_KEY, OIDC_CLIENT_SECRET, NPM_TOKEN, PACT_BROKER_TOKEN MUST be referenced via the bridge, never inline in workflow YAML.
• .env.example documents each secret's CI vs local-dev source.
• scripts/scan-secrets.ts (using @suaveplan/security-codes detectors) fails CI on inline secret patterns.

Port collision and budgets

• Static port ranges (dev 8001-8902, test 7004-7011, testbed 3200) reserved for compose stacks; testcontainers integration tests use random ephemeral ports.
• docker/scripts/preflight.ts fails loudly on bound reserved ports with a remediation hint and PORTS_OFFSET escape hatch.
• Per-spec memory cap: 1Gi default, configurable via vitest.config.ts pool.forks.maxMemory.
• Suite wall-clock ceiling: 30 min per bunx turbo test; soft warning at 20 min, abort at 30, no retries.

Network sandboxing (testing-e2e)

• createE2ETestConfig() defaults to blocking external origins via page.route. localhost, 127.0.0.1, and the configured testbed origin are auto-allowed. allowedOrigins: string[] and test.use({ allowExternalNetwork: true }) are explicit opt-outs (the latter emits a warning).

Flake quarantine and leak detection (testing)

• markFlaky({ retries, reason, ticket }) wraps a test with bounded retries (max 3), emits testing.flake.count and testing.flake.retry_used metrics, requires a non-empty issue URL.
• quarantine.json at repo root enumerates flaky tests with { path, name, ticket, addedAt, expiresAt }; expired entries fail CI.
• After every fixture teardown the runner asserts no leaked timers, sockets, or file descriptors. Leaks fail with TESTING.LEAK.{HANDLE,TIMER,SOCKET}.

Capabilities

New Capabilities

• testing-integration-verdaccio — Verdaccio v6 ServiceFixture for testing the genesis publish path end-to-end.

Modified Capabilities

• testing-integration — operational additions: image pinning, resource limits, CI matrix, airgap rebuild, secret policy, port pre-flight, OOM/wall-clock budgets, leak detection on teardown.
• testing — flake quarantine + handle/timer/socket leak assertions. (Wave-1 lift may also touch this spec; this change layers on the operational specifics without conflict.)
• testing-e2e — network sandboxing default plus allowedOrigins + allowExternalNetwork opt-outs.

Impact

• Affected code: docker-compose.{dev,test,testbed}.yml, every testing-integration-* plugin's image config, scripts/verify-image-pinning.ts (new), scripts/scan-secrets.ts (new), docker/scripts/preflight.ts (new), packages/testing/testing-integration-verdaccio/** (new package), packages/testing/testing-e2e/src/config/**, packages/testing/testing/src/quarantine/** and src/leak/**, quarantine.json (new at repo root), .env.example, GitHub Actions workflow YAML.
• Affected consumers: bun scripts/publish.ts gains an integration-test path; every existing testing-integration plugin gains digest pins and resource caps; every E2E spec inherits the network block (no source change required for the common case).
• Spec deltas: ADDED requirements in testing-integration and testing-integration-verdaccio specs; ADDED + MODIFIED requirements in testing and testing-e2e specs.
• Pre-publish gate: unchanged thresholds; coverage stays at 100% per openspec/project.md.