Initial Commit
This commit is contained in:
79
openspec/changes/enterprise-testing-operational/proposal.md
Normal file
79
openspec/changes/enterprise-testing-operational/proposal.md
Normal file
@@ -0,0 +1,79 @@
|
||||
## Why
|
||||
|
||||
The 2026-04-25 test-suite audit (`/mnt/s/web/genesis/docs/superpowers/specs/2026-04-25-test-suite-audit.md`, sections 3, 4, 5) catalogued a class of gaps that the in-flight `enterprise-testing-integration` lift does not cover: the **operational hygiene** of the test infrastructure itself. Image pinning is by tag only (`axllent/mailpit:latest` is the worst offender); compose stacks declare no resource limits; CI runs on a single Bun version and a single architecture; there is no airgap rebuild gate; secrets are referenced inline in workflow YAML; testcontainers and compose stacks share no port-collision contract; flaky tests have no quarantine path; integration teardown does not assert handle/timer/socket leak freedom; E2E specs may reach the live internet by default; and the publish flow against Verdaccio (`https://npmjs.fq.io/`) has no fixture, so `bun scripts/publish.ts` cannot be exercised end-to-end before a real publish.
|
||||
|
||||
Each gap is a near-miss for a CI outage or a non-reproducible release. Pinning by digest, resource caps, a Bun/OS/arch matrix, an airgap rebuild test, a Verdaccio fixture, secret-management policy, port pre-flight, OOM/timeout budgets, network sandboxing, and flake/leak detection are all standard at the enterprise tier but absent from the genesis testing substrate today. This change closes them in one coherent set of normative requirements.
|
||||
|
||||
This change is **scoped narrowly** to operational hygiene. The pyramid-tier additions called out in audit §2 (a11y, visual regression, mutation, contract, load/perf) are out of scope and will land as separate OpenSpec changes per audit §"Recommended sequencing".
|
||||
|
||||
## What Changes
|
||||
|
||||
### Image policy and supply chain
|
||||
|
||||
- Every image referenced in any `docker-compose.*.yml`, plugin spec, or testcontainers config SHALL be pinned by tag **and** sha256 digest (`image: foo/bar:v1.2.3@sha256:…`).
|
||||
- A repo-level script `scripts/verify-image-pinning.ts` SHALL fail CI on any tag-only reference.
|
||||
- `axllent/mailpit:latest` SHALL be replaced with a pinned `v1.x@sha256:…` form immediately.
|
||||
- Digest refresh policy is monthly dependabot-style bump with reviewer approval; CVE advisories trigger immediate refresh.
|
||||
|
||||
### Container resource limits
|
||||
|
||||
- Every service in `docker-compose.{dev,test,testbed}.yml` SHALL declare `deploy.resources.limits.{cpus,memory}`. Defaults: 1.0 cpu / 1Gi (services), 2.0 cpu / 2Gi (databases), 1.5 cpu / 1.5Gi (Garage/MinIO).
|
||||
- testcontainers-managed containers SHALL set `HostConfig.Memory` and `HostConfig.NanoCpus` via the existing config shape with the same defaults.
|
||||
|
||||
### CI matrix and airgap
|
||||
|
||||
- CI matrix: `bun-latest` × `bun-canary`, `ubuntu-22.04` × `ubuntu-24.04` × `macos-14` × `windows-latest` (Playwright-only on Windows), `amd64` × `arm64` (Linux only), with per-plugin `archSupport: ['amd64','arm64']` declaration to skip unsupported cells.
|
||||
- Node-fallback parity: a separate Node 22 LTS job runs the universal-purity suite for browser-runnable packages.
|
||||
- Bun version pin lives in root `package.json` `engines.bun`; `bun-canary` is fail-open with a warning.
|
||||
- Nightly `airgap-rebuild` job: `bun install --offline` from a pre-warmed cache, `bunx turbo build`, `bunx turbo test --filter=...[HEAD~7]` with images pre-pulled and `--network=none` during the test phase.
|
||||
|
||||
### testing-integration-verdaccio fixture (new)
|
||||
|
||||
- Image: `verdaccio/verdaccio:6@sha256:…`.
|
||||
- Env-bridge: `NPM_REGISTRY_URL`, `NPM_TOKEN`.
|
||||
- Healthcheck: HTTP `GET /-/ping`.
|
||||
- Config: `{ storage?: 'memory'|'tmpfs', users?, packages?, uplinks? }`.
|
||||
- Error prefix `INTEGRATION.VERDACCIO.*`.
|
||||
- Enables end-to-end testing of `bun scripts/publish.ts --package @suaveplan/<pkg>` in CI before real publish.
|
||||
|
||||
### Secret management
|
||||
|
||||
- CI secrets sourced from GitHub OIDC + a Vault/SOPS bridge. `GARAGE_S3_SECRET_ACCESS_KEY`, `OIDC_CLIENT_SECRET`, `NPM_TOKEN`, `PACT_BROKER_TOKEN` MUST be referenced via the bridge, never inline in workflow YAML.
|
||||
- `.env.example` documents each secret's CI vs local-dev source.
|
||||
- `scripts/scan-secrets.ts` (using `@suaveplan/security-codes` detectors) fails CI on inline secret patterns.
|
||||
|
||||
### Port collision and budgets
|
||||
|
||||
- Static port ranges (dev 8001-8902, test 7004-7011, testbed 3200) reserved for compose stacks; testcontainers integration tests use random ephemeral ports.
|
||||
- `docker/scripts/preflight.ts` fails loudly on bound reserved ports with a remediation hint and `PORTS_OFFSET` escape hatch.
|
||||
- Per-spec memory cap: 1Gi default, configurable via `vitest.config.ts` `pool.forks.maxMemory`.
|
||||
- Suite wall-clock ceiling: 30 min per `bunx turbo test`; soft warning at 20 min, abort at 30, no retries.
|
||||
|
||||
### Network sandboxing (testing-e2e)
|
||||
|
||||
- `createE2ETestConfig()` defaults to blocking external origins via `page.route`. `localhost`, `127.0.0.1`, and the configured testbed origin are auto-allowed. `allowedOrigins: string[]` and `test.use({ allowExternalNetwork: true })` are explicit opt-outs (the latter emits a warning).
|
||||
|
||||
### Flake quarantine and leak detection (testing)
|
||||
|
||||
- `markFlaky({ retries, reason, ticket })` wraps a test with bounded retries (max 3), emits `testing.flake.count` and `testing.flake.retry_used` metrics, requires a non-empty issue URL.
|
||||
- `quarantine.json` at repo root enumerates flaky tests with `{ path, name, ticket, addedAt, expiresAt }`; expired entries fail CI.
|
||||
- After every fixture teardown the runner asserts no leaked timers, sockets, or file descriptors. Leaks fail with `TESTING.LEAK.{HANDLE,TIMER,SOCKET}`.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `testing-integration-verdaccio` — Verdaccio v6 ServiceFixture for testing the genesis publish path end-to-end.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `testing-integration` — operational additions: image pinning, resource limits, CI matrix, airgap rebuild, secret policy, port pre-flight, OOM/wall-clock budgets, leak detection on teardown.
|
||||
- `testing` — flake quarantine + handle/timer/socket leak assertions. (Wave-1 lift may also touch this spec; this change layers on the operational specifics without conflict.)
|
||||
- `testing-e2e` — network sandboxing default plus `allowedOrigins` + `allowExternalNetwork` opt-outs.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Affected code:** `docker-compose.{dev,test,testbed}.yml`, every `testing-integration-*` plugin's image config, `scripts/verify-image-pinning.ts` (new), `scripts/scan-secrets.ts` (new), `docker/scripts/preflight.ts` (new), `packages/testing/testing-integration-verdaccio/**` (new package), `packages/testing/testing-e2e/src/config/**`, `packages/testing/testing/src/quarantine/**` and `src/leak/**`, `quarantine.json` (new at repo root), `.env.example`, GitHub Actions workflow YAML.
|
||||
- **Affected consumers:** `bun scripts/publish.ts` gains an integration-test path; every existing testing-integration plugin gains digest pins and resource caps; every E2E spec inherits the network block (no source change required for the common case).
|
||||
- **Spec deltas:** ADDED requirements in `testing-integration` and `testing-integration-verdaccio` specs; ADDED + MODIFIED requirements in `testing` and `testing-e2e` specs.
|
||||
- **Pre-publish gate:** unchanged thresholds; coverage stays at 100% per `openspec/project.md`.
|
||||
Reference in New Issue
Block a user