129 lines
8.1 KiB
Markdown
129 lines
8.1 KiB
Markdown
## ADDED Requirements
|
||
|
||
### Requirement: Image pinning by tag and digest
|
||
|
||
Every Docker image referenced by any `testing-integration-*` plugin, by `docker-compose.{dev,test,testbed}.yml`, or by any testcontainers config helper SHALL be pinned by both tag AND sha256 digest in the canonical form `image: <registry>/<name>:<tag>@sha256:<64-hex>`. Tag-only pinning is a defect. The pin SHALL be verified by `scripts/verify-image-pinning.ts`, which fails CI on any reference that does not match `^[a-z0-9./-]+:[A-Za-z0-9._-]+@sha256:[a-f0-9]{64}$`. Digest refresh follows a monthly dependabot-style cadence with reviewer approval; security CVE advisories trigger immediate refresh outside the monthly window.
|
||
|
||
#### Scenario: Tag-only image fails verification
|
||
|
||
- **WHEN** a plugin or compose file declares `image: foo/bar:v1.2.3` without a `@sha256:` digest
|
||
- **THEN** `scripts/verify-image-pinning.ts` exits non-zero AND CI fails the lint stage
|
||
|
||
#### Scenario: Mailpit is pinned to v1.x with a digest
|
||
|
||
- **WHEN** the Mailpit plugin or testbed compose stack references the Mailpit image
|
||
- **THEN** the reference is `axllent/mailpit:v1.<minor>@sha256:<digest>` (never `:latest`)
|
||
|
||
#### Scenario: CVE refresh bypasses monthly cadence
|
||
|
||
- **WHEN** an upstream CVE advisory is filed against a pinned image
|
||
- **THEN** a digest-bump PR SHALL be opened immediately AND the monthly cadence does not block it
|
||
|
||
### Requirement: Container resource limits
|
||
|
||
Every service in `docker-compose.{dev,test,testbed}.yml` SHALL declare `deploy.resources.limits.{cpus,memory}`. Defaults: 1.0 cpu / 1Gi for application services, 2.0 cpu / 2Gi for database services (Postgres, SurrealDB, Redis, Typesense), 1.5 cpu / 1.5Gi for object stores (Garage, MinIO). testcontainers-managed containers SHALL set `HostConfig.Memory` (bytes) and `HostConfig.NanoCpus` (cpu × 1e9) via the plugin config shape `{ memoryMiB?: number; cpuCount?: number }`, with the same defaults applied when fields are omitted.
|
||
|
||
#### Scenario: Compose service lacking limits fails verification
|
||
|
||
- **WHEN** a service in any tracked `docker-compose.*.yml` does not declare `deploy.resources.limits.cpus` AND `deploy.resources.limits.memory`
|
||
- **THEN** the compose-lint step fails
|
||
|
||
#### Scenario: testcontainer translates plugin budget to docker HostConfig
|
||
|
||
- **WHEN** a plugin starts with `{ memoryMiB: 1024, cpuCount: 1.0 }`
|
||
- **THEN** the resulting docker `HostConfig.Memory === 1073741824` AND `HostConfig.NanoCpus === 1_000_000_000`
|
||
|
||
### Requirement: CI matrix on Bun, OS, and architecture
|
||
|
||
CI SHALL execute on a matrix of `bun-latest` × `bun-canary`, `ubuntu-22.04` × `ubuntu-24.04` × `macos-14` × `windows-latest` (Playwright-only on Windows), and `amd64` × `arm64` on Linux. Each `testing-integration-*` plugin SHALL declare `archSupport: ('amd64'|'arm64')[]` and the matrix-generator SHALL skip unsupported cells. The canonical Bun version is pinned via root `package.json` `engines.bun`; the `bun-canary` job is allowed to fail-open with a warning annotation. A separate Node 22 LTS job SHALL run the universal-purity suite for browser-runnable packages (`testing`, `testing-dom`); failures here block merge.
|
||
|
||
#### Scenario: Plugin without arm64 support skips arm64 cell
|
||
|
||
- **WHEN** a plugin declares `archSupport: ['amd64']`
|
||
- **THEN** the matrix-generator emits no `arm64` job for that plugin AND CI does not flag the absence as a failure
|
||
|
||
#### Scenario: Bun canary fails
|
||
|
||
- **WHEN** the `bun-canary` job exits non-zero
|
||
- **THEN** the PR check surfaces a warning annotation AND merge is not blocked
|
||
|
||
#### Scenario: Node parity fails
|
||
|
||
- **WHEN** the Node 22 LTS parity job for browser-runnable packages exits non-zero
|
||
- **THEN** merge is blocked
|
||
|
||
### Requirement: Airgap rebuild gate
|
||
|
||
A nightly workflow `airgap-rebuild` SHALL: pre-warm the Bun cache via a normal install, run `bun install --offline` from that cache, run `bunx turbo build`, then run `bunx turbo test --filter=...[HEAD~7]` with all required images pre-pulled and the test phase executed inside `--network=none`. Failure indicates a missing offline-cache entry and SHALL block the next release.
|
||
|
||
#### Scenario: Offline install missing dep fails the job
|
||
|
||
- **WHEN** any package pulls a dependency that is not present in the pre-warmed cache during `bun install --offline`
|
||
- **THEN** the airgap job fails AND release is blocked until the offline cache is fixed
|
||
|
||
#### Scenario: Test phase isolated from network
|
||
|
||
- **WHEN** the airgap job enters the test phase
|
||
- **THEN** the Docker network for the runner is `--network=none` AND any test that attempts external network egress fails fast
|
||
|
||
### Requirement: Secret management policy
|
||
|
||
CI secrets SHALL be sourced from GitHub OIDC plus a Vault or SOPS bridge. The variables `GARAGE_S3_SECRET_ACCESS_KEY`, `OIDC_CLIENT_SECRET`, `NPM_TOKEN`, and `PACT_BROKER_TOKEN` MUST be referenced via the bridge — never inline in workflow YAML. `.env.example` SHALL document each secret's CI source and its local-dev source. `scripts/scan-secrets.ts` SHALL fail CI on any inline secret pattern using `@suaveplan/security-codes` detectors.
|
||
|
||
#### Scenario: Inline secret in workflow YAML fails CI
|
||
|
||
- **WHEN** a workflow declares `env: NPM_TOKEN: <literal-token>` (any pattern detected by `@suaveplan/security-codes`)
|
||
- **THEN** `scripts/scan-secrets.ts` exits non-zero AND CI fails
|
||
|
||
#### Scenario: Bridge resolution
|
||
|
||
- **WHEN** a workflow references `${{ secrets.NPM_TOKEN }}` via the OIDC + Vault/SOPS bridge step
|
||
- **THEN** the secret is fetched at job start AND no inline value appears in the YAML source
|
||
|
||
### Requirement: Port collision policy
|
||
|
||
Static port ranges (dev 8001-8902, test 7004-7011, testbed 3200) are reserved for compose stacks. testcontainers-managed integration tests SHALL use random ephemeral ports (the testcontainers default). `docker/scripts/preflight.ts` SHALL fail loudly if any reserved port is already bound, with a remediation hint pointing the user at either killing the conflicting process or remapping via `PORTS_OFFSET`.
|
||
|
||
#### Scenario: Bound reserved port aborts startup
|
||
|
||
- **WHEN** `docker compose -f docker-compose.dev.yml up` runs and port 8001 is already bound by another process
|
||
- **THEN** preflight exits non-zero AND prints the offending PID + a `PORTS_OFFSET` remediation hint
|
||
|
||
#### Scenario: PORTS_OFFSET shift
|
||
|
||
- **WHEN** `PORTS_OFFSET=100` is set in the environment
|
||
- **THEN** every reserved port is shifted by 100 AND preflight passes if the shifted range is free
|
||
|
||
### Requirement: OOM and suite wall-clock budgets
|
||
|
||
Per-spec memory cap SHALL default to 1Gi via the shared `createTestConfig()`'s `pool.forks.maxMemory`; packages MAY override per-package. Per-`bunx turbo test` invocation wall-clock ceiling SHALL be 30 minutes; a soft warning SHALL be emitted at 20 minutes; the run SHALL be aborted at 30 minutes. No automatic retries on timeout — timeout indicates a pathological condition that must be diagnosed.
|
||
|
||
#### Scenario: Single-spec OOM aborts that fork only
|
||
|
||
- **WHEN** a single test spec exceeds 1Gi resident
|
||
- **THEN** the forked vitest worker is killed with a `TESTING.MEMORY.LIMIT_EXCEEDED` error AND sibling specs continue
|
||
|
||
#### Scenario: Wall-clock 30-minute abort
|
||
|
||
- **WHEN** `bunx turbo test` exceeds 30 minutes wall-clock
|
||
- **THEN** the run is aborted with a non-zero exit AND no automatic retry is attempted
|
||
|
||
### Requirement: Leak detection on fixture teardown
|
||
|
||
After every `TestEnvironment.teardown()` the runner SHALL assert no leaked timers, sockets, or file descriptors via `Bun.peek` / runtime handle table on Bun or `process._getActiveHandles()` on Node fallback. Leaks SHALL fail the suite with a categorised error: `TESTING.LEAK.HANDLE`, `TESTING.LEAK.TIMER`, or `TESTING.LEAK.SOCKET`, each carrying `{ count, sample }` context.
|
||
|
||
#### Scenario: Unreleased timer leaks
|
||
|
||
- **WHEN** a fixture sets `setInterval` and does not clear it before teardown
|
||
- **THEN** teardown throws `TESTING.LEAK.TIMER` AND `context.count >= 1`
|
||
|
||
#### Scenario: Unclosed socket leaks
|
||
|
||
- **WHEN** a fixture opens a TCP socket and does not close it before teardown
|
||
- **THEN** teardown throws `TESTING.LEAK.SOCKET` AND `context.sample` contains the offending socket descriptor
|
||
|
||
#### Scenario: Clean teardown passes
|
||
|
||
- **WHEN** all timers are cleared and all sockets are closed before teardown returns
|
||
- **THEN** the leak assertion passes silently
|