Initial Commit
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Default network sandboxing
|
||||
|
||||
`createE2ETestConfig()` SHALL default to blocking external origins via a per-test `page.route('**/*')` handler that aborts requests with `'blockedbyclient'` unless their origin is allow-listed. The auto-allowed set SHALL include `localhost`, `127.0.0.1`, and the configured testbed origin (read from the `TESTBED_ORIGIN` env var, falling back to the factory's `baseURL`). The factory SHALL accept an `allowedOrigins: string[]` option (default `[]`) which lifts the block per-origin. Origin matching SHALL be performed via `URL` constructor parsing on full origin (`<scheme>://<host>:<port>`), never substring or regex matching.
|
||||
|
||||
#### Scenario: External origin blocked by default
|
||||
|
||||
- **WHEN** a spec configured by `createE2ETestConfig({ baseURL: 'http://localhost:3200' })` issues a request to `https://example.com/data`
|
||||
- **THEN** the request is aborted with reason `'blockedbyclient'` AND the assertion library surfaces the abort as a network failure
|
||||
|
||||
#### Scenario: localhost auto-allowed
|
||||
|
||||
- **WHEN** a spec under the same config issues a request to `http://localhost:3200/api/users` or `http://127.0.0.1:9000/`
|
||||
- **THEN** the request proceeds normally AND no abort is raised
|
||||
|
||||
#### Scenario: Testbed origin auto-allowed
|
||||
|
||||
- **WHEN** `TESTBED_ORIGIN=https://testbed.internal:8443` is set in the environment
|
||||
- **THEN** requests to that exact origin proceed normally
|
||||
|
||||
#### Scenario: allowedOrigins lifts the block
|
||||
|
||||
- **WHEN** `createE2ETestConfig({ allowedOrigins: ['https://api.partner.test'] })` is used and a spec issues `GET https://api.partner.test/v1/foo`
|
||||
- **THEN** the request proceeds normally AND requests to any other external origin remain blocked
|
||||
|
||||
#### Scenario: Origin matching is exact
|
||||
|
||||
- **WHEN** `allowedOrigins: ['https://api.partner.test']` is set and a spec issues `GET https://evil.api.partner.test.attacker.example/`
|
||||
- **THEN** the request is blocked AND the substring overlap does NOT lift the block
|
||||
|
||||
### Requirement: Spec-level escape hatch with warning
|
||||
|
||||
The package SHALL expose a `test.use({ allowExternalNetwork: true })` Playwright fixture extension that disables the default block for the enclosing spec only. Use of this escape hatch SHALL emit a warning at suite start naming the offending spec file path, encouraging migration to explicit `allowedOrigins`. The escape hatch MUST be implemented as a Playwright `test.extend` fixture and MUST NOT mutate other specs' route handlers.
|
||||
|
||||
#### Scenario: Escape hatch lifts the block locally
|
||||
|
||||
- **WHEN** a spec declares `test.use({ allowExternalNetwork: true })` and issues a request to `https://example.com/`
|
||||
- **THEN** the request proceeds normally AND no abort is raised within that spec
|
||||
|
||||
#### Scenario: Escape-hatch warning emitted
|
||||
|
||||
- **WHEN** the suite starts and at least one spec opts into `allowExternalNetwork: true`
|
||||
- **THEN** a warning line is written to stderr naming each offending spec file path AND the warning persists in the test report
|
||||
|
||||
#### Scenario: Escape hatch is spec-local
|
||||
|
||||
- **WHEN** spec A declares `test.use({ allowExternalNetwork: true })` and spec B does not
|
||||
- **THEN** spec B's external requests are still blocked AND spec A's opt-in does not leak across
|
||||
|
||||
### Requirement: Origin-matching algorithm is documented and tested
|
||||
|
||||
The origin-matching algorithm used by the network sandbox SHALL be exported as a pure function (e.g. `isOriginAllowed(url, allowedOrigins, autoAllowed)`) so it is testable in isolation. The function MUST: parse `url` via the `URL` constructor and reject inputs that fail to parse with `TESTING_E2E.NETWORK.URL_INVALID`; compare on the `<scheme>://<host>:<port>` triple, treating the default port for the scheme as equivalent to an explicit port; return a boolean. fast-check property tests SHALL cover round-trip parsing and the substring-attack scenario above.
|
||||
|
||||
#### Scenario: Default-port equivalence
|
||||
|
||||
- **WHEN** `isOriginAllowed('https://api.partner.test/', ['https://api.partner.test:443'], [])` is called
|
||||
- **THEN** the function returns `true`
|
||||
|
||||
#### Scenario: Invalid URL rejected
|
||||
|
||||
- **WHEN** `isOriginAllowed('not a url', [], [])` is called
|
||||
- **THEN** the function throws `TESTING_E2E.NETWORK.URL_INVALID`
|
||||
@@ -0,0 +1,75 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Verdaccio image and pinning
|
||||
|
||||
`@suaveplan/testing-integration-verdaccio` SHALL ship a `VerdaccioFixture` ServiceFixture backed by `verdaccio/verdaccio:6@sha256:<digest>`. The image MUST be pinned by both tag and sha256 digest and MUST be tracked by `scripts/verify-image-pinning.ts`.
|
||||
|
||||
#### Scenario: Image is pinned
|
||||
|
||||
- **WHEN** the fixture's image config is read
|
||||
- **THEN** the value matches `^verdaccio/verdaccio:6@sha256:[a-f0-9]{64}$`
|
||||
|
||||
### Requirement: Environment-variable bridge
|
||||
|
||||
The fixture SHALL expose its endpoint and credentials to the test environment via the variables `NPM_REGISTRY_URL` and `NPM_TOKEN`. `NPM_REGISTRY_URL` MUST be the fully-qualified `http://<host>:<port>/` form. `NPM_TOKEN` MUST be a Verdaccio-compatible bearer token generated at fixture startup.
|
||||
|
||||
#### Scenario: Env-bridge populated after start
|
||||
|
||||
- **WHEN** `VerdaccioFixture.setup()` resolves
|
||||
- **THEN** `process.env.NPM_REGISTRY_URL` is a parseable URL with `protocol === 'http:'` AND `process.env.NPM_TOKEN` has length ≥ 32
|
||||
|
||||
### Requirement: Healthcheck via /-/ping
|
||||
|
||||
Readiness SHALL be determined by a polling HTTP `GET /-/ping` against the fixture's endpoint. The poller MUST respect `HealthPoller` defaults from `@suaveplan/testing-integration` and SHALL accept an injectable `fetch` implementation for unit tests.
|
||||
|
||||
#### Scenario: Ping returns 200 → ready
|
||||
|
||||
- **WHEN** `GET /-/ping` returns HTTP 200
|
||||
- **THEN** `HealthPoller` resolves and the fixture is considered ready
|
||||
|
||||
#### Scenario: Ping never succeeds → timeout
|
||||
|
||||
- **WHEN** `/-/ping` never returns 200 within the configured `healthTimeoutMs`
|
||||
- **THEN** `setup()` rejects with `INTEGRATION.VERDACCIO.HEALTH_TIMEOUT`
|
||||
|
||||
### Requirement: Configuration shape
|
||||
|
||||
The fixture SHALL accept a config object of shape `{ storage?: 'memory'|'tmpfs'; users?: Array<{ username: string; password: string }>; packages?: Array<{ name: string; access?: string; publish?: string }>; uplinks?: Record<string, string> }`. `storage` defaults to `'memory'`. The fixture SHALL translate this object into Verdaccio's native YAML config at startup. Invalid config MUST be rejected via Zod with a `INTEGRATION.VERDACCIO.CONFIG_INVALID` error.
|
||||
|
||||
#### Scenario: Memory storage default
|
||||
|
||||
- **WHEN** the fixture is started without a `storage` field
|
||||
- **THEN** Verdaccio runs with in-memory storage AND no host filesystem path is mounted
|
||||
|
||||
#### Scenario: tmpfs storage
|
||||
|
||||
- **WHEN** the fixture is started with `storage: 'tmpfs'`
|
||||
- **THEN** the container's storage path is backed by a tmpfs mount AND publish state survives within the fixture lifetime but not across restarts
|
||||
|
||||
#### Scenario: User and package config translated
|
||||
|
||||
- **WHEN** `users: [{ username: 'ci', password: 'p' }]` and `packages: [{ name: '@suaveplan/*', access: '$authenticated', publish: 'ci' }]` are supplied
|
||||
- **THEN** the rendered Verdaccio YAML contains an htpasswd entry for `ci` AND a `packages` block matching the input
|
||||
|
||||
### Requirement: Error code prefix
|
||||
|
||||
Every error thrown by `@suaveplan/testing-integration-verdaccio` SHALL extend the package's structured error class and carry a `code` of the form `INTEGRATION.VERDACCIO.<REASON>` (uppercase ASCII, dots as separators).
|
||||
|
||||
#### Scenario: Coded error on health timeout
|
||||
|
||||
- **WHEN** the fixture's healthcheck never succeeds
|
||||
- **THEN** the thrown error has `code === 'INTEGRATION.VERDACCIO.HEALTH_TIMEOUT'`
|
||||
|
||||
#### Scenario: Coded error on invalid config
|
||||
|
||||
- **WHEN** an invalid `users` array is supplied
|
||||
- **THEN** the thrown error has `code === 'INTEGRATION.VERDACCIO.CONFIG_INVALID'` AND `cause` is the underlying `ZodError`
|
||||
|
||||
### Requirement: Publish flow integration test
|
||||
|
||||
The fixture SHALL be exercisable end-to-end against `bun scripts/publish.ts --package @suaveplan/<pkg>` so the genesis publish path is testable in CI before any real publish to the production Verdaccio at `https://npmjs.fq.io/`. A reference integration test SHALL ship in the package demonstrating: fixture startup, env-bridge wiring, `bun scripts/publish.ts` invocation against the fixture, and verification of the published tarball via `npm view --registry $NPM_REGISTRY_URL`.
|
||||
|
||||
#### Scenario: Publish round-trip
|
||||
|
||||
- **WHEN** the integration test runs `bun scripts/publish.ts --package <probe>` against the fixture
|
||||
- **THEN** `npm view <probe>@0.0.0 --registry $NPM_REGISTRY_URL` returns the package metadata AND the tarball digest matches the locally built artefact
|
||||
@@ -0,0 +1,128 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Image pinning by tag and digest
|
||||
|
||||
Every Docker image referenced by any `testing-integration-*` plugin, by `docker-compose.{dev,test,testbed}.yml`, or by any testcontainers config helper SHALL be pinned by both tag AND sha256 digest in the canonical form `image: <registry>/<name>:<tag>@sha256:<64-hex>`. Tag-only pinning is a defect. The pin SHALL be verified by `scripts/verify-image-pinning.ts`, which fails CI on any reference that does not match `^[a-z0-9./-]+:[A-Za-z0-9._-]+@sha256:[a-f0-9]{64}$`. Digest refresh follows a monthly dependabot-style cadence with reviewer approval; security CVE advisories trigger immediate refresh outside the monthly window.
|
||||
|
||||
#### Scenario: Tag-only image fails verification
|
||||
|
||||
- **WHEN** a plugin or compose file declares `image: foo/bar:v1.2.3` without a `@sha256:` digest
|
||||
- **THEN** `scripts/verify-image-pinning.ts` exits non-zero AND CI fails the lint stage
|
||||
|
||||
#### Scenario: Mailpit is pinned to v1.x with a digest
|
||||
|
||||
- **WHEN** the Mailpit plugin or testbed compose stack references the Mailpit image
|
||||
- **THEN** the reference is `axllent/mailpit:v1.<minor>@sha256:<digest>` (never `:latest`)
|
||||
|
||||
#### Scenario: CVE refresh bypasses monthly cadence
|
||||
|
||||
- **WHEN** an upstream CVE advisory is filed against a pinned image
|
||||
- **THEN** a digest-bump PR SHALL be opened immediately AND the monthly cadence does not block it
|
||||
|
||||
### Requirement: Container resource limits
|
||||
|
||||
Every service in `docker-compose.{dev,test,testbed}.yml` SHALL declare `deploy.resources.limits.{cpus,memory}`. Defaults: 1.0 cpu / 1Gi for application services, 2.0 cpu / 2Gi for database services (Postgres, SurrealDB, Redis, Typesense), 1.5 cpu / 1.5Gi for object stores (Garage, MinIO). testcontainers-managed containers SHALL set `HostConfig.Memory` (bytes) and `HostConfig.NanoCpus` (cpu × 1e9) via the plugin config shape `{ memoryMiB?: number; cpuCount?: number }`, with the same defaults applied when fields are omitted.
|
||||
|
||||
#### Scenario: Compose service lacking limits fails verification
|
||||
|
||||
- **WHEN** a service in any tracked `docker-compose.*.yml` does not declare `deploy.resources.limits.cpus` AND `deploy.resources.limits.memory`
|
||||
- **THEN** the compose-lint step fails
|
||||
|
||||
#### Scenario: testcontainer translates plugin budget to docker HostConfig
|
||||
|
||||
- **WHEN** a plugin starts with `{ memoryMiB: 1024, cpuCount: 1.0 }`
|
||||
- **THEN** the resulting docker `HostConfig.Memory === 1073741824` AND `HostConfig.NanoCpus === 1_000_000_000`
|
||||
|
||||
### Requirement: CI matrix on Bun, OS, and architecture
|
||||
|
||||
CI SHALL execute on a matrix of `bun-latest` × `bun-canary`, `ubuntu-22.04` × `ubuntu-24.04` × `macos-14` × `windows-latest` (Playwright-only on Windows), and `amd64` × `arm64` on Linux. Each `testing-integration-*` plugin SHALL declare `archSupport: ('amd64'|'arm64')[]` and the matrix-generator SHALL skip unsupported cells. The canonical Bun version is pinned via root `package.json` `engines.bun`; the `bun-canary` job is allowed to fail-open with a warning annotation. A separate Node 22 LTS job SHALL run the universal-purity suite for browser-runnable packages (`testing`, `testing-dom`); failures here block merge.
|
||||
|
||||
#### Scenario: Plugin without arm64 support skips arm64 cell
|
||||
|
||||
- **WHEN** a plugin declares `archSupport: ['amd64']`
|
||||
- **THEN** the matrix-generator emits no `arm64` job for that plugin AND CI does not flag the absence as a failure
|
||||
|
||||
#### Scenario: Bun canary fails
|
||||
|
||||
- **WHEN** the `bun-canary` job exits non-zero
|
||||
- **THEN** the PR check surfaces a warning annotation AND merge is not blocked
|
||||
|
||||
#### Scenario: Node parity fails
|
||||
|
||||
- **WHEN** the Node 22 LTS parity job for browser-runnable packages exits non-zero
|
||||
- **THEN** merge is blocked
|
||||
|
||||
### Requirement: Airgap rebuild gate
|
||||
|
||||
A nightly workflow `airgap-rebuild` SHALL: pre-warm the Bun cache via a normal install, run `bun install --offline` from that cache, run `bunx turbo build`, then run `bunx turbo test --filter=...[HEAD~7]` with all required images pre-pulled and the test phase executed inside `--network=none`. Failure indicates a missing offline-cache entry and SHALL block the next release.
|
||||
|
||||
#### Scenario: Offline install missing dep fails the job
|
||||
|
||||
- **WHEN** any package pulls a dependency that is not present in the pre-warmed cache during `bun install --offline`
|
||||
- **THEN** the airgap job fails AND release is blocked until the offline cache is fixed
|
||||
|
||||
#### Scenario: Test phase isolated from network
|
||||
|
||||
- **WHEN** the airgap job enters the test phase
|
||||
- **THEN** the Docker network for the runner is `--network=none` AND any test that attempts external network egress fails fast
|
||||
|
||||
### Requirement: Secret management policy
|
||||
|
||||
CI secrets SHALL be sourced from GitHub OIDC plus a Vault or SOPS bridge. The variables `GARAGE_S3_SECRET_ACCESS_KEY`, `OIDC_CLIENT_SECRET`, `NPM_TOKEN`, and `PACT_BROKER_TOKEN` MUST be referenced via the bridge — never inline in workflow YAML. `.env.example` SHALL document each secret's CI source and its local-dev source. `scripts/scan-secrets.ts` SHALL fail CI on any inline secret pattern using `@suaveplan/security-codes` detectors.
|
||||
|
||||
#### Scenario: Inline secret in workflow YAML fails CI
|
||||
|
||||
- **WHEN** a workflow declares `env: NPM_TOKEN: <literal-token>` (any pattern detected by `@suaveplan/security-codes`)
|
||||
- **THEN** `scripts/scan-secrets.ts` exits non-zero AND CI fails
|
||||
|
||||
#### Scenario: Bridge resolution
|
||||
|
||||
- **WHEN** a workflow references `${{ secrets.NPM_TOKEN }}` via the OIDC + Vault/SOPS bridge step
|
||||
- **THEN** the secret is fetched at job start AND no inline value appears in the YAML source
|
||||
|
||||
### Requirement: Port collision policy
|
||||
|
||||
Static port ranges (dev 8001-8902, test 7004-7011, testbed 3200) are reserved for compose stacks. testcontainers-managed integration tests SHALL use random ephemeral ports (the testcontainers default). `docker/scripts/preflight.ts` SHALL fail loudly if any reserved port is already bound, with a remediation hint pointing the user at either killing the conflicting process or remapping via `PORTS_OFFSET`.
|
||||
|
||||
#### Scenario: Bound reserved port aborts startup
|
||||
|
||||
- **WHEN** `docker compose -f docker-compose.dev.yml up` runs and port 8001 is already bound by another process
|
||||
- **THEN** preflight exits non-zero AND prints the offending PID + a `PORTS_OFFSET` remediation hint
|
||||
|
||||
#### Scenario: PORTS_OFFSET shift
|
||||
|
||||
- **WHEN** `PORTS_OFFSET=100` is set in the environment
|
||||
- **THEN** every reserved port is shifted by 100 AND preflight passes if the shifted range is free
|
||||
|
||||
### Requirement: OOM and suite wall-clock budgets
|
||||
|
||||
Per-spec memory cap SHALL default to 1Gi via the shared `createTestConfig()`'s `pool.forks.maxMemory`; packages MAY override per-package. Per-`bunx turbo test` invocation wall-clock ceiling SHALL be 30 minutes; a soft warning SHALL be emitted at 20 minutes; the run SHALL be aborted at 30 minutes. No automatic retries on timeout — timeout indicates a pathological condition that must be diagnosed.
|
||||
|
||||
#### Scenario: Single-spec OOM aborts that fork only
|
||||
|
||||
- **WHEN** a single test spec exceeds 1Gi resident
|
||||
- **THEN** the forked vitest worker is killed with a `TESTING.MEMORY.LIMIT_EXCEEDED` error AND sibling specs continue
|
||||
|
||||
#### Scenario: Wall-clock 30-minute abort
|
||||
|
||||
- **WHEN** `bunx turbo test` exceeds 30 minutes wall-clock
|
||||
- **THEN** the run is aborted with a non-zero exit AND no automatic retry is attempted
|
||||
|
||||
### Requirement: Leak detection on fixture teardown
|
||||
|
||||
After every `TestEnvironment.teardown()` the runner SHALL assert no leaked timers, sockets, or file descriptors via `Bun.peek` / runtime handle table on Bun or `process._getActiveHandles()` on Node fallback. Leaks SHALL fail the suite with a categorised error: `TESTING.LEAK.HANDLE`, `TESTING.LEAK.TIMER`, or `TESTING.LEAK.SOCKET`, each carrying `{ count, sample }` context.
|
||||
|
||||
#### Scenario: Unreleased timer leaks
|
||||
|
||||
- **WHEN** a fixture sets `setInterval` and does not clear it before teardown
|
||||
- **THEN** teardown throws `TESTING.LEAK.TIMER` AND `context.count >= 1`
|
||||
|
||||
#### Scenario: Unclosed socket leaks
|
||||
|
||||
- **WHEN** a fixture opens a TCP socket and does not close it before teardown
|
||||
- **THEN** teardown throws `TESTING.LEAK.SOCKET` AND `context.sample` contains the offending socket descriptor
|
||||
|
||||
#### Scenario: Clean teardown passes
|
||||
|
||||
- **WHEN** all timers are cleared and all sockets are closed before teardown returns
|
||||
- **THEN** the leak assertion passes silently
|
||||
@@ -0,0 +1,81 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Flake quarantine via `markFlaky`
|
||||
|
||||
The package SHALL export `markFlaky({ retries, reason, ticket })` that wraps a Vitest `it`/`test` callback with bounded retries (max 3). Each retry SHALL emit `testing.flake.retry_used` as an OTel counter with attributes `{ test, attempt }`; the wrapper SHALL emit `testing.flake.count` as a counter on initial registration. The `ticket` field MUST be a non-empty issue URL validated by Zod (`url()` + `.min(1)`); a missing or invalid ticket MUST throw `TESTING.FLAKE.TICKET_REQUIRED` at registration time, not at first run.
|
||||
|
||||
#### Scenario: Bounded retries succeed
|
||||
|
||||
- **WHEN** `markFlaky({ retries: 2, reason: 'docker DNS jitter', ticket: 'https://example.com/issues/42' })` wraps a test that fails once then passes
|
||||
- **THEN** the test reports green AND `testing.flake.retry_used` records exactly one increment
|
||||
|
||||
#### Scenario: Retries above the cap rejected
|
||||
|
||||
- **WHEN** `markFlaky({ retries: 5, ... })` is called
|
||||
- **THEN** `markFlaky` throws `TESTING.FLAKE.RETRIES_OUT_OF_RANGE` with `context.max === 3`
|
||||
|
||||
#### Scenario: Missing ticket rejected at registration
|
||||
|
||||
- **WHEN** `markFlaky({ retries: 1, reason: 'x', ticket: '' })` is called
|
||||
- **THEN** `markFlaky` throws `TESTING.FLAKE.TICKET_REQUIRED` immediately AND the test is not registered
|
||||
|
||||
### Requirement: Quarantine ledger
|
||||
|
||||
A `quarantine.json` file SHALL live at the repository root and enumerate every flaky test as `{ path: string; name: string; ticket: string; addedAt: ISO8601; expiresAt: ISO8601 }`. CI SHALL fail when any entry's `expiresAt` is in the past, forcing triage. The ledger schema MUST be exported as a Zod schema from the testing package so external tooling can validate it without re-implementing the shape.
|
||||
|
||||
#### Scenario: Expired entry fails CI
|
||||
|
||||
- **WHEN** `quarantine.json` contains an entry with `expiresAt` earlier than the current time
|
||||
- **THEN** the quarantine-lint step exits non-zero AND CI fails with a message naming the expired entries
|
||||
|
||||
#### Scenario: Schema mismatch rejected
|
||||
|
||||
- **WHEN** `quarantine.json` is missing the `ticket` field for any entry
|
||||
- **THEN** the loader throws `TESTING.QUARANTINE.SCHEMA_INVALID` AND the suite refuses to start
|
||||
|
||||
#### Scenario: Active entry passes
|
||||
|
||||
- **WHEN** every entry's `expiresAt` is in the future AND each entry validates against the Zod schema
|
||||
- **THEN** the quarantine-lint step passes silently
|
||||
|
||||
### Requirement: Handle, timer, and socket leak detection on teardown
|
||||
|
||||
After every fixture or `TestEnvironment.teardown()` call, the runner SHALL snapshot active timers, sockets, and file descriptors via `Bun.peek` / runtime handle table on Bun or `process._getActiveHandles()` on Node fallback. Any leak SHALL fail the suite with a categorised error: `TESTING.LEAK.HANDLE`, `TESTING.LEAK.TIMER`, or `TESTING.LEAK.SOCKET`, each carrying `{ count: number; sample: unknown[] }` context.
|
||||
|
||||
#### Scenario: Timer leaked
|
||||
|
||||
- **WHEN** a teardown returns while at least one `setInterval` is still active
|
||||
- **THEN** the runner throws `TESTING.LEAK.TIMER` with `context.count >= 1`
|
||||
|
||||
#### Scenario: Socket leaked
|
||||
|
||||
- **WHEN** a teardown returns while at least one TCP socket remains open
|
||||
- **THEN** the runner throws `TESTING.LEAK.SOCKET` AND `context.sample[0]` describes the offending descriptor
|
||||
|
||||
#### Scenario: Clean teardown
|
||||
|
||||
- **WHEN** all timers, sockets, and handles attributable to fixtures are released before teardown returns
|
||||
- **THEN** the leak assertion passes silently AND no error is thrown
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Coverage and docs gates
|
||||
|
||||
Istanbul coverage across `packages/universal/testing/src/` (excluding `*.bench.ts`, `*.schemas.ts`, `*.test.ts`, inner-dir `*.types.ts`, and barrel `index.ts`) SHALL be 100% on statements, branches, functions, and lines — no exceptions, no "foundation only" carve-out, no degraded-situation floor (per `openspec/project.md`). The pre-publish gate (`bunx turbo test:coverage --filter=@suaveplan/testing`) MUST hard-fail under 100%. Every implementation file MUST have a sibling `.md` of ≥ 200 non-code words containing both a simple AND an advanced usage example. Every test wrapped by `markFlaky` SHALL be exempted from coverage attribution to avoid retry-based double counting; the exemption MUST be recorded in the istanbul ignore-set automatically by the wrapper.
|
||||
|
||||
This requirement replaces the previous "≥ 95%" formulation verbatim quoted here for traceability: "Istanbul statement + branch coverage across `packages/universal/testing/src/` (excluding `*.bench.ts` and `*.schemas.ts`) MUST be ≥ 95%."
|
||||
|
||||
#### Scenario: Coverage gate in CI
|
||||
|
||||
- **WHEN** `bunx turbo test:coverage --filter=@suaveplan/testing` runs
|
||||
- **THEN** reported statement, branch, function, and line coverage are each 100% AND the gate exits non-zero on anything less
|
||||
|
||||
#### Scenario: Docs coverage enforced
|
||||
|
||||
- **WHEN** a script enumerates implementation files under `src/` (excluding `*.test.ts`, `*.bench.ts`, `*.schemas.ts`, inner-dir `*.types.ts`, and barrel `index.ts`) and checks for sibling `.md` files
|
||||
- **THEN** every implementation file has a sibling `.md` AND every `.md` is ≥ 200 non-code words AND each `.md` contains both a simple and an advanced usage example
|
||||
|
||||
#### Scenario: Flaky retries do not inflate coverage
|
||||
|
||||
- **WHEN** a `markFlaky`-wrapped test executes its body twice (initial + one retry)
|
||||
- **THEN** the second execution does NOT contribute additional counts to istanbul AND the coverage report is identical to the single-execution case
|
||||
Reference in New Issue
Block a user