82 lines
5.4 KiB
Markdown
82 lines
5.4 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: Flake quarantine via `markFlaky`
|
|
|
|
The package SHALL export `markFlaky({ retries, reason, ticket })` that wraps a Vitest `it`/`test` callback with bounded retries (max 3). Each retry SHALL emit `testing.flake.retry_used` as an OTel counter with attributes `{ test, attempt }`; the wrapper SHALL emit `testing.flake.count` as a counter on initial registration. The `ticket` field MUST be a non-empty issue URL validated by Zod (`url()` + `.min(1)`); a missing or invalid ticket MUST throw `TESTING.FLAKE.TICKET_REQUIRED` at registration time, not at first run.
|
|
|
|
#### Scenario: Bounded retries succeed
|
|
|
|
- **WHEN** `markFlaky({ retries: 2, reason: 'docker DNS jitter', ticket: 'https://example.com/issues/42' })` wraps a test that fails once then passes
|
|
- **THEN** the test reports green AND `testing.flake.retry_used` records exactly one increment
|
|
|
|
#### Scenario: Retries above the cap rejected
|
|
|
|
- **WHEN** `markFlaky({ retries: 5, ... })` is called
|
|
- **THEN** `markFlaky` throws `TESTING.FLAKE.RETRIES_OUT_OF_RANGE` with `context.max === 3`
|
|
|
|
#### Scenario: Missing ticket rejected at registration
|
|
|
|
- **WHEN** `markFlaky({ retries: 1, reason: 'x', ticket: '' })` is called
|
|
- **THEN** `markFlaky` throws `TESTING.FLAKE.TICKET_REQUIRED` immediately AND the test is not registered
|
|
|
|
### Requirement: Quarantine ledger
|
|
|
|
A `quarantine.json` file SHALL live at the repository root and enumerate every flaky test as `{ path: string; name: string; ticket: string; addedAt: ISO8601; expiresAt: ISO8601 }`. CI SHALL fail when any entry's `expiresAt` is in the past, forcing triage. The ledger schema MUST be exported as a Zod schema from the testing package so external tooling can validate it without re-implementing the shape.
|
|
|
|
#### Scenario: Expired entry fails CI
|
|
|
|
- **WHEN** `quarantine.json` contains an entry with `expiresAt` earlier than the current time
|
|
- **THEN** the quarantine-lint step exits non-zero AND CI fails with a message naming the expired entries
|
|
|
|
#### Scenario: Schema mismatch rejected
|
|
|
|
- **WHEN** `quarantine.json` is missing the `ticket` field for any entry
|
|
- **THEN** the loader throws `TESTING.QUARANTINE.SCHEMA_INVALID` AND the suite refuses to start
|
|
|
|
#### Scenario: Active entry passes
|
|
|
|
- **WHEN** every entry's `expiresAt` is in the future AND each entry validates against the Zod schema
|
|
- **THEN** the quarantine-lint step passes silently
|
|
|
|
### Requirement: Handle, timer, and socket leak detection on teardown
|
|
|
|
After every fixture or `TestEnvironment.teardown()` call, the runner SHALL snapshot active timers, sockets, and file descriptors via `Bun.peek` / runtime handle table on Bun or `process._getActiveHandles()` on Node fallback. Any leak SHALL fail the suite with a categorised error: `TESTING.LEAK.HANDLE`, `TESTING.LEAK.TIMER`, or `TESTING.LEAK.SOCKET`, each carrying `{ count: number; sample: unknown[] }` context.
|
|
|
|
#### Scenario: Timer leaked
|
|
|
|
- **WHEN** a teardown returns while at least one `setInterval` is still active
|
|
- **THEN** the runner throws `TESTING.LEAK.TIMER` with `context.count >= 1`
|
|
|
|
#### Scenario: Socket leaked
|
|
|
|
- **WHEN** a teardown returns while at least one TCP socket remains open
|
|
- **THEN** the runner throws `TESTING.LEAK.SOCKET` AND `context.sample[0]` describes the offending descriptor
|
|
|
|
#### Scenario: Clean teardown
|
|
|
|
- **WHEN** all timers, sockets, and handles attributable to fixtures are released before teardown returns
|
|
- **THEN** the leak assertion passes silently AND no error is thrown
|
|
|
|
## MODIFIED Requirements
|
|
|
|
### Requirement: Coverage and docs gates
|
|
|
|
Istanbul coverage across `packages/universal/testing/src/` (excluding `*.bench.ts`, `*.schemas.ts`, `*.test.ts`, inner-dir `*.types.ts`, and barrel `index.ts`) SHALL be 100% on statements, branches, functions, and lines — no exceptions, no "foundation only" carve-out, no degraded-situation floor (per `openspec/project.md`). The pre-publish gate (`bunx turbo test:coverage --filter=@suaveplan/testing`) MUST hard-fail under 100%. Every implementation file MUST have a sibling `.md` of ≥ 200 non-code words containing both a simple AND an advanced usage example. Every test wrapped by `markFlaky` SHALL be exempted from coverage attribution to avoid retry-based double counting; the exemption MUST be recorded in the istanbul ignore-set automatically by the wrapper.
|
|
|
|
This requirement replaces the previous "≥ 95%" formulation verbatim quoted here for traceability: "Istanbul statement + branch coverage across `packages/universal/testing/src/` (excluding `*.bench.ts` and `*.schemas.ts`) MUST be ≥ 95%."
|
|
|
|
#### Scenario: Coverage gate in CI
|
|
|
|
- **WHEN** `bunx turbo test:coverage --filter=@suaveplan/testing` runs
|
|
- **THEN** reported statement, branch, function, and line coverage are each 100% AND the gate exits non-zero on anything less
|
|
|
|
#### Scenario: Docs coverage enforced
|
|
|
|
- **WHEN** a script enumerates implementation files under `src/` (excluding `*.test.ts`, `*.bench.ts`, `*.schemas.ts`, inner-dir `*.types.ts`, and barrel `index.ts`) and checks for sibling `.md` files
|
|
- **THEN** every implementation file has a sibling `.md` AND every `.md` is ≥ 200 non-code words AND each `.md` contains both a simple and an advanced usage example
|
|
|
|
#### Scenario: Flaky retries do not inflate coverage
|
|
|
|
- **WHEN** a `markFlaky`-wrapped test executes its body twice (initial + one retry)
|
|
- **THEN** the second execution does NOT contribute additional counts to istanbul AND the coverage report is identical to the single-execution case
|