suaveplan/openspec/changes/enterprise-testing-operational/specs/testing/spec.md

ADDED Requirements

Requirement: Flake quarantine via markFlaky

The package SHALL export markFlaky({ retries, reason, ticket }) that wraps a Vitest it/test callback with bounded retries (max 3). Each retry SHALL emit testing.flake.retry_used as an OTel counter with attributes { test, attempt }; the wrapper SHALL emit testing.flake.count as a counter on initial registration. The ticket field MUST be a non-empty issue URL validated by Zod (url() + .min(1)); a missing or invalid ticket MUST throw TESTING.FLAKE.TICKET_REQUIRED at registration time, not at first run.

Scenario: Bounded retries succeed

• WHEN markFlaky({ retries: 2, reason: 'docker DNS jitter', ticket: 'https://example.com/issues/42' }) wraps a test that fails once then passes
• THEN the test reports green AND testing.flake.retry_used records exactly one increment

Scenario: Retries above the cap rejected

• WHEN markFlaky({ retries: 5, ... }) is called
• THEN markFlaky throws TESTING.FLAKE.RETRIES_OUT_OF_RANGE with context.max === 3

Scenario: Missing ticket rejected at registration

• WHEN markFlaky({ retries: 1, reason: 'x', ticket: '' }) is called
• THEN markFlaky throws TESTING.FLAKE.TICKET_REQUIRED immediately AND the test is not registered

Requirement: Quarantine ledger

A quarantine.json file SHALL live at the repository root and enumerate every flaky test as { path: string; name: string; ticket: string; addedAt: ISO8601; expiresAt: ISO8601 }. CI SHALL fail when any entry's expiresAt is in the past, forcing triage. The ledger schema MUST be exported as a Zod schema from the testing package so external tooling can validate it without re-implementing the shape.

Scenario: Expired entry fails CI

• WHEN quarantine.json contains an entry with expiresAt earlier than the current time
• THEN the quarantine-lint step exits non-zero AND CI fails with a message naming the expired entries

Scenario: Schema mismatch rejected

• WHEN quarantine.json is missing the ticket field for any entry
• THEN the loader throws TESTING.QUARANTINE.SCHEMA_INVALID AND the suite refuses to start

Scenario: Active entry passes

• WHEN every entry's expiresAt is in the future AND each entry validates against the Zod schema
• THEN the quarantine-lint step passes silently

Requirement: Handle, timer, and socket leak detection on teardown

After every fixture or TestEnvironment.teardown() call, the runner SHALL snapshot active timers, sockets, and file descriptors via Bun.peek / runtime handle table on Bun or process._getActiveHandles() on Node fallback. Any leak SHALL fail the suite with a categorised error: TESTING.LEAK.HANDLE, TESTING.LEAK.TIMER, or TESTING.LEAK.SOCKET, each carrying { count: number; sample: unknown[] } context.

Scenario: Timer leaked

• WHEN a teardown returns while at least one setInterval is still active
• THEN the runner throws TESTING.LEAK.TIMER with context.count >= 1

Scenario: Socket leaked

• WHEN a teardown returns while at least one TCP socket remains open
• THEN the runner throws TESTING.LEAK.SOCKET AND context.sample[0] describes the offending descriptor

Scenario: Clean teardown

• WHEN all timers, sockets, and handles attributable to fixtures are released before teardown returns
• THEN the leak assertion passes silently AND no error is thrown

MODIFIED Requirements

Requirement: Coverage and docs gates

Istanbul coverage across packages/universal/testing/src/ (excluding *.bench.ts, *.schemas.ts, *.test.ts, inner-dir *.types.ts, and barrel index.ts) SHALL be 100% on statements, branches, functions, and lines — no exceptions, no "foundation only" carve-out, no degraded-situation floor (per openspec/project.md). The pre-publish gate (bunx turbo test:coverage --filter=@suaveplan/testing) MUST hard-fail under 100%. Every implementation file MUST have a sibling .md of ≥ 200 non-code words containing both a simple AND an advanced usage example. Every test wrapped by markFlaky SHALL be exempted from coverage attribution to avoid retry-based double counting; the exemption MUST be recorded in the istanbul ignore-set automatically by the wrapper.

This requirement replaces the previous "≥ 95%" formulation verbatim quoted here for traceability: "Istanbul statement + branch coverage across packages/universal/testing/src/ (excluding *.bench.ts and *.schemas.ts) MUST be ≥ 95%."

Scenario: Coverage gate in CI

• WHEN bunx turbo test:coverage --filter=@suaveplan/testing runs
• THEN reported statement, branch, function, and line coverage are each 100% AND the gate exits non-zero on anything less

Scenario: Docs coverage enforced

• WHEN a script enumerates implementation files under src/ (excluding *.test.ts, *.bench.ts, *.schemas.ts, inner-dir *.types.ts, and barrel index.ts) and checks for sibling .md files
• THEN every implementation file has a sibling .md AND every .md is ≥ 200 non-code words AND each .md contains both a simple and an advanced usage example

Scenario: Flaky retries do not inflate coverage

• WHEN a markFlaky-wrapped test executes its body twice (initial + one retry)
• THEN the second execution does NOT contribute additional counts to istanbul AND the coverage report is identical to the single-execution case