187 lines
12 KiB
Markdown
187 lines
12 KiB
Markdown
## ADDED Requirements
|
|
|
|
### Requirement: ServiceFixture<TConfig> contract
|
|
|
|
The `@suaveplan/testing-integration` package SHALL export an interface `ServiceFixture<TConfig extends Record<string, unknown>>` that every plugin fixture MUST implement. The interface defines the universal lifecycle of a service used by integration tests, whether backed by a testcontainers-managed Docker container or by an env-bridge passthrough to an already-running service.
|
|
|
|
The interface MUST expose the following members:
|
|
|
|
- `readonly name: string` — globally unique among fixtures registered into one `TestEnvironment`.
|
|
- `readonly config: TConfig` — the resolved configuration after `start()`; before `start()` MAY be a partial draft.
|
|
- `readonly dependsOn?: readonly string[]` — names of fixtures that MUST start before this one.
|
|
- `start(opts: { signal: AbortSignal }): Promise<void>` — owns container startup and the health-poll loop; honours abort.
|
|
- `stop(opts: { signal: AbortSignal }): Promise<void>` — releases all resources; idempotent; no-op in env-bridge passthrough mode.
|
|
- `isHealthy(): Promise<boolean>` — single health probe; called by `HealthPoller`; never throws.
|
|
- `env(): Record<string, string>` — the full set of environment variables the fixture forwards to worker pools.
|
|
|
|
A utility type `InferConfig<F>` SHALL extract `TConfig` from a `ServiceFixture` implementation; it returns `never` for non-fixture types.
|
|
|
|
#### Scenario: Fixture exposes the full env set
|
|
|
|
- **WHEN** a plugin instance is constructed and `env()` is read
|
|
- **THEN** the returned record contains every environment variable name the plugin's spec declares — no implicit filtering
|
|
|
|
#### Scenario: Idempotent stop is safe
|
|
|
|
- **WHEN** `stop()` is awaited twice on the same fixture instance
|
|
- **THEN** the second call resolves without throwing AND does not attempt to interact with already-released resources
|
|
|
|
#### Scenario: InferConfig narrows TConfig
|
|
|
|
- **WHEN** `InferConfig<RedisFixture>` is evaluated where `RedisFixture extends ServiceFixture<{ url: string }>`
|
|
- **THEN** the resulting type is `{ url: string }`
|
|
|
|
### Requirement: TestEnvironment topological lifecycle
|
|
|
|
`TestEnvironment` SHALL accept a `Record<string, ServiceFixture<…>>` plus `TestEnvironmentOptions` and expose `setupAll()` / `teardownAll()`. `setupAll()` MUST compute a topological ordering from each fixture's `dependsOn` and start fixtures in parallel **within** each topological layer. `teardownAll()` MUST stop fixtures in reverse-topological order. A `started` flag MUST prevent double-start. A fixture that never started MUST never be stopped. A failing stop MUST NOT short-circuit teardown — `teardownAll()` MUST drain remaining fixtures and aggregate errors.
|
|
|
|
#### Scenario: Parallel start within a layer
|
|
|
|
- **WHEN** three fixtures `a`, `b`, `c` have no `dependsOn` and `setupAll()` is called
|
|
- **THEN** all three `start()` calls are in flight before any resolves
|
|
|
|
#### Scenario: Topological order respects dependsOn
|
|
|
|
- **WHEN** fixture `app` declares `dependsOn: ["oidc"]` and `setupAll()` runs
|
|
- **THEN** `oidc.start()` resolves before `app.start()` is invoked
|
|
|
|
#### Scenario: Reverse-topological stop
|
|
|
|
- **WHEN** `teardownAll()` runs against the same set
|
|
- **THEN** `app.stop()` is invoked before `oidc.stop()`
|
|
|
|
#### Scenario: Failed start does not start dependents and does not block teardown
|
|
|
|
- **WHEN** `oidc.start()` rejects
|
|
- **THEN** `app.start()` is never invoked AND `teardownAll()` calls `oidc.stop()` only if `oidc.start()` had begun, AND remaining fixtures' `stop()` calls all run regardless of any single failure
|
|
|
|
#### Scenario: Double setup throws
|
|
|
|
- **WHEN** `setupAll()` is awaited twice on the same instance
|
|
- **THEN** the second call rejects with `INTEGRATION.ALREADY_STARTED`
|
|
|
|
### Requirement: HealthPoller contract
|
|
|
|
`HealthPoller` SHALL accept `{ intervalMs?: number; timeoutMs?: number; sleep?: (ms: number) => Promise<void> }` with defaults `intervalMs = 500` and `timeoutMs = 120_000`. `poll(fixture, { signal })` MUST call `fixture.isHealthy()` repeatedly until it returns `true` or the deadline elapses. On deadline, the promise rejects with a `TestEnvironmentError` whose `code === "INTEGRATION.HEALTH_TIMEOUT"`, `context.fixtureName`, and `context.elapsedMs` are populated. The poller MUST honour `AbortSignal` and reject with `INTEGRATION.ABORTED` on first observed abort.
|
|
|
|
#### Scenario: Healthy fixture resolves promptly
|
|
|
|
- **WHEN** `fixture.isHealthy()` returns `true` on the first call
|
|
- **THEN** `poll()` resolves before a second call is scheduled
|
|
|
|
#### Scenario: Deadline elapsed
|
|
|
|
- **WHEN** `fixture.isHealthy()` always returns `false` and 120 s elapse
|
|
- **THEN** `poll()` rejects with `INTEGRATION.HEALTH_TIMEOUT` AND `context.elapsedMs >= 120000`
|
|
|
|
#### Scenario: Abort honoured
|
|
|
|
- **WHEN** the caller-provided `signal` aborts mid-poll
|
|
- **THEN** `poll()` rejects with `INTEGRATION.ABORTED` within one event-loop tick of the abort
|
|
|
|
### Requirement: skipIfUnavailable() helper
|
|
|
|
The package SHALL export `skipIfUnavailable(fixtureName: string): void` that, when called from the body of a Vitest `it`/`test`, causes the test to be **skipped** (not failed) when the named fixture failed to start during `globalSetup`. The helper MUST read the inject key `__suaveplan_integration_failed__` and gracefully no-op when the inject key is absent (e.g. unit-test mode). A bun-test mode SHALL emit the equivalent `it.skip` behaviour.
|
|
|
|
#### Scenario: Skip on partial failure
|
|
|
|
- **WHEN** `surrealdb.start()` failed during global setup, `provide("__suaveplan_integration_failed__", { surrealdb: true })` was called, and a test calls `skipIfUnavailable("surrealdb")`
|
|
- **THEN** the test is reported as skipped, not failed
|
|
|
|
#### Scenario: No-op when inject key absent
|
|
|
|
- **WHEN** the test runner has no global-setup wired and `skipIfUnavailable("anything")` is called
|
|
- **THEN** the call returns without throwing AND the test continues
|
|
|
|
### Requirement: createIntegrationSetup() factory and Vitest global-setup bridge
|
|
|
|
`createIntegrationSetup(fixtures)` SHALL register the supplied fixture map in a module-level registry and return a `globalSetup` array suitable for a Vitest config. At Vitest startup, the bridge MUST construct a single shared `TestEnvironment` from the registered fixtures, run `setupAll()`, and on partial failure call `provide("__suaveplan_integration_failed__", failures)` where `failures` is `Record<string, true>` keyed by fixture name. On full success it MUST `provide()` an empty record. Every env var returned by every fixture's `env()` MUST be set on `process.env` before the worker pool is forked.
|
|
|
|
#### Scenario: Successful setup propagates env
|
|
|
|
- **WHEN** all fixtures start successfully and their `env()` returns disjoint keys
|
|
- **THEN** every key is present in `process.env` before any test worker forks
|
|
|
|
#### Scenario: Partial failure is reported via inject key
|
|
|
|
- **WHEN** one fixture's `start()` rejects
|
|
- **THEN** `provide("__suaveplan_integration_failed__", { <failedName>: true })` is invoked AND the worker pool still starts so unaffected tests can run
|
|
|
|
### Requirement: Deterministic injection propagation via TestEnvironmentOptions
|
|
|
|
`TestEnvironmentOptions` SHALL include `clock?: Clock; rng?: Rng; signal?: AbortSignal`. When a fixture's config declares `acceptsContext: true`, `TestEnvironment` MUST forward the same `clock` / `rng` / `signal` references to that fixture before invoking `start()`. Fixtures that do not opt in MUST be unaffected. This requirement extends the deterministic-injection contract from `openspec/specs/testing/spec.md` into the integration substrate.
|
|
|
|
#### Scenario: Opt-in fixture sees the shared clock
|
|
|
|
- **WHEN** `TestEnvironment` is constructed with `{ clock: fakeClock }` and a fixture's config declares `acceptsContext: true`
|
|
- **THEN** that fixture's `config.clock === fakeClock` at the time `start()` is invoked
|
|
|
|
#### Scenario: Non-opt-in fixture is unaffected
|
|
|
|
- **WHEN** the same `TestEnvironment` includes a fixture whose config does NOT declare `acceptsContext: true`
|
|
- **THEN** that fixture's config is left untouched
|
|
|
|
### Requirement: Leak detection on teardown
|
|
|
|
After `teardownAll()` resolves, `TestEnvironment` SHALL inspect the host runtime's active-handle set (e.g. `process._getActiveHandles()` on Node, the Bun equivalent on Bun) and emit a structured warning with code `INTEGRATION.LEAK_DETECTED` plus an OTel counter increment `testing.integration.leak.count` for each handle attributable to a fixture that should have released. The warning MUST identify the fixture by name when attribution is possible.
|
|
|
|
#### Scenario: Handle leak detected
|
|
|
|
- **WHEN** a fixture's `stop()` returns but a TCP socket it owned remains open
|
|
- **THEN** `teardownAll()` emits an `INTEGRATION.LEAK_DETECTED` warning with `context.fixtureName` AND increments `testing.integration.leak.count`
|
|
|
|
### Requirement: Flake quarantine
|
|
|
|
The package SHALL export `markFlaky(testFn, { retries, reason })` that wraps a test with bounded retries and emits a `testing.integration.flake.count` metric increment per retry attempt. Quarantined tests MUST be enumerated in a `quarantine.json` at the package root listing `{ test: string; reason: string; addedAt: string }` entries. A test that exhausts its retries MUST fail with `INTEGRATION.QUARANTINED` carrying the `reason` in `context`.
|
|
|
|
#### Scenario: Flaky test passes within budget
|
|
|
|
- **WHEN** `markFlaky(fn, { retries: 3, reason: "TC-123" })` is run and the underlying test fails twice then passes
|
|
- **THEN** the test is reported as passing AND `testing.integration.flake.count` is incremented twice
|
|
|
|
#### Scenario: Flaky test exhausts budget
|
|
|
|
- **WHEN** the underlying test fails on every retry
|
|
- **THEN** the wrapping test fails with `INTEGRATION.QUARANTINED` AND `context.reason === "TC-123"`
|
|
|
|
### Requirement: Coverage gate
|
|
|
|
Istanbul statement, branch, function, and line coverage across `packages/testing/testing-integration/src/` MUST each be exactly `100 %`. The pre-publish gate (`bunx turbo test:coverage --filter=@suaveplan/testing-integration`) MUST hard-fail below `100 %` on any dimension.
|
|
|
|
#### Scenario: Coverage gate enforced
|
|
|
|
- **WHEN** the pre-publish gate runs
|
|
- **THEN** statement / branch / function / line coverage are each `100 %`
|
|
|
|
### Requirement: OTel observability
|
|
|
|
Every public entrypoint SHALL emit OTel counters (`testing.integration.<op>.count`), histograms (`testing.integration.<op>.duration_ms`), and an error counter (`testing.integration.<op>.errors{code}`). Spans SHALL be created for `setupAll`, `teardownAll`, and `HealthPoller.poll`. When no SDK is registered, all instrumentation MUST be a no-op with overhead below 5 % per op vs an unwrapped baseline.
|
|
|
|
#### Scenario: Counter increments on every setup
|
|
|
|
- **WHEN** `setupAll()` runs N times against a registered SDK
|
|
- **THEN** `testing.integration.setup.count` records N increments
|
|
|
|
#### Scenario: Error counter records code attribute
|
|
|
|
- **WHEN** `HealthPoller.poll` rejects with `INTEGRATION.HEALTH_TIMEOUT`
|
|
- **THEN** `testing.integration.health-poll.errors` records `code = "INTEGRATION.HEALTH_TIMEOUT"`
|
|
|
|
### Requirement: Banned workspace deps
|
|
|
|
`@suaveplan/testing-integration` `dependencies`, `peerDependencies`, and `devDependencies` MAY list `@suaveplan/types` (the documented types-only exception in CLAUDE.md rule 8) but MUST NOT list any other `@suaveplan/*` workspace package. `testcontainers` MUST be a peer dependency.
|
|
|
|
#### Scenario: Dep-graph guard
|
|
|
|
- **WHEN** `bunx turbo ls --filter=@suaveplan/testing-integration...` runs
|
|
- **THEN** the only `@suaveplan/*` package in the closure is `@suaveplan/types` AND `testcontainers` resolves via peer dependency
|
|
|
|
### Requirement: Inject-key namespace
|
|
|
|
The integration substrate SHALL use the inject key `__suaveplan_integration_failed__` (renamed from the development-time literal `__integration_failed__` used in the implementation plans) for partial-failure propagation. The key MUST NOT collide with any other inject key in the workspace.
|
|
|
|
#### Scenario: Renamed inject key is the contract
|
|
|
|
- **WHEN** any consumer calls `inject("__suaveplan_integration_failed__")`
|
|
- **THEN** the key resolves to a `Record<string, true>` (possibly empty) populated by `createIntegrationSetup`'s global-setup bridge
|