Initial Commit

openspec/changes/enterprise-testing-integration-otel-collector/specs/testing/spec.md

@@ -0,0 +1,53 @@
+## ADDED Requirements
+
+### Requirement: Test-execution OTel export
+
+When the test runner detects that `OTEL_EXPORTER_OTLP_ENDPOINT` is set in its environment, it SHALL emit one OpenTelemetry span per `it` / `test` invocation, one span event per assertion failure, and one span event per fixture lifecycle transition. When the env var is unset the runner MUST NOT create any spans, register any SDK, or perform any export work — the path MUST be a no-op.
+
+Per-test spans SHALL carry attributes `{ suite: string; file: string; retry: number }` and a span name equal to the test's fully-qualified description (e.g. `"FakeClock > advances time"`). Span status MUST be `OK` for passing tests, `ERROR` for failing tests, and `UNSET` for skipped tests. Failure events MUST carry attributes `exception.type`, `exception.message`, and `exception.stacktrace`. Fixture lifecycle events MUST be one of `setup-start`, `setup-end`, `teardown-start`, `teardown-end`.
+
+Implementation hooks SHALL ship at `@suaveplan/testing/reporters/otel` for both Vitest and bun-test. The reporter SHALL accept an optional `{ clock?, idGenerator? }` options bundle so span timestamps and ids are deterministic under test of the reporter itself.
+
+#### Scenario: Span emitted per test when endpoint set
+
+- **WHEN** `OTEL_EXPORTER_OTLP_ENDPOINT` is set AND a vitest suite runs `describe("X", () => it("y", …))`
+- **THEN** exactly one span with name `"X > y"` is exported AND its attributes include `suite === "X"` AND `file` is the absolute spec path AND `retry === 0` on the first attempt
+
+#### Scenario: Assertion failure recorded as span event
+
+- **WHEN** an `it` assertion throws an `AssertionError`
+- **THEN** the test's span has exactly one event whose name is `"exception"` AND whose attributes include `exception.type === "AssertionError"` AND `exception.message` matches the thrown message AND `exception.stacktrace` is the captured stack
+
+#### Scenario: Fixture lifecycle events emitted
+
+- **WHEN** a `FixtureManager` runs `setupAll` over fixture `db` followed by `teardownAll`
+- **THEN** the active span tree contains events `setup-start{name:"db"}`, `setup-end{name:"db"}`, `teardown-start{name:"db"}`, `teardown-end{name:"db"}` in that order
+
+#### Scenario: Skipped test emits UNSET span
+
+- **WHEN** a test is marked `it.skip`
+- **THEN** a span is exported with status `UNSET` AND no `exception` events
+
+#### Scenario: No-op without endpoint
+
+- **WHEN** `OTEL_EXPORTER_OTLP_ENDPOINT` is unset AND the test suite runs
+- **THEN** zero spans are exported AND the reporter MUST NOT instantiate any SDK AND benchmark deltas relative to the unwrapped runner MUST be < 5% per op
+
+### Requirement: Trace-on-failure persistence
+
+For every failed test the runner SHALL serialise the test's full span tree (including child spans from instrumented code under test) to `playwright-report/traces/<spec>.json` on the local filesystem. When `TESTING_TRACE_SINK_URL` is set the runner SHALL additionally upload that file to the supplied URL via HTTP PUT (Garage S3-compatible). Upload failures MUST be logged but MUST NOT fail the test run.
+
+#### Scenario: Failed test serialises trace to disk
+
+- **WHEN** a test in `auth.test.ts` fails
+- **THEN** the file `playwright-report/traces/auth.test.json` exists AND its parsed content contains the failed test's span AND every child span emitted while the test was on the stack
+
+#### Scenario: Trace uploaded when sink configured
+
+- **WHEN** `TESTING_TRACE_SINK_URL=https://garage.example/traces/run-123/` is set AND a test fails
+- **THEN** an HTTP PUT is issued to `https://garage.example/traces/run-123/<spec>.json` AND the request body is byte-identical to the on-disk trace file
+
+#### Scenario: Upload failure does not fail the run
+
+- **WHEN** the trace-sink PUT returns HTTP 500
+- **THEN** a warning is logged AND the test runner exit code remains determined by test outcomes only AND the on-disk trace file is preserved