Initial Commit

openspec/changes/enterprise-testbed-verification/specs/testbed/spec.md

@@ -0,0 +1,107 @@
+## ADDED Requirements
+
+### Requirement: Per-page interaction spec coverage
+
+Every route registered in `apps/testbed/src/routes.ts` SHALL have a corresponding `<route>.interaction.spec.ts` under `apps/testbed/e2e/` that:
+
+1. Navigates to the route.
+2. Asserts the element with `data-testid="page-title"` resolves and has non-empty text content.
+3. Iterates every `data-testid="demo-card"` rendered on the page; for each card, triggers its primary action (button click, form submit, or focus event as the card defines) AND asserts the card-scoped `data-testid="output"` populates with non-empty content within 5 seconds.
+4. Includes at least one demo-specific assertion beyond the title and output checks (e.g. an output regex, a computed style assertion, or a returned JSON shape assertion).
+
+A "baseline-only" spec — one that merely navigates and asserts a non-empty page title — MUST NOT be the sole coverage for a route. A baseline navigation spec MAY exist alongside an interaction spec but does not satisfy this requirement on its own.
+
+#### Scenario: New route ships with interaction spec
+
+- **WHEN** a new route `/demos/widgets` is added to `apps/testbed/src/routes.ts`
+- **AND** the only e2e file referencing the route is `widgets.baseline.spec.ts` asserting only the page title
+- **THEN** the testbed verification gate fails with a missing-coverage diagnostic naming `widgets.interaction.spec.ts`
+
+#### Scenario: Demo card without output fails the spec
+
+- **WHEN** an interaction spec triggers a demo card's primary action
+- **AND** the card-scoped `data-testid="output"` remains empty for 5 seconds
+- **THEN** the spec fails with a timeout error referencing the card's testid
+
+### Requirement: Per-page invariants attached on every interaction spec
+
+Every interaction spec SHALL attach and exercise ALL of the following invariants, in addition to its functional assertions:
+
+1. `attachConsoleMonitor(page)` followed by `assertNoErrors()` and `assertNoWarnings()` at end of test.
+2. `attachNetworkMonitor(page)` followed by `assertNoFailures()`, `assertNoMissingAssets()`, `assertNoMixedContent()`, and `assertNoCorsErrors()` at end of test.
+3. `attachAxe(page).assertNoViolations()` per the contract introduced by `enterprise-testing-a11y` (WCAG 2.1 AA).
+4. `expect(page).toHaveScreenshot()` per the contract introduced by `enterprise-testing-visual-regression`, with a stable viewport and `prefers-reduced-motion: reduce`.
+5. The interaction-coverage assertion defined in the previous requirement.
+
+A spec that omits any of these five invariants does NOT satisfy this requirement.
+
+#### Scenario: Spec missing axe assertion fails the gate
+
+- **WHEN** an interaction spec attaches console + network + visual + interaction coverage but not `attachAxe`
+- **THEN** the testbed verification gate fails with a missing-invariant diagnostic naming the omitted helper
+
+#### Scenario: All invariants attached and passing
+
+- **WHEN** an interaction spec attaches all five invariants and every assertion passes
+- **THEN** the spec passes the per-page invariant requirement
+
+### Requirement: Release-readiness CI gate
+
+A CI job named `release-readiness` SHALL run against `apps/testbed` deployed to the testbed Docker stack defined by `enterprise-testing-operational`. The job SHALL fail on any of:
+
+1. Any console error or warning (per the console invariant).
+2. Any network failure, missing asset, mixed content, CORS error, or unexpected request (per the network invariant).
+3. Any axe accessibility violation at WCAG 2.1 AA (per `enterprise-testing-a11y`).
+4. Any visual-regression diff exceeding 0.5% from the committed baseline (per `enterprise-testing-visual-regression`).
+5. Any broken internal link reported by `crawlInternalLinks(page, { maxDepth: 2, sameOriginOnly: true })`.
+6. Any HTTP 5xx response from any Hono API route under the load smoke defined by `enterprise-testing-load-perf`.
+
+The release-readiness gate is mandatory for any pull request whose change set touches `apps/testbed` or any package in its dependency closure.
+
+#### Scenario: Broken internal link blocks the gate
+
+- **WHEN** the testbed footer links to `/legal/terms` which returns 404
+- **AND** `release-readiness` runs
+- **THEN** the gate fails with a `TESTING_E2E.CRAWL.BROKEN_LINK` diagnostic
+
+#### Scenario: 5xx under load blocks the gate
+
+- **WHEN** the load smoke for an API route observes any HTTP 5xx response
+- **AND** `release-readiness` runs
+- **THEN** the gate fails citing the responsible route and status
+
+#### Scenario: Clean run passes the gate
+
+- **WHEN** all six failure conditions are absent
+- **AND** `release-readiness` runs
+- **THEN** the gate passes
+
+### Requirement: Zero-defect-to-live enforcement
+
+Failure of the `release-readiness` gate SHALL block merge to `main`, and SHALL block merge to `genesis` until graduation. There is no override path for "ship now, fix later". The `CLAUDE.md` "Web App & Page Verification" rule (bullet 6, "Zero-defect push to live") becomes machine-enforced through this requirement.
+
+#### Scenario: Failing gate blocks merge
+
+- **WHEN** a pull request touching `apps/testbed` triggers `release-readiness`
+- **AND** the gate fails for any reason listed in the previous requirement
+- **THEN** the branch protection prevents the PR from being merged to `main` (or to `genesis` pre-graduation)
+
+#### Scenario: No override token
+
+- **WHEN** a maintainer attempts to merge a PR whose `release-readiness` gate has failed
+- **THEN** no label, comment, or commit-trailer SHALL bypass the protection
+
+### Requirement: Component-on-page coverage complements Storybook coverage
+
+Storybook stories (per `enterprise-browser-storybook-test-utils`) cover components in isolation; interaction specs (defined above) cover components rendered on actual pages. The two coverage layers are complementary: a passing Storybook test SHALL NOT be accepted as a substitute for an interaction-spec assertion exercising the same component on a live page, and vice versa.
+
+#### Scenario: Component covered only by Storybook
+
+- **WHEN** a component is exercised only in `*.stories.tsx` and never rendered by any testbed route's interaction spec
+- **AND** the component appears on a testbed route in production
+- **THEN** the testbed verification gate fails with a missing-on-page-coverage diagnostic
+
+#### Scenario: Component covered only on-page
+
+- **WHEN** a component is exercised by an interaction spec but has no `*.stories.tsx`
+- **THEN** the broader package quality gate (per `enterprise-browser-storybook-test-utils`) fails with a missing-story diagnostic; the testbed gate is not the enforcement point for that case

openspec/changes/enterprise-testbed-verification/specs/testing-e2e/spec.md

@@ -0,0 +1,110 @@
+## ADDED Requirements
+
+### Requirement: NetworkMonitor MUST assert absence of missing assets
+
+`NetworkMonitor` (returned by `attachNetworkMonitor(page)`) SHALL expose `assertNoMissingAssets()` that fails when any 404 was observed for an `image`, `font`, `script`, or `stylesheet` resource type. The monitor SHALL also expose `allow404(predicate: (entry) => boolean)` to opt-out specific URLs from this assertion. Other 404s (e.g. for `xhr`/`fetch` resource types) are NOT subject to this assertion and remain governed by `assertNoFailures()`.
+
+#### Scenario: Missing image fails the assertion
+
+- **WHEN** the page references `<img src="/missing.png">` and the request returns 404
+- **AND** `monitor.assertNoMissingAssets()` is called
+- **THEN** the call throws `MissingAssetError` with `code === "TESTING_E2E.NETWORK.MISSING_ASSET"` and `context.url` containing `/missing.png`
+
+#### Scenario: Allow-listed 404 does not fail
+
+- **WHEN** `monitor.allow404((e) => e.url.endsWith("/beacon.gif"))` is called before navigation
+- **AND** `/beacon.gif` returns 404
+- **AND** `monitor.assertNoMissingAssets()` is called
+- **THEN** the call returns without throwing
+
+#### Scenario: Non-asset 404 does not fail this assertion
+
+- **WHEN** an `xhr` request to `/api/foo` returns 404
+- **AND** `monitor.assertNoMissingAssets()` is called
+- **THEN** the call returns without throwing (the failure is reported by `assertNoFailures()` instead)
+
+### Requirement: NetworkMonitor MUST assert absence of mixed content
+
+`NetworkMonitor` SHALL expose `assertNoMixedContent()` that fails when the page is loaded over `https://` AND any subresource was fetched over `http://`, OR when the browser reports a mixed-content blocking event. The assertion is a no-op when the page is loaded over `http://`.
+
+#### Scenario: HTTP subresource on HTTPS page fails
+
+- **WHEN** the page is `https://example.test/`
+- **AND** it includes `<script src="http://cdn.test/lib.js">`
+- **AND** `monitor.assertNoMixedContent()` is called
+- **THEN** the call throws `MixedContentError` with `code === "TESTING_E2E.NETWORK.MIXED_CONTENT"` and `context.url` containing `http://cdn.test/lib.js`
+
+#### Scenario: Plain HTTP page is exempt
+
+- **WHEN** the page is `http://localhost:3000/`
+- **AND** `monitor.assertNoMixedContent()` is called
+- **THEN** the call returns without throwing
+
+### Requirement: NetworkMonitor MUST assert absence of CORS errors
+
+`NetworkMonitor` SHALL expose `assertNoCorsErrors()` that fails on any of: a request that failed with a `net::ERR_CORS_*` error code, a preflight (`OPTIONS`) response that did not satisfy the subsequent request, or a response missing a required `Access-Control-Allow-Origin` header that the browser blocked.
+
+#### Scenario: CORS preflight failure fails the assertion
+
+- **WHEN** an `OPTIONS` preflight returns without `Access-Control-Allow-Origin` and the browser cancels the follow-up `POST`
+- **AND** `monitor.assertNoCorsErrors()` is called
+- **THEN** the call throws `CorsError` with `code === "TESTING_E2E.NETWORK.CORS_ERROR"`
+
+#### Scenario: Same-origin requests are unaffected
+
+- **WHEN** all requests are same-origin and complete normally
+- **AND** `monitor.assertNoCorsErrors()` is called
+- **THEN** the call returns without throwing
+
+### Requirement: NetworkMonitor MUST assert request graph against an allow-list
+
+`NetworkMonitor` SHALL expose `assertNoUnexpectedRequests({ allowed: (url: string) => boolean })` that fails when any observed request URL fails the `allowed` predicate. This assertion is opt-in and intended for tests requiring strict request-graph control.
+
+#### Scenario: Unexpected request fails the assertion
+
+- **WHEN** the page made a request to `https://tracker.test/pixel`
+- **AND** `monitor.assertNoUnexpectedRequests({ allowed: (u) => u.startsWith("https://example.test") })` is called
+- **THEN** the call throws `UnexpectedRequestError` with `code === "TESTING_E2E.NETWORK.UNEXPECTED_REQUEST"` and `context.url === "https://tracker.test/pixel"`
+
+#### Scenario: All allowed requests pass
+
+- **WHEN** every request URL satisfies the predicate
+- **AND** `monitor.assertNoUnexpectedRequests({ allowed })` is called
+- **THEN** the call returns without throwing
+
+### Requirement: `crawlInternalLinks` MUST walk and verify internal links
+
+The package SHALL export `crawlInternalLinks(page, options?: { maxDepth?: number; sameOriginOnly?: boolean; ignore?: (url: string) => boolean }): Promise<CrawlReport>` that BFS-walks all `<a href>` links from the entry page, follows internal links up to `maxDepth` (default 2), and asserts each link returns 2xx via HEAD with GET fallback. `sameOriginOnly` defaults to `true`. The returned `CrawlReport` SHALL list visited URLs, broken links, and any circular redirects encountered.
+
+#### Scenario: Crawler reports broken internal link
+
+- **WHEN** the page links to `/missing` which returns 404
+- **AND** `crawlInternalLinks(page)` is called
+- **THEN** the returned report contains an entry whose `error.code === "TESTING_E2E.CRAWL.BROKEN_LINK"` and whose `url` ends in `/missing`
+
+#### Scenario: Crawler honours sameOriginOnly default
+
+- **WHEN** the page links to `https://external.test/foo`
+- **AND** `crawlInternalLinks(page)` is called with default options
+- **THEN** `https://external.test/foo` is NOT visited and is NOT reported as broken
+
+#### Scenario: Crawler reports circular redirect
+
+- **WHEN** `/a` redirects to `/b` and `/b` redirects to `/a`
+- **AND** `crawlInternalLinks(page)` encounters this chain
+- **THEN** the report contains an entry whose `error.code === "TESTING_E2E.CRAWL.CIRCULAR_REDIRECT"`
+
+#### Scenario: Crawler reports max-depth exceeded
+
+- **WHEN** the link graph contains paths deeper than `maxDepth`
+- **AND** `crawlInternalLinks(page, { maxDepth: 1 })` is called
+- **THEN** unvisited deeper links are reported with `error.code === "TESTING_E2E.CRAWL.MAX_DEPTH_EXCEEDED"`
+
+### Requirement: Network and crawl errors MUST follow the structured-error contract
+
+All errors thrown or reported by the new NetworkMonitor methods and `crawlInternalLinks` MUST be subclasses of `TestingE2eError` carrying `{ code, cause?, context? }`. Codes MUST match `^TESTING_E2E\.(NETWORK|CRAWL)\.[A-Z_]+$`.
+
+#### Scenario: Code shape
+
+- **WHEN** any of `MissingAssetError`, `MixedContentError`, `CorsError`, `UnexpectedRequestError`, `BrokenLinkError`, `CircularRedirectError`, `MaxDepthExceededError` is thrown
+- **THEN** `err instanceof TestingE2eError` is true AND `err.code` matches the regex