Initial Commit
This commit is contained in:
34
openspec/changes/enterprise-testbed-verification/design.md
Normal file
34
openspec/changes/enterprise-testbed-verification/design.md
Normal file
@@ -0,0 +1,34 @@
|
||||
## Gate composition
|
||||
|
||||
The release-readiness gate is the union of five orthogonal layers. Each layer is owned by a separate change folder and contributes its own assertions; this change wires them into a single pass/fail outcome at the `apps/testbed` boundary.
|
||||
|
||||
| Layer | Source change | Failure condition |
|
||||
|---|---|---|
|
||||
| Console health | this change (per-page invariant requirement) | any error or warning logged during the spec |
|
||||
| Network health | this change (`NetworkMonitor` extensions) | any failure, missing asset, mixed-content, CORS error, or unexpected request |
|
||||
| Accessibility | `enterprise-testing-a11y` | any axe violation at WCAG 2.1 AA |
|
||||
| Visual regression | `enterprise-testing-visual-regression` | any pixel diff > 0.5% from the committed baseline |
|
||||
| Link reachability | this change (`crawlInternalLinks`) | any 4xx/5xx, circular redirect, or depth-exceeded |
|
||||
| Load smoke | `enterprise-testing-load-perf` | any 5xx from any Hono API route under load smoke |
|
||||
|
||||
The `release-readiness` Playwright project runs these layers in series against the Docker stack stood up per `enterprise-testing-operational`. Because the layers are orthogonal, a single spec failure surfaces the precise root cause via the structured-error code (`TESTING_E2E.NETWORK.MISSING_ASSET`, `TESTING_E2E.CRAWL.BROKEN_LINK`, etc.) — no log-spelunking required.
|
||||
|
||||
## Why a separate `testbed` capability
|
||||
|
||||
`testing-e2e` is a Layer 0 library — runner-agnostic helpers with zero `@suaveplan/*` workspace deps (per `openspec/project.md` rule 9). Per-page invariants for `apps/testbed` are application-level policy: which testids are required, how many demo cards each route must expose, what counts as "baseline-only" coverage. Mixing this into `testing-e2e` would conflate library API with consumer policy. A separate `testbed` capability keeps the library re-usable by other apps while making the testbed's verification contract explicit and auditable.
|
||||
|
||||
## Why broken-link crawling lives in `testing-e2e`
|
||||
|
||||
The crawler is generically useful — any app can call `crawlInternalLinks(page)` against its own routes. It does not embed testbed-specific knowledge. Putting it in `testing-e2e` lets future apps (`apps/docs`, future consumer apps) re-use it without copying.
|
||||
|
||||
## Component-on-page vs Storybook
|
||||
|
||||
Storybook (covered by `enterprise-browser-storybook-test-utils`) renders components in isolation with controlled props. Interaction specs render the real page composition: real router, real data fetching, real layout, real error boundaries. A button can pass every Storybook test and still fail when wired into a page that misroutes its onClick. Both layers are mandatory and neither substitutes for the other; the spec encodes this so future agents do not "consolidate" them away.
|
||||
|
||||
## Allow-listing 404s
|
||||
|
||||
`monitor.allow404(predicate)` exists for legitimate cases (e.g. ad-blocker telemetry beacons, optional CDN preloads). The default is deny-by-default; opt-outs are per-test, predicate-based, and surface the allowed URL in the structured error context for audit.
|
||||
|
||||
## Performance budget for the gate
|
||||
|
||||
The crawler is bounded by `maxDepth` (default 2) and `sameOriginOnly` (default true) to keep the gate tractable on large site maps. Anything deeper belongs to a periodic crawl job, not the per-PR gate.
|
||||
42
openspec/changes/enterprise-testbed-verification/proposal.md
Normal file
42
openspec/changes/enterprise-testbed-verification/proposal.md
Normal file
@@ -0,0 +1,42 @@
|
||||
## Why
|
||||
|
||||
`CLAUDE.md` declares a six-bullet "Web App & Page Verification" rule that any client-side page, website, or web app generated in this monorepo MUST satisfy before being declared done. The rule is normative ("verification is NOT optional"), but it is currently enforced informally — there is no spec that compels every `apps/testbed` route to attach the full invariant set, no spec that defines a release-readiness gate, and no machine-checkable mapping from each CLAUDE.md bullet to a Playwright assertion.
|
||||
|
||||
The test-suite audit at `/mnt/s/web/genesis/docs/superpowers/specs/2026-04-25-test-suite-audit.md` (sections 7 and 8) calls out three gaps that map directly to the six bullets:
|
||||
|
||||
1. `NetworkMonitor` returned by `attachNetworkMonitor(page)` lacks dedicated assertions for missing assets, mixed content, and CORS errors — bullets 4 and 5 of CLAUDE.md cannot be enforced.
|
||||
2. There is no broken-link crawler, so navigation/routing transitions between pages (bullet 2) are only spot-checked.
|
||||
3. The testbed plan (`/mnt/s/web/genesis/docs/superpowers/plans/2026-04-24-testbed-pages-and-e2e.md`) and the design at `/mnt/s/web/genesis/docs/superpowers/specs/2026-04-24-enterprise-testbed-design.md` §7.4 list the invariants every page should run, but specs allow "baseline-only" navigation tests as the sole coverage for a route — bullet 2 ("every component on every page") and bullet 3 ("never trust HTTP 200 alone") are not machine-enforced.
|
||||
|
||||
This change closes those gaps by elevating the six-bullet rule into normative requirements across two capabilities: `testing-e2e` (gains the missing assertions and the crawler) and a new `testbed` capability (defines the per-page invariant contract and the release-readiness gate).
|
||||
|
||||
## What Changes
|
||||
|
||||
### `testing-e2e` — ADDED requirements
|
||||
|
||||
- `NetworkMonitor.assertNoMissingAssets()` — fails on any 404 for image/font/script/stylesheet resource types, with `monitor.allow404(predicate)` opt-out. Maps to CLAUDE.md bullet 5 ("missing assets").
|
||||
- `NetworkMonitor.assertNoMixedContent()` — fails on any `http://` subresource on an `https://` page or any active mixed-content blocking event. Maps to CLAUDE.md bullet 5 ("mixed content").
|
||||
- `NetworkMonitor.assertNoCorsErrors()` — fails on any `Access-Control-Allow-Origin` violation, preflight failure, or `net::ERR_CORS_*`. Maps to CLAUDE.md bullet 5 ("CORS failures").
|
||||
- `NetworkMonitor.assertNoUnexpectedRequests({ allowed })` — declarative allow-list for strict request-graph control.
|
||||
- `crawlInternalLinks(page, options)` — BFS-walks `<a href>` links from an entry page (default depth 2, same-origin), asserts each link returns 2xx, returns typed broken-link reports. Maps to CLAUDE.md bullet 2 ("every navigation/routing transition").
|
||||
- All four error families use stable codes under `TESTING_E2E.NETWORK.*` and `TESTING_E2E.CRAWL.*`.
|
||||
|
||||
### `testbed` — ADDED capability
|
||||
|
||||
A new normative spec for `apps/testbed` verification:
|
||||
|
||||
- **Per-page interaction spec coverage** — every route in `apps/testbed/src/routes.ts` SHALL have a sibling `<route>.interaction.spec.ts` that exercises every `data-testid="demo-card"`, asserts non-empty output, and includes at least one demo-specific assertion. "Baseline-only" specs are forbidden as sole coverage. Maps to bullets 2 and 3.
|
||||
- **Per-page invariants** — every interaction spec SHALL attach `attachConsoleMonitor` (with `assertNoErrors` + `assertNoWarnings`), `attachNetworkMonitor` (with all four assertions above), `attachAxe` (per `enterprise-testing-a11y`), and visual regression (per `enterprise-testing-visual-regression`). Maps to bullets 3, 4, 5.
|
||||
- **Release-readiness gate** — a CI job runs against the testbed Docker stack and fails on any console error/warning, network failure, missing asset, mixed content, CORS error, axe violation, visual diff over 0.5%, broken internal link, or 5xx under load smoke (per `enterprise-testing-load-perf`). Mandatory for any merge touching `apps/testbed` or its dependency closure. Maps to bullets 1 and 6.
|
||||
- **Zero-defect-to-live enforcement** — gate failure blocks merge to `main` (and `genesis` until graduation). Bullet 6 becomes machine-enforced.
|
||||
- **Component-on-page coverage** — Storybook (per `enterprise-browser-storybook-test-utils`) covers components in isolation; interaction specs cover components on actual pages. Both required, neither substitutes for the other.
|
||||
|
||||
This change is additive. It does not modify existing `testing-e2e` requirements; it extends them. It also coordinates with `enterprise-testing-a11y` (axe assertion source), `enterprise-testing-visual-regression` (screenshot baseline policy), `enterprise-testing-load-perf` (load smoke composition), and `enterprise-testing-operational` (Docker staging stack composition).
|
||||
|
||||
## Impact
|
||||
|
||||
- **Affected code:** `packages/universal/testing-e2e/src/network/**` (new assertions), `packages/universal/testing-e2e/src/crawl/**` (new helper), `apps/testbed/e2e/**` (interaction specs), `apps/testbed/playwright.config.ts` (release-readiness project), CI workflow.
|
||||
- **New capability spec:** `openspec/specs/testbed/spec.md` (created on archive).
|
||||
- **Modified capability spec:** `openspec/specs/testing-e2e/spec.md` (folded on archive).
|
||||
- **Cross-references:** `enterprise-testing-a11y`, `enterprise-testing-visual-regression`, `enterprise-testing-load-perf`, `enterprise-testing-operational`, `enterprise-browser-storybook-test-utils`.
|
||||
- **Pre-publish gate:** unchanged — coverage and lint thresholds already 100% per `openspec/AGENTS.md` §7.
|
||||
@@ -0,0 +1,107 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Per-page interaction spec coverage
|
||||
|
||||
Every route registered in `apps/testbed/src/routes.ts` SHALL have a corresponding `<route>.interaction.spec.ts` under `apps/testbed/e2e/` that:
|
||||
|
||||
1. Navigates to the route.
|
||||
2. Asserts the element with `data-testid="page-title"` resolves and has non-empty text content.
|
||||
3. Iterates every `data-testid="demo-card"` rendered on the page; for each card, triggers its primary action (button click, form submit, or focus event as the card defines) AND asserts the card-scoped `data-testid="output"` populates with non-empty content within 5 seconds.
|
||||
4. Includes at least one demo-specific assertion beyond the title and output checks (e.g. an output regex, a computed style assertion, or a returned JSON shape assertion).
|
||||
|
||||
A "baseline-only" spec — one that merely navigates and asserts a non-empty page title — MUST NOT be the sole coverage for a route. A baseline navigation spec MAY exist alongside an interaction spec but does not satisfy this requirement on its own.
|
||||
|
||||
#### Scenario: New route ships with interaction spec
|
||||
|
||||
- **WHEN** a new route `/demos/widgets` is added to `apps/testbed/src/routes.ts`
|
||||
- **AND** the only e2e file referencing the route is `widgets.baseline.spec.ts` asserting only the page title
|
||||
- **THEN** the testbed verification gate fails with a missing-coverage diagnostic naming `widgets.interaction.spec.ts`
|
||||
|
||||
#### Scenario: Demo card without output fails the spec
|
||||
|
||||
- **WHEN** an interaction spec triggers a demo card's primary action
|
||||
- **AND** the card-scoped `data-testid="output"` remains empty for 5 seconds
|
||||
- **THEN** the spec fails with a timeout error referencing the card's testid
|
||||
|
||||
### Requirement: Per-page invariants attached on every interaction spec
|
||||
|
||||
Every interaction spec SHALL attach and exercise ALL of the following invariants, in addition to its functional assertions:
|
||||
|
||||
1. `attachConsoleMonitor(page)` followed by `assertNoErrors()` and `assertNoWarnings()` at end of test.
|
||||
2. `attachNetworkMonitor(page)` followed by `assertNoFailures()`, `assertNoMissingAssets()`, `assertNoMixedContent()`, and `assertNoCorsErrors()` at end of test.
|
||||
3. `attachAxe(page).assertNoViolations()` per the contract introduced by `enterprise-testing-a11y` (WCAG 2.1 AA).
|
||||
4. `expect(page).toHaveScreenshot()` per the contract introduced by `enterprise-testing-visual-regression`, with a stable viewport and `prefers-reduced-motion: reduce`.
|
||||
5. The interaction-coverage assertion defined in the previous requirement.
|
||||
|
||||
A spec that omits any of these five invariants does NOT satisfy this requirement.
|
||||
|
||||
#### Scenario: Spec missing axe assertion fails the gate
|
||||
|
||||
- **WHEN** an interaction spec attaches console + network + visual + interaction coverage but not `attachAxe`
|
||||
- **THEN** the testbed verification gate fails with a missing-invariant diagnostic naming the omitted helper
|
||||
|
||||
#### Scenario: All invariants attached and passing
|
||||
|
||||
- **WHEN** an interaction spec attaches all five invariants and every assertion passes
|
||||
- **THEN** the spec passes the per-page invariant requirement
|
||||
|
||||
### Requirement: Release-readiness CI gate
|
||||
|
||||
A CI job named `release-readiness` SHALL run against `apps/testbed` deployed to the testbed Docker stack defined by `enterprise-testing-operational`. The job SHALL fail on any of:
|
||||
|
||||
1. Any console error or warning (per the console invariant).
|
||||
2. Any network failure, missing asset, mixed content, CORS error, or unexpected request (per the network invariant).
|
||||
3. Any axe accessibility violation at WCAG 2.1 AA (per `enterprise-testing-a11y`).
|
||||
4. Any visual-regression diff exceeding 0.5% from the committed baseline (per `enterprise-testing-visual-regression`).
|
||||
5. Any broken internal link reported by `crawlInternalLinks(page, { maxDepth: 2, sameOriginOnly: true })`.
|
||||
6. Any HTTP 5xx response from any Hono API route under the load smoke defined by `enterprise-testing-load-perf`.
|
||||
|
||||
The release-readiness gate is mandatory for any pull request whose change set touches `apps/testbed` or any package in its dependency closure.
|
||||
|
||||
#### Scenario: Broken internal link blocks the gate
|
||||
|
||||
- **WHEN** the testbed footer links to `/legal/terms` which returns 404
|
||||
- **AND** `release-readiness` runs
|
||||
- **THEN** the gate fails with a `TESTING_E2E.CRAWL.BROKEN_LINK` diagnostic
|
||||
|
||||
#### Scenario: 5xx under load blocks the gate
|
||||
|
||||
- **WHEN** the load smoke for an API route observes any HTTP 5xx response
|
||||
- **AND** `release-readiness` runs
|
||||
- **THEN** the gate fails citing the responsible route and status
|
||||
|
||||
#### Scenario: Clean run passes the gate
|
||||
|
||||
- **WHEN** all six failure conditions are absent
|
||||
- **AND** `release-readiness` runs
|
||||
- **THEN** the gate passes
|
||||
|
||||
### Requirement: Zero-defect-to-live enforcement
|
||||
|
||||
Failure of the `release-readiness` gate SHALL block merge to `main`, and SHALL block merge to `genesis` until graduation. There is no override path for "ship now, fix later". The `CLAUDE.md` "Web App & Page Verification" rule (bullet 6, "Zero-defect push to live") becomes machine-enforced through this requirement.
|
||||
|
||||
#### Scenario: Failing gate blocks merge
|
||||
|
||||
- **WHEN** a pull request touching `apps/testbed` triggers `release-readiness`
|
||||
- **AND** the gate fails for any reason listed in the previous requirement
|
||||
- **THEN** the branch protection prevents the PR from being merged to `main` (or to `genesis` pre-graduation)
|
||||
|
||||
#### Scenario: No override token
|
||||
|
||||
- **WHEN** a maintainer attempts to merge a PR whose `release-readiness` gate has failed
|
||||
- **THEN** no label, comment, or commit-trailer SHALL bypass the protection
|
||||
|
||||
### Requirement: Component-on-page coverage complements Storybook coverage
|
||||
|
||||
Storybook stories (per `enterprise-browser-storybook-test-utils`) cover components in isolation; interaction specs (defined above) cover components rendered on actual pages. The two coverage layers are complementary: a passing Storybook test SHALL NOT be accepted as a substitute for an interaction-spec assertion exercising the same component on a live page, and vice versa.
|
||||
|
||||
#### Scenario: Component covered only by Storybook
|
||||
|
||||
- **WHEN** a component is exercised only in `*.stories.tsx` and never rendered by any testbed route's interaction spec
|
||||
- **AND** the component appears on a testbed route in production
|
||||
- **THEN** the testbed verification gate fails with a missing-on-page-coverage diagnostic
|
||||
|
||||
#### Scenario: Component covered only on-page
|
||||
|
||||
- **WHEN** a component is exercised by an interaction spec but has no `*.stories.tsx`
|
||||
- **THEN** the broader package quality gate (per `enterprise-browser-storybook-test-utils`) fails with a missing-story diagnostic; the testbed gate is not the enforcement point for that case
|
||||
@@ -0,0 +1,110 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: NetworkMonitor MUST assert absence of missing assets
|
||||
|
||||
`NetworkMonitor` (returned by `attachNetworkMonitor(page)`) SHALL expose `assertNoMissingAssets()` that fails when any 404 was observed for an `image`, `font`, `script`, or `stylesheet` resource type. The monitor SHALL also expose `allow404(predicate: (entry) => boolean)` to opt-out specific URLs from this assertion. Other 404s (e.g. for `xhr`/`fetch` resource types) are NOT subject to this assertion and remain governed by `assertNoFailures()`.
|
||||
|
||||
#### Scenario: Missing image fails the assertion
|
||||
|
||||
- **WHEN** the page references `<img src="/missing.png">` and the request returns 404
|
||||
- **AND** `monitor.assertNoMissingAssets()` is called
|
||||
- **THEN** the call throws `MissingAssetError` with `code === "TESTING_E2E.NETWORK.MISSING_ASSET"` and `context.url` containing `/missing.png`
|
||||
|
||||
#### Scenario: Allow-listed 404 does not fail
|
||||
|
||||
- **WHEN** `monitor.allow404((e) => e.url.endsWith("/beacon.gif"))` is called before navigation
|
||||
- **AND** `/beacon.gif` returns 404
|
||||
- **AND** `monitor.assertNoMissingAssets()` is called
|
||||
- **THEN** the call returns without throwing
|
||||
|
||||
#### Scenario: Non-asset 404 does not fail this assertion
|
||||
|
||||
- **WHEN** an `xhr` request to `/api/foo` returns 404
|
||||
- **AND** `monitor.assertNoMissingAssets()` is called
|
||||
- **THEN** the call returns without throwing (the failure is reported by `assertNoFailures()` instead)
|
||||
|
||||
### Requirement: NetworkMonitor MUST assert absence of mixed content
|
||||
|
||||
`NetworkMonitor` SHALL expose `assertNoMixedContent()` that fails when the page is loaded over `https://` AND any subresource was fetched over `http://`, OR when the browser reports a mixed-content blocking event. The assertion is a no-op when the page is loaded over `http://`.
|
||||
|
||||
#### Scenario: HTTP subresource on HTTPS page fails
|
||||
|
||||
- **WHEN** the page is `https://example.test/`
|
||||
- **AND** it includes `<script src="http://cdn.test/lib.js">`
|
||||
- **AND** `monitor.assertNoMixedContent()` is called
|
||||
- **THEN** the call throws `MixedContentError` with `code === "TESTING_E2E.NETWORK.MIXED_CONTENT"` and `context.url` containing `http://cdn.test/lib.js`
|
||||
|
||||
#### Scenario: Plain HTTP page is exempt
|
||||
|
||||
- **WHEN** the page is `http://localhost:3000/`
|
||||
- **AND** `monitor.assertNoMixedContent()` is called
|
||||
- **THEN** the call returns without throwing
|
||||
|
||||
### Requirement: NetworkMonitor MUST assert absence of CORS errors
|
||||
|
||||
`NetworkMonitor` SHALL expose `assertNoCorsErrors()` that fails on any of: a request that failed with a `net::ERR_CORS_*` error code, a preflight (`OPTIONS`) response that did not satisfy the subsequent request, or a response missing a required `Access-Control-Allow-Origin` header that the browser blocked.
|
||||
|
||||
#### Scenario: CORS preflight failure fails the assertion
|
||||
|
||||
- **WHEN** an `OPTIONS` preflight returns without `Access-Control-Allow-Origin` and the browser cancels the follow-up `POST`
|
||||
- **AND** `monitor.assertNoCorsErrors()` is called
|
||||
- **THEN** the call throws `CorsError` with `code === "TESTING_E2E.NETWORK.CORS_ERROR"`
|
||||
|
||||
#### Scenario: Same-origin requests are unaffected
|
||||
|
||||
- **WHEN** all requests are same-origin and complete normally
|
||||
- **AND** `monitor.assertNoCorsErrors()` is called
|
||||
- **THEN** the call returns without throwing
|
||||
|
||||
### Requirement: NetworkMonitor MUST assert request graph against an allow-list
|
||||
|
||||
`NetworkMonitor` SHALL expose `assertNoUnexpectedRequests({ allowed: (url: string) => boolean })` that fails when any observed request URL fails the `allowed` predicate. This assertion is opt-in and intended for tests requiring strict request-graph control.
|
||||
|
||||
#### Scenario: Unexpected request fails the assertion
|
||||
|
||||
- **WHEN** the page made a request to `https://tracker.test/pixel`
|
||||
- **AND** `monitor.assertNoUnexpectedRequests({ allowed: (u) => u.startsWith("https://example.test") })` is called
|
||||
- **THEN** the call throws `UnexpectedRequestError` with `code === "TESTING_E2E.NETWORK.UNEXPECTED_REQUEST"` and `context.url === "https://tracker.test/pixel"`
|
||||
|
||||
#### Scenario: All allowed requests pass
|
||||
|
||||
- **WHEN** every request URL satisfies the predicate
|
||||
- **AND** `monitor.assertNoUnexpectedRequests({ allowed })` is called
|
||||
- **THEN** the call returns without throwing
|
||||
|
||||
### Requirement: `crawlInternalLinks` MUST walk and verify internal links
|
||||
|
||||
The package SHALL export `crawlInternalLinks(page, options?: { maxDepth?: number; sameOriginOnly?: boolean; ignore?: (url: string) => boolean }): Promise<CrawlReport>` that BFS-walks all `<a href>` links from the entry page, follows internal links up to `maxDepth` (default 2), and asserts each link returns 2xx via HEAD with GET fallback. `sameOriginOnly` defaults to `true`. The returned `CrawlReport` SHALL list visited URLs, broken links, and any circular redirects encountered.
|
||||
|
||||
#### Scenario: Crawler reports broken internal link
|
||||
|
||||
- **WHEN** the page links to `/missing` which returns 404
|
||||
- **AND** `crawlInternalLinks(page)` is called
|
||||
- **THEN** the returned report contains an entry whose `error.code === "TESTING_E2E.CRAWL.BROKEN_LINK"` and whose `url` ends in `/missing`
|
||||
|
||||
#### Scenario: Crawler honours sameOriginOnly default
|
||||
|
||||
- **WHEN** the page links to `https://external.test/foo`
|
||||
- **AND** `crawlInternalLinks(page)` is called with default options
|
||||
- **THEN** `https://external.test/foo` is NOT visited and is NOT reported as broken
|
||||
|
||||
#### Scenario: Crawler reports circular redirect
|
||||
|
||||
- **WHEN** `/a` redirects to `/b` and `/b` redirects to `/a`
|
||||
- **AND** `crawlInternalLinks(page)` encounters this chain
|
||||
- **THEN** the report contains an entry whose `error.code === "TESTING_E2E.CRAWL.CIRCULAR_REDIRECT"`
|
||||
|
||||
#### Scenario: Crawler reports max-depth exceeded
|
||||
|
||||
- **WHEN** the link graph contains paths deeper than `maxDepth`
|
||||
- **AND** `crawlInternalLinks(page, { maxDepth: 1 })` is called
|
||||
- **THEN** unvisited deeper links are reported with `error.code === "TESTING_E2E.CRAWL.MAX_DEPTH_EXCEEDED"`
|
||||
|
||||
### Requirement: Network and crawl errors MUST follow the structured-error contract
|
||||
|
||||
All errors thrown or reported by the new NetworkMonitor methods and `crawlInternalLinks` MUST be subclasses of `TestingE2eError` carrying `{ code, cause?, context? }`. Codes MUST match `^TESTING_E2E\.(NETWORK|CRAWL)\.[A-Z_]+$`.
|
||||
|
||||
#### Scenario: Code shape
|
||||
|
||||
- **WHEN** any of `MissingAssetError`, `MixedContentError`, `CorsError`, `UnexpectedRequestError`, `BrokenLinkError`, `CircularRedirectError`, `MaxDepthExceededError` is thrown
|
||||
- **THEN** `err instanceof TestingE2eError` is true AND `err.code` matches the regex
|
||||
62
openspec/changes/enterprise-testbed-verification/tasks.md
Normal file
62
openspec/changes/enterprise-testbed-verification/tasks.md
Normal file
@@ -0,0 +1,62 @@
|
||||
|
||||
|
||||
## 0. Pre-flight (non-negotiable)
|
||||
|
||||
- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
|
||||
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
|
||||
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
|
||||
|
||||
## 1. NetworkMonitor extensions in `@suaveplan/testing-e2e`
|
||||
|
||||
- [ ] 1.1 Add `assertNoMissingAssets()` method to `NetworkMonitor`, with resource-type filter for image/font/script/stylesheet.
|
||||
- [ ] 1.2 Add `allow404(predicate)` opt-out registry consulted by `assertNoMissingAssets()`.
|
||||
- [ ] 1.3 Add `assertNoMixedContent()` method, gated by `page.url()` scheme.
|
||||
- [ ] 1.4 Add `assertNoCorsErrors()` method, sourcing data from `page.on('requestfailed')` plus `net::ERR_CORS_*` matching.
|
||||
- [ ] 1.5 Add `assertNoUnexpectedRequests({ allowed })` method.
|
||||
- [ ] 1.6 Define error subclasses `MissingAssetError`, `MixedContentError`, `CorsError`, `UnexpectedRequestError` under `TESTING_E2E.NETWORK.*` codes.
|
||||
- [ ] 1.7 Co-located `.test.ts` covering each method with real Playwright fixtures (no mocks).
|
||||
- [ ] 1.8 Sibling `.md` files (≥200 words, basic + advanced examples).
|
||||
- [ ] 1.9 Update `openspec/specs/testing-e2e/spec.md` deltas folded on archive.
|
||||
|
||||
## 2. Broken-link crawler in `@suaveplan/testing-e2e`
|
||||
|
||||
- [ ] 2.1 Implement `crawlInternalLinks(page, { maxDepth, sameOriginOnly, ignore })` with BFS over `<a href>`.
|
||||
- [ ] 2.2 HEAD-then-GET fallback for status checks; honour redirects with circular detection.
|
||||
- [ ] 2.3 Define error subclasses `BrokenLinkError`, `CircularRedirectError`, `MaxDepthExceededError` under `TESTING_E2E.CRAWL.*` codes.
|
||||
- [ ] 2.4 Co-located `.test.ts` against a fixture site served by Playwright's request fixture.
|
||||
- [ ] 2.5 Sibling `.md` (≥200 words, basic + advanced examples).
|
||||
|
||||
## 3. New `testbed` capability
|
||||
|
||||
- [ ] 3.1 Create `openspec/specs/testbed/` (folded on archive) seeded from this change's `specs/testbed/spec.md`.
|
||||
- [ ] 3.2 Author per-route `<route>.interaction.spec.ts` files under `apps/testbed/e2e/` for every entry in `apps/testbed/src/routes.ts`.
|
||||
- [ ] 3.3 Each interaction spec attaches console + network + axe + visual + crawler invariants per the spec contract.
|
||||
- [ ] 3.4 Remove or supplement any existing "baseline-only" specs so they are no longer the sole coverage for any route.
|
||||
|
||||
## 4. Release-readiness CI gate
|
||||
|
||||
- [ ] 4.1 Add `release-readiness` Playwright project in `apps/testbed/playwright.config.ts`.
|
||||
- [ ] 4.2 Compose the gate against the Docker stack from `enterprise-testing-operational`.
|
||||
- [ ] 4.3 Wire load-smoke step from `enterprise-testing-load-perf`.
|
||||
- [ ] 4.4 Wire visual baseline step from `enterprise-testing-visual-regression`.
|
||||
- [ ] 4.5 Wire axe step from `enterprise-testing-a11y`.
|
||||
- [ ] 4.6 Configure CI workflow to run the gate on any PR touching `apps/testbed` or its dependency closure.
|
||||
- [ ] 4.7 Configure branch protection so gate failure blocks merge to `main` and `genesis`.
|
||||
|
||||
## 5. Documentation
|
||||
|
||||
- [ ] 5.1 Update `apps/testbed/README.md` documenting the per-page invariant contract.
|
||||
- [ ] 5.2 Cross-link this change folder from `enterprise-testing-a11y`, `enterprise-testing-visual-regression`, `enterprise-testing-load-perf`, `enterprise-testing-operational`.
|
||||
|
||||
## 6. Verification
|
||||
|
||||
- [ ] 6.1 `bunx turbo lint typecheck test --filter=@suaveplan/testing-e2e` green.
|
||||
- [ ] 6.2 `bunx turbo test:coverage --filter=@suaveplan/testing-e2e` at 100% on touched modules.
|
||||
- [ ] 6.3 `release-readiness` project passes against the testbed Docker stack.
|
||||
- [ ] 6.4 Confirm every CLAUDE.md "Web App & Page Verification" bullet maps to at least one passing assertion in the gate.
|
||||
|
||||
## 7. Archive
|
||||
|
||||
- [ ] 7.1 Fold testing-e2e delta into `openspec/specs/testing-e2e/spec.md`.
|
||||
- [ ] 7.2 Create `openspec/specs/testbed/spec.md` from the new-capability delta.
|
||||
- [ ] 7.3 Move `openspec/changes/enterprise-testbed-verification/` to `openspec/archive/<yyyy-mm-dd>-enterprise-testbed-verification/`.
|
||||
Reference in New Issue
Block a user