suaveplan/openspec/changes/ab-testing-completion/tasks.md

# Tasks — A/B testing completion

> Every checkbox MUST reflect work actually done in this change. Do not retroactively tick boxes for work that was already in main when this change opened. Do not tick a box for a test that doesn't exist or doesn't pin the requirement it claims to verify. Retroactive ticking is the failure mode this template guards against.

## 0. Pre-flight (non-negotiable)

- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.

## 1. Spec authoring

- [ ] 1.1 Read `../suaveplan/packages/universal/ab-testing/src/` end-to-end (bandit/, multivariate/, segmentation/, sequential/, stats/, store/, types/, engine/)
- [ ] 1.2 Read existing `openspec/specs/ab-testing/spec.md` (none exists yet — will be created at archive)
- [ ] 1.3 Read `openspec/AGENTS.md` Sections 13, 14, and §0a (migration parity)
- [ ] 1.4 Draft `proposal.md` from `openspec/templates/proposal.md`
- [ ] 1.5 Fill in `## Source mapping` table — every requirement cites a line range OR appears in the "no source line" table with justification
- [ ] 1.6 Draft `specs/ab-testing/spec.md` from `openspec/templates/spec-delta.md`
- [ ] 1.7 Every requirement in the spec delta carries a `**Verified by:**` line naming concrete `it(...)` titles
- [ ] 1.8 `## Parity Checklist` table fully populated; every NO has architectural justification ≤1 sentence

## 2. Implementation

### 2a. Promote `AssignmentStoreLike` to `@suaveplan/types`

- [ ] 2.1 Add `AssignmentStoreLike` interface to `packages/types/types/src/storage/assignment-store.ts` with five async methods (`get`, `set`, `getByExperiment`, `getByUser`, `deleteByExperiment`)
- [ ] 2.2 Re-export `AssignmentStoreLike` from `@suaveplan/types` root barrel
- [ ] 2.3 Add `Assignment` interface alongside (or import-only re-export from `@suaveplan/ab-testing` once defined there) — pin the cross-package contract

### 2b. Bandit subsystem

- [ ] 2.4 Add `src/bandit/bandit.types.ts` with `Bandit<TArmId>` interface and `BanditConfig<TArmId>`
- [ ] 2.5 Add `src/bandit/epsilon-greedy.ts` — `EpsilonGreedyBandit<TArmId>` class; constructor accepts `{ arms, epsilon, rng }`; default `epsilon = 0.1`; require explicit `Rng` (no `Math.random` fallback at construction)
- [ ] 2.6 Add `src/bandit/ucb1.ts` — `UCB1Bandit<TArmId>` class
- [ ] 2.7 Add `src/bandit/thompson.ts` — `ThompsonSamplingBandit<TArmId>` class; depends on `betaSample` from `./stats/bayesian`
- [ ] 2.8 Add `src/bandit/index.ts` barrel
- [ ] 2.9 Throw `AbTestingError(AB_CODES.INVALID_BANDIT_CONFIG, ...)` when `arms.length < 2` or `epsilon ∉ [0, 1]`

### 2c. Multivariate engine

- [ ] 2.10 Add `src/multivariate/multivariate.types.ts` with `Factor`, `TreatmentCombination`, `TreatmentResult`, `FactorEffect`, `LevelEffect`, `InteractionEffect`, `MultivariateResult`
- [ ] 2.11 Add `src/multivariate/factorial.ts` — `fullFactorial`, `fractionalFactorial`
- [ ] 2.12 Add `src/multivariate/multivariate-engine.ts` — `analyzeMultivariate` + factor-effect / interaction-effect helpers; split helpers into `src/multivariate/factor-effects.ts` and `src/multivariate/interactions.ts` if file > 250 L
- [ ] 2.13 Add `src/multivariate/index.ts` barrel
- [ ] 2.14 Throw `AbTestingError(AB_CODES.INVALID_FACTOR_DESIGN, ...)` when `factors.length === 0` or any factor has `levels.length === 0`

### 2d. Segment analysis

- [ ] 2.15 Add `src/segments/segments.types.ts` with `SegmentResult`, `SegmentConfidenceInterval`, `SegmentInteraction`
- [ ] 2.16 Add `src/segments/segment-analyzer.ts` — `analyzeBySegments`, `segmentConfidenceIntervals`, `detectSegmentInteraction`
- [ ] 2.17 Move `normalQuantile` helper to `src/stats/normal.ts` (shared by segments and sequential)
- [ ] 2.18 Add `src/segments/index.ts` barrel
- [ ] 2.19 Throw `AbTestingError(AB_CODES.INVALID_SEGMENT_ATTRIBUTE, ...)` when `assignments.every(a => a.attributes?.[segmentAttribute] === undefined)`

### 2e. Sequential testing

- [ ] 2.20 Add `src/sequential/sequential.types.ts` with `SequentialTestConfig` (extended with `boundaryType`), `InterimResult`
- [ ] 2.21 Add `src/sequential/boundaries.ts` — `obrienFlemingBoundary`, `pocockBoundary`, dispatch on `boundaryType`
- [ ] 2.22 Add `src/sequential/sequential-test.ts` — `alphaSpent`, `interimAnalysis`, `runSequentialTest`, `checkFutility`
- [ ] 2.23 Add `src/sequential/index.ts` barrel
- [ ] 2.24 Throw `AbTestingError(AB_CODES.INVALID_SEQUENTIAL_CONFIG, ...)` when `alpha ∉ (0, 1)`, `maxLooks < 1`, `mde ≤ 0`, or `power ∉ (0, 1)`

### 2f. Bayesian stats

- [ ] 2.25 Add `src/stats/beta.ts` — `BetaParams`, `betaPdf`, `betaCdf`, `betaQuantile`, `betaSample`, `posteriorBeta`, internal `logBeta` / `logGamma` / `continuedFraction` / `gammaSample` / `normalSample`
- [ ] 2.26 Add `src/stats/bayesian.ts` — `BayesianResult`, `thompsonSample`, `probabilityBGreaterA`, `credibleInterval`, `expectedLoss`, `bayesianAnalysis`
- [ ] 2.27 Add `src/stats/normal.ts` — `normalCdf`, `normalQuantile` (Beasley-Springer-Moro) shared with sequential and segments
- [ ] 2.28 Add `src/stats/index.ts` barrel
- [ ] 2.29 Replace every `rng: () => number = Math.random` default with required `rng: Rng` param; expose `createDefaultRng()` from `@suaveplan/ab-testing/stats` for callers wanting a `crypto.getRandomValues`-backed default

### 2g. Assignment store

- [ ] 2.30 Add `src/store/memory.ts` — `MemoryAssignmentStore` class implementing `AssignmentStoreLike` (imported from `@suaveplan/types`)
- [ ] 2.31 Add `createMemoryAssignmentStore()` factory
- [ ] 2.32 Add `src/store/index.ts` barrel
- [ ] 2.33 Wire `store?: AssignmentStoreLike` field through `ExperimentManagerOptions` in `src/types/common.ts`; update `createExperimentManager` to consult and persist via the store when provided

### 2h. Errors and exports

- [ ] 2.34 Add `INVALID_BANDIT_CONFIG`, `INVALID_FACTOR_DESIGN`, `INVALID_SEGMENT_ATTRIBUTE`, `INSUFFICIENT_SAMPLE_SIZE`, `INVALID_SEQUENTIAL_CONFIG` to `AB_CODES` in `src/error-codes.ts`
- [ ] 2.35 Update `src/index.ts` root barrel to re-export from `./bandit`, `./multivariate`, `./segments`, `./sequential`, `./stats`, `./store`
- [ ] 2.36 Add subpath entries to `package.json` `exports` map for the six new subpaths (matches existing `./bucketing` / `./targeting` / `./results` / `./providers` / `./errors` shape)
- [ ] 2.37 Update `package.json` `description` field if needed (currently mentions only deterministic bucketing + targeting + variants + conversions; extend to mention bandits / multivariate / sequential / Bayesian)

## 3. Tests

### 3a. Bandit subsystem

- [ ] 3.1 `src/bandit/bandit.test.ts` — `EpsilonGreedyBandit` exploration (when rng < epsilon) vs. exploitation (else); `UCB1Bandit` plays each arm once before applying score; `ThompsonSamplingBandit` posterior update on success vs. failure
- [ ] 3.2 `src/bandit/bandit.spec-conformance.test.ts` — injected `Rng` is the only randomness source; `INVALID_BANDIT_CONFIG` thrown for invalid configs

### 3b. Multivariate engine

- [ ] 3.3 `src/multivariate/multivariate-engine.test.ts` — `fullFactorial` count = `Π levels`; `fractionalFactorial(fraction = 0.5)` returns half; factor importance sum-of-squares ranking
- [ ] 3.4 `src/multivariate/multivariate-engine.spec-conformance.test.ts` — interaction strength is non-negative; pairwise interactions cover every `(i, j)` with `i < j`; `INVALID_FACTOR_DESIGN` thrown for empty factors

### 3c. Segment analysis

- [ ] 3.5 `src/segments/segment-analyzer.test.ts` — segment grouping by attribute; significance with min sample size 10; CI is `[0, 1]`-clamped
- [ ] 3.6 `src/segments/segment-analyzer.spec-conformance.test.ts` — interaction-strength threshold; `INVALID_SEGMENT_ATTRIBUTE` thrown when attribute is absent

### 3d. Sequential testing

- [ ] 3.7 `src/sequential/sequential-test.test.ts` — O'Brien-Fleming boundary monotone-decreasing in `informationFraction`; Pocock boundary constant across looks; `runSequentialTest` stops at first `shouldStop`
- [ ] 3.8 `src/sequential/sequential-test.spec-conformance.test.ts` — alpha-spent monotone-increasing; futility only fires when `informationFraction ≥ 0.75`; `INVALID_SEQUENTIAL_CONFIG` thrown for invalid configs

### 3e. Bayesian stats

- [ ] 3.9 `src/stats/bayesian.test.ts` — `betaPdf` / `betaCdf` / `betaQuantile` round-trip; `posteriorBeta` conjugate update (priors + data); `probabilityBGreaterA` converges to closed-form `0.5` for symmetric posteriors
- [ ] 3.10 `src/stats/bayesian.spec-conformance.test.ts` — same `Rng` seed produces identical `betaSample` / `thompsonSample` / `expectedLoss` outputs across runs (reproducibility pin)

### 3f. Assignment store

- [ ] 3.11 `src/store/memory-assignment-store.test.ts` — `set`/`get` round-trip; `getByExperiment` / `getByUser` filtering; `deleteByExperiment` cascade; `size` and `clear`
- [ ] 3.12 `src/store/memory-assignment-store.spec-conformance.test.ts` — `AssignmentStoreLike` contract: every method returns a `Promise`; `get` resolves `undefined` for missing keys; `set` is idempotent on identical assignments

### 3g. ExperimentManager integration with store

- [ ] 3.13 `src/core/experiment-manager/experiment-manager.store.test.ts` — when `options.store` is provided, `assignVariant` checks store first and writes through on miss; second call for same `(experimentId, userId)` returns the persisted variant without re-bucketing

### 3h. Cross-package

- [ ] 3.14 `tests/ab-testing-public-surface.test.ts` — assert root barrel re-exports include `EpsilonGreedyBandit`, `UCB1Bandit`, `ThompsonSamplingBandit`, `fullFactorial`, `fractionalFactorial`, `analyzeMultivariate`, `analyzeBySegments`, `segmentConfidenceIntervals`, `detectSegmentInteraction`, `obrienFlemingBoundary`, `pocockBoundary`, `interimAnalysis`, `runSequentialTest`, `posteriorBeta`, `bayesianAnalysis`, `MemoryAssignmentStore`, `createMemoryAssignmentStore`
- [ ] 3.15 Every `it(...)` carries `{ timeout: 5000 }` (Bayesian Monte Carlo tests use 30000)

## 4. Documentation

- [ ] 4.1 Sibling `.md` for every new implementation `.ts` (≥200 words, basic + advanced examples, full API reference): one per file under `bandit/`, `multivariate/`, `segments/`, `sequential/`, `stats/`, `store/`
- [ ] 4.2 README links every new co-located `.md` under `## Modules`
- [ ] 4.3 Update root `README.md` quick-start with three new sections: bandit selection, multivariate analysis, sequential testing
- [ ] 4.4 Update `CLAUDE.md` `@suaveplan/ab-testing` package listing to include bandits, multivariate, segment analysis, sequential testing, Bayesian stats, assignment store
- [ ] 4.5 Update `@suaveplan/types` listing in `CLAUDE.md` to mention `AssignmentStoreLike` if the listing is detailed enough

## 5. Pre-archive verification (the 5-gate triple)

- [ ] 5.1 `bun run check` — Biome 0 errors / 0 warnings / 0 info across the whole monorepo
- [ ] 5.2 `bunx turbo lint typecheck test --filter=@suaveplan/ab-testing` — green
- [ ] 5.3 `bunx turbo test:coverage --filter=@suaveplan/ab-testing` — 100/100/100/100
- [ ] 5.4 `bun scripts/openspec-conformance.ts @suaveplan/ab-testing` — exit 0, no `error` findings
- [ ] 5.5 `bunx turbo build --filter=@suaveplan/ab-testing` — green
- [ ] 5.6 `bunx turbo lint typecheck test --filter=@suaveplan/types` — green (foundation-package side-effect from `AssignmentStoreLike` promotion)

## 6. Pre-archive 7-row checklist (AGENTS.md Section 13)

- [ ] 6.1 Generics threaded — `Bandit<TArmId>`, `EpsilonGreedyBandit<TArmId>`, `UCB1Bandit<TArmId>`, `ThompsonSamplingBandit<TArmId>` infer arm-id type from constructor `arms` array; `expectTypeOf` assertion that `bandit.selectArm()` returns `TArmId`, not `number`
- [ ] 6.2 Zod schemas wired at every declared boundary — `BanditConfigSchema`, `FactorSchema`, `SequentialTestConfigSchema` validate at construction
- [ ] 6.3 Every error code in `error-codes.ts` thrown by at least one production path (the five new codes each tested in spec-conformance suites)
- [ ] 6.4 Strict-mode behaviors tested both on and off — `Bandit` with `strict: true` throws on unknown arm IDs; `Bandit` with `strict: false` ignores
- [ ] 6.5 OTel instruments emitted from production paths and asserted with a real in-memory `MeterProvider` — `ab.bandit.arm.selected`, `ab.multivariate.combinations.evaluated`, `ab.sequential.alpha.spent`, `ab.bayesian.simulations.duration_ms`, `ab.assignment.store.lookup`
- [ ] 6.6 Lifecycle dedup tested where claimed — `MemoryAssignmentStore.set` is idempotent; `ExperimentManager` with `store` option does not re-bucket after a hit
- [ ] 6.7 No retroactive tick — every box above reflects work done in this change

## 7. Archive

- [ ] 7.1 Fold spec delta into `openspec/specs/ab-testing/spec.md` (file does not exist yet — this change creates it)
- [ ] 7.2 Move `openspec/changes/ab-testing-completion/` → `openspec/archive/<yyyy-mm-dd>-ab-testing-completion/`
- [ ] 7.3 Refresh `MANIFEST.md` if package set or deps changed (no new deps; `@suaveplan/types` gains `AssignmentStoreLike` interface)

## Audit Note (2026-05-02)

Audit verdict: **DEFERRED — 0/81 tasks evidenced in code**.

Evidence reviewed:

- `packages/testing/ab-testing/src/` contains `bucketing/`, `targeting/`, `providers/`, `results/`, `core/experiment-manager/`, `error-codes.ts`, `types/`, `constants/`. No `bandit/`, no `multivariate/`, no `segments/`, no `sequential/`, no `stats/`, no `store/`.
- `error-codes.ts` does not declare the five new codes (`INVALID_BANDIT_CONFIG`, `INVALID_FACTOR_DESIGN`, `INVALID_SEGMENT_ATTRIBUTE`, `INSUFFICIENT_SAMPLE_SIZE`, `INVALID_SEQUENTIAL_CONFIG`).
- `@suaveplan/types` has no `AssignmentStoreLike` interface yet (the §2a promotion is not done).

Missing pieces:

1. Bandit subsystem (`epsilon-greedy`, `ucb1`, `thompson` with explicit `Rng` injection) and Bayesian stats module (`stats/{beta,bayesian,normal}.ts`).
2. Multivariate engine (`fullFactorial` / `fractionalFactorial` / `analyzeMultivariate` + factor-effect / interaction helpers).
3. Segment analyzer + sequential testing (O'Brien-Fleming + Pocock boundaries) + `MemoryAssignmentStore` plus `AssignmentStoreLike` promotion to `@suaveplan/types` and wiring through `ExperimentManagerOptions`.

Decision: leave change open. Six new feature blocks across statistics + storage + cross-package type-promotion is a multi-day implementation, not an audit pass.