suaveplan/openspec/changes/ab-testing-completion/tasks.md

Tasks — A/B testing completion

Every checkbox MUST reflect work actually done in this change. Do not retroactively tick boxes for work that was already in main when this change opened. Do not tick a box for a test that doesn't exist or doesn't pin the requirement it claims to verify. Retroactive ticking is the failure mode this template guards against.

0. Pre-flight (non-negotiable)

• 0.1 Confirm clean working tree (git status reports nothing) and no in-flight worktrees with uncommitted work (git worktree list then git status in each)
• 0.2 If any worktrees exist under .claude/worktrees/, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (git worktree remove) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
• 0.3 Confirm the package set in MANIFEST.md matches find packages -maxdepth 3 -name package.json count. Drift here means a prior change archived without regenerating the manifest.

1. Spec authoring

• 1.1 Read ../suaveplan/packages/universal/ab-testing/src/ end-to-end (bandit/, multivariate/, segmentation/, sequential/, stats/, store/, types/, engine/)
• 1.2 Read existing openspec/specs/ab-testing/spec.md (none exists yet — will be created at archive)
• 1.3 Read openspec/AGENTS.md Sections 13, 14, and §0a (migration parity)
• 1.4 Draft proposal.md from openspec/templates/proposal.md
• 1.5 Fill in ## Source mapping table — every requirement cites a line range OR appears in the "no source line" table with justification
• 1.6 Draft specs/ab-testing/spec.md from openspec/templates/spec-delta.md
• 1.7 Every requirement in the spec delta carries a **Verified by:** line naming concrete it(...) titles
• 1.8 ## Parity Checklist table fully populated; every NO has architectural justification ≤1 sentence

2. Implementation

2a. Promote AssignmentStoreLike to @suaveplan/types

• 2.1 Add AssignmentStoreLike interface to packages/types/types/src/storage/assignment-store.ts with five async methods (get, set, getByExperiment, getByUser, deleteByExperiment)
• 2.2 Re-export AssignmentStoreLike from @suaveplan/types root barrel
• 2.3 Add Assignment interface alongside (or import-only re-export from @suaveplan/ab-testing once defined there) — pin the cross-package contract

2b. Bandit subsystem

• 2.4 Add src/bandit/bandit.types.ts with Bandit<TArmId> interface and BanditConfig<TArmId>
• 2.5 Add src/bandit/epsilon-greedy.ts — EpsilonGreedyBandit<TArmId> class; constructor accepts { arms, epsilon, rng }; default epsilon = 0.1; require explicit Rng (no Math.random fallback at construction)
• 2.6 Add src/bandit/ucb1.ts — UCB1Bandit<TArmId> class
• 2.7 Add src/bandit/thompson.ts — ThompsonSamplingBandit<TArmId> class; depends on betaSample from ./stats/bayesian
• 2.8 Add src/bandit/index.ts barrel
• 2.9 Throw AbTestingError(AB_CODES.INVALID_BANDIT_CONFIG, ...) when arms.length < 2 or epsilon ∉ [0, 1]

2c. Multivariate engine

• 2.10 Add src/multivariate/multivariate.types.ts with Factor, TreatmentCombination, TreatmentResult, FactorEffect, LevelEffect, InteractionEffect, MultivariateResult
• 2.11 Add src/multivariate/factorial.ts — fullFactorial, fractionalFactorial
• 2.12 Add src/multivariate/multivariate-engine.ts — analyzeMultivariate + factor-effect / interaction-effect helpers; split helpers into src/multivariate/factor-effects.ts and src/multivariate/interactions.ts if file > 250 L
• 2.13 Add src/multivariate/index.ts barrel
• 2.14 Throw AbTestingError(AB_CODES.INVALID_FACTOR_DESIGN, ...) when factors.length === 0 or any factor has levels.length === 0

2d. Segment analysis

• 2.15 Add src/segments/segments.types.ts with SegmentResult, SegmentConfidenceInterval, SegmentInteraction
• 2.16 Add src/segments/segment-analyzer.ts — analyzeBySegments, segmentConfidenceIntervals, detectSegmentInteraction
• 2.17 Move normalQuantile helper to src/stats/normal.ts (shared by segments and sequential)
• 2.18 Add src/segments/index.ts barrel
• 2.19 Throw AbTestingError(AB_CODES.INVALID_SEGMENT_ATTRIBUTE, ...) when assignments.every(a => a.attributes?.[segmentAttribute] === undefined)

2e. Sequential testing

• 2.20 Add src/sequential/sequential.types.ts with SequentialTestConfig (extended with boundaryType), InterimResult
• 2.21 Add src/sequential/boundaries.ts — obrienFlemingBoundary, pocockBoundary, dispatch on boundaryType
• 2.22 Add src/sequential/sequential-test.ts — alphaSpent, interimAnalysis, runSequentialTest, checkFutility
• 2.23 Add src/sequential/index.ts barrel
• 2.24 Throw AbTestingError(AB_CODES.INVALID_SEQUENTIAL_CONFIG, ...) when alpha ∉ (0, 1), maxLooks < 1, mde ≤ 0, or power ∉ (0, 1)

2f. Bayesian stats

• 2.25 Add src/stats/beta.ts — BetaParams, betaPdf, betaCdf, betaQuantile, betaSample, posteriorBeta, internal logBeta / logGamma / continuedFraction / gammaSample / normalSample
• 2.26 Add src/stats/bayesian.ts — BayesianResult, thompsonSample, probabilityBGreaterA, credibleInterval, expectedLoss, bayesianAnalysis
• 2.27 Add src/stats/normal.ts — normalCdf, normalQuantile (Beasley-Springer-Moro) shared with sequential and segments
• 2.28 Add src/stats/index.ts barrel
• 2.29 Replace every rng: () => number = Math.random default with required rng: Rng param; expose createDefaultRng() from @suaveplan/ab-testing/stats for callers wanting a crypto.getRandomValues-backed default

2g. Assignment store

• 2.30 Add src/store/memory.ts — MemoryAssignmentStore class implementing AssignmentStoreLike (imported from @suaveplan/types)
• 2.31 Add createMemoryAssignmentStore() factory
• 2.32 Add src/store/index.ts barrel
• 2.33 Wire store?: AssignmentStoreLike field through ExperimentManagerOptions in src/types/common.ts; update createExperimentManager to consult and persist via the store when provided

2h. Errors and exports

• 2.34 Add INVALID_BANDIT_CONFIG, INVALID_FACTOR_DESIGN, INVALID_SEGMENT_ATTRIBUTE, INSUFFICIENT_SAMPLE_SIZE, INVALID_SEQUENTIAL_CONFIG to AB_CODES in src/error-codes.ts
• 2.35 Update src/index.ts root barrel to re-export from ./bandit, ./multivariate, ./segments, ./sequential, ./stats, ./store
• 2.36 Add subpath entries to package.json exports map for the six new subpaths (matches existing ./bucketing / ./targeting / ./results / ./providers / ./errors shape)
• 2.37 Update package.json description field if needed (currently mentions only deterministic bucketing + targeting + variants + conversions; extend to mention bandits / multivariate / sequential / Bayesian)

3. Tests

3a. Bandit subsystem

• 3.1 src/bandit/bandit.test.ts — EpsilonGreedyBandit exploration (when rng < epsilon) vs. exploitation (else); UCB1Bandit plays each arm once before applying score; ThompsonSamplingBandit posterior update on success vs. failure
• 3.2 src/bandit/bandit.spec-conformance.test.ts — injected Rng is the only randomness source; INVALID_BANDIT_CONFIG thrown for invalid configs

3b. Multivariate engine

• 3.3 src/multivariate/multivariate-engine.test.ts — fullFactorial count = Π levels; fractionalFactorial(fraction = 0.5) returns half; factor importance sum-of-squares ranking
• 3.4 src/multivariate/multivariate-engine.spec-conformance.test.ts — interaction strength is non-negative; pairwise interactions cover every (i, j) with i < j; INVALID_FACTOR_DESIGN thrown for empty factors

3c. Segment analysis

• 3.5 src/segments/segment-analyzer.test.ts — segment grouping by attribute; significance with min sample size 10; CI is [0, 1]-clamped
• 3.6 src/segments/segment-analyzer.spec-conformance.test.ts — interaction-strength threshold; INVALID_SEGMENT_ATTRIBUTE thrown when attribute is absent

3d. Sequential testing

• 3.7 src/sequential/sequential-test.test.ts — O'Brien-Fleming boundary monotone-decreasing in informationFraction; Pocock boundary constant across looks; runSequentialTest stops at first shouldStop
• 3.8 src/sequential/sequential-test.spec-conformance.test.ts — alpha-spent monotone-increasing; futility only fires when informationFraction ≥ 0.75; INVALID_SEQUENTIAL_CONFIG thrown for invalid configs

3e. Bayesian stats

• 3.9 src/stats/bayesian.test.ts — betaPdf / betaCdf / betaQuantile round-trip; posteriorBeta conjugate update (priors + data); probabilityBGreaterA converges to closed-form 0.5 for symmetric posteriors
• 3.10 src/stats/bayesian.spec-conformance.test.ts — same Rng seed produces identical betaSample / thompsonSample / expectedLoss outputs across runs (reproducibility pin)

3f. Assignment store

• 3.11 src/store/memory-assignment-store.test.ts — set/get round-trip; getByExperiment / getByUser filtering; deleteByExperiment cascade; size and clear
• 3.12 src/store/memory-assignment-store.spec-conformance.test.ts — AssignmentStoreLike contract: every method returns a Promise; get resolves undefined for missing keys; set is idempotent on identical assignments

3g. ExperimentManager integration with store

• 3.13 src/core/experiment-manager/experiment-manager.store.test.ts — when options.store is provided, assignVariant checks store first and writes through on miss; second call for same (experimentId, userId) returns the persisted variant without re-bucketing

3h. Cross-package

• 3.14 tests/ab-testing-public-surface.test.ts — assert root barrel re-exports include EpsilonGreedyBandit, UCB1Bandit, ThompsonSamplingBandit, fullFactorial, fractionalFactorial, analyzeMultivariate, analyzeBySegments, segmentConfidenceIntervals, detectSegmentInteraction, obrienFlemingBoundary, pocockBoundary, interimAnalysis, runSequentialTest, posteriorBeta, bayesianAnalysis, MemoryAssignmentStore, createMemoryAssignmentStore
• 3.15 Every it(...) carries { timeout: 5000 } (Bayesian Monte Carlo tests use 30000)

4. Documentation

• 4.1 Sibling .md for every new implementation .ts (≥200 words, basic + advanced examples, full API reference): one per file under bandit/, multivariate/, segments/, sequential/, stats/, store/
• 4.2 README links every new co-located .md under ## Modules
• 4.3 Update root README.md quick-start with three new sections: bandit selection, multivariate analysis, sequential testing
• 4.4 Update CLAUDE.md @suaveplan/ab-testing package listing to include bandits, multivariate, segment analysis, sequential testing, Bayesian stats, assignment store
• 4.5 Update @suaveplan/types listing in CLAUDE.md to mention AssignmentStoreLike if the listing is detailed enough

5. Pre-archive verification (the 5-gate triple)

• 5.1 bun run check — Biome 0 errors / 0 warnings / 0 info across the whole monorepo
• 5.2 bunx turbo lint typecheck test --filter=@suaveplan/ab-testing — green
• 5.3 bunx turbo test:coverage --filter=@suaveplan/ab-testing — 100/100/100/100
• 5.4 bun scripts/openspec-conformance.ts @suaveplan/ab-testing — exit 0, no error findings
• 5.5 bunx turbo build --filter=@suaveplan/ab-testing — green
• 5.6 bunx turbo lint typecheck test --filter=@suaveplan/types — green (foundation-package side-effect from AssignmentStoreLike promotion)

6. Pre-archive 7-row checklist (AGENTS.md Section 13)

• 6.1 Generics threaded — Bandit<TArmId>, EpsilonGreedyBandit<TArmId>, UCB1Bandit<TArmId>, ThompsonSamplingBandit<TArmId> infer arm-id type from constructor arms array; expectTypeOf assertion that bandit.selectArm() returns TArmId, not number
• 6.2 Zod schemas wired at every declared boundary — BanditConfigSchema, FactorSchema, SequentialTestConfigSchema validate at construction
• 6.3 Every error code in error-codes.ts thrown by at least one production path (the five new codes each tested in spec-conformance suites)
• 6.4 Strict-mode behaviors tested both on and off — Bandit with strict: true throws on unknown arm IDs; Bandit with strict: false ignores
• 6.5 OTel instruments emitted from production paths and asserted with a real in-memory MeterProvider — ab.bandit.arm.selected, ab.multivariate.combinations.evaluated, ab.sequential.alpha.spent, ab.bayesian.simulations.duration_ms, ab.assignment.store.lookup
• 6.6 Lifecycle dedup tested where claimed — MemoryAssignmentStore.set is idempotent; ExperimentManager with store option does not re-bucket after a hit
• 6.7 No retroactive tick — every box above reflects work done in this change

7. Archive

• 7.1 Fold spec delta into openspec/specs/ab-testing/spec.md (file does not exist yet — this change creates it)
• 7.2 Move openspec/changes/ab-testing-completion/ → openspec/archive/<yyyy-mm-dd>-ab-testing-completion/
• 7.3 Refresh MANIFEST.md if package set or deps changed (no new deps; @suaveplan/types gains AssignmentStoreLike interface)

Audit Note (2026-05-02)

Audit verdict: DEFERRED — 0/81 tasks evidenced in code.

Evidence reviewed:

• packages/testing/ab-testing/src/ contains bucketing/, targeting/, providers/, results/, core/experiment-manager/, error-codes.ts, types/, constants/. No bandit/, no multivariate/, no segments/, no sequential/, no stats/, no store/.
• error-codes.ts does not declare the five new codes (INVALID_BANDIT_CONFIG, INVALID_FACTOR_DESIGN, INVALID_SEGMENT_ATTRIBUTE, INSUFFICIENT_SAMPLE_SIZE, INVALID_SEQUENTIAL_CONFIG).
• @suaveplan/types has no AssignmentStoreLike interface yet (the §2a promotion is not done).

Missing pieces:

1. Bandit subsystem (epsilon-greedy, ucb1, thompson with explicit Rng injection) and Bayesian stats module (stats/{beta,bayesian,normal}.ts).
2. Multivariate engine (fullFactorial / fractionalFactorial / analyzeMultivariate + factor-effect / interaction helpers).
3. Segment analyzer + sequential testing (O'Brien-Fleming + Pocock boundaries) + MemoryAssignmentStore plus AssignmentStoreLike promotion to @suaveplan/types and wiring through ExperimentManagerOptions.

Decision: leave change open. Six new feature blocks across statistics + storage + cross-package type-promotion is a multi-day implementation, not an audit pass.