Initial Commit

openspec/changes/ab-testing-completion/specs/ab-testing/spec.md

@@ -0,0 +1,203 @@
+<!-- markdownlint-disable MD024 -->
+
+# Capability Spec Delta: ab-testing (ab-testing-completion)
+
+**Package**: `@suaveplan/ab-testing`
+**Status**: PROPOSED
+
+> Every requirement below carries a `**Verified by:**` line at authoring time. No exceptions. (See `openspec/AGENTS.md` Section 13.)
+
+## ADDED Requirements
+
+### Requirement: Bandit algorithms with shared `Bandit<TArmId>` interface
+
+The package SHALL provide three multi-armed bandit algorithms — `EpsilonGreedyBandit<TArmId>`, `UCB1Bandit<TArmId>`, `ThompsonSamplingBandit<TArmId>` — all implementing a shared `Bandit<TArmId>` interface with `selectArm(): TArmId`, `update(arm: TArmId, reward: number): void`, `getValues(): readonly number[]`, `getCounts(): readonly number[]`. The generic `TArmId` parameter SHALL default to `number` (array index) so existing-shape upstream code continues to work; callers MAY pass string variant IDs as arms. Every algorithm SHALL accept an explicit `Rng` from `@suaveplan/types::Rng` (no module-scope `Math.random`). Construction SHALL throw `AbTestingError(AB_CODES.INVALID_BANDIT_CONFIG, ...)` when `arms.length < 2` or, for `EpsilonGreedyBandit`, `epsilon ∉ [0, 1]`.
+
+#### Scenario: epsilon-greedy explores when rng < epsilon
+
+- **WHEN** `EpsilonGreedyBandit` is constructed with `arms: ['a', 'b', 'c']`, `epsilon: 0.5`, and an `Rng` that returns `0.1` then `0.7`
+- **THEN** `selectArm()` returns the result of `rng() * 3` floored as a uniform random arm (exploration branch)
+
+#### Scenario: UCB1 plays each arm once before applying score
+
+- **WHEN** `UCB1Bandit` is constructed with three arms and `selectArm()` is called three times with no intervening `update`
+- **THEN** the first three returns are `arms[0]`, `arms[1]`, `arms[2]` in order
+
+#### Scenario: Thompson sampling updates posterior on success
+
+- **WHEN** `ThompsonSamplingBandit.update(arm, 1)` is called
+- **THEN** the underlying `successes[arm]` increments by 1; `update(arm, 0)` increments `failures[arm]` by 1
+
+#### Scenario: invalid bandit config throws AbTestingError
+
+- **WHEN** `new EpsilonGreedyBandit({ arms: ['x'], epsilon: 0.1, rng })` is called (only one arm)
+- **THEN** the constructor throws `AbTestingError` with code `AB_CODES.INVALID_BANDIT_CONFIG`
+
+**Verified by:** `src/bandit/bandit.test.ts::"EpsilonGreedyBandit explores when rng draw is below epsilon"` AND `src/bandit/bandit.test.ts::"UCB1Bandit plays each arm once before scoring"` AND `src/bandit/bandit.test.ts::"ThompsonSamplingBandit updates posterior on success and failure"` AND `src/bandit/bandit.spec-conformance.test.ts::"throws INVALID_BANDIT_CONFIG when arms.length < 2"`
+
+---
+
+### Requirement: Multivariate engine with full + fractional factorial design
+
+The package SHALL provide a multivariate-experiment design engine. `fullFactorial(factors)` SHALL return every combination of factor levels (Cartesian product). `fractionalFactorial(factors, fraction)` SHALL return every Nth combination where `N = round(1 / fraction)`. `analyzeMultivariate(factors, data)` SHALL return a `MultivariateResult` containing per-treatment conversion rates, per-factor `FactorEffect` objects with `LevelEffect[]` and a sum-of-squares `importance` score, pairwise `InteractionEffect[]` covering every `(i, j)` pair with `i < j`, and a `bestCombination` (highest-rate treatment with `participants > 0`). Construction with `factors.length === 0` or any factor having `levels.length === 0` SHALL throw `AbTestingError(AB_CODES.INVALID_FACTOR_DESIGN, ...)`.
+
+#### Scenario: full factorial enumerates Π levels
+
+- **WHEN** `fullFactorial([{ name: 'a', levels: ['x', 'y'] }, { name: 'b', levels: ['1', '2', '3'] }])` is called
+- **THEN** the result has length `2 * 3 = 6` and each combination has unique `id`
+
+#### Scenario: fractional factorial samples every Nth combination
+
+- **WHEN** `fractionalFactorial(factors, 0.5)` is called against a 6-combination factor set
+- **THEN** the result length is 3 and the kept combinations are at indices 0, 2, 4 in `fullFactorial(factors)`
+
+#### Scenario: factor importance is a non-negative sum-of-squares score
+
+- **WHEN** `analyzeMultivariate` is called with treatments where one factor has level rates `[0.1, 0.5, 0.9]` and another has `[0.5, 0.5, 0.5]`
+- **THEN** the first factor's `importance` strictly exceeds the second factor's `importance`
+
+#### Scenario: bestCombination is the highest-rate treatment with participants
+
+- **WHEN** treatments include one combination with `conversionRate: 0.9` but `participants: 0` and another with `0.5` and `participants: 100`
+- **THEN** `bestCombination.combinationId` is the latter
+
+#### Scenario: empty factor set throws
+
+- **WHEN** `analyzeMultivariate([], data)` is called
+- **THEN** `AbTestingError` with code `AB_CODES.INVALID_FACTOR_DESIGN` is thrown
+
+**Verified by:** `src/multivariate/multivariate-engine.test.ts::"fullFactorial enumerates the Cartesian product of levels"` AND `src/multivariate/multivariate-engine.test.ts::"fractionalFactorial keeps every Nth combination"` AND `src/multivariate/multivariate-engine.spec-conformance.test.ts::"factor importance ranks higher-variance factors first"` AND `src/multivariate/multivariate-engine.test.ts::"bestCombination ignores treatments with zero participants"` AND `src/multivariate/multivariate-engine.spec-conformance.test.ts::"throws INVALID_FACTOR_DESIGN for empty factor set"`
+
+---
+
+### Requirement: Segment analysis with per-segment lift and interaction terms
+
+The package SHALL provide segment-level analyses. `analyzeBySegments(assignments, conversions, variantIds, segmentAttribute)` SHALL group assignments by `attributes[segmentAttribute]` value, compute per-segment `VariantResult[]`, attach a two-proportion z-test significance flag (z > 1.96 with both arms n ≥ 10), and report `conversionRateDiff` between best and worst variant. `segmentConfidenceIntervals(...)` SHALL return Wald intervals on conversion rates, `[0, 1]`-clamped. `detectSegmentInteraction(...)` SHALL return interaction-strength scores per segment value relative to global per-variant rates, flagged as `hasInteraction: true` when strength exceeds 0.01. When no assignment exposes the requested attribute, the call SHALL throw `AbTestingError(AB_CODES.INVALID_SEGMENT_ATTRIBUTE, ...)`.
+
+#### Scenario: segments grouped by attribute value
+
+- **WHEN** assignments include 50 with `attributes.country === 'US'` and 50 with `'CA'`, and `analyzeBySegments(..., 'country')` is called
+- **THEN** the result has two `SegmentResult` entries with `segmentValue: 'US'` and `'CA'` respectively
+
+#### Scenario: significance gates on min sample size
+
+- **WHEN** a segment has variant A with 5 assignments and variant B with 5 assignments
+- **THEN** `isSignificant` is `false` regardless of conversion rates
+
+#### Scenario: confidence intervals are clamped to [0, 1]
+
+- **WHEN** a tiny sample produces a Wald interval with lower bound `-0.05` or upper bound `1.10`
+- **THEN** the returned `lowerBound` is `0` and `upperBound` is `1`
+
+#### Scenario: interaction flagged above 0.01 strength
+
+- **WHEN** a segment value has variant rates that differ from global rates such that sum-of-squared-weighted-diff > 0.01
+- **THEN** `hasInteraction` is `true`
+
+#### Scenario: missing attribute throws
+
+- **WHEN** `analyzeBySegments(..., 'nonexistent')` is called and no assignment carries that attribute
+- **THEN** `AbTestingError` with code `AB_CODES.INVALID_SEGMENT_ATTRIBUTE` is thrown
+
+**Verified by:** `src/segments/segment-analyzer.test.ts::"groups assignments by segment attribute value"` AND `src/segments/segment-analyzer.test.ts::"isSignificant is false when sample size is below minimum"` AND `src/segments/segment-analyzer.test.ts::"clamps confidence intervals to [0, 1]"` AND `src/segments/segment-analyzer.spec-conformance.test.ts::"flags interaction when strength exceeds threshold"` AND `src/segments/segment-analyzer.spec-conformance.test.ts::"throws INVALID_SEGMENT_ATTRIBUTE when attribute is absent"`
+
+---
+
+### Requirement: Sequential testing with alpha-spending and early-stop rules
+
+The package SHALL provide group-sequential test machinery for safe early stopping. `obrienFlemingBoundary(informationFraction, alpha)` SHALL return the z-boundary `z(1 - alpha/2) / sqrt(informationFraction)`. `pocockBoundary(informationFraction, alpha, maxLooks)` SHALL return a constant z across looks (Pocock spending). `alphaSpent(informationFraction, alpha, boundaryType?)` SHALL return cumulative alpha spent under the chosen spending function and SHALL be monotone-non-decreasing in `informationFraction`. `interimAnalysis(...)` SHALL compute a single look and report `shouldStop: true` with `stopReason: 'significance'` when `testStatistic >= spendingBoundary`, or `'futility'` when `informationFraction >= 0.75` and the projected final z falls below 30% of the final boundary. `runSequentialTest(...)` SHALL drive the sequence and stop on first `shouldStop`. Construction validates `alpha ∈ (0, 1)`, `maxLooks ≥ 1`, `mde > 0`, `power ∈ (0, 1)` else throws `AbTestingError(AB_CODES.INVALID_SEQUENTIAL_CONFIG, ...)`.
+
+#### Scenario: O'Brien-Fleming boundary is monotone-decreasing
+
+- **WHEN** `obrienFlemingBoundary(0.25, 0.05)` and `obrienFlemingBoundary(1, 0.05)` are computed
+- **THEN** the first value is strictly greater than the second
+
+#### Scenario: Pocock boundary is constant across looks
+
+- **WHEN** `pocockBoundary(0.5, 0.05, 4)` and `pocockBoundary(1, 0.05, 4)` are computed
+- **THEN** both values are equal within numerical tolerance
+
+#### Scenario: alpha-spent is monotone-non-decreasing
+
+- **WHEN** `alphaSpent` is called at fractions `[0.1, 0.25, 0.5, 0.75, 1.0]`
+- **THEN** the returned sequence is non-decreasing and ends at exactly `alpha`
+
+#### Scenario: futility check fires only after 75% information
+
+- **WHEN** `interimAnalysis` is called at `informationFraction = 0.5` with a tiny test statistic
+- **THEN** `stopReason` is `null` (futility branch suppressed)
+
+#### Scenario: runSequentialTest stops on first shouldStop
+
+- **WHEN** the data array has four looks and the second look returns `shouldStop: true`
+- **THEN** `runSequentialTest` returns exactly two `InterimResult` entries
+
+**Verified by:** `src/sequential/sequential-test.test.ts::"obrienFlemingBoundary is monotone-decreasing in informationFraction"` AND `src/sequential/sequential-test.test.ts::"pocockBoundary is constant across looks"` AND `src/sequential/sequential-test.spec-conformance.test.ts::"alphaSpent is monotone-non-decreasing and ends at alpha"` AND `src/sequential/sequential-test.spec-conformance.test.ts::"futility branch is suppressed below 75% informationFraction"` AND `src/sequential/sequential-test.test.ts::"runSequentialTest stops on first shouldStop"`
+
+---
+
+### Requirement: Bayesian stats — Beta-Binomial conversion model
+
+The package SHALL provide Beta-Binomial Bayesian primitives. `posteriorBeta(conversions, participants, priorAlpha?, priorBeta?)` SHALL return the conjugate posterior `{ alpha: priorAlpha + conversions, beta: priorBeta + (participants - conversions) }`. `betaPdf` / `betaCdf` / `betaQuantile` SHALL be self-consistent (`betaCdf(betaQuantile(p, a, b), a, b) ≈ p` within bisection tolerance). `betaSample(alpha, beta, rng)` SHALL draw a Beta sample via two Gamma draws (Marsaglia-Tsang). `probabilityBGreaterA(a, b, simulations?, rng)` SHALL return a Monte Carlo estimate of `P(B > A)`. `credibleInterval(params, width?)` SHALL return a quantile-based interval at the requested width (default 95%). `expectedLoss(chosen, other, simulations?, rng)` SHALL return the expected forfeited lift from picking `chosen`. `bayesianAnalysis(results, priorAlpha?, priorBeta?, rng)` SHALL run the full pairwise comparison on two `VariantResult` objects. All RNG-dependent functions SHALL accept an injected `Rng` (no module-scope `Math.random`).
+
+#### Scenario: posteriorBeta is the conjugate update
+
+- **WHEN** `posteriorBeta(10, 100, 1, 1)` is called
+- **THEN** the result is `{ alpha: 11, beta: 91 }`
+
+#### Scenario: betaCdf and betaQuantile are inverses
+
+- **WHEN** `betaQuantile(0.5, 5, 5)` then `betaCdf(<that>, 5, 5)` is computed
+- **THEN** the round-trip value is within `1e-3` of `0.5`
+
+#### Scenario: probabilityBGreaterA converges to 0.5 for symmetric posteriors
+
+- **WHEN** `probabilityBGreaterA({ alpha: 5, beta: 5 }, { alpha: 5, beta: 5 }, 20000, rng)` is called
+- **THEN** the result is within `0.02` of `0.5`
+
+#### Scenario: credibleInterval respects width parameter
+
+- **WHEN** `credibleInterval(params, 0.5)` and `credibleInterval(params, 0.95)` are computed for the same `params`
+- **THEN** the 95% interval contains the 50% interval
+
+#### Scenario: same Rng seed produces deterministic samples
+
+- **WHEN** `betaSample(2, 3, rng)` is called twice with the same `Rng` state
+- **THEN** both calls return the same numeric value (reproducibility pin)
+
+**Verified by:** `src/stats/bayesian.test.ts::"posteriorBeta returns the conjugate update"` AND `src/stats/bayesian.test.ts::"betaCdf and betaQuantile are mutual inverses"` AND `src/stats/bayesian.test.ts::"probabilityBGreaterA converges to 0.5 for symmetric posteriors"` AND `src/stats/bayesian.test.ts::"credibleInterval width parameter widens the interval"` AND `src/stats/bayesian.spec-conformance.test.ts::"injected Rng produces reproducible betaSample output"`
+
+---
+
+### Requirement: Assignment store — interface and in-memory adapter
+
+The package SHALL persist variant assignments via an `AssignmentStoreLike` interface (defined in `@suaveplan/types`). The interface SHALL expose five async methods: `get(experimentId, userId): Promise<Assignment | undefined>`, `set(assignment): Promise<void>`, `getByExperiment(experimentId): Promise<readonly Assignment[]>`, `getByUser(userId): Promise<readonly Assignment[]>`, `deleteByExperiment(experimentId): Promise<void>`. The package SHALL ship a `MemoryAssignmentStore` class implementing this interface and a `createMemoryAssignmentStore()` factory. `ExperimentManagerOptions` SHALL accept an optional `store?: AssignmentStoreLike`; when provided, `assignVariant` SHALL consult the store before re-bucketing and write through on miss. `MemoryAssignmentStore.set` SHALL be idempotent on identical-key assignments (last-writer-wins).
+
+#### Scenario: set/get round-trip
+
+- **WHEN** `store.set({ experimentId: 'e1', userId: 'u1', variantId: 'v1', assignedAt: '2026-05-02T00:00:00Z' })` is called and then `store.get('e1', 'u1')` is awaited
+- **THEN** the resolved value is the same `Assignment` object shape
+
+#### Scenario: getByExperiment filters to one experiment
+
+- **WHEN** the store contains assignments for `e1` and `e2` and `getByExperiment('e1')` is awaited
+- **THEN** the result contains only `e1` assignments
+
+#### Scenario: deleteByExperiment cascades
+
+- **WHEN** the store contains 100 assignments for `e1` and `deleteByExperiment('e1')` is awaited
+- **THEN** subsequent `getByExperiment('e1')` returns an empty array
+
+#### Scenario: ExperimentManager with store does not re-bucket on hit
+
+- **WHEN** `createExperimentManager({ store })` is called, `assignVariant(exp, ctx)` runs once and persists, then `assignVariant(exp, ctx)` runs again with the same `ctx`
+- **THEN** the second call returns the persisted variant without invoking `bucketUser` (verified by an `Rng` spy that increments only on miss)
+
+#### Scenario: missing key resolves undefined
+
+- **WHEN** `store.get('does-not-exist', 'u1')` is awaited
+- **THEN** the resolved value is `undefined` (no error thrown)
+
+**Verified by:** `src/store/memory-assignment-store.test.ts::"set and get round-trip an Assignment"` AND `src/store/memory-assignment-store.test.ts::"getByExperiment returns only matching experiment ids"` AND `src/store/memory-assignment-store.test.ts::"deleteByExperiment removes all assignments for that experiment"` AND `src/core/experiment-manager/experiment-manager.store.test.ts::"second assignVariant call with store option returns persisted variant without re-bucketing"` AND `src/store/memory-assignment-store.spec-conformance.test.ts::"get resolves undefined for missing keys"`
+
+<!-- markdownlint-enable MD024 -->