# Capability Spec Delta: ab-testing (ab-testing-completion) **Package**: `@suaveplan/ab-testing` **Status**: PROPOSED > Every requirement below carries a `**Verified by:**` line at authoring time. No exceptions. (See `openspec/AGENTS.md` Section 13.) ## ADDED Requirements ### Requirement: Bandit algorithms with shared `Bandit` interface The package SHALL provide three multi-armed bandit algorithms — `EpsilonGreedyBandit`, `UCB1Bandit`, `ThompsonSamplingBandit` — all implementing a shared `Bandit` interface with `selectArm(): TArmId`, `update(arm: TArmId, reward: number): void`, `getValues(): readonly number[]`, `getCounts(): readonly number[]`. The generic `TArmId` parameter SHALL default to `number` (array index) so existing-shape upstream code continues to work; callers MAY pass string variant IDs as arms. Every algorithm SHALL accept an explicit `Rng` from `@suaveplan/types::Rng` (no module-scope `Math.random`). Construction SHALL throw `AbTestingError(AB_CODES.INVALID_BANDIT_CONFIG, ...)` when `arms.length < 2` or, for `EpsilonGreedyBandit`, `epsilon ∉ [0, 1]`. #### Scenario: epsilon-greedy explores when rng < epsilon - **WHEN** `EpsilonGreedyBandit` is constructed with `arms: ['a', 'b', 'c']`, `epsilon: 0.5`, and an `Rng` that returns `0.1` then `0.7` - **THEN** `selectArm()` returns the result of `rng() * 3` floored as a uniform random arm (exploration branch) #### Scenario: UCB1 plays each arm once before applying score - **WHEN** `UCB1Bandit` is constructed with three arms and `selectArm()` is called three times with no intervening `update` - **THEN** the first three returns are `arms[0]`, `arms[1]`, `arms[2]` in order #### Scenario: Thompson sampling updates posterior on success - **WHEN** `ThompsonSamplingBandit.update(arm, 1)` is called - **THEN** the underlying `successes[arm]` increments by 1; `update(arm, 0)` increments `failures[arm]` by 1 #### Scenario: invalid bandit config throws AbTestingError - **WHEN** `new EpsilonGreedyBandit({ arms: ['x'], epsilon: 0.1, rng })` is called (only one arm) - **THEN** the constructor throws `AbTestingError` with code `AB_CODES.INVALID_BANDIT_CONFIG` **Verified by:** `src/bandit/bandit.test.ts::"EpsilonGreedyBandit explores when rng draw is below epsilon"` AND `src/bandit/bandit.test.ts::"UCB1Bandit plays each arm once before scoring"` AND `src/bandit/bandit.test.ts::"ThompsonSamplingBandit updates posterior on success and failure"` AND `src/bandit/bandit.spec-conformance.test.ts::"throws INVALID_BANDIT_CONFIG when arms.length < 2"` --- ### Requirement: Multivariate engine with full + fractional factorial design The package SHALL provide a multivariate-experiment design engine. `fullFactorial(factors)` SHALL return every combination of factor levels (Cartesian product). `fractionalFactorial(factors, fraction)` SHALL return every Nth combination where `N = round(1 / fraction)`. `analyzeMultivariate(factors, data)` SHALL return a `MultivariateResult` containing per-treatment conversion rates, per-factor `FactorEffect` objects with `LevelEffect[]` and a sum-of-squares `importance` score, pairwise `InteractionEffect[]` covering every `(i, j)` pair with `i < j`, and a `bestCombination` (highest-rate treatment with `participants > 0`). Construction with `factors.length === 0` or any factor having `levels.length === 0` SHALL throw `AbTestingError(AB_CODES.INVALID_FACTOR_DESIGN, ...)`. #### Scenario: full factorial enumerates Π levels - **WHEN** `fullFactorial([{ name: 'a', levels: ['x', 'y'] }, { name: 'b', levels: ['1', '2', '3'] }])` is called - **THEN** the result has length `2 * 3 = 6` and each combination has unique `id` #### Scenario: fractional factorial samples every Nth combination - **WHEN** `fractionalFactorial(factors, 0.5)` is called against a 6-combination factor set - **THEN** the result length is 3 and the kept combinations are at indices 0, 2, 4 in `fullFactorial(factors)` #### Scenario: factor importance is a non-negative sum-of-squares score - **WHEN** `analyzeMultivariate` is called with treatments where one factor has level rates `[0.1, 0.5, 0.9]` and another has `[0.5, 0.5, 0.5]` - **THEN** the first factor's `importance` strictly exceeds the second factor's `importance` #### Scenario: bestCombination is the highest-rate treatment with participants - **WHEN** treatments include one combination with `conversionRate: 0.9` but `participants: 0` and another with `0.5` and `participants: 100` - **THEN** `bestCombination.combinationId` is the latter #### Scenario: empty factor set throws - **WHEN** `analyzeMultivariate([], data)` is called - **THEN** `AbTestingError` with code `AB_CODES.INVALID_FACTOR_DESIGN` is thrown **Verified by:** `src/multivariate/multivariate-engine.test.ts::"fullFactorial enumerates the Cartesian product of levels"` AND `src/multivariate/multivariate-engine.test.ts::"fractionalFactorial keeps every Nth combination"` AND `src/multivariate/multivariate-engine.spec-conformance.test.ts::"factor importance ranks higher-variance factors first"` AND `src/multivariate/multivariate-engine.test.ts::"bestCombination ignores treatments with zero participants"` AND `src/multivariate/multivariate-engine.spec-conformance.test.ts::"throws INVALID_FACTOR_DESIGN for empty factor set"` --- ### Requirement: Segment analysis with per-segment lift and interaction terms The package SHALL provide segment-level analyses. `analyzeBySegments(assignments, conversions, variantIds, segmentAttribute)` SHALL group assignments by `attributes[segmentAttribute]` value, compute per-segment `VariantResult[]`, attach a two-proportion z-test significance flag (z > 1.96 with both arms n ≥ 10), and report `conversionRateDiff` between best and worst variant. `segmentConfidenceIntervals(...)` SHALL return Wald intervals on conversion rates, `[0, 1]`-clamped. `detectSegmentInteraction(...)` SHALL return interaction-strength scores per segment value relative to global per-variant rates, flagged as `hasInteraction: true` when strength exceeds 0.01. When no assignment exposes the requested attribute, the call SHALL throw `AbTestingError(AB_CODES.INVALID_SEGMENT_ATTRIBUTE, ...)`. #### Scenario: segments grouped by attribute value - **WHEN** assignments include 50 with `attributes.country === 'US'` and 50 with `'CA'`, and `analyzeBySegments(..., 'country')` is called - **THEN** the result has two `SegmentResult` entries with `segmentValue: 'US'` and `'CA'` respectively #### Scenario: significance gates on min sample size - **WHEN** a segment has variant A with 5 assignments and variant B with 5 assignments - **THEN** `isSignificant` is `false` regardless of conversion rates #### Scenario: confidence intervals are clamped to [0, 1] - **WHEN** a tiny sample produces a Wald interval with lower bound `-0.05` or upper bound `1.10` - **THEN** the returned `lowerBound` is `0` and `upperBound` is `1` #### Scenario: interaction flagged above 0.01 strength - **WHEN** a segment value has variant rates that differ from global rates such that sum-of-squared-weighted-diff > 0.01 - **THEN** `hasInteraction` is `true` #### Scenario: missing attribute throws - **WHEN** `analyzeBySegments(..., 'nonexistent')` is called and no assignment carries that attribute - **THEN** `AbTestingError` with code `AB_CODES.INVALID_SEGMENT_ATTRIBUTE` is thrown **Verified by:** `src/segments/segment-analyzer.test.ts::"groups assignments by segment attribute value"` AND `src/segments/segment-analyzer.test.ts::"isSignificant is false when sample size is below minimum"` AND `src/segments/segment-analyzer.test.ts::"clamps confidence intervals to [0, 1]"` AND `src/segments/segment-analyzer.spec-conformance.test.ts::"flags interaction when strength exceeds threshold"` AND `src/segments/segment-analyzer.spec-conformance.test.ts::"throws INVALID_SEGMENT_ATTRIBUTE when attribute is absent"` --- ### Requirement: Sequential testing with alpha-spending and early-stop rules The package SHALL provide group-sequential test machinery for safe early stopping. `obrienFlemingBoundary(informationFraction, alpha)` SHALL return the z-boundary `z(1 - alpha/2) / sqrt(informationFraction)`. `pocockBoundary(informationFraction, alpha, maxLooks)` SHALL return a constant z across looks (Pocock spending). `alphaSpent(informationFraction, alpha, boundaryType?)` SHALL return cumulative alpha spent under the chosen spending function and SHALL be monotone-non-decreasing in `informationFraction`. `interimAnalysis(...)` SHALL compute a single look and report `shouldStop: true` with `stopReason: 'significance'` when `testStatistic >= spendingBoundary`, or `'futility'` when `informationFraction >= 0.75` and the projected final z falls below 30% of the final boundary. `runSequentialTest(...)` SHALL drive the sequence and stop on first `shouldStop`. Construction validates `alpha ∈ (0, 1)`, `maxLooks ≥ 1`, `mde > 0`, `power ∈ (0, 1)` else throws `AbTestingError(AB_CODES.INVALID_SEQUENTIAL_CONFIG, ...)`. #### Scenario: O'Brien-Fleming boundary is monotone-decreasing - **WHEN** `obrienFlemingBoundary(0.25, 0.05)` and `obrienFlemingBoundary(1, 0.05)` are computed - **THEN** the first value is strictly greater than the second #### Scenario: Pocock boundary is constant across looks - **WHEN** `pocockBoundary(0.5, 0.05, 4)` and `pocockBoundary(1, 0.05, 4)` are computed - **THEN** both values are equal within numerical tolerance #### Scenario: alpha-spent is monotone-non-decreasing - **WHEN** `alphaSpent` is called at fractions `[0.1, 0.25, 0.5, 0.75, 1.0]` - **THEN** the returned sequence is non-decreasing and ends at exactly `alpha` #### Scenario: futility check fires only after 75% information - **WHEN** `interimAnalysis` is called at `informationFraction = 0.5` with a tiny test statistic - **THEN** `stopReason` is `null` (futility branch suppressed) #### Scenario: runSequentialTest stops on first shouldStop - **WHEN** the data array has four looks and the second look returns `shouldStop: true` - **THEN** `runSequentialTest` returns exactly two `InterimResult` entries **Verified by:** `src/sequential/sequential-test.test.ts::"obrienFlemingBoundary is monotone-decreasing in informationFraction"` AND `src/sequential/sequential-test.test.ts::"pocockBoundary is constant across looks"` AND `src/sequential/sequential-test.spec-conformance.test.ts::"alphaSpent is monotone-non-decreasing and ends at alpha"` AND `src/sequential/sequential-test.spec-conformance.test.ts::"futility branch is suppressed below 75% informationFraction"` AND `src/sequential/sequential-test.test.ts::"runSequentialTest stops on first shouldStop"` --- ### Requirement: Bayesian stats — Beta-Binomial conversion model The package SHALL provide Beta-Binomial Bayesian primitives. `posteriorBeta(conversions, participants, priorAlpha?, priorBeta?)` SHALL return the conjugate posterior `{ alpha: priorAlpha + conversions, beta: priorBeta + (participants - conversions) }`. `betaPdf` / `betaCdf` / `betaQuantile` SHALL be self-consistent (`betaCdf(betaQuantile(p, a, b), a, b) ≈ p` within bisection tolerance). `betaSample(alpha, beta, rng)` SHALL draw a Beta sample via two Gamma draws (Marsaglia-Tsang). `probabilityBGreaterA(a, b, simulations?, rng)` SHALL return a Monte Carlo estimate of `P(B > A)`. `credibleInterval(params, width?)` SHALL return a quantile-based interval at the requested width (default 95%). `expectedLoss(chosen, other, simulations?, rng)` SHALL return the expected forfeited lift from picking `chosen`. `bayesianAnalysis(results, priorAlpha?, priorBeta?, rng)` SHALL run the full pairwise comparison on two `VariantResult` objects. All RNG-dependent functions SHALL accept an injected `Rng` (no module-scope `Math.random`). #### Scenario: posteriorBeta is the conjugate update - **WHEN** `posteriorBeta(10, 100, 1, 1)` is called - **THEN** the result is `{ alpha: 11, beta: 91 }` #### Scenario: betaCdf and betaQuantile are inverses - **WHEN** `betaQuantile(0.5, 5, 5)` then `betaCdf(, 5, 5)` is computed - **THEN** the round-trip value is within `1e-3` of `0.5` #### Scenario: probabilityBGreaterA converges to 0.5 for symmetric posteriors - **WHEN** `probabilityBGreaterA({ alpha: 5, beta: 5 }, { alpha: 5, beta: 5 }, 20000, rng)` is called - **THEN** the result is within `0.02` of `0.5` #### Scenario: credibleInterval respects width parameter - **WHEN** `credibleInterval(params, 0.5)` and `credibleInterval(params, 0.95)` are computed for the same `params` - **THEN** the 95% interval contains the 50% interval #### Scenario: same Rng seed produces deterministic samples - **WHEN** `betaSample(2, 3, rng)` is called twice with the same `Rng` state - **THEN** both calls return the same numeric value (reproducibility pin) **Verified by:** `src/stats/bayesian.test.ts::"posteriorBeta returns the conjugate update"` AND `src/stats/bayesian.test.ts::"betaCdf and betaQuantile are mutual inverses"` AND `src/stats/bayesian.test.ts::"probabilityBGreaterA converges to 0.5 for symmetric posteriors"` AND `src/stats/bayesian.test.ts::"credibleInterval width parameter widens the interval"` AND `src/stats/bayesian.spec-conformance.test.ts::"injected Rng produces reproducible betaSample output"` --- ### Requirement: Assignment store — interface and in-memory adapter The package SHALL persist variant assignments via an `AssignmentStoreLike` interface (defined in `@suaveplan/types`). The interface SHALL expose five async methods: `get(experimentId, userId): Promise`, `set(assignment): Promise`, `getByExperiment(experimentId): Promise`, `getByUser(userId): Promise`, `deleteByExperiment(experimentId): Promise`. The package SHALL ship a `MemoryAssignmentStore` class implementing this interface and a `createMemoryAssignmentStore()` factory. `ExperimentManagerOptions` SHALL accept an optional `store?: AssignmentStoreLike`; when provided, `assignVariant` SHALL consult the store before re-bucketing and write through on miss. `MemoryAssignmentStore.set` SHALL be idempotent on identical-key assignments (last-writer-wins). #### Scenario: set/get round-trip - **WHEN** `store.set({ experimentId: 'e1', userId: 'u1', variantId: 'v1', assignedAt: '2026-05-02T00:00:00Z' })` is called and then `store.get('e1', 'u1')` is awaited - **THEN** the resolved value is the same `Assignment` object shape #### Scenario: getByExperiment filters to one experiment - **WHEN** the store contains assignments for `e1` and `e2` and `getByExperiment('e1')` is awaited - **THEN** the result contains only `e1` assignments #### Scenario: deleteByExperiment cascades - **WHEN** the store contains 100 assignments for `e1` and `deleteByExperiment('e1')` is awaited - **THEN** subsequent `getByExperiment('e1')` returns an empty array #### Scenario: ExperimentManager with store does not re-bucket on hit - **WHEN** `createExperimentManager({ store })` is called, `assignVariant(exp, ctx)` runs once and persists, then `assignVariant(exp, ctx)` runs again with the same `ctx` - **THEN** the second call returns the persisted variant without invoking `bucketUser` (verified by an `Rng` spy that increments only on miss) #### Scenario: missing key resolves undefined - **WHEN** `store.get('does-not-exist', 'u1')` is awaited - **THEN** the resolved value is `undefined` (no error thrown) **Verified by:** `src/store/memory-assignment-store.test.ts::"set and get round-trip an Assignment"` AND `src/store/memory-assignment-store.test.ts::"getByExperiment returns only matching experiment ids"` AND `src/store/memory-assignment-store.test.ts::"deleteByExperiment removes all assignments for that experiment"` AND `src/core/experiment-manager/experiment-manager.store.test.ts::"second assignVariant call with store option returns persisted variant without re-bucketing"` AND `src/store/memory-assignment-store.spec-conformance.test.ts::"get resolves undefined for missing keys"`