suaveplan/openspec/changes/ab-testing-completion/proposal.md

# A/B testing completion — bandits, multivariate, segments, sequential, Bayesian, store

**Package**: `@suaveplan/ab-testing`
**Status**: PROPOSED
**Type**: spec-conformance + feature

## Why

A 2026-05-02 audit of the genesis `@suaveplan/ab-testing` package against the upstream library at `../suaveplan/packages/universal/ab-testing/src` and against this package's archived migration `openspec/archive/2026-04-24-enterprise-ab-testing/` found six entire capability surfaces silently scoped out without justification. The archived proposal's parity table either omits these features or marks them shipped without delivering them; the current genesis implementation ships only the frequentist subset (deterministic FNV-1a bucketing, targeting, conversion-rate results, assignment via providers).

Six feature surfaces are missing:

1. **Bandits** — `EpsilonGreedy`, `UCB1`, `ThompsonSamplingBandit` (153 L upstream `bandit/bandit.ts`). Without them, callers wanting exploration/exploitation must hand-roll the algorithms or import a third-party — which then can't reuse the assignment store, error codes, or OTel instrumentation already in genesis.
2. **Multivariate engine** — `fullFactorial`, `fractionalFactorial`, `analyzeMultivariate`, factor-effect / interaction-effect analysis (226 L upstream `multivariate/multivariate-engine.ts`). The frequentist `computeResults` only handles A/B/C variant comparisons, not factor × level designs.
3. **Segment analysis** — `analyzeBySegments`, `segmentConfidenceIntervals`, `detectSegmentInteraction` (268 L upstream `segmentation/segment-analyzer.ts`). Per-segment lift and segment × variant interactions are how product teams actually read experiment results; without them, the package only answers "did variant B win overall?".
4. **Sequential testing** — `obrienFlemingBoundary`, `alphaSpent`, `interimAnalysis`, `runSequentialTest` (221 L upstream `sequential/sequential-test.ts`). Fixed-horizon hypothesis testing inflates Type I error when peeked at — alpha-spending boundaries are how production experimentation platforms permit early stopping safely.
5. **Bayesian stats** — `betaPdf` / `betaCdf` / `betaQuantile` / `betaSample`, `posteriorBeta`, `thompsonSample`, `probabilityBGreaterA`, `credibleInterval`, `expectedLoss`, `bayesianAnalysis` (263 L upstream `stats/bayesian.ts`). The package ships pure point-estimate conversion rates with no posterior, no credible intervals, no probability-of-being-best — which the bandit and Bayesian analysis paths both require.
6. **Assignment store** — `AssignmentStore` interface + `MemoryAssignmentStore` class (78 L upstream `store/assignment-store.ts`). Genesis currently re-derives assignments from providers on every read, which makes consistent assignment across server restarts impossible without each consumer re-implementing the persistence contract.

The audit found "These were silently dropped from `openspec/archive/2026-04-24-enterprise-ab-testing` scope (no archive justification)." — this change closes the gap and ratifies a parity table that explicitly admits these features are required.

## What changes

### `@suaveplan/ab-testing` — additions

#### Bandit subsystem

- ADDED: `Bandit<TArmId extends string | number = number>` interface — `selectArm(): TArmId`, `update(arm: TArmId, reward: number): void`, `getValues(): readonly number[]`, `getCounts(): readonly number[]`. Generic over arm-id type so callers can use string variant IDs directly instead of a `number → variantId` map.
- ADDED: `EpsilonGreedyBandit<TArmId>` class — explore with probability `epsilon`, exploit best-known arm otherwise; injectable `Rng` from `@suaveplan/types::Rng`.
- ADDED: `UCB1Bandit<TArmId>` class — Upper Confidence Bound, plays each arm once, then `value + sqrt(2 ln(N) / n)`.
- ADDED: `ThompsonSamplingBandit<TArmId>` class — Beta-Bernoulli posterior sampling with injectable `Rng`.
- ADDED: `BanditConfig<TArmId>` interface — `arms: readonly TArmId[]`, optional `epsilon`, `rng`, `priorAlpha`, `priorBeta`.

#### Multivariate engine

- ADDED: `Factor` interface — `name: string`, `levels: readonly string[]`.
- ADDED: `TreatmentCombination`, `TreatmentResult`, `FactorEffect`, `LevelEffect`, `InteractionEffect`, `MultivariateResult` interfaces.
- ADDED: `fullFactorial(factors)` — generate every combination.
- ADDED: `fractionalFactorial(factors, fraction)` — sample every Nth combination.
- ADDED: `analyzeMultivariate(factors, data)` — produce `MultivariateResult` with treatment-level rates, factor-effect importance scores, and pairwise interaction-strength scores.

#### Segment analysis

- ADDED: `SegmentResult`, `SegmentConfidenceInterval`, `SegmentInteraction` interfaces.
- ADDED: `analyzeBySegments(assignments, conversions, variantIds, segmentAttribute)` — variant rates per segment value, with simple two-proportion z-test significance flag.
- ADDED: `segmentConfidenceIntervals(assignments, conversions, variantIds, segmentAttribute, confidence?)` — Wald intervals on conversion rates per segment × variant.
- ADDED: `detectSegmentInteraction(assignments, conversions, variantIds, segmentAttribute)` — interaction-strength score between segment value and variant assignment.

#### Sequential testing

- ADDED: `SequentialTestConfig` — `maxLooks`, `alpha`, `mde`, `power`, optional `boundaryType: 'pocock' | 'obrien-fleming'` (default `'obrien-fleming'`).
- ADDED: `InterimResult` — `lookNumber`, `totalLooks`, `informationFraction`, `spendingBoundary`, `testStatistic`, `pValue`, `shouldStop`, `stopReason: 'significance' | 'futility' | null`, `cumulativeAlphaSpent`.
- ADDED: `obrienFlemingBoundary(informationFraction, alpha)` — z-boundary at fractional time.
- ADDED: `pocockBoundary(informationFraction, alpha, maxLooks)` — constant-z Pocock alternative (genesis enhancement; upstream ships only O'Brien-Fleming).
- ADDED: `alphaSpent(informationFraction, alpha, boundaryType?)` — cumulative alpha spent under the chosen spending function.
- ADDED: `interimAnalysis(config, lookNumber, controlConv, controlPart, treatmentConv, treatmentPart)` — single look.
- ADDED: `runSequentialTest(config, dataAtLooks)` — drive the sequence, stop on first `shouldStop`.

#### Bayesian stats

- ADDED: `BetaParams` interface — `alpha`, `beta`.
- ADDED: `BayesianResult` interface — `posteriors`, `probabilityBIsGreater`, `credibleIntervalA`, `credibleIntervalB`, `expectedLossA`, `expectedLossB`.
- ADDED: `betaPdf(x, alpha, beta)` — Beta probability density.
- ADDED: `betaCdf(x, alpha, beta)` — Beta cumulative distribution via regularized incomplete beta + continued fraction.
- ADDED: `betaQuantile(p, alpha, beta)` — inverse CDF via bisection.
- ADDED: `betaSample(alpha, beta, rng?)` — Beta draw via two Gamma draws (Marsaglia-Tsang).
- ADDED: `posteriorBeta(conversions, participants, priorAlpha?, priorBeta?)` — conjugate Beta-Binomial posterior.
- ADDED: `thompsonSample(posteriors, rng?)` — pick the variant with highest sampled posterior value.
- ADDED: `probabilityBGreaterA(a, b, simulations?, rng?)` — Monte Carlo `P(B > A)`.
- ADDED: `credibleInterval(params, width?)` — quantile-based credible interval.
- ADDED: `expectedLoss(chosen, other, simulations?, rng?)` — expected lift forfeited by choosing `chosen`.
- ADDED: `bayesianAnalysis(results, priorAlpha?, priorBeta?, rng?)` — full pairwise analysis on two `VariantResults`.

#### Assignment store

- ADDED: `AssignmentStoreLike` interface — `get(experimentId, userId)`, `set(assignment)`, `getByExperiment(experimentId)`, `getByUser(userId)`, `deleteByExperiment(experimentId)`. Async throughout. Promoted to `@suaveplan/types` per AGENTS.md §0 rule 5 (referenced by ≥2 packages once consumers wire it up).
- ADDED: `MemoryAssignmentStore` class implementing `AssignmentStoreLike`.
- ADDED: `createMemoryAssignmentStore()` factory.
- ADDED: optional `store?: AssignmentStoreLike` field on `ExperimentManagerOptions`. When present, the manager SHALL persist via the store and check the store before re-bucketing.

#### Errors

- ADDED: `AB_CODES` entries — `INVALID_BANDIT_CONFIG`, `INVALID_FACTOR_DESIGN`, `INVALID_SEGMENT_ATTRIBUTE`, `INSUFFICIENT_SAMPLE_SIZE`, `INVALID_SEQUENTIAL_CONFIG`.

### Subpath exports (additive)

- ADDED: `./bandit`, `./multivariate`, `./segments`, `./sequential`, `./stats`, `./store`.

### Existing surface — preserved

- Bucketing, targeting, results, providers, errors, types, schemas, `createExperimentManager` — unchanged.

## Out of scope

- **Multi-armed contextual bandits** (LinUCB, Thompson with feature vectors) — upstream does not ship these; genesis will not introduce them without a separate proposal and a real consumer asking for them. The three context-free algorithms cover the documented use cases.
- **Frequency-domain / time-series adjustments** (CUPED, Bayesian state-space models) — not in upstream; orthogonal feature better served by a dedicated `@suaveplan/experiment-stats` follow-up if demand emerges.
- **Persistent assignment-store adapters** beyond `MemoryAssignmentStore` (Redis, SurrealDB, SQLite) — kept out of this proposal to keep dep-graph blast radius small. Each adapter will land as its own package (`@suaveplan/ab-testing-redis`, etc.) once `MemoryAssignmentStore` and the `AssignmentStoreLike` interface ship and the contract is pinned by `testing-integration-redis` etc.
- **Variance-reduced bandits** (linear / kernelized regression bandits) — large surface, not in upstream; deferred.
- **Multi-objective optimization** (Pareto-frontier multi-arm bandits) — not in upstream; deferred.
- **Mutual-information-based factor pruning** for the multivariate engine — not in upstream; the importance-by-sum-of-squares heuristic is sufficient for the documented use cases. Architectural reason: a real multi-objective design would change `MultivariateResult.bestCombination` semantics from "highest mean rate" to "Pareto-optimal set", which is a breaking shape change that needs its own proposal.
- **Continuous monitoring / always-valid p-values** (mSPRT, e-values) — distinct statistical regime from alpha-spending; the chosen O'Brien-Fleming + Pocock pair covers the documented "interim look" use case and is what upstream ships.

## Source mapping

This change is a migration from `../suaveplan/packages/universal/ab-testing/src/`. There is no genesis `enterprise/ab-testing.md` audit doc, so source mapping cites the upstream package source directly.

| Requirement | Source file | Source lines | Notes |
|---|---|---|---|
| `BanditAlgorithm` interface | `packages/universal/ab-testing/src/bandit/bandit.ts` | L3-L13 | Renamed to `Bandit<TArmId>` and made generic over arm id |
| `EpsilonGreedy` class | `packages/universal/ab-testing/src/bandit/bandit.ts` | L15-L57 | Renamed to `EpsilonGreedyBandit<TArmId>`; replace `Math.random` default with explicit `Rng` injection |
| `UCB1` class | `packages/universal/ab-testing/src/bandit/bandit.ts` | L59-L105 | Renamed to `UCB1Bandit<TArmId>` |
| `ThompsonSamplingBandit` class | `packages/universal/ab-testing/src/bandit/bandit.ts` | L107-L153 | Verbatim (already correctly named); replace `Math.random` default with explicit `Rng` injection |
| `Factor` / `TreatmentCombination` / `TreatmentResult` / `FactorEffect` / `LevelEffect` / `InteractionEffect` / `MultivariateResult` interfaces | `packages/universal/ab-testing/src/multivariate/multivariate-engine.ts` | L1-L50 | Verbatim |
| `fullFactorial` | `packages/universal/ab-testing/src/multivariate/multivariate-engine.ts` | L52-L82 | Verbatim |
| `fractionalFactorial` | `packages/universal/ab-testing/src/multivariate/multivariate-engine.ts` | L84-L93 | Verbatim |
| `analyzeMultivariate` + factor-effect / interaction-effect helpers | `packages/universal/ab-testing/src/multivariate/multivariate-engine.ts` | L95-L226 | Verbatim |
| `SegmentResult` / `SegmentConfidenceInterval` / `SegmentInteraction` interfaces | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L7-L33 | Verbatim |
| `analyzeBySegments` | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L35-L85 | Verbatim |
| `segmentConfidenceIntervals` | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L87-L123 | Verbatim |
| `detectSegmentInteraction` | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L125-L162 | Verbatim |
| `normalQuantile` helper | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L226-L268 | Move to shared `stats/normal.ts` (used by both segmentation and sequential) |
| `SequentialTestConfig` / `InterimResult` interfaces | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L1-L24 | Verbatim |
| `obrienFlemingBoundary` | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L26-L35 | Verbatim |
| `alphaSpent` | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L37-L43 | Extend with `boundaryType` parameter |
| `interimAnalysis` | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L45-L114 | Verbatim |
| `runSequentialTest` | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L129-L154 | Verbatim |
| `BetaParams` / `BayesianResult` interfaces | `packages/universal/ab-testing/src/stats/bayesian.ts` | L3-L17 | Verbatim |
| `betaPdf` / `betaCdf` / `betaQuantile` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L48-L116 | Verbatim |
| `betaSample` (Marsaglia-Tsang) | `packages/universal/ab-testing/src/stats/bayesian.ts` | L118-L159 | Verbatim; replace `Math.random` default with `Rng` injection |
| `posteriorBeta` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L161-L172 | Verbatim |
| `thompsonSample` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L174-L189 | Verbatim |
| `probabilityBGreaterA` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L191-L205 | Verbatim |
| `credibleInterval` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L207-L216 | Verbatim |
| `expectedLoss` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L218-L232 | Verbatim |
| `bayesianAnalysis` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L234-L263 | Verbatim |
| `AssignmentStore` interface | `packages/universal/ab-testing/src/store/assignment-store.ts` | L3-L15 | Renamed to `AssignmentStoreLike`; promoted to `@suaveplan/types` |
| `MemoryAssignmentStore` class | `packages/universal/ab-testing/src/store/assignment-store.ts` | L17-L78 | Verbatim |

### Requirements with no source line

| Requirement | Justification |
|---|---|
| Generic `TArmId` parameter on `Bandit<TArmId>` | Ergonomic improvement over upstream's positional `number` arms — callers pass string variant IDs directly, eliminating an error-prone `number → variantId` lookup in user code. Upstream uses array indices; genesis preserves array indices as the default `TArmId = number` so existing-shape code continues to work. |
| Pocock spending boundary alternative | Upstream ships only O'Brien-Fleming. Pocock's constant-z spending is the canonical alternative; offering both lets callers pick the conservative-early (O'Brien-Fleming) vs. liberal-early (Pocock) trade-off. Adds < 30 lines and zero deps. |
| Inject `Rng` instead of defaulting to `Math.random` | CLAUDE.md determinism rule (Critical rule 9) requires no module-scope `Math.random()`. Upstream defaults all `rng` parameters to `Math.random`; genesis SHALL require an explicit `Rng` from `@suaveplan/types` at call boundary, with `crypto.getRandomValues`-based default available via `createDefaultRng()`. |
| `AssignmentStoreLike` promoted to `@suaveplan/types` | AGENTS.md §0 rule 5: shared interface across ≥2 packages (`@suaveplan/ab-testing` defines, future `@suaveplan/ab-testing-redis` / `-surrealdb` consume) must live in the types-only foundation to prevent circular deps. Upstream defines it inline. |
| New `AB_CODES` entries — `INVALID_BANDIT_CONFIG`, `INVALID_FACTOR_DESIGN`, `INVALID_SEGMENT_ATTRIBUTE`, `INSUFFICIENT_SAMPLE_SIZE`, `INVALID_SEQUENTIAL_CONFIG` | Upstream throws plain `Error`; genesis-wide policy (CLAUDE.md Critical rule 2 "every thrown error route through `@suaveplan/error`") requires registered codes for each public failure path. |
| OTel counters / histograms — `ab.bandit.arm.selected` (counter, by arm_id), `ab.multivariate.combinations.evaluated` (counter), `ab.sequential.alpha.spent` (histogram), `ab.bayesian.simulations.duration_ms` (histogram), `ab.assignment.store.lookup` (counter, by hit/miss) | Genesis-wide instrumentation policy. Upstream ships zero observability. |

## Verification approach

Each requirement in the spec delta carries a `**Verified by:**` line pointing at concrete `it(...)` titles. Test files that will exist after this change ships:

- `packages/testing/ab-testing/src/bandit/bandit.test.ts` — `EpsilonGreedyBandit` exploration vs. exploitation distribution; `UCB1Bandit` plays each arm once; `ThompsonSamplingBandit` posterior update on success vs. failure
- `packages/testing/ab-testing/src/bandit/bandit.spec-conformance.test.ts` — injected `Rng` is the only randomness source; `INVALID_BANDIT_CONFIG` thrown when `arms.length < 2`
- `packages/testing/ab-testing/src/multivariate/multivariate-engine.test.ts` — `fullFactorial` exhaustive enumeration; `fractionalFactorial` step semantics; factor importance ordering; `bestCombination` is the highest-rate treatment with > 0 participants
- `packages/testing/ab-testing/src/multivariate/multivariate-engine.spec-conformance.test.ts` — interaction strength is non-negative; pairwise interactions cover every `(i, j)` with `i < j`
- `packages/testing/ab-testing/src/segments/segment-analyzer.test.ts` — segment grouping by attribute value; significance gates on min sample size (≥10); confidence-interval coverage is `[0, 1]`-clamped
- `packages/testing/ab-testing/src/segments/segment-analyzer.spec-conformance.test.ts` — segment × variant interaction-strength threshold; `INVALID_SEGMENT_ATTRIBUTE` thrown for attributes absent from every assignment
- `packages/testing/ab-testing/src/sequential/sequential-test.test.ts` — O'Brien-Fleming boundary monotone-decreasing in informationFraction; Pocock boundary constant; `runSequentialTest` stops at first `shouldStop`
- `packages/testing/ab-testing/src/sequential/sequential-test.spec-conformance.test.ts` — alpha-spent monotone-increasing; futility check only fires when `informationFraction ≥ 0.75`; `INVALID_SEQUENTIAL_CONFIG` thrown for `alpha ∉ (0, 1)` or `maxLooks < 1`
- `packages/testing/ab-testing/src/stats/bayesian.test.ts` — `betaPdf` / `betaCdf` / `betaQuantile` round-trip; `posteriorBeta` conjugate update; `probabilityBGreaterA` Monte Carlo converges to closed-form for known cases; `credibleInterval` width parameter
- `packages/testing/ab-testing/src/stats/bayesian.spec-conformance.test.ts` — injected `Rng` reproducibility for `betaSample` / `thompsonSample` / `expectedLoss`
- `packages/testing/ab-testing/src/store/memory-assignment-store.test.ts` — `set`/`get` round-trip; `getByExperiment` / `getByUser` filtering; `deleteByExperiment` cascade; `clear`
- `packages/testing/ab-testing/src/store/memory-assignment-store.spec-conformance.test.ts` — `AssignmentStoreLike` contract pinned (will be re-used to verify future Redis/SurrealDB adapters)
- `tests/ab-testing-public-surface.test.ts` (cross-package) — pins the additive subpath-export surface so removal is a deliberate spec change

## Migration impact

- **Existing `createExperimentManager` consumers** — no behavior change. The manager gains an optional `store?: AssignmentStoreLike` field on `ExperimentManagerOptions`; existing consumers that don't pass it keep the in-memory derivation path.
- **`@suaveplan/ab-testing` direct dependents** — none in the genesis monorepo (`bunx turbo ls --filter=...@suaveplan/ab-testing` returns the package itself only).
- **Subpath imports** — six new subpaths added (`./bandit`, `./multivariate`, `./segments`, `./sequential`, `./stats`, `./store`). The root barrel re-exports from each. No removed or renamed exports.
- **`@suaveplan/types`** — the `AssignmentStoreLike` interface is promoted into `@suaveplan/types` per AGENTS.md §0 rule 5. This is additive to `@suaveplan/types` and does not affect existing consumers.
- **Error-code consumers** — additive only; existing `AB_CODES` entries preserved.

## Parity Checklist

| Capability | Upstream | Migrated | Notes |
|---|---|---|---|
| Deterministic FNV-1a bucketing | YES | YES | Already on genesis; preserved |
| Targeting rules | YES | YES | Already on genesis; preserved |
| Variant assignment & providers | YES | YES | Already on genesis; preserved |
| Frequentist results (`computeResults`) | YES | YES | Already on genesis; preserved |
| Bandits (`EpsilonGreedy`, `UCB1`, `ThompsonSampling`) | YES | YES | This change |
| Multivariate engine (full + fractional factorial, factor effects, interactions) | YES | YES | This change |
| Segment analysis (per-segment lift, CI, interaction) | YES | YES | This change |
| Sequential testing (O'Brien-Fleming alpha-spending, futility, early stopping) | YES | YES | This change |
| Sequential testing (Pocock boundary alternative) | NO | YES | Genesis enhancement; documented in "no source line" table |
| Bayesian stats (Beta-Binomial posteriors, credible intervals, expected loss) | YES | YES | This change |
| Assignment store (`AssignmentStore` + `MemoryAssignmentStore`) | YES | YES | This change |
| Persistent assignment-store adapters (Redis / SurrealDB / SQLite) | NO | NO | Scoped out — each adapter ships as its own package once `AssignmentStoreLike` is pinned |
| Contextual bandits (LinUCB, Thompson with features) | NO | NO | Scoped out — not in upstream, no consumer asking |
| Multi-objective / Pareto bandits | NO | NO | Scoped out — not in upstream |
| Continuous-monitoring p-values (mSPRT, e-values) | NO | NO | Scoped out — distinct statistical regime; alpha-spending covers the documented use case |

## Affected dependents

`bunx turbo ls --filter=...@suaveplan/ab-testing` enumerates only the package itself (no in-repo dependents).

- `@suaveplan/ab-testing` (self) — [x] tests run / [x] docs reviewed / [ ] no action needed

`@suaveplan/types` is touched additively (new `AssignmentStoreLike` interface). Verify the dep graph after promotion:

- `@suaveplan/types` — [x] tests run / [x] docs reviewed / [ ] no action needed (purely additive types-only export; `byte-size` test guards no runtime code is shipped)

No external dependents on genesis. The proposal is additive (no removed or modified exports), so no downstream sweep is required.