suaveplan/openspec/changes/ab-testing-completion/proposal.md

A/B testing completion — bandits, multivariate, segments, sequential, Bayesian, store

Package: @suaveplan/ab-testing Status: PROPOSED Type: spec-conformance + feature

Why

A 2026-05-02 audit of the genesis @suaveplan/ab-testing package against the upstream library at ../suaveplan/packages/universal/ab-testing/src and against this package's archived migration openspec/archive/2026-04-24-enterprise-ab-testing/ found six entire capability surfaces silently scoped out without justification. The archived proposal's parity table either omits these features or marks them shipped without delivering them; the current genesis implementation ships only the frequentist subset (deterministic FNV-1a bucketing, targeting, conversion-rate results, assignment via providers).

Six feature surfaces are missing:

1. Bandits — EpsilonGreedy, UCB1, ThompsonSamplingBandit (153 L upstream bandit/bandit.ts). Without them, callers wanting exploration/exploitation must hand-roll the algorithms or import a third-party — which then can't reuse the assignment store, error codes, or OTel instrumentation already in genesis.
2. Multivariate engine — fullFactorial, fractionalFactorial, analyzeMultivariate, factor-effect / interaction-effect analysis (226 L upstream multivariate/multivariate-engine.ts). The frequentist computeResults only handles A/B/C variant comparisons, not factor × level designs.
3. Segment analysis — analyzeBySegments, segmentConfidenceIntervals, detectSegmentInteraction (268 L upstream segmentation/segment-analyzer.ts). Per-segment lift and segment × variant interactions are how product teams actually read experiment results; without them, the package only answers "did variant B win overall?".
4. Sequential testing — obrienFlemingBoundary, alphaSpent, interimAnalysis, runSequentialTest (221 L upstream sequential/sequential-test.ts). Fixed-horizon hypothesis testing inflates Type I error when peeked at — alpha-spending boundaries are how production experimentation platforms permit early stopping safely.
5. Bayesian stats — betaPdf / betaCdf / betaQuantile / betaSample, posteriorBeta, thompsonSample, probabilityBGreaterA, credibleInterval, expectedLoss, bayesianAnalysis (263 L upstream stats/bayesian.ts). The package ships pure point-estimate conversion rates with no posterior, no credible intervals, no probability-of-being-best — which the bandit and Bayesian analysis paths both require.
6. Assignment store — AssignmentStore interface + MemoryAssignmentStore class (78 L upstream store/assignment-store.ts). Genesis currently re-derives assignments from providers on every read, which makes consistent assignment across server restarts impossible without each consumer re-implementing the persistence contract.

The audit found "These were silently dropped from openspec/archive/2026-04-24-enterprise-ab-testing scope (no archive justification)." — this change closes the gap and ratifies a parity table that explicitly admits these features are required.

What changes

@suaveplan/ab-testing — additions

Bandit subsystem

• ADDED: Bandit<TArmId extends string | number = number> interface — selectArm(): TArmId, update(arm: TArmId, reward: number): void, getValues(): readonly number[], getCounts(): readonly number[]. Generic over arm-id type so callers can use string variant IDs directly instead of a number → variantId map.
• ADDED: EpsilonGreedyBandit<TArmId> class — explore with probability epsilon, exploit best-known arm otherwise; injectable Rng from @suaveplan/types::Rng.
• ADDED: UCB1Bandit<TArmId> class — Upper Confidence Bound, plays each arm once, then value + sqrt(2 ln(N) / n).
• ADDED: ThompsonSamplingBandit<TArmId> class — Beta-Bernoulli posterior sampling with injectable Rng.
• ADDED: BanditConfig<TArmId> interface — arms: readonly TArmId[], optional epsilon, rng, priorAlpha, priorBeta.

Multivariate engine

• ADDED: Factor interface — name: string, levels: readonly string[].
• ADDED: TreatmentCombination, TreatmentResult, FactorEffect, LevelEffect, InteractionEffect, MultivariateResult interfaces.
• ADDED: fullFactorial(factors) — generate every combination.
• ADDED: fractionalFactorial(factors, fraction) — sample every Nth combination.
• ADDED: analyzeMultivariate(factors, data) — produce MultivariateResult with treatment-level rates, factor-effect importance scores, and pairwise interaction-strength scores.

Segment analysis

• ADDED: SegmentResult, SegmentConfidenceInterval, SegmentInteraction interfaces.
• ADDED: analyzeBySegments(assignments, conversions, variantIds, segmentAttribute) — variant rates per segment value, with simple two-proportion z-test significance flag.
• ADDED: segmentConfidenceIntervals(assignments, conversions, variantIds, segmentAttribute, confidence?) — Wald intervals on conversion rates per segment × variant.
• ADDED: detectSegmentInteraction(assignments, conversions, variantIds, segmentAttribute) — interaction-strength score between segment value and variant assignment.

Sequential testing

• ADDED: SequentialTestConfig — maxLooks, alpha, mde, power, optional boundaryType: 'pocock' | 'obrien-fleming' (default 'obrien-fleming').
• ADDED: InterimResult — lookNumber, totalLooks, informationFraction, spendingBoundary, testStatistic, pValue, shouldStop, stopReason: 'significance' | 'futility' | null, cumulativeAlphaSpent.
• ADDED: obrienFlemingBoundary(informationFraction, alpha) — z-boundary at fractional time.
• ADDED: pocockBoundary(informationFraction, alpha, maxLooks) — constant-z Pocock alternative (genesis enhancement; upstream ships only O'Brien-Fleming).
• ADDED: alphaSpent(informationFraction, alpha, boundaryType?) — cumulative alpha spent under the chosen spending function.
• ADDED: interimAnalysis(config, lookNumber, controlConv, controlPart, treatmentConv, treatmentPart) — single look.
• ADDED: runSequentialTest(config, dataAtLooks) — drive the sequence, stop on first shouldStop.

Bayesian stats

• ADDED: BetaParams interface — alpha, beta.
• ADDED: BayesianResult interface — posteriors, probabilityBIsGreater, credibleIntervalA, credibleIntervalB, expectedLossA, expectedLossB.
• ADDED: betaPdf(x, alpha, beta) — Beta probability density.
• ADDED: betaCdf(x, alpha, beta) — Beta cumulative distribution via regularized incomplete beta + continued fraction.
• ADDED: betaQuantile(p, alpha, beta) — inverse CDF via bisection.
• ADDED: betaSample(alpha, beta, rng?) — Beta draw via two Gamma draws (Marsaglia-Tsang).
• ADDED: posteriorBeta(conversions, participants, priorAlpha?, priorBeta?) — conjugate Beta-Binomial posterior.
• ADDED: thompsonSample(posteriors, rng?) — pick the variant with highest sampled posterior value.
• ADDED: probabilityBGreaterA(a, b, simulations?, rng?) — Monte Carlo P(B > A).
• ADDED: credibleInterval(params, width?) — quantile-based credible interval.
• ADDED: expectedLoss(chosen, other, simulations?, rng?) — expected lift forfeited by choosing chosen.
• ADDED: bayesianAnalysis(results, priorAlpha?, priorBeta?, rng?) — full pairwise analysis on two VariantResults.

Assignment store

• ADDED: AssignmentStoreLike interface — get(experimentId, userId), set(assignment), getByExperiment(experimentId), getByUser(userId), deleteByExperiment(experimentId). Async throughout. Promoted to @suaveplan/types per AGENTS.md §0 rule 5 (referenced by ≥2 packages once consumers wire it up).
• ADDED: MemoryAssignmentStore class implementing AssignmentStoreLike.
• ADDED: createMemoryAssignmentStore() factory.
• ADDED: optional store?: AssignmentStoreLike field on ExperimentManagerOptions. When present, the manager SHALL persist via the store and check the store before re-bucketing.

Errors

• ADDED: AB_CODES entries — INVALID_BANDIT_CONFIG, INVALID_FACTOR_DESIGN, INVALID_SEGMENT_ATTRIBUTE, INSUFFICIENT_SAMPLE_SIZE, INVALID_SEQUENTIAL_CONFIG.

Subpath exports (additive)

• ADDED: ./bandit, ./multivariate, ./segments, ./sequential, ./stats, ./store.

Existing surface — preserved

• Bucketing, targeting, results, providers, errors, types, schemas, createExperimentManager — unchanged.

Out of scope

• Multi-armed contextual bandits (LinUCB, Thompson with feature vectors) — upstream does not ship these; genesis will not introduce them without a separate proposal and a real consumer asking for them. The three context-free algorithms cover the documented use cases.
• Frequency-domain / time-series adjustments (CUPED, Bayesian state-space models) — not in upstream; orthogonal feature better served by a dedicated @suaveplan/experiment-stats follow-up if demand emerges.
• Persistent assignment-store adapters beyond MemoryAssignmentStore (Redis, SurrealDB, SQLite) — kept out of this proposal to keep dep-graph blast radius small. Each adapter will land as its own package (@suaveplan/ab-testing-redis, etc.) once MemoryAssignmentStore and the AssignmentStoreLike interface ship and the contract is pinned by testing-integration-redis etc.
• Variance-reduced bandits (linear / kernelized regression bandits) — large surface, not in upstream; deferred.
• Multi-objective optimization (Pareto-frontier multi-arm bandits) — not in upstream; deferred.
• Mutual-information-based factor pruning for the multivariate engine — not in upstream; the importance-by-sum-of-squares heuristic is sufficient for the documented use cases. Architectural reason: a real multi-objective design would change MultivariateResult.bestCombination semantics from "highest mean rate" to "Pareto-optimal set", which is a breaking shape change that needs its own proposal.
• Continuous monitoring / always-valid p-values (mSPRT, e-values) — distinct statistical regime from alpha-spending; the chosen O'Brien-Fleming + Pocock pair covers the documented "interim look" use case and is what upstream ships.

Source mapping

This change is a migration from ../suaveplan/packages/universal/ab-testing/src/. There is no genesis enterprise/ab-testing.md audit doc, so source mapping cites the upstream package source directly.

Requirement	Source file	Source lines	Notes
BanditAlgorithm interface	packages/universal/ab-testing/src/bandit/bandit.ts	L3-L13	Renamed to Bandit<TArmId> and made generic over arm id
EpsilonGreedy class	packages/universal/ab-testing/src/bandit/bandit.ts	L15-L57	Renamed to EpsilonGreedyBandit<TArmId>; replace Math.random default with explicit Rng injection
UCB1 class	packages/universal/ab-testing/src/bandit/bandit.ts	L59-L105	Renamed to UCB1Bandit<TArmId>
ThompsonSamplingBandit class	packages/universal/ab-testing/src/bandit/bandit.ts	L107-L153	Verbatim (already correctly named); replace Math.random default with explicit Rng injection
Factor / TreatmentCombination / TreatmentResult / FactorEffect / LevelEffect / InteractionEffect / MultivariateResult interfaces	packages/universal/ab-testing/src/multivariate/multivariate-engine.ts	L1-L50	Verbatim
fullFactorial	packages/universal/ab-testing/src/multivariate/multivariate-engine.ts	L52-L82	Verbatim
fractionalFactorial	packages/universal/ab-testing/src/multivariate/multivariate-engine.ts	L84-L93	Verbatim
analyzeMultivariate + factor-effect / interaction-effect helpers	packages/universal/ab-testing/src/multivariate/multivariate-engine.ts	L95-L226	Verbatim
SegmentResult / SegmentConfidenceInterval / SegmentInteraction interfaces	packages/universal/ab-testing/src/segmentation/segment-analyzer.ts	L7-L33	Verbatim
analyzeBySegments	packages/universal/ab-testing/src/segmentation/segment-analyzer.ts	L35-L85	Verbatim
segmentConfidenceIntervals	packages/universal/ab-testing/src/segmentation/segment-analyzer.ts	L87-L123	Verbatim
detectSegmentInteraction	packages/universal/ab-testing/src/segmentation/segment-analyzer.ts	L125-L162	Verbatim
normalQuantile helper	packages/universal/ab-testing/src/segmentation/segment-analyzer.ts	L226-L268	Move to shared stats/normal.ts (used by both segmentation and sequential)
SequentialTestConfig / InterimResult interfaces	packages/universal/ab-testing/src/sequential/sequential-test.ts	L1-L24	Verbatim
obrienFlemingBoundary	packages/universal/ab-testing/src/sequential/sequential-test.ts	L26-L35	Verbatim
alphaSpent	packages/universal/ab-testing/src/sequential/sequential-test.ts	L37-L43	Extend with boundaryType parameter
interimAnalysis	packages/universal/ab-testing/src/sequential/sequential-test.ts	L45-L114	Verbatim
runSequentialTest	packages/universal/ab-testing/src/sequential/sequential-test.ts	L129-L154	Verbatim
BetaParams / BayesianResult interfaces	packages/universal/ab-testing/src/stats/bayesian.ts	L3-L17	Verbatim
betaPdf / betaCdf / betaQuantile	packages/universal/ab-testing/src/stats/bayesian.ts	L48-L116	Verbatim
betaSample (Marsaglia-Tsang)	packages/universal/ab-testing/src/stats/bayesian.ts	L118-L159	Verbatim; replace Math.random default with Rng injection
posteriorBeta	packages/universal/ab-testing/src/stats/bayesian.ts	L161-L172	Verbatim
thompsonSample	packages/universal/ab-testing/src/stats/bayesian.ts	L174-L189	Verbatim
probabilityBGreaterA	packages/universal/ab-testing/src/stats/bayesian.ts	L191-L205	Verbatim
credibleInterval	packages/universal/ab-testing/src/stats/bayesian.ts	L207-L216	Verbatim
expectedLoss	packages/universal/ab-testing/src/stats/bayesian.ts	L218-L232	Verbatim
bayesianAnalysis	packages/universal/ab-testing/src/stats/bayesian.ts	L234-L263	Verbatim
AssignmentStore interface	packages/universal/ab-testing/src/store/assignment-store.ts	L3-L15	Renamed to AssignmentStoreLike; promoted to @suaveplan/types
MemoryAssignmentStore class	packages/universal/ab-testing/src/store/assignment-store.ts	L17-L78	Verbatim

Requirements with no source line

Requirement	Justification
Generic TArmId parameter on Bandit<TArmId>	Ergonomic improvement over upstream's positional number arms — callers pass string variant IDs directly, eliminating an error-prone number → variantId lookup in user code. Upstream uses array indices; genesis preserves array indices as the default TArmId = number so existing-shape code continues to work.
Pocock spending boundary alternative	Upstream ships only O'Brien-Fleming. Pocock's constant-z spending is the canonical alternative; offering both lets callers pick the conservative-early (O'Brien-Fleming) vs. liberal-early (Pocock) trade-off. Adds < 30 lines and zero deps.
Inject Rng instead of defaulting to Math.random	CLAUDE.md determinism rule (Critical rule 9) requires no module-scope Math.random(). Upstream defaults all rng parameters to Math.random; genesis SHALL require an explicit Rng from @suaveplan/types at call boundary, with crypto.getRandomValues-based default available via createDefaultRng().
AssignmentStoreLike promoted to @suaveplan/types	AGENTS.md §0 rule 5: shared interface across ≥2 packages (@suaveplan/ab-testing defines, future @suaveplan/ab-testing-redis / -surrealdb consume) must live in the types-only foundation to prevent circular deps. Upstream defines it inline.
New AB_CODES entries — INVALID_BANDIT_CONFIG, INVALID_FACTOR_DESIGN, INVALID_SEGMENT_ATTRIBUTE, INSUFFICIENT_SAMPLE_SIZE, INVALID_SEQUENTIAL_CONFIG	Upstream throws plain Error; genesis-wide policy (CLAUDE.md Critical rule 2 "every thrown error route through @suaveplan/error") requires registered codes for each public failure path.
OTel counters / histograms — ab.bandit.arm.selected (counter, by arm_id), ab.multivariate.combinations.evaluated (counter), ab.sequential.alpha.spent (histogram), ab.bayesian.simulations.duration_ms (histogram), ab.assignment.store.lookup (counter, by hit/miss)	Genesis-wide instrumentation policy. Upstream ships zero observability.

Verification approach

Each requirement in the spec delta carries a **Verified by:** line pointing at concrete it(...) titles. Test files that will exist after this change ships:

• packages/testing/ab-testing/src/bandit/bandit.test.ts — EpsilonGreedyBandit exploration vs. exploitation distribution; UCB1Bandit plays each arm once; ThompsonSamplingBandit posterior update on success vs. failure
• packages/testing/ab-testing/src/bandit/bandit.spec-conformance.test.ts — injected Rng is the only randomness source; INVALID_BANDIT_CONFIG thrown when arms.length < 2
• packages/testing/ab-testing/src/multivariate/multivariate-engine.test.ts — fullFactorial exhaustive enumeration; fractionalFactorial step semantics; factor importance ordering; bestCombination is the highest-rate treatment with > 0 participants
• packages/testing/ab-testing/src/multivariate/multivariate-engine.spec-conformance.test.ts — interaction strength is non-negative; pairwise interactions cover every (i, j) with i < j
• packages/testing/ab-testing/src/segments/segment-analyzer.test.ts — segment grouping by attribute value; significance gates on min sample size (≥10); confidence-interval coverage is [0, 1]-clamped
• packages/testing/ab-testing/src/segments/segment-analyzer.spec-conformance.test.ts — segment × variant interaction-strength threshold; INVALID_SEGMENT_ATTRIBUTE thrown for attributes absent from every assignment
• packages/testing/ab-testing/src/sequential/sequential-test.test.ts — O'Brien-Fleming boundary monotone-decreasing in informationFraction; Pocock boundary constant; runSequentialTest stops at first shouldStop
• packages/testing/ab-testing/src/sequential/sequential-test.spec-conformance.test.ts — alpha-spent monotone-increasing; futility check only fires when informationFraction ≥ 0.75; INVALID_SEQUENTIAL_CONFIG thrown for alpha ∉ (0, 1) or maxLooks < 1
• packages/testing/ab-testing/src/stats/bayesian.test.ts — betaPdf / betaCdf / betaQuantile round-trip; posteriorBeta conjugate update; probabilityBGreaterA Monte Carlo converges to closed-form for known cases; credibleInterval width parameter
• packages/testing/ab-testing/src/stats/bayesian.spec-conformance.test.ts — injected Rng reproducibility for betaSample / thompsonSample / expectedLoss
• packages/testing/ab-testing/src/store/memory-assignment-store.test.ts — set/get round-trip; getByExperiment / getByUser filtering; deleteByExperiment cascade; clear
• packages/testing/ab-testing/src/store/memory-assignment-store.spec-conformance.test.ts — AssignmentStoreLike contract pinned (will be re-used to verify future Redis/SurrealDB adapters)
• tests/ab-testing-public-surface.test.ts (cross-package) — pins the additive subpath-export surface so removal is a deliberate spec change

Migration impact

• Existing createExperimentManager consumers — no behavior change. The manager gains an optional store?: AssignmentStoreLike field on ExperimentManagerOptions; existing consumers that don't pass it keep the in-memory derivation path.
• @suaveplan/ab-testing direct dependents — none in the genesis monorepo (bunx turbo ls --filter=...@suaveplan/ab-testing returns the package itself only).
• Subpath imports — six new subpaths added (./bandit, ./multivariate, ./segments, ./sequential, ./stats, ./store). The root barrel re-exports from each. No removed or renamed exports.
• @suaveplan/types — the AssignmentStoreLike interface is promoted into @suaveplan/types per AGENTS.md §0 rule 5. This is additive to @suaveplan/types and does not affect existing consumers.
• Error-code consumers — additive only; existing AB_CODES entries preserved.

Parity Checklist

Capability	Upstream	Migrated	Notes
Deterministic FNV-1a bucketing	YES	YES	Already on genesis; preserved
Targeting rules	YES	YES	Already on genesis; preserved
Variant assignment & providers	YES	YES	Already on genesis; preserved
Frequentist results (computeResults)	YES	YES	Already on genesis; preserved
Bandits (EpsilonGreedy, UCB1, ThompsonSampling)	YES	YES	This change
Multivariate engine (full + fractional factorial, factor effects, interactions)	YES	YES	This change
Segment analysis (per-segment lift, CI, interaction)	YES	YES	This change
Sequential testing (O'Brien-Fleming alpha-spending, futility, early stopping)	YES	YES	This change
Sequential testing (Pocock boundary alternative)	NO	YES	Genesis enhancement; documented in "no source line" table
Bayesian stats (Beta-Binomial posteriors, credible intervals, expected loss)	YES	YES	This change
Assignment store (AssignmentStore + MemoryAssignmentStore)	YES	YES	This change
Persistent assignment-store adapters (Redis / SurrealDB / SQLite)	NO	NO	Scoped out — each adapter ships as its own package once AssignmentStoreLike is pinned
Contextual bandits (LinUCB, Thompson with features)	NO	NO	Scoped out — not in upstream, no consumer asking
Multi-objective / Pareto bandits	NO	NO	Scoped out — not in upstream
Continuous-monitoring p-values (mSPRT, e-values)	NO	NO	Scoped out — distinct statistical regime; alpha-spending covers the documented use case

Affected dependents

bunx turbo ls --filter=...@suaveplan/ab-testing enumerates only the package itself (no in-repo dependents).

• @suaveplan/ab-testing (self) — [x] tests run / [x] docs reviewed / [ ] no action needed

@suaveplan/types is touched additively (new AssignmentStoreLike interface). Verify the dep graph after promotion:

• @suaveplan/types — [x] tests run / [x] docs reviewed / [ ] no action needed (purely additive types-only export; byte-size test guards no runtime code is shipped)

No external dependents on genesis. The proposal is additive (no removed or modified exports), so no downstream sweep is required.