Initial Commit
This commit is contained in:
206
openspec/changes/ab-testing-completion/proposal.md
Normal file
206
openspec/changes/ab-testing-completion/proposal.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# A/B testing completion — bandits, multivariate, segments, sequential, Bayesian, store
|
||||
|
||||
**Package**: `@suaveplan/ab-testing`
|
||||
**Status**: PROPOSED
|
||||
**Type**: spec-conformance + feature
|
||||
|
||||
## Why
|
||||
|
||||
A 2026-05-02 audit of the genesis `@suaveplan/ab-testing` package against the upstream library at `../suaveplan/packages/universal/ab-testing/src` and against this package's archived migration `openspec/archive/2026-04-24-enterprise-ab-testing/` found six entire capability surfaces silently scoped out without justification. The archived proposal's parity table either omits these features or marks them shipped without delivering them; the current genesis implementation ships only the frequentist subset (deterministic FNV-1a bucketing, targeting, conversion-rate results, assignment via providers).
|
||||
|
||||
Six feature surfaces are missing:
|
||||
|
||||
1. **Bandits** — `EpsilonGreedy`, `UCB1`, `ThompsonSamplingBandit` (153 L upstream `bandit/bandit.ts`). Without them, callers wanting exploration/exploitation must hand-roll the algorithms or import a third-party — which then can't reuse the assignment store, error codes, or OTel instrumentation already in genesis.
|
||||
2. **Multivariate engine** — `fullFactorial`, `fractionalFactorial`, `analyzeMultivariate`, factor-effect / interaction-effect analysis (226 L upstream `multivariate/multivariate-engine.ts`). The frequentist `computeResults` only handles A/B/C variant comparisons, not factor × level designs.
|
||||
3. **Segment analysis** — `analyzeBySegments`, `segmentConfidenceIntervals`, `detectSegmentInteraction` (268 L upstream `segmentation/segment-analyzer.ts`). Per-segment lift and segment × variant interactions are how product teams actually read experiment results; without them, the package only answers "did variant B win overall?".
|
||||
4. **Sequential testing** — `obrienFlemingBoundary`, `alphaSpent`, `interimAnalysis`, `runSequentialTest` (221 L upstream `sequential/sequential-test.ts`). Fixed-horizon hypothesis testing inflates Type I error when peeked at — alpha-spending boundaries are how production experimentation platforms permit early stopping safely.
|
||||
5. **Bayesian stats** — `betaPdf` / `betaCdf` / `betaQuantile` / `betaSample`, `posteriorBeta`, `thompsonSample`, `probabilityBGreaterA`, `credibleInterval`, `expectedLoss`, `bayesianAnalysis` (263 L upstream `stats/bayesian.ts`). The package ships pure point-estimate conversion rates with no posterior, no credible intervals, no probability-of-being-best — which the bandit and Bayesian analysis paths both require.
|
||||
6. **Assignment store** — `AssignmentStore` interface + `MemoryAssignmentStore` class (78 L upstream `store/assignment-store.ts`). Genesis currently re-derives assignments from providers on every read, which makes consistent assignment across server restarts impossible without each consumer re-implementing the persistence contract.
|
||||
|
||||
The audit found "These were silently dropped from `openspec/archive/2026-04-24-enterprise-ab-testing` scope (no archive justification)." — this change closes the gap and ratifies a parity table that explicitly admits these features are required.
|
||||
|
||||
## What changes
|
||||
|
||||
### `@suaveplan/ab-testing` — additions
|
||||
|
||||
#### Bandit subsystem
|
||||
|
||||
- ADDED: `Bandit<TArmId extends string | number = number>` interface — `selectArm(): TArmId`, `update(arm: TArmId, reward: number): void`, `getValues(): readonly number[]`, `getCounts(): readonly number[]`. Generic over arm-id type so callers can use string variant IDs directly instead of a `number → variantId` map.
|
||||
- ADDED: `EpsilonGreedyBandit<TArmId>` class — explore with probability `epsilon`, exploit best-known arm otherwise; injectable `Rng` from `@suaveplan/types::Rng`.
|
||||
- ADDED: `UCB1Bandit<TArmId>` class — Upper Confidence Bound, plays each arm once, then `value + sqrt(2 ln(N) / n)`.
|
||||
- ADDED: `ThompsonSamplingBandit<TArmId>` class — Beta-Bernoulli posterior sampling with injectable `Rng`.
|
||||
- ADDED: `BanditConfig<TArmId>` interface — `arms: readonly TArmId[]`, optional `epsilon`, `rng`, `priorAlpha`, `priorBeta`.
|
||||
|
||||
#### Multivariate engine
|
||||
|
||||
- ADDED: `Factor` interface — `name: string`, `levels: readonly string[]`.
|
||||
- ADDED: `TreatmentCombination`, `TreatmentResult`, `FactorEffect`, `LevelEffect`, `InteractionEffect`, `MultivariateResult` interfaces.
|
||||
- ADDED: `fullFactorial(factors)` — generate every combination.
|
||||
- ADDED: `fractionalFactorial(factors, fraction)` — sample every Nth combination.
|
||||
- ADDED: `analyzeMultivariate(factors, data)` — produce `MultivariateResult` with treatment-level rates, factor-effect importance scores, and pairwise interaction-strength scores.
|
||||
|
||||
#### Segment analysis
|
||||
|
||||
- ADDED: `SegmentResult`, `SegmentConfidenceInterval`, `SegmentInteraction` interfaces.
|
||||
- ADDED: `analyzeBySegments(assignments, conversions, variantIds, segmentAttribute)` — variant rates per segment value, with simple two-proportion z-test significance flag.
|
||||
- ADDED: `segmentConfidenceIntervals(assignments, conversions, variantIds, segmentAttribute, confidence?)` — Wald intervals on conversion rates per segment × variant.
|
||||
- ADDED: `detectSegmentInteraction(assignments, conversions, variantIds, segmentAttribute)` — interaction-strength score between segment value and variant assignment.
|
||||
|
||||
#### Sequential testing
|
||||
|
||||
- ADDED: `SequentialTestConfig` — `maxLooks`, `alpha`, `mde`, `power`, optional `boundaryType: 'pocock' | 'obrien-fleming'` (default `'obrien-fleming'`).
|
||||
- ADDED: `InterimResult` — `lookNumber`, `totalLooks`, `informationFraction`, `spendingBoundary`, `testStatistic`, `pValue`, `shouldStop`, `stopReason: 'significance' | 'futility' | null`, `cumulativeAlphaSpent`.
|
||||
- ADDED: `obrienFlemingBoundary(informationFraction, alpha)` — z-boundary at fractional time.
|
||||
- ADDED: `pocockBoundary(informationFraction, alpha, maxLooks)` — constant-z Pocock alternative (genesis enhancement; upstream ships only O'Brien-Fleming).
|
||||
- ADDED: `alphaSpent(informationFraction, alpha, boundaryType?)` — cumulative alpha spent under the chosen spending function.
|
||||
- ADDED: `interimAnalysis(config, lookNumber, controlConv, controlPart, treatmentConv, treatmentPart)` — single look.
|
||||
- ADDED: `runSequentialTest(config, dataAtLooks)` — drive the sequence, stop on first `shouldStop`.
|
||||
|
||||
#### Bayesian stats
|
||||
|
||||
- ADDED: `BetaParams` interface — `alpha`, `beta`.
|
||||
- ADDED: `BayesianResult` interface — `posteriors`, `probabilityBIsGreater`, `credibleIntervalA`, `credibleIntervalB`, `expectedLossA`, `expectedLossB`.
|
||||
- ADDED: `betaPdf(x, alpha, beta)` — Beta probability density.
|
||||
- ADDED: `betaCdf(x, alpha, beta)` — Beta cumulative distribution via regularized incomplete beta + continued fraction.
|
||||
- ADDED: `betaQuantile(p, alpha, beta)` — inverse CDF via bisection.
|
||||
- ADDED: `betaSample(alpha, beta, rng?)` — Beta draw via two Gamma draws (Marsaglia-Tsang).
|
||||
- ADDED: `posteriorBeta(conversions, participants, priorAlpha?, priorBeta?)` — conjugate Beta-Binomial posterior.
|
||||
- ADDED: `thompsonSample(posteriors, rng?)` — pick the variant with highest sampled posterior value.
|
||||
- ADDED: `probabilityBGreaterA(a, b, simulations?, rng?)` — Monte Carlo `P(B > A)`.
|
||||
- ADDED: `credibleInterval(params, width?)` — quantile-based credible interval.
|
||||
- ADDED: `expectedLoss(chosen, other, simulations?, rng?)` — expected lift forfeited by choosing `chosen`.
|
||||
- ADDED: `bayesianAnalysis(results, priorAlpha?, priorBeta?, rng?)` — full pairwise analysis on two `VariantResults`.
|
||||
|
||||
#### Assignment store
|
||||
|
||||
- ADDED: `AssignmentStoreLike` interface — `get(experimentId, userId)`, `set(assignment)`, `getByExperiment(experimentId)`, `getByUser(userId)`, `deleteByExperiment(experimentId)`. Async throughout. Promoted to `@suaveplan/types` per AGENTS.md §0 rule 5 (referenced by ≥2 packages once consumers wire it up).
|
||||
- ADDED: `MemoryAssignmentStore` class implementing `AssignmentStoreLike`.
|
||||
- ADDED: `createMemoryAssignmentStore()` factory.
|
||||
- ADDED: optional `store?: AssignmentStoreLike` field on `ExperimentManagerOptions`. When present, the manager SHALL persist via the store and check the store before re-bucketing.
|
||||
|
||||
#### Errors
|
||||
|
||||
- ADDED: `AB_CODES` entries — `INVALID_BANDIT_CONFIG`, `INVALID_FACTOR_DESIGN`, `INVALID_SEGMENT_ATTRIBUTE`, `INSUFFICIENT_SAMPLE_SIZE`, `INVALID_SEQUENTIAL_CONFIG`.
|
||||
|
||||
### Subpath exports (additive)
|
||||
|
||||
- ADDED: `./bandit`, `./multivariate`, `./segments`, `./sequential`, `./stats`, `./store`.
|
||||
|
||||
### Existing surface — preserved
|
||||
|
||||
- Bucketing, targeting, results, providers, errors, types, schemas, `createExperimentManager` — unchanged.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Multi-armed contextual bandits** (LinUCB, Thompson with feature vectors) — upstream does not ship these; genesis will not introduce them without a separate proposal and a real consumer asking for them. The three context-free algorithms cover the documented use cases.
|
||||
- **Frequency-domain / time-series adjustments** (CUPED, Bayesian state-space models) — not in upstream; orthogonal feature better served by a dedicated `@suaveplan/experiment-stats` follow-up if demand emerges.
|
||||
- **Persistent assignment-store adapters** beyond `MemoryAssignmentStore` (Redis, SurrealDB, SQLite) — kept out of this proposal to keep dep-graph blast radius small. Each adapter will land as its own package (`@suaveplan/ab-testing-redis`, etc.) once `MemoryAssignmentStore` and the `AssignmentStoreLike` interface ship and the contract is pinned by `testing-integration-redis` etc.
|
||||
- **Variance-reduced bandits** (linear / kernelized regression bandits) — large surface, not in upstream; deferred.
|
||||
- **Multi-objective optimization** (Pareto-frontier multi-arm bandits) — not in upstream; deferred.
|
||||
- **Mutual-information-based factor pruning** for the multivariate engine — not in upstream; the importance-by-sum-of-squares heuristic is sufficient for the documented use cases. Architectural reason: a real multi-objective design would change `MultivariateResult.bestCombination` semantics from "highest mean rate" to "Pareto-optimal set", which is a breaking shape change that needs its own proposal.
|
||||
- **Continuous monitoring / always-valid p-values** (mSPRT, e-values) — distinct statistical regime from alpha-spending; the chosen O'Brien-Fleming + Pocock pair covers the documented "interim look" use case and is what upstream ships.
|
||||
|
||||
## Source mapping
|
||||
|
||||
This change is a migration from `../suaveplan/packages/universal/ab-testing/src/`. There is no genesis `enterprise/ab-testing.md` audit doc, so source mapping cites the upstream package source directly.
|
||||
|
||||
| Requirement | Source file | Source lines | Notes |
|
||||
|---|---|---|---|
|
||||
| `BanditAlgorithm` interface | `packages/universal/ab-testing/src/bandit/bandit.ts` | L3-L13 | Renamed to `Bandit<TArmId>` and made generic over arm id |
|
||||
| `EpsilonGreedy` class | `packages/universal/ab-testing/src/bandit/bandit.ts` | L15-L57 | Renamed to `EpsilonGreedyBandit<TArmId>`; replace `Math.random` default with explicit `Rng` injection |
|
||||
| `UCB1` class | `packages/universal/ab-testing/src/bandit/bandit.ts` | L59-L105 | Renamed to `UCB1Bandit<TArmId>` |
|
||||
| `ThompsonSamplingBandit` class | `packages/universal/ab-testing/src/bandit/bandit.ts` | L107-L153 | Verbatim (already correctly named); replace `Math.random` default with explicit `Rng` injection |
|
||||
| `Factor` / `TreatmentCombination` / `TreatmentResult` / `FactorEffect` / `LevelEffect` / `InteractionEffect` / `MultivariateResult` interfaces | `packages/universal/ab-testing/src/multivariate/multivariate-engine.ts` | L1-L50 | Verbatim |
|
||||
| `fullFactorial` | `packages/universal/ab-testing/src/multivariate/multivariate-engine.ts` | L52-L82 | Verbatim |
|
||||
| `fractionalFactorial` | `packages/universal/ab-testing/src/multivariate/multivariate-engine.ts` | L84-L93 | Verbatim |
|
||||
| `analyzeMultivariate` + factor-effect / interaction-effect helpers | `packages/universal/ab-testing/src/multivariate/multivariate-engine.ts` | L95-L226 | Verbatim |
|
||||
| `SegmentResult` / `SegmentConfidenceInterval` / `SegmentInteraction` interfaces | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L7-L33 | Verbatim |
|
||||
| `analyzeBySegments` | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L35-L85 | Verbatim |
|
||||
| `segmentConfidenceIntervals` | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L87-L123 | Verbatim |
|
||||
| `detectSegmentInteraction` | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L125-L162 | Verbatim |
|
||||
| `normalQuantile` helper | `packages/universal/ab-testing/src/segmentation/segment-analyzer.ts` | L226-L268 | Move to shared `stats/normal.ts` (used by both segmentation and sequential) |
|
||||
| `SequentialTestConfig` / `InterimResult` interfaces | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L1-L24 | Verbatim |
|
||||
| `obrienFlemingBoundary` | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L26-L35 | Verbatim |
|
||||
| `alphaSpent` | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L37-L43 | Extend with `boundaryType` parameter |
|
||||
| `interimAnalysis` | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L45-L114 | Verbatim |
|
||||
| `runSequentialTest` | `packages/universal/ab-testing/src/sequential/sequential-test.ts` | L129-L154 | Verbatim |
|
||||
| `BetaParams` / `BayesianResult` interfaces | `packages/universal/ab-testing/src/stats/bayesian.ts` | L3-L17 | Verbatim |
|
||||
| `betaPdf` / `betaCdf` / `betaQuantile` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L48-L116 | Verbatim |
|
||||
| `betaSample` (Marsaglia-Tsang) | `packages/universal/ab-testing/src/stats/bayesian.ts` | L118-L159 | Verbatim; replace `Math.random` default with `Rng` injection |
|
||||
| `posteriorBeta` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L161-L172 | Verbatim |
|
||||
| `thompsonSample` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L174-L189 | Verbatim |
|
||||
| `probabilityBGreaterA` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L191-L205 | Verbatim |
|
||||
| `credibleInterval` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L207-L216 | Verbatim |
|
||||
| `expectedLoss` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L218-L232 | Verbatim |
|
||||
| `bayesianAnalysis` | `packages/universal/ab-testing/src/stats/bayesian.ts` | L234-L263 | Verbatim |
|
||||
| `AssignmentStore` interface | `packages/universal/ab-testing/src/store/assignment-store.ts` | L3-L15 | Renamed to `AssignmentStoreLike`; promoted to `@suaveplan/types` |
|
||||
| `MemoryAssignmentStore` class | `packages/universal/ab-testing/src/store/assignment-store.ts` | L17-L78 | Verbatim |
|
||||
|
||||
### Requirements with no source line
|
||||
|
||||
| Requirement | Justification |
|
||||
|---|---|
|
||||
| Generic `TArmId` parameter on `Bandit<TArmId>` | Ergonomic improvement over upstream's positional `number` arms — callers pass string variant IDs directly, eliminating an error-prone `number → variantId` lookup in user code. Upstream uses array indices; genesis preserves array indices as the default `TArmId = number` so existing-shape code continues to work. |
|
||||
| Pocock spending boundary alternative | Upstream ships only O'Brien-Fleming. Pocock's constant-z spending is the canonical alternative; offering both lets callers pick the conservative-early (O'Brien-Fleming) vs. liberal-early (Pocock) trade-off. Adds < 30 lines and zero deps. |
|
||||
| Inject `Rng` instead of defaulting to `Math.random` | CLAUDE.md determinism rule (Critical rule 9) requires no module-scope `Math.random()`. Upstream defaults all `rng` parameters to `Math.random`; genesis SHALL require an explicit `Rng` from `@suaveplan/types` at call boundary, with `crypto.getRandomValues`-based default available via `createDefaultRng()`. |
|
||||
| `AssignmentStoreLike` promoted to `@suaveplan/types` | AGENTS.md §0 rule 5: shared interface across ≥2 packages (`@suaveplan/ab-testing` defines, future `@suaveplan/ab-testing-redis` / `-surrealdb` consume) must live in the types-only foundation to prevent circular deps. Upstream defines it inline. |
|
||||
| New `AB_CODES` entries — `INVALID_BANDIT_CONFIG`, `INVALID_FACTOR_DESIGN`, `INVALID_SEGMENT_ATTRIBUTE`, `INSUFFICIENT_SAMPLE_SIZE`, `INVALID_SEQUENTIAL_CONFIG` | Upstream throws plain `Error`; genesis-wide policy (CLAUDE.md Critical rule 2 "every thrown error route through `@suaveplan/error`") requires registered codes for each public failure path. |
|
||||
| OTel counters / histograms — `ab.bandit.arm.selected` (counter, by arm_id), `ab.multivariate.combinations.evaluated` (counter), `ab.sequential.alpha.spent` (histogram), `ab.bayesian.simulations.duration_ms` (histogram), `ab.assignment.store.lookup` (counter, by hit/miss) | Genesis-wide instrumentation policy. Upstream ships zero observability. |
|
||||
|
||||
## Verification approach
|
||||
|
||||
Each requirement in the spec delta carries a `**Verified by:**` line pointing at concrete `it(...)` titles. Test files that will exist after this change ships:
|
||||
|
||||
- `packages/testing/ab-testing/src/bandit/bandit.test.ts` — `EpsilonGreedyBandit` exploration vs. exploitation distribution; `UCB1Bandit` plays each arm once; `ThompsonSamplingBandit` posterior update on success vs. failure
|
||||
- `packages/testing/ab-testing/src/bandit/bandit.spec-conformance.test.ts` — injected `Rng` is the only randomness source; `INVALID_BANDIT_CONFIG` thrown when `arms.length < 2`
|
||||
- `packages/testing/ab-testing/src/multivariate/multivariate-engine.test.ts` — `fullFactorial` exhaustive enumeration; `fractionalFactorial` step semantics; factor importance ordering; `bestCombination` is the highest-rate treatment with > 0 participants
|
||||
- `packages/testing/ab-testing/src/multivariate/multivariate-engine.spec-conformance.test.ts` — interaction strength is non-negative; pairwise interactions cover every `(i, j)` with `i < j`
|
||||
- `packages/testing/ab-testing/src/segments/segment-analyzer.test.ts` — segment grouping by attribute value; significance gates on min sample size (≥10); confidence-interval coverage is `[0, 1]`-clamped
|
||||
- `packages/testing/ab-testing/src/segments/segment-analyzer.spec-conformance.test.ts` — segment × variant interaction-strength threshold; `INVALID_SEGMENT_ATTRIBUTE` thrown for attributes absent from every assignment
|
||||
- `packages/testing/ab-testing/src/sequential/sequential-test.test.ts` — O'Brien-Fleming boundary monotone-decreasing in informationFraction; Pocock boundary constant; `runSequentialTest` stops at first `shouldStop`
|
||||
- `packages/testing/ab-testing/src/sequential/sequential-test.spec-conformance.test.ts` — alpha-spent monotone-increasing; futility check only fires when `informationFraction ≥ 0.75`; `INVALID_SEQUENTIAL_CONFIG` thrown for `alpha ∉ (0, 1)` or `maxLooks < 1`
|
||||
- `packages/testing/ab-testing/src/stats/bayesian.test.ts` — `betaPdf` / `betaCdf` / `betaQuantile` round-trip; `posteriorBeta` conjugate update; `probabilityBGreaterA` Monte Carlo converges to closed-form for known cases; `credibleInterval` width parameter
|
||||
- `packages/testing/ab-testing/src/stats/bayesian.spec-conformance.test.ts` — injected `Rng` reproducibility for `betaSample` / `thompsonSample` / `expectedLoss`
|
||||
- `packages/testing/ab-testing/src/store/memory-assignment-store.test.ts` — `set`/`get` round-trip; `getByExperiment` / `getByUser` filtering; `deleteByExperiment` cascade; `clear`
|
||||
- `packages/testing/ab-testing/src/store/memory-assignment-store.spec-conformance.test.ts` — `AssignmentStoreLike` contract pinned (will be re-used to verify future Redis/SurrealDB adapters)
|
||||
- `tests/ab-testing-public-surface.test.ts` (cross-package) — pins the additive subpath-export surface so removal is a deliberate spec change
|
||||
|
||||
## Migration impact
|
||||
|
||||
- **Existing `createExperimentManager` consumers** — no behavior change. The manager gains an optional `store?: AssignmentStoreLike` field on `ExperimentManagerOptions`; existing consumers that don't pass it keep the in-memory derivation path.
|
||||
- **`@suaveplan/ab-testing` direct dependents** — none in the genesis monorepo (`bunx turbo ls --filter=...@suaveplan/ab-testing` returns the package itself only).
|
||||
- **Subpath imports** — six new subpaths added (`./bandit`, `./multivariate`, `./segments`, `./sequential`, `./stats`, `./store`). The root barrel re-exports from each. No removed or renamed exports.
|
||||
- **`@suaveplan/types`** — the `AssignmentStoreLike` interface is promoted into `@suaveplan/types` per AGENTS.md §0 rule 5. This is additive to `@suaveplan/types` and does not affect existing consumers.
|
||||
- **Error-code consumers** — additive only; existing `AB_CODES` entries preserved.
|
||||
|
||||
## Parity Checklist
|
||||
|
||||
| Capability | Upstream | Migrated | Notes |
|
||||
|---|---|---|---|
|
||||
| Deterministic FNV-1a bucketing | YES | YES | Already on genesis; preserved |
|
||||
| Targeting rules | YES | YES | Already on genesis; preserved |
|
||||
| Variant assignment & providers | YES | YES | Already on genesis; preserved |
|
||||
| Frequentist results (`computeResults`) | YES | YES | Already on genesis; preserved |
|
||||
| Bandits (`EpsilonGreedy`, `UCB1`, `ThompsonSampling`) | YES | YES | This change |
|
||||
| Multivariate engine (full + fractional factorial, factor effects, interactions) | YES | YES | This change |
|
||||
| Segment analysis (per-segment lift, CI, interaction) | YES | YES | This change |
|
||||
| Sequential testing (O'Brien-Fleming alpha-spending, futility, early stopping) | YES | YES | This change |
|
||||
| Sequential testing (Pocock boundary alternative) | NO | YES | Genesis enhancement; documented in "no source line" table |
|
||||
| Bayesian stats (Beta-Binomial posteriors, credible intervals, expected loss) | YES | YES | This change |
|
||||
| Assignment store (`AssignmentStore` + `MemoryAssignmentStore`) | YES | YES | This change |
|
||||
| Persistent assignment-store adapters (Redis / SurrealDB / SQLite) | NO | NO | Scoped out — each adapter ships as its own package once `AssignmentStoreLike` is pinned |
|
||||
| Contextual bandits (LinUCB, Thompson with features) | NO | NO | Scoped out — not in upstream, no consumer asking |
|
||||
| Multi-objective / Pareto bandits | NO | NO | Scoped out — not in upstream |
|
||||
| Continuous-monitoring p-values (mSPRT, e-values) | NO | NO | Scoped out — distinct statistical regime; alpha-spending covers the documented use case |
|
||||
|
||||
## Affected dependents
|
||||
|
||||
`bunx turbo ls --filter=...@suaveplan/ab-testing` enumerates only the package itself (no in-repo dependents).
|
||||
|
||||
- `@suaveplan/ab-testing` (self) — [x] tests run / [x] docs reviewed / [ ] no action needed
|
||||
|
||||
`@suaveplan/types` is touched additively (new `AssignmentStoreLike` interface). Verify the dep graph after promotion:
|
||||
|
||||
- `@suaveplan/types` — [x] tests run / [x] docs reviewed / [ ] no action needed (purely additive types-only export; `byte-size` test guards no runtime code is shipped)
|
||||
|
||||
No external dependents on genesis. The proposal is additive (no removed or modified exports), so no downstream sweep is required.
|
||||
203
openspec/changes/ab-testing-completion/specs/ab-testing/spec.md
Normal file
203
openspec/changes/ab-testing-completion/specs/ab-testing/spec.md
Normal file
@@ -0,0 +1,203 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: ab-testing (ab-testing-completion)
|
||||
|
||||
**Package**: `@suaveplan/ab-testing`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. No exceptions. (See `openspec/AGENTS.md` Section 13.)
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Bandit algorithms with shared `Bandit<TArmId>` interface
|
||||
|
||||
The package SHALL provide three multi-armed bandit algorithms — `EpsilonGreedyBandit<TArmId>`, `UCB1Bandit<TArmId>`, `ThompsonSamplingBandit<TArmId>` — all implementing a shared `Bandit<TArmId>` interface with `selectArm(): TArmId`, `update(arm: TArmId, reward: number): void`, `getValues(): readonly number[]`, `getCounts(): readonly number[]`. The generic `TArmId` parameter SHALL default to `number` (array index) so existing-shape upstream code continues to work; callers MAY pass string variant IDs as arms. Every algorithm SHALL accept an explicit `Rng` from `@suaveplan/types::Rng` (no module-scope `Math.random`). Construction SHALL throw `AbTestingError(AB_CODES.INVALID_BANDIT_CONFIG, ...)` when `arms.length < 2` or, for `EpsilonGreedyBandit`, `epsilon ∉ [0, 1]`.
|
||||
|
||||
#### Scenario: epsilon-greedy explores when rng < epsilon
|
||||
|
||||
- **WHEN** `EpsilonGreedyBandit` is constructed with `arms: ['a', 'b', 'c']`, `epsilon: 0.5`, and an `Rng` that returns `0.1` then `0.7`
|
||||
- **THEN** `selectArm()` returns the result of `rng() * 3` floored as a uniform random arm (exploration branch)
|
||||
|
||||
#### Scenario: UCB1 plays each arm once before applying score
|
||||
|
||||
- **WHEN** `UCB1Bandit` is constructed with three arms and `selectArm()` is called three times with no intervening `update`
|
||||
- **THEN** the first three returns are `arms[0]`, `arms[1]`, `arms[2]` in order
|
||||
|
||||
#### Scenario: Thompson sampling updates posterior on success
|
||||
|
||||
- **WHEN** `ThompsonSamplingBandit.update(arm, 1)` is called
|
||||
- **THEN** the underlying `successes[arm]` increments by 1; `update(arm, 0)` increments `failures[arm]` by 1
|
||||
|
||||
#### Scenario: invalid bandit config throws AbTestingError
|
||||
|
||||
- **WHEN** `new EpsilonGreedyBandit({ arms: ['x'], epsilon: 0.1, rng })` is called (only one arm)
|
||||
- **THEN** the constructor throws `AbTestingError` with code `AB_CODES.INVALID_BANDIT_CONFIG`
|
||||
|
||||
**Verified by:** `src/bandit/bandit.test.ts::"EpsilonGreedyBandit explores when rng draw is below epsilon"` AND `src/bandit/bandit.test.ts::"UCB1Bandit plays each arm once before scoring"` AND `src/bandit/bandit.test.ts::"ThompsonSamplingBandit updates posterior on success and failure"` AND `src/bandit/bandit.spec-conformance.test.ts::"throws INVALID_BANDIT_CONFIG when arms.length < 2"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Multivariate engine with full + fractional factorial design
|
||||
|
||||
The package SHALL provide a multivariate-experiment design engine. `fullFactorial(factors)` SHALL return every combination of factor levels (Cartesian product). `fractionalFactorial(factors, fraction)` SHALL return every Nth combination where `N = round(1 / fraction)`. `analyzeMultivariate(factors, data)` SHALL return a `MultivariateResult` containing per-treatment conversion rates, per-factor `FactorEffect` objects with `LevelEffect[]` and a sum-of-squares `importance` score, pairwise `InteractionEffect[]` covering every `(i, j)` pair with `i < j`, and a `bestCombination` (highest-rate treatment with `participants > 0`). Construction with `factors.length === 0` or any factor having `levels.length === 0` SHALL throw `AbTestingError(AB_CODES.INVALID_FACTOR_DESIGN, ...)`.
|
||||
|
||||
#### Scenario: full factorial enumerates Π levels
|
||||
|
||||
- **WHEN** `fullFactorial([{ name: 'a', levels: ['x', 'y'] }, { name: 'b', levels: ['1', '2', '3'] }])` is called
|
||||
- **THEN** the result has length `2 * 3 = 6` and each combination has unique `id`
|
||||
|
||||
#### Scenario: fractional factorial samples every Nth combination
|
||||
|
||||
- **WHEN** `fractionalFactorial(factors, 0.5)` is called against a 6-combination factor set
|
||||
- **THEN** the result length is 3 and the kept combinations are at indices 0, 2, 4 in `fullFactorial(factors)`
|
||||
|
||||
#### Scenario: factor importance is a non-negative sum-of-squares score
|
||||
|
||||
- **WHEN** `analyzeMultivariate` is called with treatments where one factor has level rates `[0.1, 0.5, 0.9]` and another has `[0.5, 0.5, 0.5]`
|
||||
- **THEN** the first factor's `importance` strictly exceeds the second factor's `importance`
|
||||
|
||||
#### Scenario: bestCombination is the highest-rate treatment with participants
|
||||
|
||||
- **WHEN** treatments include one combination with `conversionRate: 0.9` but `participants: 0` and another with `0.5` and `participants: 100`
|
||||
- **THEN** `bestCombination.combinationId` is the latter
|
||||
|
||||
#### Scenario: empty factor set throws
|
||||
|
||||
- **WHEN** `analyzeMultivariate([], data)` is called
|
||||
- **THEN** `AbTestingError` with code `AB_CODES.INVALID_FACTOR_DESIGN` is thrown
|
||||
|
||||
**Verified by:** `src/multivariate/multivariate-engine.test.ts::"fullFactorial enumerates the Cartesian product of levels"` AND `src/multivariate/multivariate-engine.test.ts::"fractionalFactorial keeps every Nth combination"` AND `src/multivariate/multivariate-engine.spec-conformance.test.ts::"factor importance ranks higher-variance factors first"` AND `src/multivariate/multivariate-engine.test.ts::"bestCombination ignores treatments with zero participants"` AND `src/multivariate/multivariate-engine.spec-conformance.test.ts::"throws INVALID_FACTOR_DESIGN for empty factor set"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Segment analysis with per-segment lift and interaction terms
|
||||
|
||||
The package SHALL provide segment-level analyses. `analyzeBySegments(assignments, conversions, variantIds, segmentAttribute)` SHALL group assignments by `attributes[segmentAttribute]` value, compute per-segment `VariantResult[]`, attach a two-proportion z-test significance flag (z > 1.96 with both arms n ≥ 10), and report `conversionRateDiff` between best and worst variant. `segmentConfidenceIntervals(...)` SHALL return Wald intervals on conversion rates, `[0, 1]`-clamped. `detectSegmentInteraction(...)` SHALL return interaction-strength scores per segment value relative to global per-variant rates, flagged as `hasInteraction: true` when strength exceeds 0.01. When no assignment exposes the requested attribute, the call SHALL throw `AbTestingError(AB_CODES.INVALID_SEGMENT_ATTRIBUTE, ...)`.
|
||||
|
||||
#### Scenario: segments grouped by attribute value
|
||||
|
||||
- **WHEN** assignments include 50 with `attributes.country === 'US'` and 50 with `'CA'`, and `analyzeBySegments(..., 'country')` is called
|
||||
- **THEN** the result has two `SegmentResult` entries with `segmentValue: 'US'` and `'CA'` respectively
|
||||
|
||||
#### Scenario: significance gates on min sample size
|
||||
|
||||
- **WHEN** a segment has variant A with 5 assignments and variant B with 5 assignments
|
||||
- **THEN** `isSignificant` is `false` regardless of conversion rates
|
||||
|
||||
#### Scenario: confidence intervals are clamped to [0, 1]
|
||||
|
||||
- **WHEN** a tiny sample produces a Wald interval with lower bound `-0.05` or upper bound `1.10`
|
||||
- **THEN** the returned `lowerBound` is `0` and `upperBound` is `1`
|
||||
|
||||
#### Scenario: interaction flagged above 0.01 strength
|
||||
|
||||
- **WHEN** a segment value has variant rates that differ from global rates such that sum-of-squared-weighted-diff > 0.01
|
||||
- **THEN** `hasInteraction` is `true`
|
||||
|
||||
#### Scenario: missing attribute throws
|
||||
|
||||
- **WHEN** `analyzeBySegments(..., 'nonexistent')` is called and no assignment carries that attribute
|
||||
- **THEN** `AbTestingError` with code `AB_CODES.INVALID_SEGMENT_ATTRIBUTE` is thrown
|
||||
|
||||
**Verified by:** `src/segments/segment-analyzer.test.ts::"groups assignments by segment attribute value"` AND `src/segments/segment-analyzer.test.ts::"isSignificant is false when sample size is below minimum"` AND `src/segments/segment-analyzer.test.ts::"clamps confidence intervals to [0, 1]"` AND `src/segments/segment-analyzer.spec-conformance.test.ts::"flags interaction when strength exceeds threshold"` AND `src/segments/segment-analyzer.spec-conformance.test.ts::"throws INVALID_SEGMENT_ATTRIBUTE when attribute is absent"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Sequential testing with alpha-spending and early-stop rules
|
||||
|
||||
The package SHALL provide group-sequential test machinery for safe early stopping. `obrienFlemingBoundary(informationFraction, alpha)` SHALL return the z-boundary `z(1 - alpha/2) / sqrt(informationFraction)`. `pocockBoundary(informationFraction, alpha, maxLooks)` SHALL return a constant z across looks (Pocock spending). `alphaSpent(informationFraction, alpha, boundaryType?)` SHALL return cumulative alpha spent under the chosen spending function and SHALL be monotone-non-decreasing in `informationFraction`. `interimAnalysis(...)` SHALL compute a single look and report `shouldStop: true` with `stopReason: 'significance'` when `testStatistic >= spendingBoundary`, or `'futility'` when `informationFraction >= 0.75` and the projected final z falls below 30% of the final boundary. `runSequentialTest(...)` SHALL drive the sequence and stop on first `shouldStop`. Construction validates `alpha ∈ (0, 1)`, `maxLooks ≥ 1`, `mde > 0`, `power ∈ (0, 1)` else throws `AbTestingError(AB_CODES.INVALID_SEQUENTIAL_CONFIG, ...)`.
|
||||
|
||||
#### Scenario: O'Brien-Fleming boundary is monotone-decreasing
|
||||
|
||||
- **WHEN** `obrienFlemingBoundary(0.25, 0.05)` and `obrienFlemingBoundary(1, 0.05)` are computed
|
||||
- **THEN** the first value is strictly greater than the second
|
||||
|
||||
#### Scenario: Pocock boundary is constant across looks
|
||||
|
||||
- **WHEN** `pocockBoundary(0.5, 0.05, 4)` and `pocockBoundary(1, 0.05, 4)` are computed
|
||||
- **THEN** both values are equal within numerical tolerance
|
||||
|
||||
#### Scenario: alpha-spent is monotone-non-decreasing
|
||||
|
||||
- **WHEN** `alphaSpent` is called at fractions `[0.1, 0.25, 0.5, 0.75, 1.0]`
|
||||
- **THEN** the returned sequence is non-decreasing and ends at exactly `alpha`
|
||||
|
||||
#### Scenario: futility check fires only after 75% information
|
||||
|
||||
- **WHEN** `interimAnalysis` is called at `informationFraction = 0.5` with a tiny test statistic
|
||||
- **THEN** `stopReason` is `null` (futility branch suppressed)
|
||||
|
||||
#### Scenario: runSequentialTest stops on first shouldStop
|
||||
|
||||
- **WHEN** the data array has four looks and the second look returns `shouldStop: true`
|
||||
- **THEN** `runSequentialTest` returns exactly two `InterimResult` entries
|
||||
|
||||
**Verified by:** `src/sequential/sequential-test.test.ts::"obrienFlemingBoundary is monotone-decreasing in informationFraction"` AND `src/sequential/sequential-test.test.ts::"pocockBoundary is constant across looks"` AND `src/sequential/sequential-test.spec-conformance.test.ts::"alphaSpent is monotone-non-decreasing and ends at alpha"` AND `src/sequential/sequential-test.spec-conformance.test.ts::"futility branch is suppressed below 75% informationFraction"` AND `src/sequential/sequential-test.test.ts::"runSequentialTest stops on first shouldStop"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Bayesian stats — Beta-Binomial conversion model
|
||||
|
||||
The package SHALL provide Beta-Binomial Bayesian primitives. `posteriorBeta(conversions, participants, priorAlpha?, priorBeta?)` SHALL return the conjugate posterior `{ alpha: priorAlpha + conversions, beta: priorBeta + (participants - conversions) }`. `betaPdf` / `betaCdf` / `betaQuantile` SHALL be self-consistent (`betaCdf(betaQuantile(p, a, b), a, b) ≈ p` within bisection tolerance). `betaSample(alpha, beta, rng)` SHALL draw a Beta sample via two Gamma draws (Marsaglia-Tsang). `probabilityBGreaterA(a, b, simulations?, rng)` SHALL return a Monte Carlo estimate of `P(B > A)`. `credibleInterval(params, width?)` SHALL return a quantile-based interval at the requested width (default 95%). `expectedLoss(chosen, other, simulations?, rng)` SHALL return the expected forfeited lift from picking `chosen`. `bayesianAnalysis(results, priorAlpha?, priorBeta?, rng)` SHALL run the full pairwise comparison on two `VariantResult` objects. All RNG-dependent functions SHALL accept an injected `Rng` (no module-scope `Math.random`).
|
||||
|
||||
#### Scenario: posteriorBeta is the conjugate update
|
||||
|
||||
- **WHEN** `posteriorBeta(10, 100, 1, 1)` is called
|
||||
- **THEN** the result is `{ alpha: 11, beta: 91 }`
|
||||
|
||||
#### Scenario: betaCdf and betaQuantile are inverses
|
||||
|
||||
- **WHEN** `betaQuantile(0.5, 5, 5)` then `betaCdf(<that>, 5, 5)` is computed
|
||||
- **THEN** the round-trip value is within `1e-3` of `0.5`
|
||||
|
||||
#### Scenario: probabilityBGreaterA converges to 0.5 for symmetric posteriors
|
||||
|
||||
- **WHEN** `probabilityBGreaterA({ alpha: 5, beta: 5 }, { alpha: 5, beta: 5 }, 20000, rng)` is called
|
||||
- **THEN** the result is within `0.02` of `0.5`
|
||||
|
||||
#### Scenario: credibleInterval respects width parameter
|
||||
|
||||
- **WHEN** `credibleInterval(params, 0.5)` and `credibleInterval(params, 0.95)` are computed for the same `params`
|
||||
- **THEN** the 95% interval contains the 50% interval
|
||||
|
||||
#### Scenario: same Rng seed produces deterministic samples
|
||||
|
||||
- **WHEN** `betaSample(2, 3, rng)` is called twice with the same `Rng` state
|
||||
- **THEN** both calls return the same numeric value (reproducibility pin)
|
||||
|
||||
**Verified by:** `src/stats/bayesian.test.ts::"posteriorBeta returns the conjugate update"` AND `src/stats/bayesian.test.ts::"betaCdf and betaQuantile are mutual inverses"` AND `src/stats/bayesian.test.ts::"probabilityBGreaterA converges to 0.5 for symmetric posteriors"` AND `src/stats/bayesian.test.ts::"credibleInterval width parameter widens the interval"` AND `src/stats/bayesian.spec-conformance.test.ts::"injected Rng produces reproducible betaSample output"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Assignment store — interface and in-memory adapter
|
||||
|
||||
The package SHALL persist variant assignments via an `AssignmentStoreLike` interface (defined in `@suaveplan/types`). The interface SHALL expose five async methods: `get(experimentId, userId): Promise<Assignment | undefined>`, `set(assignment): Promise<void>`, `getByExperiment(experimentId): Promise<readonly Assignment[]>`, `getByUser(userId): Promise<readonly Assignment[]>`, `deleteByExperiment(experimentId): Promise<void>`. The package SHALL ship a `MemoryAssignmentStore` class implementing this interface and a `createMemoryAssignmentStore()` factory. `ExperimentManagerOptions` SHALL accept an optional `store?: AssignmentStoreLike`; when provided, `assignVariant` SHALL consult the store before re-bucketing and write through on miss. `MemoryAssignmentStore.set` SHALL be idempotent on identical-key assignments (last-writer-wins).
|
||||
|
||||
#### Scenario: set/get round-trip
|
||||
|
||||
- **WHEN** `store.set({ experimentId: 'e1', userId: 'u1', variantId: 'v1', assignedAt: '2026-05-02T00:00:00Z' })` is called and then `store.get('e1', 'u1')` is awaited
|
||||
- **THEN** the resolved value is the same `Assignment` object shape
|
||||
|
||||
#### Scenario: getByExperiment filters to one experiment
|
||||
|
||||
- **WHEN** the store contains assignments for `e1` and `e2` and `getByExperiment('e1')` is awaited
|
||||
- **THEN** the result contains only `e1` assignments
|
||||
|
||||
#### Scenario: deleteByExperiment cascades
|
||||
|
||||
- **WHEN** the store contains 100 assignments for `e1` and `deleteByExperiment('e1')` is awaited
|
||||
- **THEN** subsequent `getByExperiment('e1')` returns an empty array
|
||||
|
||||
#### Scenario: ExperimentManager with store does not re-bucket on hit
|
||||
|
||||
- **WHEN** `createExperimentManager({ store })` is called, `assignVariant(exp, ctx)` runs once and persists, then `assignVariant(exp, ctx)` runs again with the same `ctx`
|
||||
- **THEN** the second call returns the persisted variant without invoking `bucketUser` (verified by an `Rng` spy that increments only on miss)
|
||||
|
||||
#### Scenario: missing key resolves undefined
|
||||
|
||||
- **WHEN** `store.get('does-not-exist', 'u1')` is awaited
|
||||
- **THEN** the resolved value is `undefined` (no error thrown)
|
||||
|
||||
**Verified by:** `src/store/memory-assignment-store.test.ts::"set and get round-trip an Assignment"` AND `src/store/memory-assignment-store.test.ts::"getByExperiment returns only matching experiment ids"` AND `src/store/memory-assignment-store.test.ts::"deleteByExperiment removes all assignments for that experiment"` AND `src/core/experiment-manager/experiment-manager.store.test.ts::"second assignVariant call with store option returns persisted variant without re-bucketing"` AND `src/store/memory-assignment-store.spec-conformance.test.ts::"get resolves undefined for missing keys"`
|
||||
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
175
openspec/changes/ab-testing-completion/tasks.md
Normal file
175
openspec/changes/ab-testing-completion/tasks.md
Normal file
@@ -0,0 +1,175 @@
|
||||
# Tasks — A/B testing completion
|
||||
|
||||
> Every checkbox MUST reflect work actually done in this change. Do not retroactively tick boxes for work that was already in main when this change opened. Do not tick a box for a test that doesn't exist or doesn't pin the requirement it claims to verify. Retroactive ticking is the failure mode this template guards against.
|
||||
|
||||
## 0. Pre-flight (non-negotiable)
|
||||
|
||||
- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
|
||||
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
|
||||
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
|
||||
|
||||
## 1. Spec authoring
|
||||
|
||||
- [ ] 1.1 Read `../suaveplan/packages/universal/ab-testing/src/` end-to-end (bandit/, multivariate/, segmentation/, sequential/, stats/, store/, types/, engine/)
|
||||
- [ ] 1.2 Read existing `openspec/specs/ab-testing/spec.md` (none exists yet — will be created at archive)
|
||||
- [ ] 1.3 Read `openspec/AGENTS.md` Sections 13, 14, and §0a (migration parity)
|
||||
- [ ] 1.4 Draft `proposal.md` from `openspec/templates/proposal.md`
|
||||
- [ ] 1.5 Fill in `## Source mapping` table — every requirement cites a line range OR appears in the "no source line" table with justification
|
||||
- [ ] 1.6 Draft `specs/ab-testing/spec.md` from `openspec/templates/spec-delta.md`
|
||||
- [ ] 1.7 Every requirement in the spec delta carries a `**Verified by:**` line naming concrete `it(...)` titles
|
||||
- [ ] 1.8 `## Parity Checklist` table fully populated; every NO has architectural justification ≤1 sentence
|
||||
|
||||
## 2. Implementation
|
||||
|
||||
### 2a. Promote `AssignmentStoreLike` to `@suaveplan/types`
|
||||
|
||||
- [ ] 2.1 Add `AssignmentStoreLike` interface to `packages/types/types/src/storage/assignment-store.ts` with five async methods (`get`, `set`, `getByExperiment`, `getByUser`, `deleteByExperiment`)
|
||||
- [ ] 2.2 Re-export `AssignmentStoreLike` from `@suaveplan/types` root barrel
|
||||
- [ ] 2.3 Add `Assignment` interface alongside (or import-only re-export from `@suaveplan/ab-testing` once defined there) — pin the cross-package contract
|
||||
|
||||
### 2b. Bandit subsystem
|
||||
|
||||
- [ ] 2.4 Add `src/bandit/bandit.types.ts` with `Bandit<TArmId>` interface and `BanditConfig<TArmId>`
|
||||
- [ ] 2.5 Add `src/bandit/epsilon-greedy.ts` — `EpsilonGreedyBandit<TArmId>` class; constructor accepts `{ arms, epsilon, rng }`; default `epsilon = 0.1`; require explicit `Rng` (no `Math.random` fallback at construction)
|
||||
- [ ] 2.6 Add `src/bandit/ucb1.ts` — `UCB1Bandit<TArmId>` class
|
||||
- [ ] 2.7 Add `src/bandit/thompson.ts` — `ThompsonSamplingBandit<TArmId>` class; depends on `betaSample` from `./stats/bayesian`
|
||||
- [ ] 2.8 Add `src/bandit/index.ts` barrel
|
||||
- [ ] 2.9 Throw `AbTestingError(AB_CODES.INVALID_BANDIT_CONFIG, ...)` when `arms.length < 2` or `epsilon ∉ [0, 1]`
|
||||
|
||||
### 2c. Multivariate engine
|
||||
|
||||
- [ ] 2.10 Add `src/multivariate/multivariate.types.ts` with `Factor`, `TreatmentCombination`, `TreatmentResult`, `FactorEffect`, `LevelEffect`, `InteractionEffect`, `MultivariateResult`
|
||||
- [ ] 2.11 Add `src/multivariate/factorial.ts` — `fullFactorial`, `fractionalFactorial`
|
||||
- [ ] 2.12 Add `src/multivariate/multivariate-engine.ts` — `analyzeMultivariate` + factor-effect / interaction-effect helpers; split helpers into `src/multivariate/factor-effects.ts` and `src/multivariate/interactions.ts` if file > 250 L
|
||||
- [ ] 2.13 Add `src/multivariate/index.ts` barrel
|
||||
- [ ] 2.14 Throw `AbTestingError(AB_CODES.INVALID_FACTOR_DESIGN, ...)` when `factors.length === 0` or any factor has `levels.length === 0`
|
||||
|
||||
### 2d. Segment analysis
|
||||
|
||||
- [ ] 2.15 Add `src/segments/segments.types.ts` with `SegmentResult`, `SegmentConfidenceInterval`, `SegmentInteraction`
|
||||
- [ ] 2.16 Add `src/segments/segment-analyzer.ts` — `analyzeBySegments`, `segmentConfidenceIntervals`, `detectSegmentInteraction`
|
||||
- [ ] 2.17 Move `normalQuantile` helper to `src/stats/normal.ts` (shared by segments and sequential)
|
||||
- [ ] 2.18 Add `src/segments/index.ts` barrel
|
||||
- [ ] 2.19 Throw `AbTestingError(AB_CODES.INVALID_SEGMENT_ATTRIBUTE, ...)` when `assignments.every(a => a.attributes?.[segmentAttribute] === undefined)`
|
||||
|
||||
### 2e. Sequential testing
|
||||
|
||||
- [ ] 2.20 Add `src/sequential/sequential.types.ts` with `SequentialTestConfig` (extended with `boundaryType`), `InterimResult`
|
||||
- [ ] 2.21 Add `src/sequential/boundaries.ts` — `obrienFlemingBoundary`, `pocockBoundary`, dispatch on `boundaryType`
|
||||
- [ ] 2.22 Add `src/sequential/sequential-test.ts` — `alphaSpent`, `interimAnalysis`, `runSequentialTest`, `checkFutility`
|
||||
- [ ] 2.23 Add `src/sequential/index.ts` barrel
|
||||
- [ ] 2.24 Throw `AbTestingError(AB_CODES.INVALID_SEQUENTIAL_CONFIG, ...)` when `alpha ∉ (0, 1)`, `maxLooks < 1`, `mde ≤ 0`, or `power ∉ (0, 1)`
|
||||
|
||||
### 2f. Bayesian stats
|
||||
|
||||
- [ ] 2.25 Add `src/stats/beta.ts` — `BetaParams`, `betaPdf`, `betaCdf`, `betaQuantile`, `betaSample`, `posteriorBeta`, internal `logBeta` / `logGamma` / `continuedFraction` / `gammaSample` / `normalSample`
|
||||
- [ ] 2.26 Add `src/stats/bayesian.ts` — `BayesianResult`, `thompsonSample`, `probabilityBGreaterA`, `credibleInterval`, `expectedLoss`, `bayesianAnalysis`
|
||||
- [ ] 2.27 Add `src/stats/normal.ts` — `normalCdf`, `normalQuantile` (Beasley-Springer-Moro) shared with sequential and segments
|
||||
- [ ] 2.28 Add `src/stats/index.ts` barrel
|
||||
- [ ] 2.29 Replace every `rng: () => number = Math.random` default with required `rng: Rng` param; expose `createDefaultRng()` from `@suaveplan/ab-testing/stats` for callers wanting a `crypto.getRandomValues`-backed default
|
||||
|
||||
### 2g. Assignment store
|
||||
|
||||
- [ ] 2.30 Add `src/store/memory.ts` — `MemoryAssignmentStore` class implementing `AssignmentStoreLike` (imported from `@suaveplan/types`)
|
||||
- [ ] 2.31 Add `createMemoryAssignmentStore()` factory
|
||||
- [ ] 2.32 Add `src/store/index.ts` barrel
|
||||
- [ ] 2.33 Wire `store?: AssignmentStoreLike` field through `ExperimentManagerOptions` in `src/types/common.ts`; update `createExperimentManager` to consult and persist via the store when provided
|
||||
|
||||
### 2h. Errors and exports
|
||||
|
||||
- [ ] 2.34 Add `INVALID_BANDIT_CONFIG`, `INVALID_FACTOR_DESIGN`, `INVALID_SEGMENT_ATTRIBUTE`, `INSUFFICIENT_SAMPLE_SIZE`, `INVALID_SEQUENTIAL_CONFIG` to `AB_CODES` in `src/error-codes.ts`
|
||||
- [ ] 2.35 Update `src/index.ts` root barrel to re-export from `./bandit`, `./multivariate`, `./segments`, `./sequential`, `./stats`, `./store`
|
||||
- [ ] 2.36 Add subpath entries to `package.json` `exports` map for the six new subpaths (matches existing `./bucketing` / `./targeting` / `./results` / `./providers` / `./errors` shape)
|
||||
- [ ] 2.37 Update `package.json` `description` field if needed (currently mentions only deterministic bucketing + targeting + variants + conversions; extend to mention bandits / multivariate / sequential / Bayesian)
|
||||
|
||||
## 3. Tests
|
||||
|
||||
### 3a. Bandit subsystem
|
||||
|
||||
- [ ] 3.1 `src/bandit/bandit.test.ts` — `EpsilonGreedyBandit` exploration (when rng < epsilon) vs. exploitation (else); `UCB1Bandit` plays each arm once before applying score; `ThompsonSamplingBandit` posterior update on success vs. failure
|
||||
- [ ] 3.2 `src/bandit/bandit.spec-conformance.test.ts` — injected `Rng` is the only randomness source; `INVALID_BANDIT_CONFIG` thrown for invalid configs
|
||||
|
||||
### 3b. Multivariate engine
|
||||
|
||||
- [ ] 3.3 `src/multivariate/multivariate-engine.test.ts` — `fullFactorial` count = `Π levels`; `fractionalFactorial(fraction = 0.5)` returns half; factor importance sum-of-squares ranking
|
||||
- [ ] 3.4 `src/multivariate/multivariate-engine.spec-conformance.test.ts` — interaction strength is non-negative; pairwise interactions cover every `(i, j)` with `i < j`; `INVALID_FACTOR_DESIGN` thrown for empty factors
|
||||
|
||||
### 3c. Segment analysis
|
||||
|
||||
- [ ] 3.5 `src/segments/segment-analyzer.test.ts` — segment grouping by attribute; significance with min sample size 10; CI is `[0, 1]`-clamped
|
||||
- [ ] 3.6 `src/segments/segment-analyzer.spec-conformance.test.ts` — interaction-strength threshold; `INVALID_SEGMENT_ATTRIBUTE` thrown when attribute is absent
|
||||
|
||||
### 3d. Sequential testing
|
||||
|
||||
- [ ] 3.7 `src/sequential/sequential-test.test.ts` — O'Brien-Fleming boundary monotone-decreasing in `informationFraction`; Pocock boundary constant across looks; `runSequentialTest` stops at first `shouldStop`
|
||||
- [ ] 3.8 `src/sequential/sequential-test.spec-conformance.test.ts` — alpha-spent monotone-increasing; futility only fires when `informationFraction ≥ 0.75`; `INVALID_SEQUENTIAL_CONFIG` thrown for invalid configs
|
||||
|
||||
### 3e. Bayesian stats
|
||||
|
||||
- [ ] 3.9 `src/stats/bayesian.test.ts` — `betaPdf` / `betaCdf` / `betaQuantile` round-trip; `posteriorBeta` conjugate update (priors + data); `probabilityBGreaterA` converges to closed-form `0.5` for symmetric posteriors
|
||||
- [ ] 3.10 `src/stats/bayesian.spec-conformance.test.ts` — same `Rng` seed produces identical `betaSample` / `thompsonSample` / `expectedLoss` outputs across runs (reproducibility pin)
|
||||
|
||||
### 3f. Assignment store
|
||||
|
||||
- [ ] 3.11 `src/store/memory-assignment-store.test.ts` — `set`/`get` round-trip; `getByExperiment` / `getByUser` filtering; `deleteByExperiment` cascade; `size` and `clear`
|
||||
- [ ] 3.12 `src/store/memory-assignment-store.spec-conformance.test.ts` — `AssignmentStoreLike` contract: every method returns a `Promise`; `get` resolves `undefined` for missing keys; `set` is idempotent on identical assignments
|
||||
|
||||
### 3g. ExperimentManager integration with store
|
||||
|
||||
- [ ] 3.13 `src/core/experiment-manager/experiment-manager.store.test.ts` — when `options.store` is provided, `assignVariant` checks store first and writes through on miss; second call for same `(experimentId, userId)` returns the persisted variant without re-bucketing
|
||||
|
||||
### 3h. Cross-package
|
||||
|
||||
- [ ] 3.14 `tests/ab-testing-public-surface.test.ts` — assert root barrel re-exports include `EpsilonGreedyBandit`, `UCB1Bandit`, `ThompsonSamplingBandit`, `fullFactorial`, `fractionalFactorial`, `analyzeMultivariate`, `analyzeBySegments`, `segmentConfidenceIntervals`, `detectSegmentInteraction`, `obrienFlemingBoundary`, `pocockBoundary`, `interimAnalysis`, `runSequentialTest`, `posteriorBeta`, `bayesianAnalysis`, `MemoryAssignmentStore`, `createMemoryAssignmentStore`
|
||||
- [ ] 3.15 Every `it(...)` carries `{ timeout: 5000 }` (Bayesian Monte Carlo tests use 30000)
|
||||
|
||||
## 4. Documentation
|
||||
|
||||
- [ ] 4.1 Sibling `.md` for every new implementation `.ts` (≥200 words, basic + advanced examples, full API reference): one per file under `bandit/`, `multivariate/`, `segments/`, `sequential/`, `stats/`, `store/`
|
||||
- [ ] 4.2 README links every new co-located `.md` under `## Modules`
|
||||
- [ ] 4.3 Update root `README.md` quick-start with three new sections: bandit selection, multivariate analysis, sequential testing
|
||||
- [ ] 4.4 Update `CLAUDE.md` `@suaveplan/ab-testing` package listing to include bandits, multivariate, segment analysis, sequential testing, Bayesian stats, assignment store
|
||||
- [ ] 4.5 Update `@suaveplan/types` listing in `CLAUDE.md` to mention `AssignmentStoreLike` if the listing is detailed enough
|
||||
|
||||
## 5. Pre-archive verification (the 5-gate triple)
|
||||
|
||||
- [ ] 5.1 `bun run check` — Biome 0 errors / 0 warnings / 0 info across the whole monorepo
|
||||
- [ ] 5.2 `bunx turbo lint typecheck test --filter=@suaveplan/ab-testing` — green
|
||||
- [ ] 5.3 `bunx turbo test:coverage --filter=@suaveplan/ab-testing` — 100/100/100/100
|
||||
- [ ] 5.4 `bun scripts/openspec-conformance.ts @suaveplan/ab-testing` — exit 0, no `error` findings
|
||||
- [ ] 5.5 `bunx turbo build --filter=@suaveplan/ab-testing` — green
|
||||
- [ ] 5.6 `bunx turbo lint typecheck test --filter=@suaveplan/types` — green (foundation-package side-effect from `AssignmentStoreLike` promotion)
|
||||
|
||||
## 6. Pre-archive 7-row checklist (AGENTS.md Section 13)
|
||||
|
||||
- [ ] 6.1 Generics threaded — `Bandit<TArmId>`, `EpsilonGreedyBandit<TArmId>`, `UCB1Bandit<TArmId>`, `ThompsonSamplingBandit<TArmId>` infer arm-id type from constructor `arms` array; `expectTypeOf` assertion that `bandit.selectArm()` returns `TArmId`, not `number`
|
||||
- [ ] 6.2 Zod schemas wired at every declared boundary — `BanditConfigSchema`, `FactorSchema`, `SequentialTestConfigSchema` validate at construction
|
||||
- [ ] 6.3 Every error code in `error-codes.ts` thrown by at least one production path (the five new codes each tested in spec-conformance suites)
|
||||
- [ ] 6.4 Strict-mode behaviors tested both on and off — `Bandit` with `strict: true` throws on unknown arm IDs; `Bandit` with `strict: false` ignores
|
||||
- [ ] 6.5 OTel instruments emitted from production paths and asserted with a real in-memory `MeterProvider` — `ab.bandit.arm.selected`, `ab.multivariate.combinations.evaluated`, `ab.sequential.alpha.spent`, `ab.bayesian.simulations.duration_ms`, `ab.assignment.store.lookup`
|
||||
- [ ] 6.6 Lifecycle dedup tested where claimed — `MemoryAssignmentStore.set` is idempotent; `ExperimentManager` with `store` option does not re-bucket after a hit
|
||||
- [ ] 6.7 No retroactive tick — every box above reflects work done in this change
|
||||
|
||||
## 7. Archive
|
||||
|
||||
- [ ] 7.1 Fold spec delta into `openspec/specs/ab-testing/spec.md` (file does not exist yet — this change creates it)
|
||||
- [ ] 7.2 Move `openspec/changes/ab-testing-completion/` → `openspec/archive/<yyyy-mm-dd>-ab-testing-completion/`
|
||||
- [ ] 7.3 Refresh `MANIFEST.md` if package set or deps changed (no new deps; `@suaveplan/types` gains `AssignmentStoreLike` interface)
|
||||
|
||||
## Audit Note (2026-05-02)
|
||||
|
||||
Audit verdict: **DEFERRED — 0/81 tasks evidenced in code**.
|
||||
|
||||
Evidence reviewed:
|
||||
|
||||
- `packages/testing/ab-testing/src/` contains `bucketing/`, `targeting/`, `providers/`, `results/`, `core/experiment-manager/`, `error-codes.ts`, `types/`, `constants/`. No `bandit/`, no `multivariate/`, no `segments/`, no `sequential/`, no `stats/`, no `store/`.
|
||||
- `error-codes.ts` does not declare the five new codes (`INVALID_BANDIT_CONFIG`, `INVALID_FACTOR_DESIGN`, `INVALID_SEGMENT_ATTRIBUTE`, `INSUFFICIENT_SAMPLE_SIZE`, `INVALID_SEQUENTIAL_CONFIG`).
|
||||
- `@suaveplan/types` has no `AssignmentStoreLike` interface yet (the §2a promotion is not done).
|
||||
|
||||
Missing pieces:
|
||||
|
||||
1. Bandit subsystem (`epsilon-greedy`, `ucb1`, `thompson` with explicit `Rng` injection) and Bayesian stats module (`stats/{beta,bayesian,normal}.ts`).
|
||||
2. Multivariate engine (`fullFactorial` / `fractionalFactorial` / `analyzeMultivariate` + factor-effect / interaction helpers).
|
||||
3. Segment analyzer + sequential testing (O'Brien-Fleming + Pocock boundaries) + `MemoryAssignmentStore` plus `AssignmentStoreLike` promotion to `@suaveplan/types` and wiring through `ExperimentManagerOptions`.
|
||||
|
||||
Decision: leave change open. Six new feature blocks across statistics + storage + cross-package type-promotion is a multi-day implementation, not an audit pass.
|
||||
Reference in New Issue
Block a user