SuavePlan/suaveplan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
suaveplan/openspec/changes/ab-testing-completion/specs/ab-testing/spec.md
admin fbc70f96e6 Initial Commit
2026-05-27 23:29:53 +01:00

16 KiB
Raw Permalink Blame History

Capability Spec Delta: ab-testing (ab-testing-completion)

Package: @suaveplan/ab-testing Status: PROPOSED

Every requirement below carries a **Verified by:** line at authoring time. No exceptions. (See openspec/AGENTS.md Section 13.)

ADDED Requirements

Requirement: Bandit algorithms with shared Bandit<TArmId> interface

The package SHALL provide three multi-armed bandit algorithms — EpsilonGreedyBandit<TArmId>, UCB1Bandit<TArmId>, ThompsonSamplingBandit<TArmId> — all implementing a shared Bandit<TArmId> interface with selectArm(): TArmId, update(arm: TArmId, reward: number): void, getValues(): readonly number[], getCounts(): readonly number[]. The generic TArmId parameter SHALL default to number (array index) so existing-shape upstream code continues to work; callers MAY pass string variant IDs as arms. Every algorithm SHALL accept an explicit Rng from @suaveplan/types::Rng (no module-scope Math.random). Construction SHALL throw AbTestingError(AB_CODES.INVALID_BANDIT_CONFIG, ...) when arms.length < 2 or, for EpsilonGreedyBandit, epsilon ∉ [0, 1].

Scenario: epsilon-greedy explores when rng < epsilon

  • WHEN EpsilonGreedyBandit is constructed with arms: ['a', 'b', 'c'], epsilon: 0.5, and an Rng that returns 0.1 then 0.7
  • THEN selectArm() returns the result of rng() * 3 floored as a uniform random arm (exploration branch)

Scenario: UCB1 plays each arm once before applying score

  • WHEN UCB1Bandit is constructed with three arms and selectArm() is called three times with no intervening update
  • THEN the first three returns are arms[0], arms[1], arms[2] in order

Scenario: Thompson sampling updates posterior on success

  • WHEN ThompsonSamplingBandit.update(arm, 1) is called
  • THEN the underlying successes[arm] increments by 1; update(arm, 0) increments failures[arm] by 1

Scenario: invalid bandit config throws AbTestingError

  • WHEN new EpsilonGreedyBandit({ arms: ['x'], epsilon: 0.1, rng }) is called (only one arm)
  • THEN the constructor throws AbTestingError with code AB_CODES.INVALID_BANDIT_CONFIG

Verified by: src/bandit/bandit.test.ts::"EpsilonGreedyBandit explores when rng draw is below epsilon" AND src/bandit/bandit.test.ts::"UCB1Bandit plays each arm once before scoring" AND src/bandit/bandit.test.ts::"ThompsonSamplingBandit updates posterior on success and failure" AND src/bandit/bandit.spec-conformance.test.ts::"throws INVALID_BANDIT_CONFIG when arms.length < 2"

Requirement: Multivariate engine with full + fractional factorial design

The package SHALL provide a multivariate-experiment design engine. fullFactorial(factors) SHALL return every combination of factor levels (Cartesian product). fractionalFactorial(factors, fraction) SHALL return every Nth combination where N = round(1 / fraction). analyzeMultivariate(factors, data) SHALL return a MultivariateResult containing per-treatment conversion rates, per-factor FactorEffect objects with LevelEffect[] and a sum-of-squares importance score, pairwise InteractionEffect[] covering every (i, j) pair with i < j, and a bestCombination (highest-rate treatment with participants > 0). Construction with factors.length === 0 or any factor having levels.length === 0 SHALL throw AbTestingError(AB_CODES.INVALID_FACTOR_DESIGN, ...).

Scenario: full factorial enumerates Π levels

  • WHEN fullFactorial([{ name: 'a', levels: ['x', 'y'] }, { name: 'b', levels: ['1', '2', '3'] }]) is called
  • THEN the result has length 2 * 3 = 6 and each combination has unique id

Scenario: fractional factorial samples every Nth combination

  • WHEN fractionalFactorial(factors, 0.5) is called against a 6-combination factor set
  • THEN the result length is 3 and the kept combinations are at indices 0, 2, 4 in fullFactorial(factors)

Scenario: factor importance is a non-negative sum-of-squares score

  • WHEN analyzeMultivariate is called with treatments where one factor has level rates [0.1, 0.5, 0.9] and another has [0.5, 0.5, 0.5]
  • THEN the first factor's importance strictly exceeds the second factor's importance

Scenario: bestCombination is the highest-rate treatment with participants

  • WHEN treatments include one combination with conversionRate: 0.9 but participants: 0 and another with 0.5 and participants: 100
  • THEN bestCombination.combinationId is the latter

Scenario: empty factor set throws

  • WHEN analyzeMultivariate([], data) is called
  • THEN AbTestingError with code AB_CODES.INVALID_FACTOR_DESIGN is thrown

Verified by: src/multivariate/multivariate-engine.test.ts::"fullFactorial enumerates the Cartesian product of levels" AND src/multivariate/multivariate-engine.test.ts::"fractionalFactorial keeps every Nth combination" AND src/multivariate/multivariate-engine.spec-conformance.test.ts::"factor importance ranks higher-variance factors first" AND src/multivariate/multivariate-engine.test.ts::"bestCombination ignores treatments with zero participants" AND src/multivariate/multivariate-engine.spec-conformance.test.ts::"throws INVALID_FACTOR_DESIGN for empty factor set"

Requirement: Segment analysis with per-segment lift and interaction terms

The package SHALL provide segment-level analyses. analyzeBySegments(assignments, conversions, variantIds, segmentAttribute) SHALL group assignments by attributes[segmentAttribute] value, compute per-segment VariantResult[], attach a two-proportion z-test significance flag (z > 1.96 with both arms n ≥ 10), and report conversionRateDiff between best and worst variant. segmentConfidenceIntervals(...) SHALL return Wald intervals on conversion rates, [0, 1]-clamped. detectSegmentInteraction(...) SHALL return interaction-strength scores per segment value relative to global per-variant rates, flagged as hasInteraction: true when strength exceeds 0.01. When no assignment exposes the requested attribute, the call SHALL throw AbTestingError(AB_CODES.INVALID_SEGMENT_ATTRIBUTE, ...).

Scenario: segments grouped by attribute value

  • WHEN assignments include 50 with attributes.country === 'US' and 50 with 'CA', and analyzeBySegments(..., 'country') is called
  • THEN the result has two SegmentResult entries with segmentValue: 'US' and 'CA' respectively

Scenario: significance gates on min sample size

  • WHEN a segment has variant A with 5 assignments and variant B with 5 assignments
  • THEN isSignificant is false regardless of conversion rates

Scenario: confidence intervals are clamped to [0, 1]

  • WHEN a tiny sample produces a Wald interval with lower bound -0.05 or upper bound 1.10
  • THEN the returned lowerBound is 0 and upperBound is 1

Scenario: interaction flagged above 0.01 strength

  • WHEN a segment value has variant rates that differ from global rates such that sum-of-squared-weighted-diff > 0.01
  • THEN hasInteraction is true

Scenario: missing attribute throws

  • WHEN analyzeBySegments(..., 'nonexistent') is called and no assignment carries that attribute
  • THEN AbTestingError with code AB_CODES.INVALID_SEGMENT_ATTRIBUTE is thrown

Verified by: src/segments/segment-analyzer.test.ts::"groups assignments by segment attribute value" AND src/segments/segment-analyzer.test.ts::"isSignificant is false when sample size is below minimum" AND src/segments/segment-analyzer.test.ts::"clamps confidence intervals to [0, 1]" AND src/segments/segment-analyzer.spec-conformance.test.ts::"flags interaction when strength exceeds threshold" AND src/segments/segment-analyzer.spec-conformance.test.ts::"throws INVALID_SEGMENT_ATTRIBUTE when attribute is absent"

Requirement: Sequential testing with alpha-spending and early-stop rules

The package SHALL provide group-sequential test machinery for safe early stopping. obrienFlemingBoundary(informationFraction, alpha) SHALL return the z-boundary z(1 - alpha/2) / sqrt(informationFraction). pocockBoundary(informationFraction, alpha, maxLooks) SHALL return a constant z across looks (Pocock spending). alphaSpent(informationFraction, alpha, boundaryType?) SHALL return cumulative alpha spent under the chosen spending function and SHALL be monotone-non-decreasing in informationFraction. interimAnalysis(...) SHALL compute a single look and report shouldStop: true with stopReason: 'significance' when testStatistic >= spendingBoundary, or 'futility' when informationFraction >= 0.75 and the projected final z falls below 30% of the final boundary. runSequentialTest(...) SHALL drive the sequence and stop on first shouldStop. Construction validates alpha ∈ (0, 1), maxLooks ≥ 1, mde > 0, power ∈ (0, 1) else throws AbTestingError(AB_CODES.INVALID_SEQUENTIAL_CONFIG, ...).

Scenario: O'Brien-Fleming boundary is monotone-decreasing

  • WHEN obrienFlemingBoundary(0.25, 0.05) and obrienFlemingBoundary(1, 0.05) are computed
  • THEN the first value is strictly greater than the second

Scenario: Pocock boundary is constant across looks

  • WHEN pocockBoundary(0.5, 0.05, 4) and pocockBoundary(1, 0.05, 4) are computed
  • THEN both values are equal within numerical tolerance

Scenario: alpha-spent is monotone-non-decreasing

  • WHEN alphaSpent is called at fractions [0.1, 0.25, 0.5, 0.75, 1.0]
  • THEN the returned sequence is non-decreasing and ends at exactly alpha

Scenario: futility check fires only after 75% information

  • WHEN interimAnalysis is called at informationFraction = 0.5 with a tiny test statistic
  • THEN stopReason is null (futility branch suppressed)

Scenario: runSequentialTest stops on first shouldStop

  • WHEN the data array has four looks and the second look returns shouldStop: true
  • THEN runSequentialTest returns exactly two InterimResult entries

Verified by: src/sequential/sequential-test.test.ts::"obrienFlemingBoundary is monotone-decreasing in informationFraction" AND src/sequential/sequential-test.test.ts::"pocockBoundary is constant across looks" AND src/sequential/sequential-test.spec-conformance.test.ts::"alphaSpent is monotone-non-decreasing and ends at alpha" AND src/sequential/sequential-test.spec-conformance.test.ts::"futility branch is suppressed below 75% informationFraction" AND src/sequential/sequential-test.test.ts::"runSequentialTest stops on first shouldStop"

Requirement: Bayesian stats — Beta-Binomial conversion model

The package SHALL provide Beta-Binomial Bayesian primitives. posteriorBeta(conversions, participants, priorAlpha?, priorBeta?) SHALL return the conjugate posterior { alpha: priorAlpha + conversions, beta: priorBeta + (participants - conversions) }. betaPdf / betaCdf / betaQuantile SHALL be self-consistent (betaCdf(betaQuantile(p, a, b), a, b) ≈ p within bisection tolerance). betaSample(alpha, beta, rng) SHALL draw a Beta sample via two Gamma draws (Marsaglia-Tsang). probabilityBGreaterA(a, b, simulations?, rng) SHALL return a Monte Carlo estimate of P(B > A). credibleInterval(params, width?) SHALL return a quantile-based interval at the requested width (default 95%). expectedLoss(chosen, other, simulations?, rng) SHALL return the expected forfeited lift from picking chosen. bayesianAnalysis(results, priorAlpha?, priorBeta?, rng) SHALL run the full pairwise comparison on two VariantResult objects. All RNG-dependent functions SHALL accept an injected Rng (no module-scope Math.random).

Scenario: posteriorBeta is the conjugate update

  • WHEN posteriorBeta(10, 100, 1, 1) is called
  • THEN the result is { alpha: 11, beta: 91 }

Scenario: betaCdf and betaQuantile are inverses

  • WHEN betaQuantile(0.5, 5, 5) then betaCdf(<that>, 5, 5) is computed
  • THEN the round-trip value is within 1e-3 of 0.5

Scenario: probabilityBGreaterA converges to 0.5 for symmetric posteriors

  • WHEN probabilityBGreaterA({ alpha: 5, beta: 5 }, { alpha: 5, beta: 5 }, 20000, rng) is called
  • THEN the result is within 0.02 of 0.5

Scenario: credibleInterval respects width parameter

  • WHEN credibleInterval(params, 0.5) and credibleInterval(params, 0.95) are computed for the same params
  • THEN the 95% interval contains the 50% interval

Scenario: same Rng seed produces deterministic samples

  • WHEN betaSample(2, 3, rng) is called twice with the same Rng state
  • THEN both calls return the same numeric value (reproducibility pin)

Verified by: src/stats/bayesian.test.ts::"posteriorBeta returns the conjugate update" AND src/stats/bayesian.test.ts::"betaCdf and betaQuantile are mutual inverses" AND src/stats/bayesian.test.ts::"probabilityBGreaterA converges to 0.5 for symmetric posteriors" AND src/stats/bayesian.test.ts::"credibleInterval width parameter widens the interval" AND src/stats/bayesian.spec-conformance.test.ts::"injected Rng produces reproducible betaSample output"

Requirement: Assignment store — interface and in-memory adapter

The package SHALL persist variant assignments via an AssignmentStoreLike interface (defined in @suaveplan/types). The interface SHALL expose five async methods: get(experimentId, userId): Promise<Assignment | undefined>, set(assignment): Promise<void>, getByExperiment(experimentId): Promise<readonly Assignment[]>, getByUser(userId): Promise<readonly Assignment[]>, deleteByExperiment(experimentId): Promise<void>. The package SHALL ship a MemoryAssignmentStore class implementing this interface and a createMemoryAssignmentStore() factory. ExperimentManagerOptions SHALL accept an optional store?: AssignmentStoreLike; when provided, assignVariant SHALL consult the store before re-bucketing and write through on miss. MemoryAssignmentStore.set SHALL be idempotent on identical-key assignments (last-writer-wins).

Scenario: set/get round-trip

  • WHEN store.set({ experimentId: 'e1', userId: 'u1', variantId: 'v1', assignedAt: '2026-05-02T00:00:00Z' }) is called and then store.get('e1', 'u1') is awaited
  • THEN the resolved value is the same Assignment object shape

Scenario: getByExperiment filters to one experiment

  • WHEN the store contains assignments for e1 and e2 and getByExperiment('e1') is awaited
  • THEN the result contains only e1 assignments

Scenario: deleteByExperiment cascades

  • WHEN the store contains 100 assignments for e1 and deleteByExperiment('e1') is awaited
  • THEN subsequent getByExperiment('e1') returns an empty array

Scenario: ExperimentManager with store does not re-bucket on hit

  • WHEN createExperimentManager({ store }) is called, assignVariant(exp, ctx) runs once and persists, then assignVariant(exp, ctx) runs again with the same ctx
  • THEN the second call returns the persisted variant without invoking bucketUser (verified by an Rng spy that increments only on miss)

Scenario: missing key resolves undefined

  • WHEN store.get('does-not-exist', 'u1') is awaited
  • THEN the resolved value is undefined (no error thrown)

Verified by: src/store/memory-assignment-store.test.ts::"set and get round-trip an Assignment" AND src/store/memory-assignment-store.test.ts::"getByExperiment returns only matching experiment ids" AND src/store/memory-assignment-store.test.ts::"deleteByExperiment removes all assignments for that experiment" AND src/core/experiment-manager/experiment-manager.store.test.ts::"second assignVariant call with store option returns persisted variant without re-bucketing" AND src/store/memory-assignment-store.spec-conformance.test.ts::"get resolves undefined for missing keys"

Reference in New Issue View Git Blame Copy Permalink