OpenSpec Change: enterprise-ab-testing

Change ID: enterprise-ab-testing
Package: @suaveplan/ab-testing
Category: universal/workflow
Target path: universal/workflow/ab-testing
Status: PROPOSED
Date: 2026-04-24
Reference spec: /mnt/s/web/suaveplan/enterprise/ab-testing.md

Summary

Deliver @suaveplan/ab-testing on the genesis branch from scratch at enterprise quality. The package provides an A/B testing engine with deterministic assignment, audience targeting, experiment lifecycle management, metric tracking, and results analysis. Genesis adds structured errors via @suaveplan/error, Zod schemas for all experiment types, OTel instrumentation on assignment and conversion events, injectable Clock and Rng for deterministic bucketing, integration with @suaveplan/feature-flags as an optional provider, event-bus emission for assignment/conversion events, and property-based tests for bucketing distribution invariants.

Current-state audit (genesis)

The package does not exist on genesis yet — this is a fresh build.

Available workspace deps on genesis

Package	Role
`@suaveplan/error` (`workspace:^`)	Base class for `AbTestingError`; stable error codes
`@suaveplan/types` (`workspace:^`)	`LoggerLike`, `Clock`, `Rng`, `EventBusLike`
`@suaveplan/events` (`workspace:^`)	Event bus for assignment and conversion events
`@suaveplan/logger` (`workspace:^`)	Structured logging with OTel trace correlation
`@suaveplan/feature-flags` (`workspace:^`)	Optional feature-flag provider for experiment gating
`zod` (`catalog:forms`)	Zod schemas for all public types
`@opentelemetry/api` (`catalog:observability`)	Spans + counters on assignment/conversion
`tslib` (`catalog:build`)	Runtime helpers

Dev deps: vitest, @vitest/coverage-istanbul, fast-check, mitata (all catalog:testing).

Public API surface

Core

function createExperimentManager(options?: ExperimentManagerOptions): ExperimentManager

interface ExperimentManager {
  defineExperiment(experiment: ExperimentDefinition): void
  getExperiment(id: string): ExperimentDefinition | null
  listExperiments(): ReadonlyArray<ExperimentDefinition>
  assign(experimentId: string, userId: string, attributes?: UserAttributes): Assignment
  track(experimentId: string, userId: string, metric: MetricEvent): void
  getResults(experimentId: string): ExperimentResults
  use(provider: ExperimentProvider): this
}

Experiment definition

interface ExperimentDefinition {
  id: string
  name: string
  status: 'draft' | 'running' | 'paused' | 'concluded'
  variants: ReadonlyArray<Variant>
  targeting?: TargetingRule[]
  startDate?: string
  endDate?: string
  sampleRate?: number  // 0–1, default 1.0
  primaryMetric?: string
  secondaryMetrics?: string[]
}

interface Variant {
  id: string
  name: string
  weight: number  // relative weight, will be normalised
  isControl?: boolean
}

interface TargetingRule {
  attribute: string
  operator: 'eq' | 'neq' | 'in' | 'notIn' | 'gt' | 'lt' | 'gte' | 'lte'
  value: unknown
}

Assignment

interface Assignment {
  readonly experimentId: string
  readonly userId: string
  readonly variantId: string
  readonly isControl: boolean
  readonly timestamp: number
  readonly inExperiment: boolean  // false if excluded by targeting or sample rate
}

Metrics

interface MetricEvent {
  name: string
  value?: number
  timestamp?: number
  properties?: Record<string, unknown>
}

interface ExperimentResults {
  readonly experimentId: string
  readonly variants: ReadonlyArray<VariantResults>
  readonly totalAssignments: number
  readonly totalConversions: number
}

interface VariantResults {
  readonly variantId: string
  readonly assignments: number
  readonly conversions: number
  readonly conversionRate: number
  readonly metrics: ReadonlyArray<{ name: string; count: number; sum: number; avg: number }>
}

Provider interface

interface ExperimentProvider {
  readonly name: string
  getAssignment?(experimentId: string, userId: string): string | null
  trackAssignment?(assignment: Assignment): void
  trackConversion?(experimentId: string, userId: string, metric: MetricEvent): void
}

type UserAttributes = Record<string, string | number | boolean>

interface ExperimentManagerOptions {
  logger?: LoggerLike
  clock?: Clock
  rng?: Rng
  eventBus?: EventBusLike
  hashSeed?: number
}

Bucketing utilities

function bucketUser(userId: string, experimentId: string, seed?: number): number  // 0–1
function normaliseWeights(variants: Variant[]): Array<Variant & { normalizedWeight: number }>
function selectVariant(variants: Variant[], bucket: number): Variant

Error codes

type AbTestingErrorCode =
  | 'AB_EXPERIMENT_NOT_FOUND'
  | 'AB_EXPERIMENT_INACTIVE'
  | 'AB_INVALID_DEFINITION'
  | 'AB_VARIANT_NOT_FOUND'
  | 'AB_METRIC_INVALID'

function registerAbTestingErrorCodes(): void

Genesis additions

Structured AbTestingError extending SuaveplanError with stable AB.* codes
Zod schemas for ExperimentDefinition, Variant, TargetingRule, MetricEvent
Deterministic FNV-1a bucketing — injectable Rng for testing; FNV-1a hash by default for consistency with @suaveplan/feature-flags
Targeting rule evaluation — 8 operators, evaluated before assignment
Provider plugin interface — use() for external integrations
AbortSignal available on async provider calls
OTel instrumentation — span ab.assign, counter ab.assignments_total, counter ab.conversions_total, attributes include experiment_id, variant_id
Property-based tests — bucketing distribution, targeting rule evaluation, weight normalisation

File layout

universal/workflow/ab-testing/
  package.json
  tsconfig.json
  vitest.config.ts
  src/
    index.ts
    error-codes.ts
    constants/
      index.ts
    types/
      common.ts
      common.md
    core/
      experiment-manager/
        experiment-manager.ts
        experiment-manager.md
        experiment-manager.types.ts
        experiment-manager.test.ts
        index.ts
    bucketing/
      bucketing.ts
      bucketing.md
      bucketing.test.ts
      index.ts
    targeting/
      targeting.ts
      targeting.md
      targeting.test.ts
      index.ts
    results/
      results.ts
      results.md
      results.test.ts
      index.ts
    providers/
      provider.ts
      provider.md
      provider.test.ts
      index.ts
    __benchmarks__/
      ab-testing.bench.ts
  tests/
    integration.test.ts
    property.test.ts

Impact

Downstream packages: Any application layer that needs experiment assignment. No existing genesis package currently depends on it.

Workspace dep prerequisites: @suaveplan/error, @suaveplan/types, @suaveplan/events, @suaveplan/logger, @suaveplan/feature-flags must be published.

6.9 KiB Raw Permalink Blame History Unescape Escape