Move docs/ package mirror into document-navigator plugin
Relocate the ~6k-file docs tree from the repo root into plugins/document-navigator/docs/ so the plugin is self-contained and portable. docs/superpowers/ (specs and plans) stays at the repo root. Moved via cp+rm rather than git mv because directory rename() fails with EPERM on this WSL /mnt DrvFs mount; git rename-detection pairs the identical-content files at commit time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,116 @@
|
||||
# bucketing.ts — Deterministic User Bucketing
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides the core hashing and variant selection primitives that map a
|
||||
`(userId, experimentId)` pair to a stable, deterministic variant. The
|
||||
implementation uses the FNV-1a 32-bit algorithm — a non-cryptographic hash
|
||||
chosen for its excellent distribution characteristics, extremely low
|
||||
computational overhead, and zero external dependencies.
|
||||
|
||||
Determinism is a hard requirement for A/B testing: the same user must always
|
||||
receive the same variant for the lifetime of the experiment. This module
|
||||
achieves that by combining the user and experiment identifiers with a null-byte
|
||||
separator before hashing, preventing cross-experiment correlation while
|
||||
remaining completely stateless.
|
||||
|
||||
## Features
|
||||
|
||||
- `bucketUser` — FNV-1a 32-bit hash of `userId\0experimentId` with optional
|
||||
seed XOR offset; output is a float in `[0, 1)`
|
||||
- `normaliseWeights` — converts arbitrary positive weights to cumulative
|
||||
normalised slices in `[0, 1]`; zero/negative-weight variants are excluded
|
||||
- `selectVariant` — binary search over normalised cumulative weights to find
|
||||
the owning variant for a bucket value
|
||||
- `NormalisedVariant` — typed extension of `Variant` with `normalizedWeight`
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { bucketUser, normaliseWeights, selectVariant } from '@suaveplan/ab-testing/bucketing';
|
||||
|
||||
const variants = [
|
||||
{ id: 'control', name: 'Control', weight: 50, isControl: true },
|
||||
{ id: 'treatment', name: 'Treatment', weight: 50 },
|
||||
];
|
||||
|
||||
const normalised = normaliseWeights(variants);
|
||||
const bucket = bucketUser('user-123', 'checkout-cta');
|
||||
const chosen = selectVariant(normalised, bucket);
|
||||
|
||||
console.log(chosen.id); // deterministic: 'control' or 'treatment'
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { bucketUser, normaliseWeights, selectVariant } from '@suaveplan/ab-testing/bucketing';
|
||||
|
||||
// Unequal weights: 10% treatment A, 20% treatment B, 70% control
|
||||
const variants = [
|
||||
{ id: 'ctrl', name: 'Control', weight: 70, isControl: true },
|
||||
{ id: 'treat-a', name: 'Treatment A', weight: 10 },
|
||||
{ id: 'treat-b', name: 'Treatment B', weight: 20 },
|
||||
];
|
||||
|
||||
// Custom seed differentiates this experiment's buckets from other experiments
|
||||
// even when the same userId is used across both.
|
||||
const normalised = normaliseWeights(variants);
|
||||
const bucket = bucketUser('user-456', 'homepage-hero', 0xdeadbeef);
|
||||
const chosen = selectVariant(normalised, bucket);
|
||||
|
||||
// Verify: generate 10 000 users and check distribution
|
||||
let counts: Record<string, number> = {};
|
||||
for (let i = 0; i < 10_000; i++) {
|
||||
const b = bucketUser(`u-${i}`, 'homepage-hero', 0xdeadbeef);
|
||||
const v = selectVariant(normalised, b);
|
||||
counts[v.id] = (counts[v.id] ?? 0) + 1;
|
||||
}
|
||||
// Expected: ctrl ≈ 7000, treat-a ≈ 1000, treat-b ≈ 2000
|
||||
console.log(counts);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `bucketUser(userId, experimentId, seed?): number`
|
||||
|
||||
- **`userId`** — `string` — user or entity identifier
|
||||
- **`experimentId`** — `string` — experiment identifier; scopes the bucket
|
||||
- **`seed`** — `number` (default `0`) — XOR'd into the FNV offset basis
|
||||
- **Returns** — `number` in `[0, 1)`
|
||||
- **Deterministic** — identical inputs always produce identical output
|
||||
|
||||
### `normaliseWeights(variants): NormalisedVariant[]`
|
||||
|
||||
- **`variants`** — `ReadonlyArray<Variant>`
|
||||
- **Returns** — `NormalisedVariant[]` sorted in original order with `normalizedWeight`
|
||||
set to cumulative upper bounds. Variants with `weight <= 0` are excluded.
|
||||
Returns `[]` when all variants have non-positive weight.
|
||||
|
||||
### `selectVariant(variants, bucket): Variant`
|
||||
|
||||
- **`variants`** — `ReadonlyArray<NormalisedVariant>` from `normaliseWeights`
|
||||
- **`bucket`** — `number` in `[0, 1)` from `bucketUser`
|
||||
- **Returns** — the `Variant` whose cumulative slice contains `bucket`
|
||||
- **Throws** — `RangeError` when `variants` is empty
|
||||
|
||||
### `NormalisedVariant`
|
||||
|
||||
Extends `Variant` with `normalizedWeight: number` representing the cumulative
|
||||
upper bound of the variant's slice. The last variant always has
|
||||
`normalizedWeight === 1.0`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The FNV-1a algorithm is applied with `Math.imul` for correct 32-bit wrapping
|
||||
on all JavaScript engines (V8, JavaScriptCore, SpiderMonkey). The hash is
|
||||
mapped to `[0, 1)` by dividing by `0x1_0000_0000` (`2^32`), preserving the
|
||||
full resolution of the 32-bit output.
|
||||
|
||||
The null-byte separator (`\0`) between `userId` and `experimentId` prevents
|
||||
hash collisions between adjacent key combinations such as `('ab', 'c')` and
|
||||
`('a', 'bc')`. The seed XOR is applied to the FNV offset basis before
|
||||
iterating so that the seed affects every step of the hash, not just the first.
|
||||
@@ -0,0 +1,148 @@
|
||||
# experiment-manager.ts — Experiment Manager
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Implements the `ExperimentManager` interface as a stateful, in-process service
|
||||
that manages the full A/B experiment lifecycle: defining experiments, assigning
|
||||
users to variants, recording conversion events, and computing results. All
|
||||
state is held in memory, making the manager suitable for server-side
|
||||
applications where state can be injected at startup and reset between test runs.
|
||||
|
||||
The manager integrates with the OTel API for counter and span instrumentation,
|
||||
emits events on an injectable `EventBusLike` for downstream consumers, and
|
||||
delegates to registered `ExperimentProvider` instances for forced assignments
|
||||
and platform-specific event forwarding.
|
||||
|
||||
## Features
|
||||
|
||||
- `defineExperiment` — registers or replaces an experiment definition with
|
||||
Zod validation
|
||||
- `getExperiment` / `listExperiments` — read-only accessors
|
||||
- `assign` — deterministic variant assignment with targeting evaluation,
|
||||
sample-rate gating, provider override, FNV-1a bucketing, assignment caching,
|
||||
OTel counter, event-bus emission, and provider tracking
|
||||
- `track` — conversion event recording with Zod validation, provider tracking,
|
||||
and OTel counter
|
||||
- `getResults` — on-demand result aggregation via `computeResults`
|
||||
- `use` — chainable provider registration
|
||||
- Injectable `clock`, `rng`, `logger`, `eventBus`, `hashSeed`
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createExperimentManager } from '@suaveplan/ab-testing';
|
||||
|
||||
const manager = createExperimentManager();
|
||||
|
||||
manager.defineExperiment({
|
||||
id: 'checkout-cta',
|
||||
name: 'Checkout CTA Colour',
|
||||
status: 'running',
|
||||
variants: [
|
||||
{ id: 'ctrl', name: 'Blue', weight: 1, isControl: true },
|
||||
{ id: 'treat', name: 'Green', weight: 1 },
|
||||
],
|
||||
primaryMetric: 'checkout_started',
|
||||
});
|
||||
|
||||
const a = manager.assign('checkout-cta', 'user-abc');
|
||||
if (a.inExperiment) {
|
||||
console.log('variant:', a.variantId);
|
||||
}
|
||||
|
||||
manager.track('checkout-cta', 'user-abc', { name: 'checkout_started' });
|
||||
|
||||
const results = manager.getResults('checkout-cta');
|
||||
console.log(results.totalConversions); // 1
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { createExperimentManager } from '@suaveplan/ab-testing';
|
||||
import { ForceVariantProvider, LoggingProvider } from '@suaveplan/ab-testing/providers';
|
||||
|
||||
const manager = createExperimentManager({
|
||||
hashSeed: 0xcafe, // scope buckets to this deployment
|
||||
sampleRate: undefined, // per-experiment
|
||||
});
|
||||
|
||||
manager
|
||||
.use(new ForceVariantProvider({ 'my-exp': { 'qa-1': 'treatment' } }))
|
||||
.use(new LoggingProvider({ onAssignment: (a) => metrics.record(a) }));
|
||||
|
||||
manager.defineExperiment({
|
||||
id: 'my-exp',
|
||||
name: 'Dashboard Redesign',
|
||||
status: 'running',
|
||||
variants: [
|
||||
{ id: 'control', name: 'Old', weight: 50, isControl: true },
|
||||
{ id: 'treatment', name: 'New', weight: 50 },
|
||||
],
|
||||
targeting: [
|
||||
{ attribute: 'plan', operator: 'eq', value: 'premium' },
|
||||
{ attribute: 'country', operator: 'in', value: ['US', 'CA'] },
|
||||
],
|
||||
sampleRate: 0.1,
|
||||
primaryMetric: 'feature_click',
|
||||
});
|
||||
|
||||
// qa-1 always gets treatment (forced by provider)
|
||||
const qa = manager.assign('my-exp', 'qa-1', { plan: 'premium', country: 'US' });
|
||||
console.log(qa.variantId); // 'treatment'
|
||||
|
||||
// Random user goes through bucketing + sample-rate
|
||||
const user = manager.assign('my-exp', 'user-999', { plan: 'premium', country: 'CA' });
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `createExperimentManager(options?): ExperimentManager`
|
||||
|
||||
Factory function. All options are optional.
|
||||
|
||||
| Option | Type | Default |
|
||||
|---|---|---|
|
||||
| `logger` | `LoggerLike` | none |
|
||||
| `clock` | `Clock` | `{ now: () => Date.now() }` |
|
||||
| `rng` | `Rng` | `{ next: () => Math.random() }` |
|
||||
| `eventBus` | `EventBusLike` | none |
|
||||
| `hashSeed` | `number` | `0` |
|
||||
|
||||
### `ExperimentManager` interface
|
||||
|
||||
| Method | Signature | Description |
|
||||
|---|---|---|
|
||||
| `defineExperiment` | `(def) => void` | Register/replace experiment |
|
||||
| `getExperiment` | `(id) => ExperimentDefinition \| null` | Lookup by id |
|
||||
| `listExperiments` | `() => ReadonlyArray<ExperimentDefinition>` | All registered |
|
||||
| `assign` | `(expId, userId, attrs?) => Assignment` | Assign user to variant |
|
||||
| `track` | `(expId, userId, metric) => void` | Record conversion event |
|
||||
| `getResults` | `(expId) => ExperimentResults` | Aggregate results |
|
||||
| `use` | `(provider) => this` | Register provider (chainable) |
|
||||
|
||||
### Throws
|
||||
|
||||
- `AB_EXPERIMENT_NOT_FOUND` — `assign`, `track`, `getResults` when experiment
|
||||
is not registered
|
||||
- `AB_EXPERIMENT_INACTIVE` — `assign` when `status !== 'running'`
|
||||
- `AB_INVALID_DEFINITION` — `defineExperiment` when Zod validation fails, or
|
||||
when all variants have weight ≤ 0 at assign time
|
||||
- `AB_METRIC_INVALID` — `track` when Zod validation of the metric event fails
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Assignments are cached by `${experimentId}:${userId}` key so repeated calls
|
||||
to `assign` for the same user always return the same result. This is critical
|
||||
for correctness — a user must not "jump" between variants within an experiment.
|
||||
|
||||
The OTel tracer wraps `_assign` in a span named `ab.assign` so that
|
||||
distributed traces include experiment assignment latency. The span is closed
|
||||
in a `finally` block to ensure it is always ended even when exceptions occur.
|
||||
|
||||
Providers are iterated in registration order. For `getAssignment`, the first
|
||||
non-null result wins (short-circuit). For `trackAssignment` and
|
||||
`trackConversion`, all providers are notified.
|
||||
@@ -0,0 +1,116 @@
|
||||
# provider.ts — Experiment Providers
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides two concrete implementations of the `ExperimentProvider` interface:
|
||||
`LoggingProvider` for forwarding assignment and conversion events to custom
|
||||
callbacks, and `ForceVariantProvider` for overriding variant assignment in
|
||||
QA, staging, and testing environments.
|
||||
|
||||
Providers are the extension point for integrating `@suaveplan/ab-testing`
|
||||
with external platforms (LaunchDarkly, Optimizely, Amplitude, custom analytics
|
||||
pipelines) without coupling the core engine to any specific SDK.
|
||||
|
||||
## Features
|
||||
|
||||
- `LoggingProvider` — configurable callbacks for assignment and conversion
|
||||
events; defaults to `console.log` for zero-config debugging
|
||||
- `ForceVariantProvider` — experiment-scoped per-user variant overrides with
|
||||
wildcard `'*'` support for forcing all users in an experiment to a variant
|
||||
- Both classes implement `ExperimentProvider` so they can be passed to
|
||||
`manager.use(provider)` for chaining
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { LoggingProvider } from '@suaveplan/ab-testing/providers';
|
||||
import { createExperimentManager } from '@suaveplan/ab-testing';
|
||||
|
||||
const manager = createExperimentManager();
|
||||
|
||||
manager.use(new LoggingProvider({
|
||||
onAssignment: (a) => console.info('assigned', a.variantId, 'to', a.userId),
|
||||
}));
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
ForceVariantProvider,
|
||||
LoggingProvider,
|
||||
} from '@suaveplan/ab-testing/providers';
|
||||
import { createExperimentManager } from '@suaveplan/ab-testing';
|
||||
|
||||
const manager = createExperimentManager();
|
||||
|
||||
// Force QA users to treatment, but let normal users bucket normally
|
||||
const forceProvider = new ForceVariantProvider({
|
||||
'checkout-redesign': {
|
||||
'qa-user-1': 'treatment',
|
||||
'qa-user-2': 'control',
|
||||
// '*': 'treatment', // would force everyone not explicitly listed
|
||||
},
|
||||
});
|
||||
|
||||
// Send events to an analytics platform
|
||||
const analyticsProvider = new LoggingProvider({
|
||||
onAssignment: (a) => externalAnalytics.track('experiment_assigned', a),
|
||||
onConversion: (expId, userId, metric) =>
|
||||
externalAnalytics.track('experiment_converted', { expId, userId, ...metric }),
|
||||
});
|
||||
|
||||
// Chain multiple providers
|
||||
manager.use(forceProvider).use(analyticsProvider);
|
||||
|
||||
// For the QA user, forceProvider.getAssignment returns 'treatment'
|
||||
// and the bucketing step is skipped
|
||||
const assignment = manager.assign('checkout-redesign', 'qa-user-1');
|
||||
console.log(assignment.variantId); // 'treatment'
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `LoggingProvider`
|
||||
|
||||
Constructor: `new LoggingProvider(opts?: LoggingProviderOptions)`
|
||||
|
||||
#### `LoggingProviderOptions`
|
||||
|
||||
| Property | Type | Default |
|
||||
|---|---|---|
|
||||
| `onAssignment` | `(assignment: Assignment) => void` | `console.log` |
|
||||
| `onConversion` | `(experimentId, userId, metric) => void` | `console.log` |
|
||||
|
||||
Methods:
|
||||
|
||||
- `trackAssignment(assignment)` — invokes `onAssignment`
|
||||
- `trackConversion(experimentId, userId, metric)` — invokes `onConversion`
|
||||
|
||||
### `ForceVariantProvider`
|
||||
|
||||
Constructor: `new ForceVariantProvider(overrides?)`
|
||||
|
||||
Where `overrides` is `Record<string, Record<string, string>>` mapping
|
||||
`experimentId → { userId → variantId }`. Use `'*'` as the `userId` key to
|
||||
force all users not explicitly listed.
|
||||
|
||||
Methods:
|
||||
|
||||
- `getAssignment(experimentId, userId): string | null` — returns the forced
|
||||
variant ID or `null` to fall through to bucketing
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`LoggingProvider` stores the resolved callbacks in a `protected readonly opts`
|
||||
object so that subclasses can override or extend the default behaviour.
|
||||
`ForceVariantProvider` checks user-specific overrides before the wildcard to
|
||||
allow precise QA control even when a global experiment override is active.
|
||||
|
||||
Neither provider implements both `getAssignment` and `trackAssignment` /
|
||||
`trackConversion` simultaneously — they are intentionally specialised. For a
|
||||
production integration, compose multiple providers via `manager.use()` or
|
||||
implement a custom class that merges the two concerns.
|
||||
@@ -0,0 +1,118 @@
|
||||
# results.ts — Experiment Results Computation
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides the `computeResults` function, which aggregates raw assignment and
|
||||
conversion data into structured per-variant statistics. This module is a pure
|
||||
function with no side effects, making it straightforward to test exhaustively
|
||||
and to call from any context — the manager uses it on demand, but it can also
|
||||
be called directly from analytics pipelines or reporting jobs.
|
||||
|
||||
The function produces assignment counts, conversion counts, conversion rates,
|
||||
and per-metric aggregates (count, sum, average) for every variant that received
|
||||
at least one in-experiment assignment.
|
||||
|
||||
## Features
|
||||
|
||||
- Groups assignments by `variantId`, excluding out-of-experiment users
|
||||
- Counts unique conversions per user (one conversion per user, regardless of
|
||||
how many metric events they fire)
|
||||
- Aggregates named metric events into `count`, `sum`, and `avg` per metric
|
||||
per variant
|
||||
- Computes `conversionRate = conversions / assignments` (0 when no assignments)
|
||||
- Returns `totalAssignments` and `totalConversions` rolled up across all
|
||||
variants for dashboard summary views
|
||||
- Fully deterministic — given the same inputs, produces byte-identical output
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { computeResults } from '@suaveplan/ab-testing/results';
|
||||
import type { Assignment, MetricEvent } from '@suaveplan/ab-testing';
|
||||
|
||||
const assignments: Assignment[] = [
|
||||
{ experimentId: 'exp-1', userId: 'u1', variantId: 'ctrl', isControl: true, timestamp: 1, inExperiment: true },
|
||||
{ experimentId: 'exp-1', userId: 'u2', variantId: 'treat', isControl: false, timestamp: 2, inExperiment: true },
|
||||
];
|
||||
const conversions = new Map<string, MetricEvent[]>([
|
||||
['u1', [{ name: 'checkout_started', value: 1 }]],
|
||||
]);
|
||||
|
||||
const results = computeResults('exp-1', assignments, conversions);
|
||||
console.log(results.totalAssignments); // 2
|
||||
console.log(results.totalConversions); // 1
|
||||
console.log(results.variants[0]?.conversionRate); // 1.0 (ctrl: 1/1)
|
||||
console.log(results.variants[1]?.conversionRate); // 0.0 (treat: 0/1)
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { computeResults } from '@suaveplan/ab-testing/results';
|
||||
import type { Assignment, MetricEvent } from '@suaveplan/ab-testing';
|
||||
|
||||
// Multiple metric events per user (only first triggers conversion flag)
|
||||
const assignments: Assignment[] = [
|
||||
{ experimentId: 'exp-2', userId: 'u1', variantId: 'v1', isControl: true, timestamp: 1, inExperiment: true },
|
||||
{ experimentId: 'exp-2', userId: 'u2', variantId: 'v1', isControl: true, timestamp: 2, inExperiment: true },
|
||||
{ experimentId: 'exp-2', userId: 'u3', variantId: 'v2', isControl: false, timestamp: 3, inExperiment: true },
|
||||
// excluded user — not counted
|
||||
{ experimentId: 'exp-2', userId: 'u4', variantId: '', isControl: false, timestamp: 4, inExperiment: false },
|
||||
];
|
||||
const conversions = new Map<string, MetricEvent[]>([
|
||||
['u1', [
|
||||
{ name: 'revenue', value: 49.99 },
|
||||
{ name: 'revenue', value: 19.99 },
|
||||
]],
|
||||
['u3', [{ name: 'revenue', value: 99.00 }]],
|
||||
]);
|
||||
|
||||
const results = computeResults('exp-2', assignments, conversions);
|
||||
// v1: 2 assignments, 1 conversion (u1), revenue avg = (49.99 + 19.99) / 2
|
||||
// v2: 1 assignment, 1 conversion (u3), revenue avg = 99.00
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `computeResults(experimentId, assignments, conversions): ExperimentResults`
|
||||
|
||||
- **`experimentId`** — `string` — stable experiment identifier copied to result
|
||||
- **`assignments`** — `Assignment[]` — all assignments for the experiment
|
||||
- **`conversions`** — `Map<string, MetricEvent[]>` — keyed by `userId`
|
||||
- **Returns** — `ExperimentResults` with `variants`, `totalAssignments`,
|
||||
`totalConversions`
|
||||
|
||||
#### `ExperimentResults`
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `experimentId` | `string` | Echoed from input |
|
||||
| `variants` | `VariantResults[]` | Per-variant aggregates |
|
||||
| `totalAssignments` | `number` | Sum across all variants |
|
||||
| `totalConversions` | `number` | Sum across all variants |
|
||||
|
||||
#### `VariantResults`
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `variantId` | `string` | Variant identifier |
|
||||
| `assignments` | `number` | In-experiment assignment count |
|
||||
| `conversions` | `number` | Unique-user conversion count |
|
||||
| `conversionRate` | `number` | `conversions / assignments` |
|
||||
| `metrics` | `Array<{name, count, sum, avg}>` | Per-metric aggregates |
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Conversion counting follows the convention that a user can convert at most
|
||||
once per experiment — the first metric event for each user triggers the
|
||||
conversion flag, but all subsequent events for that user are still accumulated
|
||||
into the metric aggregates. This avoids double-counting in funnel analyses
|
||||
while preserving the full metric history for revenue and value calculations.
|
||||
|
||||
Out-of-experiment assignments (`inExperiment: false`) are excluded entirely
|
||||
from all counts. They are recorded so that the manager can return a stable
|
||||
assignment for a user across multiple calls, but they should not pollute the
|
||||
statistical results.
|
||||
@@ -0,0 +1,92 @@
|
||||
# targeting.ts — Targeting Rule Evaluation
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides the `evaluateTargeting` function that evaluates an ordered list of
|
||||
targeting rules against a set of user attributes. The function implements AND
|
||||
semantics — every rule in the list must pass for the user to be considered
|
||||
eligible for the experiment. This single-function module is intentionally
|
||||
small and pure: it has no side effects, no I/O, no external dependencies, and
|
||||
is trivially unit-testable.
|
||||
|
||||
Targeting rules are the primary mechanism for gating experiment exposure to
|
||||
specific user segments. Common use cases include restricting to paid users,
|
||||
specific geographic regions, device types, or beta-program membership.
|
||||
|
||||
## Features
|
||||
|
||||
- Eight comparison operators: `eq`, `neq`, `in`, `notIn`, `gt`, `lt`, `gte`, `lte`
|
||||
- Numeric ordering operators (`gt`, `lt`, `gte`, `lte`) require both sides to
|
||||
be numbers; any type mismatch returns `false` rather than throwing
|
||||
- `in` / `notIn` accept any array on the right-hand side
|
||||
- An empty rules array always returns `true` (no restriction)
|
||||
- Absent attributes evaluate to `undefined` and fail equality/ordering tests
|
||||
gracefully without exceptions
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { evaluateTargeting } from '@suaveplan/ab-testing/targeting';
|
||||
|
||||
const eligible = evaluateTargeting(
|
||||
[{ attribute: 'plan', operator: 'eq', value: 'premium' }],
|
||||
{ plan: 'premium', country: 'US' }
|
||||
);
|
||||
console.log(eligible); // true
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { evaluateTargeting } from '@suaveplan/ab-testing/targeting';
|
||||
|
||||
// Multi-rule targeting: premium users in North America with high session count
|
||||
const rules = [
|
||||
{ attribute: 'plan', operator: 'eq' as const, value: 'premium' },
|
||||
{ attribute: 'country', operator: 'in' as const, value: ['US', 'CA', 'MX'] },
|
||||
{ attribute: 'sessionCount', operator: 'gte' as const, value: 10 },
|
||||
{ attribute: 'blocklisted', operator: 'neq' as const, value: true },
|
||||
];
|
||||
|
||||
const user1 = { plan: 'premium', country: 'CA', sessionCount: 15, blocklisted: false };
|
||||
const user2 = { plan: 'free', country: 'US', sessionCount: 30, blocklisted: false };
|
||||
|
||||
console.log(evaluateTargeting(rules, user1)); // true (all 4 rules pass)
|
||||
console.log(evaluateTargeting(rules, user2)); // false (plan !== 'premium')
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `evaluateTargeting(rules, attributes): boolean`
|
||||
|
||||
- **`rules`** — `ReadonlyArray<TargetingRule>` — ordered list of predicates
|
||||
- **`attributes`** — `UserAttributes` (`Record<string, string | number | boolean>`)
|
||||
- **Returns** — `boolean` — `true` iff every rule is satisfied (or `rules` is empty)
|
||||
|
||||
#### Supported operators
|
||||
|
||||
| Operator | Description | RHS type |
|
||||
|---|---|---|
|
||||
| `eq` | Strict equality (`===`) | any |
|
||||
| `neq` | Strict inequality (`!==`) | any |
|
||||
| `in` | LHS is contained in RHS array | `unknown[]` |
|
||||
| `notIn` | LHS is not contained in RHS array | `unknown[]` |
|
||||
| `gt` | LHS > RHS | `number` |
|
||||
| `lt` | LHS < RHS | `number` |
|
||||
| `gte` | LHS >= RHS | `number` |
|
||||
| `lte` | LHS <= RHS | `number` |
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
All comparisons are intentionally strict (`===` / `!==`) to avoid surprising
|
||||
truthy/falsy coercions. When the RHS is not of the expected type for ordering
|
||||
operators (e.g. a string is passed as the threshold for `gt`), the function
|
||||
returns `false` rather than throwing, making it safe to evaluate partially
|
||||
invalid rule sets without crashing the assignment pipeline.
|
||||
|
||||
The `in` / `notIn` operators use `Array.isArray` to check the RHS type before
|
||||
calling `Array.prototype.includes`. When the RHS is not an array, the rule
|
||||
evaluates to `false`.
|
||||
@@ -0,0 +1,151 @@
|
||||
# common.ts — Core Domain Types
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines the shared TypeScript interfaces and type aliases that form the public
|
||||
contract of `@suaveplan/ab-testing`. Every other module in the package imports
|
||||
from this file rather than re-declaring shapes, ensuring a single source of
|
||||
truth for the domain model.
|
||||
|
||||
The types are designed to be structurally compatible with common external A/B
|
||||
testing platforms (LaunchDarkly, Optimizely, Split.io) to make provider
|
||||
adapters straightforward to write.
|
||||
|
||||
## Features
|
||||
|
||||
- `ExperimentDefinition` — full lifecycle descriptor: id, name, status, variants,
|
||||
targeting rules, date range, sample rate, and metric names
|
||||
- `Variant` — a single treatment group with stable id, display name, relative
|
||||
traffic weight, and optional control flag
|
||||
- `TargetingRule` — a single eligibility predicate with eight comparison
|
||||
operators (eq, neq, in, notIn, gt, lt, gte, lte)
|
||||
- `Assignment` — immutable record capturing which variant a user received,
|
||||
whether they are in-experiment, and when the assignment occurred
|
||||
- `MetricEvent` — a named conversion event with optional numeric value,
|
||||
timestamp, and arbitrary properties
|
||||
- `VariantResults` / `ExperimentResults` — aggregated assignment and conversion
|
||||
statistics ready for significance testing
|
||||
- `UserAttributes` — free-form attribute map used in targeting evaluation
|
||||
- `ExperimentManagerOptions` — dependency-injection surface for clock, RNG,
|
||||
logger, event bus, and hash seed
|
||||
- `ExperimentProvider` — plug-in interface for external platform adapters
|
||||
- `ExperimentManager` — primary management interface for defining experiments,
|
||||
assigning users, tracking conversions, and reading results
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type {
|
||||
ExperimentDefinition,
|
||||
Variant,
|
||||
} from '@suaveplan/ab-testing';
|
||||
|
||||
const control: Variant = { id: 'control', name: 'Control', weight: 50, isControl: true };
|
||||
const treatment: Variant = { id: 'treatment', name: 'Treatment', weight: 50 };
|
||||
|
||||
const exp: ExperimentDefinition = {
|
||||
id: 'checkout-cta',
|
||||
name: 'Checkout CTA colour',
|
||||
status: 'running',
|
||||
variants: [control, treatment],
|
||||
sampleRate: 0.5,
|
||||
primaryMetric: 'checkout_started',
|
||||
};
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import type {
|
||||
ExperimentDefinition,
|
||||
TargetingRule,
|
||||
ExperimentManagerOptions,
|
||||
ExperimentProvider,
|
||||
Assignment,
|
||||
} from '@suaveplan/ab-testing';
|
||||
|
||||
// Targeting: only premium users in the US
|
||||
const rules: TargetingRule[] = [
|
||||
{ attribute: 'plan', operator: 'eq', value: 'premium' },
|
||||
{ attribute: 'country', operator: 'in', value: ['US', 'CA'] },
|
||||
];
|
||||
|
||||
const exp: ExperimentDefinition = {
|
||||
id: 'dashboard-redesign',
|
||||
name: 'Dashboard redesign',
|
||||
status: 'running',
|
||||
variants: [
|
||||
{ id: 'v0', name: 'Old', weight: 1, isControl: true },
|
||||
{ id: 'v1', name: 'New', weight: 1 },
|
||||
],
|
||||
targeting: rules,
|
||||
sampleRate: 0.1,
|
||||
primaryMetric: 'session_duration',
|
||||
secondaryMetrics: ['page_views', 'feature_usage'],
|
||||
};
|
||||
|
||||
// Custom provider that logs assignments to an analytics platform
|
||||
const analyticsProvider: ExperimentProvider = {
|
||||
name: 'analytics',
|
||||
trackAssignment(assignment: Assignment) {
|
||||
// forward to external analytics service
|
||||
console.log('assigned', assignment.variantId, 'to', assignment.userId);
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `Variant`
|
||||
|
||||
| Property | Type | Description |
|
||||
|---|---|---|
|
||||
| `id` | `string` | Stable identifier |
|
||||
| `name` | `string` | Display name |
|
||||
| `weight` | `number` | Relative traffic weight (≤ 0 skipped) |
|
||||
| `isControl` | `boolean?` | Marks the baseline variant |
|
||||
|
||||
### `TargetingRule`
|
||||
|
||||
| Property | Type | Description |
|
||||
|---|---|---|
|
||||
| `attribute` | `string` | Key into `UserAttributes` |
|
||||
| `operator` | `'eq' \| 'neq' \| 'in' \| 'notIn' \| 'gt' \| 'lt' \| 'gte' \| 'lte'` | Comparison operator |
|
||||
| `value` | `unknown` | Right-hand side value |
|
||||
|
||||
### `ExperimentDefinition`
|
||||
|
||||
Full descriptor of an experiment. Only `'running'` experiments accept new
|
||||
assignments; all other statuses throw `AB_EXPERIMENT_INACTIVE`.
|
||||
|
||||
### `Assignment`
|
||||
|
||||
Immutable result of `ExperimentManager.assign()`. When `inExperiment` is
|
||||
`false` (due to targeting or sample-rate exclusion), `variantId` is `''` and
|
||||
`isControl` is `false`.
|
||||
|
||||
### `MetricEvent`
|
||||
|
||||
Conversion event passed to `ExperimentManager.track()`. `value` defaults to
|
||||
`1` and `timestamp` defaults to `Date.now()` when omitted by the caller.
|
||||
|
||||
### `ExperimentManagerOptions`
|
||||
|
||||
Dependency-injection bag. All fields are optional; sensible defaults are
|
||||
provided by the manager implementation.
|
||||
|
||||
### `ExperimentProvider`
|
||||
|
||||
Plug-in interface. `getAssignment` can return a forced variant ID (useful for
|
||||
QA overrides) or `null` to fall through to normal bucketing.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
All interfaces use `readonly` modifiers on every property to prevent accidental
|
||||
mutation of assignment or experiment records after creation. The `variants`
|
||||
array on `ExperimentDefinition` is typed as `ReadonlyArray<Variant>` so that
|
||||
freezing at the API boundary requires no explicit cast.
|
||||
|
||||
`UserAttributes` intentionally allows only `string | number | boolean` values
|
||||
to keep targeting evaluation simple and deterministic without JSON serialisation
|
||||
of complex objects.
|
||||
@@ -0,0 +1,107 @@
|
||||
# schemas.ts — Zod Validation Schemas
|
||||
|
||||
## Purpose
|
||||
|
||||
Provides Zod validation schemas for all public input types in
|
||||
`@suaveplan/ab-testing`. These schemas serve as the single runtime validation
|
||||
layer between untrusted external input (e.g. feature-flag JSON files, API
|
||||
payloads, user-supplied configs) and the strongly-typed domain objects used
|
||||
internally by the experiment manager.
|
||||
|
||||
By keeping schemas in a dedicated module, the validation logic is reusable
|
||||
across REST adapters, config loaders, and import utilities without coupling
|
||||
any of them to each other.
|
||||
|
||||
## Features
|
||||
|
||||
- `VariantSchema` — validates variant id, name, weight, and optional isControl
|
||||
- `TargetingRuleSchema` — validates attribute, all eight operator literals, and
|
||||
an open `unknown` value field
|
||||
- `ExperimentDefinitionSchema` — full experiment definition with all optional
|
||||
fields validated (sampleRate clamped to `[0, 1]`, variants array non-empty)
|
||||
- `MetricEventSchema` — metric name, optional numeric value, optional
|
||||
timestamp, and optional free-form properties map
|
||||
- Inferred TypeScript types (`VariantInput`, `TargetingRuleInput`,
|
||||
`ExperimentDefinitionInput`, `MetricEventInput`) derived directly from
|
||||
the schemas to eliminate duplication
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { ExperimentDefinitionSchema } from '@suaveplan/ab-testing';
|
||||
|
||||
const raw = JSON.parse(fs.readFileSync('experiment.json', 'utf-8'));
|
||||
const result = ExperimentDefinitionSchema.safeParse(raw);
|
||||
|
||||
if (!result.success) {
|
||||
console.error(result.error.format());
|
||||
} else {
|
||||
manager.defineExperiment(result.data);
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import z from 'zod';
|
||||
import {
|
||||
ExperimentDefinitionSchema,
|
||||
MetricEventSchema,
|
||||
} from '@suaveplan/ab-testing';
|
||||
|
||||
// Extend the base schema with domain-specific constraints
|
||||
const StrictExperimentSchema = ExperimentDefinitionSchema.refine(
|
||||
(exp) => exp.variants.some((v) => v.isControl),
|
||||
{ message: 'At least one variant must be the control' }
|
||||
).refine(
|
||||
(exp) => exp.variants.every((v) => v.weight >= 0),
|
||||
{ message: 'All variant weights must be non-negative' }
|
||||
);
|
||||
|
||||
// Parse a metric event from an incoming webhook body
|
||||
function parseMetric(body: unknown) {
|
||||
return MetricEventSchema.parse(body);
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `VariantSchema`
|
||||
|
||||
Zod object schema. Requires `id` (non-empty string), `name` (non-empty
|
||||
string), `weight` (any number — negative values are valid input but skipped
|
||||
during bucketing), and optional `isControl` boolean.
|
||||
|
||||
### `TargetingRuleSchema`
|
||||
|
||||
Validates `attribute` (non-empty string), `operator` (one of the eight
|
||||
supported literals), and `value` (any type — runtime comparison is handled
|
||||
by the targeting module).
|
||||
|
||||
### `ExperimentDefinitionSchema`
|
||||
|
||||
Full experiment descriptor. `sampleRate` is constrained to `[0, 1]` by the
|
||||
schema. `variants` must contain at least one entry. All date fields are plain
|
||||
strings; format validation (ISO-8601) is left to the application layer.
|
||||
|
||||
### `MetricEventSchema`
|
||||
|
||||
`name` is required and non-empty. `value` defaults to `undefined` (the
|
||||
manager treats this as `1` for conversion counting). `properties` is an
|
||||
open record accepting any JSON-compatible value.
|
||||
|
||||
### Inferred Types
|
||||
|
||||
`VariantInput`, `TargetingRuleInput`, `ExperimentDefinitionInput`, and
|
||||
`MetricEventInput` are `z.infer<>` aliases of the corresponding schemas.
|
||||
They are interchangeable with the handwritten interfaces in `common.ts`
|
||||
because Zod inference produces structurally compatible types.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
All schemas use `.optional()` for nullable/absent fields rather than
|
||||
`.nullable()` because the domain model uses `undefined` rather than `null`
|
||||
for absent fields, keeping serialisation round-trips clean. The `value` field
|
||||
on `TargetingRuleSchema` intentionally uses `z.unknown()` to allow the full
|
||||
range of operator comparisons (arrays for `in`/`notIn`, numbers for
|
||||
`gt`/`lt`/`gte`/`lte`, and arbitrary scalars for `eq`/`neq`).
|
||||
Reference in New Issue
Block a user