# Capability Spec Delta: storage-mux (storage-mux) **Package**: `@suaveplan/storage-mux` **Status**: PROPOSED > Every requirement below carries a `**Verified by:**` line at authoring time. No exceptions. ## ADDED Requirements ### Requirement: StorageMux extends AbstractStorage The package SHALL provide a `StorageMux` class extending `AbstractStorage` from `@suaveplan/storage`. The constructor SHALL accept `StorageMuxOptions` containing an ordered non-empty `backends: BackendEntry[]` and a `mode: MuxMode`. The class SHALL implement every method of the `StorageLike` surface (`get`, `set`, `setPartial`, `has`, `delete`, `list`, `stat`, `getStream`, `setStream`) by delegating to a `BackendSelector` chosen from the configured mode. The class SHALL expose `capabilities` reflecting the intersection of backend capabilities. #### Scenario: subclass is an instance of AbstractStorage - **WHEN** a `StorageMux` is constructed with at least one `BackendEntry` - **THEN** the instance is `instanceof AbstractStorage` AND exposes every `StorageLike` method #### Scenario: empty backend list is rejected at construction - **WHEN** a caller invokes `new StorageMux({ backends: [], mode: "failover" })` - **THEN** the constructor throws `StorageMuxError` with `code === "STORAGE_MUX.NO_BACKENDS_CONFIGURED"` #### Scenario: duplicate backend names rejected - **WHEN** two `BackendEntry` values share the same `name` - **THEN** the constructor throws `StorageMuxError` with `code === "STORAGE_MUX.BACKEND_NAME_CONFLICT"` **Verified by:** `src/mux/mux.test.ts::"is instance of AbstractStorage and exposes StorageLike surface"` AND `src/mux/mux.test.ts::"rejects empty backend list with NO_BACKENDS_CONFIGURED"` AND `src/mux/mux.test.ts::"rejects duplicate backend names with BACKEND_NAME_CONFLICT"` --- ### Requirement: MuxMode discriminated union with six modes The package SHALL declare `MuxMode` as the string union `"failover" | "round-robin" | "weighted" | "sticky-by-key" | "shield" | "mirror"`. Each mode SHALL be implemented as a `BackendSelector` strategy class in its own subdirectory under `src/modes/`. The `StorageMuxOptions.mode` field SHALL be validated by a Zod schema that rejects any value outside this whitelist with `STORAGE_MUX.INVALID_MODE`. #### Scenario: unknown mode string is rejected - **WHEN** a caller invokes `new StorageMux({ backends, mode: "magic" as MuxMode })` - **THEN** the constructor throws `StorageMuxError` with `code === "STORAGE_MUX.INVALID_MODE"` **Verified by:** `src/mux/mux.test.ts::"rejects unknown mode with INVALID_MODE"` --- ### Requirement: failover mode falls through on transient errors and surfaces semantic errors Under `failover` the mux SHALL invoke `backends[0]` first. On a transient classified error (per `design.md` §4) the mux SHALL advance to `backends[1]`, then `backends[2]`, and so on. On a semantic error (`STORAGE.KEY_NOT_FOUND`, `STORAGE.INVALID_KEY`, 4xx other than 408/429, `CREDENTIAL_EXPIRED`) the mux SHALL surface the error directly without advancing. When every backend has been tried and the last attempt was transient, the mux SHALL throw `StorageMuxError` with `code === "STORAGE_MUX.ALL_BACKENDS_DOWN"`, the prior error as `cause`, and `attemptCount` equal to the number of backends tried. #### Scenario: transient error advances to next backend - **WHEN** `backends[0].get(key)` rejects with a 503 and `backends[1].get(key)` resolves with bytes - **THEN** the mux returns `backends[1]`'s bytes AND emits a `storage-mux.failover` event naming both backends #### Scenario: semantic KEY_NOT_FOUND is not retried across backends - **WHEN** `backends[0].get(key)` throws `STORAGE.KEY_NOT_FOUND` - **THEN** the mux rethrows `KEY_NOT_FOUND` without invoking `backends[1]` #### Scenario: exhausting all backends throws ALL_BACKENDS_DOWN with cause chain - **WHEN** every backend rejects with a transient error - **THEN** the mux throws `StorageMuxError` with `code === "STORAGE_MUX.ALL_BACKENDS_DOWN"` AND `cause` is the last backend's error AND `attemptCount === backends.length` **Verified by:** `src/modes/failover/failover.test.ts::"falls through to next backend on transient 503"` AND `src/modes/failover/failover.test.ts::"surfaces semantic KEY_NOT_FOUND without fall-through"` AND `src/modes/failover/failover.test.ts::"throws ALL_BACKENDS_DOWN after exhausting list"` --- ### Requirement: round-robin mode distributes across healthy backends Under `round-robin` the mux SHALL maintain a monotonic counter and select `healthyBackends[counter mod healthyBackends.length]` for each call. Backends marked `down` or `draining` by `BackendHealth` SHALL be skipped. When zero backends are healthy the mux SHALL throw `STORAGE_MUX.ALL_BACKENDS_DOWN`. #### Scenario: nine calls across three healthy backends distribute evenly - **WHEN** the caller issues 9 `get` calls against a mux with 3 healthy backends - **THEN** each backend receives exactly 3 calls #### Scenario: down backend is skipped - **WHEN** `backends[1]` is marked down and the caller issues 4 calls - **THEN** `backends[0]` and `backends[2]` each receive 2 calls AND `backends[1]` receives 0 **Verified by:** `src/modes/round-robin/round-robin.test.ts::"distributes nine calls evenly across three backends"` AND `src/modes/round-robin/round-robin.test.ts::"skips backends marked down"` --- ### Requirement: weighted mode selects via Walker alias-method Under `weighted` the mux SHALL build a Walker alias-method table over `BackendEntry.weight` values and select a backend in O(1) per call via two random draws (slot + alias coin). The mux SHALL rebuild the alias table on `addBackend`, `removeBackend`, `setWeight`, and any health transition. Construction SHALL throw `STORAGE_MUX.WEIGHTED_REQUIRES_WEIGHTS` when `mode === "weighted"` AND any `BackendEntry.weight` is `undefined`. The mux SHALL throw `STORAGE_MUX.INVALID_WEIGHT` when any `weight < 0` or non-finite. On a transient error the mux SHALL re-roll up to `healthy.length` times before throwing `ALL_BACKENDS_DOWN`. #### Scenario: weights 1/3/6 distribute roughly 10/30/60 percent over 1000 calls - **WHEN** 1000 calls are issued under a `Rng` seeded for determinism with weights `[1, 3, 6]` - **THEN** each backend's call count is within ±5% of its expected share (100/300/600) #### Scenario: weighted without weights rejects at construction - **WHEN** a caller invokes `new StorageMux({ backends: [{ name: "a", storage }, { name: "b", storage }], mode: "weighted" })` - **THEN** the constructor throws `STORAGE_MUX.WEIGHTED_REQUIRES_WEIGHTS` #### Scenario: negative weight rejected - **WHEN** any backend's weight is `-1` - **THEN** the constructor throws `STORAGE_MUX.INVALID_WEIGHT` **Verified by:** `src/modes/weighted/weighted.test.ts::"distributes calls roughly per weight under seeded rng"` AND `src/modes/weighted/weighted.test.ts::"rejects mode=weighted with any undefined weight as WEIGHTED_REQUIRES_WEIGHTS"` AND `src/modes/weighted/weighted.test.ts::"rejects negative weight with INVALID_WEIGHT"` --- ### Requirement: sticky-by-key mode is deterministic and skips down backends Under `sticky-by-key` the mux SHALL compute `index = fnv1a(utf8(key)) mod healthyBackends.length` and route every operation for that key to the same backend. The hash SHALL be pure-JS FNV-1a (no native or crypto dep). When a backend goes down its keys SHALL be redistributed across the remaining healthy backends via the same formula recomputed against the shrunken set. Operations without a key (e.g. `list`) SHALL fall back to round-robin selection. #### Scenario: same key resolves to same backend across 100 calls - **WHEN** the caller issues 100 `get("alpha")` calls against a mux with 3 healthy backends - **THEN** exactly one backend receives all 100 calls #### Scenario: removed backend triggers redistribution - **WHEN** the backend currently serving `"alpha"` is marked down - **THEN** subsequent `get("alpha")` calls land on one of the remaining healthy backends AND the choice is stable until the health set changes again **Verified by:** `src/modes/sticky-by-key/sticky-by-key.test.ts::"routes the same key to the same backend across 100 calls"` AND `src/modes/sticky-by-key/sticky-by-key.test.ts::"redistributes keys when a backend goes down"` --- ### Requirement: shield mode cascades reads and fans out writes with origin priority Under `shield` the mux SHALL treat `backends[0]` as the origin and `backends[1..n]` as mirrors. Construction SHALL throw `STORAGE_MUX.MIRROR_ORIGIN_REQUIRED` when fewer than two backends are configured. For read operations (`get`, `has`, `stat`, `getStream`) the mux SHALL invoke `backends[0]` first and on transient error or timeout fall through to mirrors in order. For write operations (`set`, `setPartial`, `delete`, `setStream`) the mux SHALL invoke `backends[0]` AND every mirror via `Promise.allSettled`. If the origin write rejects the mux SHALL throw with the origin error as `cause`. If the origin write succeeds but one or more mirror writes reject the mux SHALL emit `storage-mux.fanout-partial` carrying every failed mirror name AND return successfully — unless `StorageMuxOptions.throwOnFanoutPartial === true`, in which case the mux SHALL additionally throw `StorageMuxError` with `code === "STORAGE_MUX.WRITE_FANOUT_PARTIAL"`. #### Scenario: shield with single backend rejected at construction - **WHEN** `new StorageMux({ backends: [origin], mode: "shield" })` - **THEN** the constructor throws `STORAGE_MUX.MIRROR_ORIGIN_REQUIRED` #### Scenario: read falls through to mirror after origin timeout - **WHEN** `backends[0].get(key)` hangs past `backendTimeoutMs` and `backends[1].get(key)` resolves with bytes - **THEN** the mux returns `backends[1]`'s bytes #### Scenario: write succeeds when origin accepts and one mirror rejects (default) - **WHEN** `backends[0].set` resolves and `backends[1].set` rejects with a transient error - **THEN** the mux returns successfully AND emits `storage-mux.fanout-partial` naming `backends[1]` #### Scenario: write throws WRITE_FANOUT_PARTIAL when caller opts in - **WHEN** `throwOnFanoutPartial: true` AND origin succeeds while a mirror rejects - **THEN** the mux throws `StorageMuxError` with `code === "STORAGE_MUX.WRITE_FANOUT_PARTIAL"` #### Scenario: write fails when origin rejects regardless of mirror outcomes - **WHEN** `backends[0].set` rejects with any error - **THEN** the mux throws AND `cause` is the origin error AND mirror outcomes are not consulted **Verified by:** `src/modes/shield/shield.test.ts::"rejects single-backend shield with MIRROR_ORIGIN_REQUIRED"` AND `src/modes/shield/shield.test.ts::"reads fall through to mirror after origin timeout"` AND `src/modes/shield/shield.test.ts::"write succeeds when origin accepts and one mirror rejects"` AND `src/modes/shield/shield.test.ts::"write throws WRITE_FANOUT_PARTIAL when opt-in flag is set"` AND `src/modes/shield/shield.test.ts::"write fails when origin rejects regardless of mirror outcomes"` --- ### Requirement: mirror mode races reads and fans out writes across symmetric peers Under `mirror` every backend is a symmetric peer — no designated origin. Read operations SHALL race healthy backends via `Promise.any`; the first successful result wins; all rejections SHALL produce `STORAGE_MUX.ALL_BACKENDS_DOWN` (cause = `AggregateError` of underlying errors). Write operations SHALL invoke every healthy backend via `Promise.allSettled`; any single rejection SHALL emit `storage-mux.fanout-partial`; all rejections SHALL throw `STORAGE_MUX.ALL_BACKENDS_DOWN`. The `MIRROR_ORIGIN_REQUIRED` code SHALL NOT be thrown in this mode. #### Scenario: read returns first successful peer - **WHEN** `backends[0].get` resolves at 30ms and `backends[1].get` resolves at 5ms - **THEN** the mux returns `backends[1]`'s value (no wait for `backends[0]`) #### Scenario: write fans out to all peers and emits partial when one rejects - **WHEN** writes to backends 0 and 1 succeed but backend 2 rejects - **THEN** the mux returns successfully AND emits `storage-mux.fanout-partial` naming `backends[2]` #### Scenario: all peers reject a read throws ALL_BACKENDS_DOWN - **WHEN** every peer rejects a `get` - **THEN** the mux throws `STORAGE_MUX.ALL_BACKENDS_DOWN` AND `cause instanceof AggregateError` **Verified by:** `src/modes/mirror/mirror.test.ts::"read returns first successful peer"` AND `src/modes/mirror/mirror.test.ts::"write fans out and emits partial on single rejection"` AND `src/modes/mirror/mirror.test.ts::"all peers rejecting read throws ALL_BACKENDS_DOWN with AggregateError cause"` --- ### Requirement: bulkhead — per-backend timeout via Promise.race Every backend invocation SHALL be wrapped in `Promise.race` against a `Clock`-driven timeout of `backendTimeoutMs` (default 30000, configurable per-mux). When the race rejects due to timeout the mux SHALL classify the timeout as transient AND proceed per mode policy. The underlying backend promise MAY remain pending; the mux SHALL NOT await it. A backend whose timeout fires consecutively without recovery SHALL surface `STORAGE_MUX.BACKEND_TIMEOUT` on exhaustion. #### Scenario: hung primary does not delay fallback beyond backendTimeoutMs - **WHEN** `backends[0].get` returns a never-resolving promise and `backends[1].get` resolves in 10ms - **THEN** the caller receives `backends[1]`'s value within `backendTimeoutMs + 50ms` measured against the injected `FakeClock` #### Scenario: exhausted timeouts surface BACKEND_TIMEOUT - **WHEN** every backend hangs past `backendTimeoutMs` - **THEN** the mux throws `STORAGE_MUX.ALL_BACKENDS_DOWN` AND `cause.code === "STORAGE_MUX.BACKEND_TIMEOUT"` **Verified by:** `src/bulkhead/bulkhead.test.ts::"hung primary does not block fallback past backendTimeoutMs"` AND `src/bulkhead/bulkhead.test.ts::"exhausted timeouts surface BACKEND_TIMEOUT in cause chain"` --- ### Requirement: AbortSignal.any composition surfaces BACKEND_ABORTED The per-attempt signal forwarded to each backend SHALL be composed via `AbortSignal.any([userSignal, timeoutSignal, drainSignal])`. When the caller's `AbortSignal` aborts during a mux operation, the mux SHALL surface a `StorageMuxError` with `code === "STORAGE_MUX.BACKEND_ABORTED"` rather than falling through to subsequent backends. The mux SHALL NOT classify a caller-driven abort as a transient backend failure. #### Scenario: caller abort surfaces BACKEND_ABORTED and does not advance - **WHEN** the caller passes a pre-aborted signal to `mux.get(key, signal)` - **THEN** the mux throws `StorageMuxError` with `code === "STORAGE_MUX.BACKEND_ABORTED"` AND `backends[1].get` is NEVER invoked #### Scenario: composed signal aborts backend mid-flight on user cancellation - **WHEN** the user aborts `signal` 5ms after `backends[0].get` begins - **THEN** `backends[0].get`'s received `AbortSignal` reports `aborted === true` within 1 tick **Verified by:** `src/mux/mux.test.ts::"caller abort surfaces BACKEND_ABORTED without falling through"` AND `src/mux/mux.test.ts::"composed AbortSignal propagates user cancellation to in-flight backend"` --- ### Requirement: BackendHealth — passive degrade with capped exponential cooldown The package SHALL maintain a `BackendHealth` tracker per backend. A backend SHALL transition from `up` to `down` on the first classified-transient error. The cooldown SHALL be `min(cooldownMs * 2 ** min(consecutiveFailures, 6), maxCooldownMs)` (default `cooldownMs` 1000, default `maxCooldownMs` 64000). A `down` backend SHALL be skipped by every selector. Once `Clock.now() >= cooldownUntil` the next operation SHALL probe the backend; success transitions it back to `up` and resets `consecutiveFailures`. #### Scenario: failing backend cools down for 2000ms after second failure - **WHEN** a backend rejects twice in succession (default `cooldownMs` 1000) - **THEN** the backend is `down` AND remains skipped until `FakeClock` advances by 2000ms #### Scenario: cooldown is capped at maxCooldownMs - **WHEN** a backend rejects 20 consecutive times with `cooldownMs: 1000, maxCooldownMs: 5000` - **THEN** the cooldown never exceeds 5000ms #### Scenario: successful probe after cooldown restores backend and emits recovered event - **WHEN** `FakeClock` advances past `cooldownUntil` AND the next probe succeeds - **THEN** the backend is `up` AND a `storage-mux.backend.recovered` event is emitted AND `consecutiveFailures` is reset to 0 **Verified by:** `src/health/health.test.ts::"exponential cooldown after consecutive failures"` AND `src/health/health.test.ts::"cooldown capped at maxCooldownMs"` AND `src/health/health.test.ts::"successful probe after cooldown restores backend and emits recovered"` --- ### Requirement: Active healthchecks via injected Clock When `BackendEntry.healthcheck` is provided AND `StorageMuxOptions.healthcheckIntervalMs` is set, the mux SHALL schedule the healthcheck via the injected `Clock.setInterval` and invoke it with a derived `AbortSignal`. A rejecting healthcheck SHALL mark the backend `down` AND emit `storage-mux.backend.down` carrying `STORAGE_MUX.HEALTHCHECK_FAILED` as the cause. The mux SHALL clear all schedules when disposed. #### Scenario: rejecting active healthcheck marks backend down - **WHEN** an active healthcheck rejects with a network error - **THEN** the backend transitions to `down` AND a `storage-mux.backend.down` event is emitted AND the failure cause is `STORAGE_MUX.HEALTHCHECK_FAILED` #### Scenario: dispose cancels scheduled healthchecks - **WHEN** `mux[Symbol.asyncDispose]()` is invoked - **THEN** `FakeClock` advancing past the next interval does NOT invoke the healthcheck **Verified by:** `src/health/health.test.ts::"rejecting active healthcheck marks backend down with HEALTHCHECK_FAILED"` AND `src/health/health.test.ts::"dispose cancels scheduled healthchecks"` --- ### Requirement: Retry composition via @suaveplan/retry — no inline retry Every per-backend invocation SHALL be wrapped by `runWithRetry(retryStrategy, ...)` from `@suaveplan/retry` using the strategy supplied in `StorageMuxOptions.retry` (defaulting to `exponentialBackoff()` from `@suaveplan/retry`). The mux SHALL NOT reimplement backoff, jitter, or attempt counting; all in-backend retry behavior MUST be expressed by composing the retry strategy. Cross-backend fall-through is policy logic; in-backend retry is `runWithRetry`. #### Scenario: backend that fails twice then succeeds is retried in place before fall-through - **WHEN** `backends[0].get` rejects twice then resolves on the third attempt under a 3-attempt retry strategy - **THEN** the mux returns `backends[0]`'s value AND `backends[1].get` is NEVER invoked #### Scenario: no inline retry implementation in package - **WHEN** the package source is grepped for `backoff`, `setTimeout`, or `retryCount` outside `src/health/` - **THEN** the only references are imports from `@suaveplan/retry` **Verified by:** `src/modes/failover/failover.test.ts::"retries in place before falling through to next backend"` AND `src/modes/failover/failover.test.ts::"package contains no inline retry implementation outside health"` --- ### Requirement: Backend isolation — one backend's failure cannot corrupt mux state A throw, hang, or rejection inside one backend SHALL NOT leave shared mux state (`BackendHealth` table, round-robin counter, weighted alias table, stats counters) in a partially-mutated state observable to any other backend. State updates SHALL be made through dedicated keyed helpers; no helper SHALL mutate state for backend B while handling an error from backend A. #### Scenario: throw from backend A leaves backend B health unchanged - **WHEN** `backends[0].get` throws a transient error and `backends[1]` has never been called - **THEN** `mux.stats().backends[1]` shows `status === "up"`, `consecutiveFailures === 0`, `errors === 0` #### Scenario: concurrent in-flight ops on different backends do not interleave state writes - **WHEN** 100 concurrent `get` calls fan out over 4 round-robin backends with one of them throwing - **THEN** `mux.stats().backends` totals exactly equal the issued call count (no lost or double-counted attempts) **Verified by:** `src/mux/mux.test.ts::"throw from one backend leaves siblings health untouched"` AND `src/mux/mux.test.ts::"concurrent calls across backends produce consistent stats totals"` --- ### Requirement: Recursive composition — StorageMux acceptable as a backend A `StorageMux` SHALL itself satisfy `StorageLike` AND be acceptable as a `BackendEntry.storage`. Configurations like `Mux(failover, [Mux(mirror, [...]), Mux(mirror, [...])])` SHALL function correctly. Recursion depth SHALL be capped at 8; constructing a mux whose backend tree exceeds the cap SHALL throw `STORAGE_MUX.RECURSION_DEPTH_EXCEEDED`. #### Scenario: two-level recursive mux passes basic round-trip - **WHEN** a `Mux(failover, [Mux(mirror, [a, b]), Mux(mirror, [c, d])])` is constructed and a value is set then read - **THEN** the round-trip succeeds AND OTel spans show a parent-child hierarchy with both mux levels named #### Scenario: deeply nested mux beyond depth 8 throws RECURSION_DEPTH_EXCEEDED - **WHEN** a caller constructs a mux tree 9 levels deep - **THEN** the outermost constructor throws `STORAGE_MUX.RECURSION_DEPTH_EXCEEDED` **Verified by:** `src/mux/mux.spec-conformance.test.ts::"two-level recursive mux passes round-trip and emits nested spans"` AND `src/mux/mux.test.ts::"nine-level recursive construction throws RECURSION_DEPTH_EXCEEDED"` --- ### Requirement: Runtime mutation API — add / remove / setWeight / drain The mux SHALL expose `addBackend(entry)`, `removeBackend(name)`, `setWeight(name, weight)`, and `drain(name, options?)` methods. Mutations SHALL be atomic relative to in-flight operations: in-flight requests targeting a draining backend complete normally; new requests skip it. `addBackend` SHALL throw `BACKEND_NAME_CONFLICT` on duplicate names. `setWeight` and `removeBackend` SHALL throw `BACKEND_NOT_FOUND` for unknown names. `setWeight` SHALL reject `weight < 0` with `INVALID_WEIGHT`. Each mutation SHALL emit a corresponding event (`backend.added`, `backend.removed`, `weight.changed`). #### Scenario: addBackend with duplicate name rejected - **WHEN** `mux.addBackend({ name: "existing", storage })` is called for a name already present - **THEN** the call throws `STORAGE_MUX.BACKEND_NAME_CONFLICT` AND no event is emitted #### Scenario: setWeight rebuilds alias table under weighted mode - **WHEN** the mode is `weighted` AND `mux.setWeight("b", 10)` doubles backend b's weight - **THEN** subsequent 1000 calls under the same seeded `Rng` show b's share roughly doubled **Verified by:** `src/mutate/mutate.test.ts::"addBackend rejects duplicate name with BACKEND_NAME_CONFLICT"` AND `src/mutate/mutate.test.ts::"setWeight rebuilds weighted alias and shifts distribution"` --- ### Requirement: Drain — graceful backend removal `mux.drain(name, { signal? }): Promise` SHALL mark the named backend as `draining`, remove it from new-selection eligibility, await all in-flight operations to complete, then transition the backend to `down` and remove from the active set. The returned promise SHALL resolve when in-flight count reaches zero (or rejects when the caller-supplied signal aborts). `storage-mux.drain.started` SHALL be emitted on transition; `.drain.completed` on resolution. A backend in `draining` status SHALL surface `BACKEND_DRAINING` to selectors as a re-selection signal (classified transient). #### Scenario: drain awaits in-flight calls before completing - **WHEN** `mux.drain("primary")` is called while one `get` is in flight on `primary` - **THEN** the drain promise does NOT resolve until that `get` settles AND the `storage-mux.drain.completed` event carries a non-zero `drainedDurationMs` #### Scenario: draining backend is skipped by new selections - **WHEN** `mux.drain("primary")` is in progress under `round-robin` - **THEN** new calls bypass `primary` AND distribute across remaining healthy backends **Verified by:** `src/mutate/mutate.test.ts::"drain awaits in-flight operations before completing"` AND `src/mutate/mutate.test.ts::"draining backend is skipped by new round-robin selections"` --- ### Requirement: Stats API — per-backend counters and sampled latencies The mux SHALL expose `mux.stats(): MuxStats` returning a snapshot containing the configured `mode`, aggregate totals (`totalCalls`, `totalFailovers`, `totalFanoutPartials`), and per-backend records with `name`, `status`, `hits`, `misses`, `errors`, `timeouts`, `lastLatencyMs`, `p50LatencyMs`, `p99LatencyMs`, `consecutiveFailures`, `lastTransitionAt`, `inFlightCount`. Latency percentiles SHALL be computed from a 256-slot Vitter Algorithm R reservoir sample to bound memory. #### Scenario: stats reflect a successful call and a failed call - **WHEN** one successful `get` and one failing `get` are issued against a single-backend mux - **THEN** `mux.stats().backends[0].hits === 1` AND `errors === 1` AND `lastLatencyMs > 0` #### Scenario: reservoir bounds memory under high call volume - **WHEN** 100000 calls are issued - **THEN** internal latency-sample arrays never exceed 256 entries per backend **Verified by:** `src/stats/stats.test.ts::"reflects one success and one failure in counters"` AND `src/stats/stats.test.ts::"latency reservoir stays bounded at 256 samples under 100k calls"` --- ### Requirement: STORAGE_MUX error registry — all 14 codes reachable The package SHALL declare `STORAGE_MUX_CODES` via `defineCodes("STORAGE_MUX", [...])` with exactly these 14 codes: `ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_MODE`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`, `BACKEND_NOT_FOUND`, `BACKEND_NAME_CONFLICT`, `BACKEND_DRAINING`, `WEIGHTED_REQUIRES_WEIGHTS`, `MIRROR_ORIGIN_REQUIRED`, `RECURSION_DEPTH_EXCEEDED`. The package SHALL declare `StorageMuxError extends SuaveplanError` with optional `backendName`, `attemptIndex`, `attemptCount`, and `mode` discriminants. Every code SHALL be reachable from a real call site exercised by tests in this package. #### Scenario: every declared code is reachable from a real call site - **WHEN** the test suite is run - **THEN** every code in `STORAGE_MUX_CODES` has at least one test that triggers the call site that throws or surfaces it **Verified by:** `src/error-codes/error-codes.test.ts::"every STORAGE_MUX code is reachable from a real call site"` --- ### Requirement: Observability — EventBusLike events on lifecycle transitions The mux SHALL emit events through an injected `EventBusLike` (from `@suaveplan/types`) on the following lifecycle transitions: `storage-mux.backend.down`, `storage-mux.backend.recovered`, `storage-mux.failover`, `storage-mux.fanout-partial`, `storage-mux.drain.started`, `storage-mux.drain.completed`, `storage-mux.backend.added`, `storage-mux.backend.removed`, `storage-mux.weight.changed`. When no `EventBusLike` is provided the mux SHALL operate normally without emitting (silent fallback). #### Scenario: failover emits storage-mux.failover with both backend names - **WHEN** the failover mode advances from `backends[0]` to `backends[1]` due to a 503 - **THEN** exactly one `storage-mux.failover` event is emitted carrying both backend names AND the underlying error #### Scenario: no bus configured does not throw - **WHEN** the mux is constructed without `events` and a failover occurs - **THEN** the operation completes successfully AND no error is thrown for the missing bus **Verified by:** `src/observability/events.test.ts::"failover emits storage-mux.failover with from and to backend names"` AND `src/observability/events.test.ts::"absent EventBus does not throw on lifecycle transitions"` --- ### Requirement: Observability — OTel parent + per-attempt child spans Every public operation SHALL open a parent OTel span named `storage-mux.` with attributes `mux.mode`, `mux.key` (when present). Every backend attempt SHALL open a child span named `storage-mux.attempt` carrying `mux.backend.name`, `mux.attempt.index` (0-based), `mux.mode`, `mux.outcome` ∈ `{ok, transient, semantic, abort, timeout}`, and `mux.latency.ms`. When no tracer (`TracerLike` from `@suaveplan/types`) is provided the mux SHALL operate normally without recording spans. #### Scenario: failover attempt records two child spans under one parent - **WHEN** the failover mode invokes `backends[0]` (rejects) then `backends[1]` (succeeds) - **THEN** the test tracer records one parent `storage-mux.get` span AND two child `storage-mux.attempt` spans with `mux.attempt.index` `0` and `1` AND `mux.outcome` `transient` and `ok` **Verified by:** `src/observability/otel.test.ts::"records parent op span and one child span per backend attempt with outcome"` --- ### Requirement: StorageLike conformance under every mode and recursive composition The mux SHALL pass `runStorageConformance` (from `@suaveplan/storage`) when configured with three `MemoryStorage` backends under each of the six modes (`failover`, `round-robin`, `weighted`, `sticky-by-key`, `shield`, `mirror`). The mux SHALL additionally pass conformance when configured recursively as `Mux(failover, [Mux(mirror, mems), Mux(mirror, mems)])`. The conformance suite SHALL include `setPartial` sparse-extension and overwrite semantics, capability-honest reporting, and AbortSignal propagation. #### Scenario: conformance passes under all six modes - **WHEN** `runStorageConformance(mux)` runs six times — once per mode — against three MemoryStorage backends - **THEN** every conformance assertion passes for every mode #### Scenario: conformance passes under recursive composition - **WHEN** `runStorageConformance(recursiveMux)` runs against the two-level configuration - **THEN** every conformance assertion passes **Verified by:** `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under failover"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under round-robin"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under weighted"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under sticky-by-key"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under shield"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under mirror"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under recursive failover-of-mirrors"` --- ### Requirement: Tier purity — universal, no Node or DOM imports The package SHALL NOT import from `node:*`, `bun:*`, or any DOM-only API. Hashing SHALL be pure-JS (FNV-1a). Randomness SHALL come from an injectable `Rng` (`@suaveplan/types`). Timeouts SHALL use the injected `Clock`. The package SHALL run unmodified under Node 18+, Bun, Deno, modern browsers, and Cloudflare Workers. #### Scenario: tier-purity grep finds no Node or DOM imports - **WHEN** `grep -rnE "from ['\"](node:|bun:)" packages/storage/storage-mux/src/` is run - **THEN** the output is empty **Verified by:** `src/mux/mux.spec-conformance.test.ts::"src tree contains no node: or bun: imports"`