suaveplan/openspec/changes/complete-storage-mux/design.md

Design — @suaveplan/storage-mux (complete)

Non-normative companion to proposal.md and specs/storage-mux/spec.md.

1. Role in the family

@suaveplan/storage                 AbstractStorage core (key/value, in-memory)
  ├─ @suaveplan/storage-cdn        CdnAdapter contract
  │    ├─ @suaveplan/storage-cdn-bunnycdn
  │    └─ @suaveplan/storage-cdn-rackspace
  ├─ @suaveplan/storage-s3
  ├─ @suaveplan/storage-http
  ├─ @suaveplan/storage-file
  └─ @suaveplan/storage-mux        Multi-backend composition (THIS PACKAGE)
                                    └─ wraps N StorageLike (incl. other Mux), exposes 1 StorageLike

storage-mux is a composition package, not a transport. A StorageMux is itself a StorageLike, so a sharded-then-failed-over deployment is Mux(round-robin, [Mux(failover, regionA), Mux(failover, regionB)]). Recursion depth is capped at 8 (RECURSION_DEPTH_EXCEEDED) so a configuration cycle cannot DoS the process.

2. File layout

src/
  index.ts                              # explicit named re-exports
  error-codes/
    error-codes.ts                      # defineCodes("STORAGE_MUX", [...]) + StorageMuxError
    error-codes.md
    error-codes.test.ts                 # reachability sweep — every code triggered
  types/
    backend-entry.types.ts              # BackendEntry interface
    mux-options.types.ts                # StorageMuxOptions, MuxMode union, MuxStats
    mux-options.schemas.ts              # Zod boundary schemas
  classify/
    classify.ts                         # error → "transient" | "semantic" | "abort"
    classify.md
    classify.test.ts
  health/
    health.ts                           # BackendHealth tracker + cooldown + active probe scheduler
    health.md
    health.test.ts
  bulkhead/
    bulkhead.ts                         # Promise.race timeout + AbortSignal.any composition
    bulkhead.md
    bulkhead.test.ts
  stats/
    stats.ts                            # per-backend counters + reservoir-sampled latency
    stats.md
    stats.test.ts
  modes/
    selector.types.ts                   # BackendSelector + BackendExecutor interfaces
    failover/
      failover.ts
      failover.md
      failover.test.ts
    round-robin/
      round-robin.ts
      round-robin.md
      round-robin.test.ts
    weighted/
      weighted.ts                       # Walker alias-method
      weighted.md
      weighted.test.ts
    sticky-by-key/
      sticky-by-key.ts                  # FNV-1a → backend index
      sticky-by-key.md
      sticky-by-key.test.ts
    shield/
      shield.ts                         # origin-first reads + best-effort mirror writes
      shield.md
      shield.test.ts
    mirror/
      mirror.ts                         # parallel writes (Promise.allSettled) + read-first-success
      mirror.md
      mirror.test.ts
  mutate/
    mutate.ts                           # addBackend / removeBackend / setWeight / drain
    mutate.md
    mutate.test.ts
  observability/
    events.ts                           # event topic constants + emit helpers
    events.md
    events.test.ts
    otel.ts                             # per-attempt span hierarchy + attributes
    otel.md
    otel.test.ts
  mux/
    mux.ts                              # StorageMux extends AbstractStorage
    mux.md
    mux.test.ts
    mux.spec-conformance.test.ts        # runStorageConformance × 6 modes + 1 recursive

3. Mode state machines

3.1 failover

Sequential fall-through. Per-attempt: runWithRetry(retryStrategy, () => bulkhead(backend[idx].op(...))). Transient errors advance idx; semantic errors surface; abort surfaces BACKEND_ABORTED. Exhausting the list throws ALL_BACKENDS_DOWN with the prior error as cause and attemptCount = backends.length.

3.2 round-robin

Atomic counter modulo healthy.length. Down/draining backends skipped. Zero healthy → ALL_BACKENDS_DOWN.

3.3 weighted

Walker alias-method table rebuilt on addBackend / removeBackend / setWeight / health transition. Selection is O(1) — one random byte for the slot, one for the alias coin. weight < 0 throws INVALID_WEIGHT. Construction without weights on any backend throws WEIGHTED_REQUIRES_WEIGHTS. Fall-through on transient: re-roll up to healthy.length times, then ALL_BACKENDS_DOWN.

3.4 sticky-by-key

index = fnv1a(utf8(key)) % healthy.length. Same key → same backend. Backend removal redistributes via the same formula recomputed against the shrunken set (acceptable cache-warmth trade-off). Operations without a key (e.g. list) round-robin internally.

3.5 shield

backends[0] = origin, backends[1..n] = mirrors.

• read: origin first; on transient/timeout, cascade to mirrors in order; first success wins.
• write: Promise.allSettled over all backends. Origin rejection throws (cause = origin error, mirror outcomes discarded). Origin success + any mirror rejection emits storage-mux.fanout-partial and returns successfully — unless throwOnFanoutPartial: true, then throws WRITE_FANOUT_PARTIAL.

3.6 mirror

Symmetric peers — no designated origin.

• read: Promise.any race across healthy backends; first success wins; all-reject → ALL_BACKENDS_DOWN.
• write: Promise.allSettled across all healthy backends; any rejection emits storage-mux.fanout-partial; all rejections → ALL_BACKENDS_DOWN. MIRROR_ORIGIN_REQUIRED is not thrown in this mode — every backend is equal.

The difference vs shield: shield privileges backends[0] (writes must succeed there); mirror treats every backend identically. Both fan out writes; only mirror races reads in parallel.

4. Error classification

Error shape	Class	Triggers fall-through?	Code surfaced
AbortError / signal.aborted from caller	abort	NO	STORAGE_MUX.BACKEND_ABORTED
STORAGE.KEY_NOT_FOUND	semantic	NO	rethrow as-is
STORAGE.INVALID_KEY	semantic	NO	rethrow as-is
STORAGE_CDN.CREDENTIAL_EXPIRED	semantic	NO	rethrow (caller must rotate)
HTTP 4xx (except 408 / 429)	semantic	NO	rethrow
backendTimeoutMs race wins	transient	YES	wrapped, last in cause chain (BACKEND_TIMEOUT if exhausted)
ECONNRESET / ETIMEDOUT / fetch network error	transient	YES	wrapped
HTTP 5xx	transient	YES	wrapped
HTTP 408 / 429	transient	YES	wrapped
Unknown thrown value (non-Error)	transient	YES	wrapped
BACKEND_DRAINING re-selection signal (internal)	transient	YES	re-select next backend

classify is a pure function exported from src/classify/classify.ts and tested independently with property tests over the boundary shape.

5. Backend isolation guarantees

A slow / hung / throwing backend MUST NOT corrupt mux-wide state.

1. Per-backend timeout — every backend call wrapped in Promise.race([call, timeoutPromise(backendTimeoutMs)]). Default 30 s. Underlying promise is allowed to settle in the background.
2. AbortSignal.any composition — per-attempt signal = AbortSignal.any([userSignal, timeoutSignal, drainSignal]). Forwarded to backend; well-behaved backends abort promptly.
3. State writes are atomic per backend — health table, weighted alias table, round-robin counter, and stats counters mutate through dedicated helpers that take the backend name as a key. A throw inside backend N can never leave backend M in a partially-updated state.
4. mirror / shield writes use Promise.allSettled — never Promise.all; one slow mirror cannot reject the operation early.
5. Recursion safety — when a backend is itself a mux, the parent passes its composite AbortSignal. The child mux is a black box; its internal state is its own.

6. Health tracking

BackendState:

type Status = "up" | "down" | "draining";
interface BackendState {
  status: Status;
  consecutiveFailures: number;
  cooldownUntil: number;          // Clock.now() millis
  lastError?: unknown;
  lastTransitionAt: number;
  inFlightCount: number;          // for drain accounting
}

Transitions:

• up → down: any classified-transient error. cooldownUntil = now + min(cooldownMs * 2 ** min(consecutiveFailures, 6), maxCooldownMs). Emits storage-mux.backend.down.
• down → up: now >= cooldownUntil AND next selection probes AND probe succeeds. Reset consecutiveFailures = 0. Emits storage-mux.backend.recovered.
• * → draining: mux.drain(name). Removed from new selection. When inFlightCount === 0, transitions to down and removes from active set. Emits storage-mux.drain.started and .drain.completed.
• Active healthcheck: if BackendEntry.healthcheck is provided AND StorageMuxOptions.healthcheckIntervalMs is set, the mux schedules Clock.setInterval to invoke it with a derived AbortSignal. A rejecting healthcheck marks down with cause HEALTHCHECK_FAILED.

Clock is injected (@suaveplan/types#Clock); tests use FakeClock.

7. Stats API

interface BackendStats {
  name: string;
  status: "up" | "down" | "draining";
  hits: number;
  misses: number;
  errors: number;
  timeouts: number;
  lastLatencyMs: number;
  p50LatencyMs: number;
  p99LatencyMs: number;
  consecutiveFailures: number;
  lastTransitionAt: number;
  inFlightCount: number;
}
interface MuxStats {
  mode: MuxMode;
  totalCalls: number;
  totalFailovers: number;
  totalFanoutPartials: number;
  backends: readonly BackendStats[];
}

Latencies use a 256-slot reservoir sample (Vitter Algorithm R) so memory is bounded regardless of call volume. Counters use plain number increments — JS is single-threaded; no atomics needed.

8. Runtime mutation

API	Effect
addBackend(entry)	Append to backend list; rebuild weighted alias table if mode is weighted; emit backend.added. Conflict on name → BACKEND_NAME_CONFLICT.
removeBackend(name)	Synonym for drain(name) then immediate purge from list.
setWeight(name, weight)	Mutate BackendEntry.weight; rebuild alias table; emit weight.changed. weight < 0 → INVALID_WEIGHT.
drain(name, { signal? })	Mark draining; await inFlightCount === 0 (with optional caller abort); transition to removed. Returns Promise<void>.

Mutation is atomic relative to in-flight ops: in-flight requests targeting a draining backend complete normally; new requests skip the backend.

9. Observability

9.1 OTel span hierarchy

storage-mux.<operation>          (parent span, attrs: mux.mode, mux.key, mux.policy)
  └─ storage-mux.attempt         (child span per backend attempt)
        attrs: mux.backend.name, mux.attempt.index, mux.mode,
               mux.outcome ∈ {ok, transient, semantic, abort, timeout},
               mux.latency.ms

No-op when no TracerLike is provided.

9.2 Events (via EventBusLike from @suaveplan/types)

Topic	Payload
storage-mux.backend.down	{ name, cause, code }
storage-mux.backend.recovered	{ name, downForMs }
storage-mux.failover	{ from, to, attemptIndex, cause }
storage-mux.fanout-partial	{ mode, failed: readonly string[], causes: readonly unknown[] }
storage-mux.drain.started	{ name, inFlightCount }
storage-mux.drain.completed	{ name, drainedDurationMs }
storage-mux.backend.added	{ name, weight? }
storage-mux.backend.removed	{ name }
storage-mux.weight.changed	{ name, from, to }

No-op when no bus is provided.

10. Tier purity

Universal — no node:*, no bun:*, no DOM. Hashing is pure-JS FNV-1a. Randomness for weighted uses an injectable Rng from @suaveplan/types (defaults to Math.random). Timeouts use injected Clock. AbortSignal.any is standard Web API (ES2024); a tiny shim is included for runtimes that lack it.

11. Trade-offs

• Six modes vs. four. Adding weighted and mirror covers the canary and multi-region-write use cases without forcing callers to wrap failover in custom logic. Each mode is ≤ 250 lines; the cost is six test files instead of four.
• Walker alias-method for weighted. O(1) selection beats roulette-wheel O(n) under runtime mutation. The cost is O(n) rebuild on weight changes — acceptable for backend counts ≤ 100.
• Recursive composition. Allowed because StorageMux is itself StorageLike. Depth capped at 8 to bound debugging surface and prevent configuration loops. Span hierarchy makes nested traces legible.
• Stats use plain counters. JS single-threaded execution makes atomics unnecessary; the reservoir sample bounds memory at 256 floats per backend.
• Drain returns a promise. Callers can await mux.drain("old-backend") before tearing down credentials — no race between drain completion and credential rotation.