suaveplan/openspec/changes/_superseded-narrow-deferrals-2026-05-17/add-storage-mux/design.md

Design — @suaveplan/storage-mux

Non-normative companion to proposal.md and specs/storage-mux/spec.md.

1. Role in the family

@suaveplan/storage                 AbstractStorage core (key/value, in-memory)
  ├─ @suaveplan/storage-cdn        CdnAdapter contract
  │    ├─ @suaveplan/storage-cdn-bunnycdn
  │    └─ @suaveplan/storage-cdn-rackspace
  ├─ @suaveplan/storage-s3
  ├─ @suaveplan/storage-http
  ├─ @suaveplan/storage-file
  └─ @suaveplan/storage-mux        Multi-backend composition (THIS PACKAGE)
                                    └─ wraps N StorageLike, exposes 1 StorageLike

storage-mux is a composition package, not a transport. It accepts any list of StorageLike (cdn / s3 / http / file / memory / other mux instances) and produces a single StorageLike. Recursion is intentional — a sharded-then-failed-over deployment is Mux(round-robin, [Mux(failover, shard1), Mux(failover, shard2)]).

2. File layout

src/
  index.ts                                # explicit named re-exports
  error-codes.ts                          # defineCodes("STORAGE_MUX", [...]) + StorageMuxError
  types/
    backend-entry.types.ts                # BackendEntry interface
    mux-options.types.ts                  # StorageMuxOptions, MuxPolicy union
    mux-options.schemas.ts                # Zod boundary schemas
  mux/
    mux.ts                                # StorageMux extends AbstractStorage
    mux.md
    mux.test.ts
    mux.spec-conformance.test.ts          # runStorageConformance × 4 policies
  policies/
    selector.types.ts                     # BackendSelector interface
    failover/
      failover.ts
      failover.md
      failover.test.ts
    round-robin/
      round-robin.ts
      round-robin.md
      round-robin.test.ts
    sticky-by-key/
      sticky-by-key.ts                    # stable hash → backend index
      sticky-by-key.md
      sticky-by-key.test.ts
    shield/
      shield.ts                           # reads cascade, writes fan-out
      shield.md
      shield.test.ts
  health/
    health.ts                             # BackendHealth tracker + cooldown
    health.md
    health.test.ts
  observability/
    events.ts                             # event topic constants + emit helpers
    events.md
    events.test.ts
    otel.ts                               # span attribute helpers
    otel.md
    otel.test.ts
  error-codes/
    error-codes.test.ts                   # reachability sweep

3. Policy state machines

3.1 failover

            ┌──────────────────────────────────────────┐
            ▼                                          │
[idx=0] ─ call backend[idx] ── ok ──► return           │
            │                                          │
            ├── transient error ─────► idx++ ──────────┘
            │
            ├── non-transient error ──► throw (surface as-is)
            │
            └── idx == backends.length ─► throw STORAGE_MUX.ALL_BACKENDS_DOWN
                                          (cause = last transient error)

Per-attempt: runWithRetry(retryStrategy, () => backend[idx].op(...)) so transient errors get the configured retry inside a single backend before falling through.

3.2 round-robin

Atomic counter modulo healthyBackends.length. On ALL_BACKENDS_DOWN, returns the failover error. Down backends skipped via BackendHealth.isUp(name).

3.3 sticky-by-key

const h = stableHash(key) % healthyBackends.length;
return healthyBackends[h];

Hash is FNV-1a over UTF-8 bytes (universal, no crypto dep). When a backend goes down, the modulo redistributes — same key may move to a different backend; this is acceptable for cache workloads and documented.

3.4 shield

read:  backend[0] (origin) ?? backend[1..n] (edge mirrors) ── first success wins
write: Promise.allSettled(backend[0..n].set(...))
       ├── all ok                      ─► return
       ├── primary ok, mirror failed   ─► return + emit WRITE_FANOUT_PARTIAL event
       └── primary failed              ─► throw (cause = primary error)

Distinct from failover because shield reads must hit origin first (origin is canonical), and writes must fan out (mirrors only serve what was written to them). failover is symmetric — any backend can serve any operation.

4. Error classification

Which errors trigger failover vs surface directly:

Error shape	Classification	Triggers fall-through?	Code surfaced
AbortError / signal.aborted	non-transient	NO	STORAGE_MUX.BACKEND_ABORTED
STORAGE.KEY_NOT_FOUND (from get/stat)	semantic	NO	rethrow as-is
STORAGE.INVALID_KEY	semantic	NO	rethrow as-is
Network timeout (configurable backendTimeoutMs)	transient	YES	wrapped, last in cause chain
ECONNRESET / ETIMEDOUT / fetch network error	transient	YES	wrapped
HTTP 5xx from backend	transient	YES	wrapped
HTTP 4xx (except 408/429)	non-transient	NO	rethrow
HTTP 408 / 429	transient	YES	wrapped
STORAGE_CDN.CREDENTIAL_EXPIRED	non-transient	NO	rethrow (caller must rotate)
Unknown thrown value (non-Error)	transient	YES	wrapped

Classification lives in src/policies/classify.ts, pure function over (error: unknown) => "transient" | "semantic" | "abort". Tested independently.

5. Backend-isolation guarantees

A slow primary MUST NOT block a fallback. Mechanism:

1. Per-backend timeout — every backend call wrapped in Promise.race([call, timeoutPromise(backendTimeoutMs)]). Default 30 s, configurable per-mux.
2. AbortSignal forwarding — the per-attempt signal is linked (AbortSignal.any([userSignal, timeoutSignal])) and passed to the backend. A well-behaved backend respects it; a misbehaved backend will eventually reject via the timeout race.
3. shield reads cascade sequentially with timeout — origin gets backendTimeoutMs; on timeout/error, the mux moves to mirror[0], etc. The next backend's call starts before the previous backend's underlying promise has settled (it is left to its own cleanup).
4. shield writes are Promise.allSettled — one slow mirror cannot starve the others. Caller sees the result when all settle OR backendTimeoutMs * 2 elapses (guard against a permanently-hung mirror).

The mux NEVER waits for a hung promise indefinitely — Promise.race with a clock-driven timeout is the contract.

6. Health tracking

BackendHealth per-backend state:

interface BackendState {
  status: "up" | "down";
  consecutiveFailures: number;
  cooldownUntil: number;      // Clock.now() millis
  lastError?: unknown;
}

Transitions:

• up → down: triggered by any classified-transient error. cooldownUntil = now + cooldownMs * 2 ** min(consecutiveFailures, 6) (exponential capped at 64× base). Emits storage-mux.backend.down.
• down → up: triggered when now >= cooldownUntil AND the next selection call probes the backend AND the probe succeeds. Resets consecutiveFailures. Emits storage-mux.backend.recovered.
• active healthcheck: if BackendEntry.healthcheck is provided AND healthcheckIntervalMs is set, the mux schedules Clock.setTimeout to invoke it; failure marks the backend down without waiting for a real request.

Clock is injected (@suaveplan/types#Clock), so tests use FakeClock for deterministic cooldown / healthcheck advancement.

7. Tier purity

Universal — no node:*, no bun:*, no DOM. Hash uses pure-JS FNV-1a. Timeouts use injected Clock. AbortSignal is standard Web API. Conformance harness runs under Node, Bun, browsers, Workers.

8. Trade-offs

• Strategy classes vs. switch statement. Chose strategy classes (BackendSelector interface). Each policy is a separate file ≤ 250 lines, individually tested, and new policies (e.g. weighted-random later) plug in without editing the core StorageMux.
• FNV-1a vs. xxhash for sticky-by-key. FNV-1a is 20 lines of pure JS with no dep; xxhash needs a wasm or noble dep. For backend selection (low-cardinality output), FNV's distribution is sufficient. If a future workload needs better distribution, the hash function is injectable via options.
• shield writes fan out vs. primary-only. Fan-out is the right default because mirrors that don't see writes can't serve reads. Callers who want primary-only writes should use failover instead — the policy choice is the API.
• Recursive Mux. Allowed because StorageMux is itself a StorageLike. The conformance harness verifies recursion does not break invariants. The cost — debugging a deeply-nested mux tree — is mitigated by per-call OTel span attributes naming every backend hop.