Add @suaveplan/storage-mux — multi-backend storage failover / shielding mux

Package: @suaveplan/storage-mux Status: PROPOSED Type: new-package (genesis-original)

Design: ./design.md
Spec delta: ./specs/storage-mux/spec.md

Why

@suaveplan/storage-cdn (archived 2026-05-16) explicitly deferred origin shielding and multi-origin failover as out-of-scope, naming storage-mux as the future home (openspec/archive/2026-05-16-add-storage-cdn/proposal.md "Out of scope": "Origin shielding / multi-origin failover. A failover/shielding layer is a separate concern composed via storage-mux (future) or application logic."). The deferral was correct — failover semantics are orthogonal to CDN-specific surface (purge/signing/edgeUrl) and belong in a composable layer over the bare AbstractStorage contract.

storage-mux is a universal-tier composition layer: it accepts an ordered list of StorageLike backends and a policy (failover / round-robin / sticky-by-key / shield) and exposes a single StorageLike to consumers. Backends can be any storage adapter — storage-cdn-*, storage-s3, storage-http, storage-file, in-memory — making the mux the canonical primitive for CDN failover, origin shielding, mirror reads, and shard distribution.

≥2-consumer justification (per `openspec/AGENTS.md` §0 step 6)

Three forward-declared scenarios consume storage-mux directly:

CDN failover for user-uploaded content — primary storage-cdn-bunnycdn with storage-cdn-rackspace fallback. When BunnyCDN edge degrades, reads transparently fall through. The failover policy.
Cache mirroring for storage-plugin-cache — local storage-file cache layered in front of remote storage-s3 origin via the shield policy. Hits the fast tier first, falls through to origin on miss, writes to both.
Drive chunk shielding — drive-plugin-chunks reads chunks from a regional mirror set with sticky-by-key so the same chunk always hits the same backend for cache warmth, plus failover when a mirror is down.

All three scenarios share the same primitive (ordered backend list + policy + health), differing only in policy. Without a shared mux, each consumer reinvents the failover state machine and the registry of STORAGE_MUX.* codes would be scattered.

What changes

ADDED: StorageMux class extending AbstractStorage from @suaveplan/storage. Wraps an ordered backend list and a policy and exposes the full StorageLike surface (get, set, setPartial, has, delete, list, stat, getStream, setStream).
ADDED: MuxPolicy discriminated union — "failover" | "round-robin" | "sticky-by-key" | "shield". Each policy is implemented as a strategy class behind a common BackendSelector interface.
ADDED: BackendEntry interface — { name: string; storage: StorageLike; weight?: number; healthcheck?: (signal: AbortSignal) => Promise<void> }.
ADDED: BackendHealth tracker — passive degrade (mark down on transient error, exponential cooldown via injected Clock) + optional active polling via healthcheck callbacks scheduled on Clock.
ADDED: StorageMuxOptions — { backends: BackendEntry[]; policy: MuxPolicy; retry?: RetryStrategy; clock?: Clock; events?: EventBusLike; logger?: LoggerLike; tracer?: TracerLike; cooldownMs?: number; healthcheckIntervalMs?: number }.
ADDED: STORAGE_MUX_CODES via defineCodes("STORAGE_MUX", [...]) — 8 codes: ALL_BACKENDS_DOWN, NO_BACKENDS_CONFIGURED, HEALTHCHECK_FAILED, BACKEND_TIMEOUT, INVALID_POLICY, INVALID_WEIGHT, WRITE_FANOUT_PARTIAL, BACKEND_ABORTED.
ADDED: StorageMuxError extends SuaveplanError<StorageMuxErrorCode> with backendName, attemptIndex, attemptCount discriminants.
ADDED: Retry composition — every backend call routed through @suaveplan/retry (runWithRetry + injected strategy). The mux never reimplements backoff; it composes.
ADDED: AbortSignal threaded through every backend call. A caller-aborted operation must surface BACKEND_ABORTED, not ALL_BACKENDS_DOWN.
ADDED: Observability — EventBusLike emissions on backend down/recovery/failover; OTel span attributes per call (mux.backend.name, mux.attempt.index, mux.policy).
ADDED: Conformance harness integration — runStorageConformance (from @suaveplan/storage) executed against a StorageMux configured with N MemoryStorage backends and each policy.

Out of scope

Provider-specific backends. The mux composes StorageLike values handed in by the caller; it does not instantiate CDN clients or know provider authentication.
Cross-backend replication / consistency. Writes under shield are best-effort fan-out with WRITE_FANOUT_PARTIAL surfacing partial failures — there is no two-phase commit, no quorum, no vector clocks. Stronger consistency is @suaveplan/vault-sync territory.
Geo-aware routing. Backend selection is by ordered list + policy. Geographic affinity belongs in a higher-level router that constructs the backend list per-request.
Adaptive policy switching. The policy is fixed at construction; the mux never silently switches failover → round-robin based on load.

Source mapping

Genesis-original. No upstream in ../suaveplan/. The design draws from:

openspec/archive/2026-05-16-add-storage-cdn/proposal.md — the explicit deferral naming storage-mux.
Industry conventions: HAProxy / Envoy passive+active health checks; HashiCorp Consul service-mesh failover; AWS Route 53 health-check failover routing; NGINX upstream max_fails / fail_timeout. The four policies (failover, round-robin, sticky-by-key, shield) are the union of what those tools offer in the storage-IO dimension.

Requirements with no source line

Requirement	Justification
Pluggable `RetryStrategy` via `@suaveplan/retry`	Reimplementing retry in the mux would duplicate `retry`'s state machine and diverge from the rest of the monorepo. Composition keeps the contract uniform.
Backend isolation (slow primary cannot block fallback)	A timeout-aware race or per-backend abort is required because a hung primary that never rejects starves the failover chain. The design uses `Promise.race` against a clock-driven timeout, not a parallel call — parallel-call shielding (`shield` writes) is opt-in.
`WRITE_FANOUT_PARTIAL` distinct from `ALL_BACKENDS_DOWN`	`shield` writes intentionally fan out to mirrors; one mirror down does not fail the operation, but the caller deserves a signal. A separate code lets callers choose to surface or swallow it.
`sticky-by-key` uses a stable hash, not a Map cache	Cache memory grows unbounded under high-cardinality keys. Hashing keeps the policy O(1) memory regardless of key population.

Verification approach

src/mux/mux.test.ts — construction validates non-empty backend list, policy whitelist, weights.
src/policies/failover/failover.test.ts — fall-through on transient error; surface non-transient; respect AbortSignal; ALL_BACKENDS_DOWN after exhausting list.
src/policies/round-robin/round-robin.test.ts — even distribution under N calls; skip down backends.
src/policies/sticky-by-key/sticky-by-key.test.ts — same key → same backend across calls; redistribute when backend down.
src/policies/shield/shield.test.ts — reads hit primary first then mirrors; writes fan out; WRITE_FANOUT_PARTIAL on partial mirror failure.
src/health/health.test.ts — passive degrade + exponential cooldown; active polling via injected Clock; recovery emits event.
src/mux/mux.spec-conformance.test.ts — runStorageConformance passes with 3× MemoryStorage backends under every policy.
src/error-codes/error-codes.test.ts — every STORAGE_MUX_* code reachable from a real call site.

Impact

Affected dependents

Run bunx turbo ls --filter=...@suaveplan/storage-mux to enumerate. The package is new — no current dependents.

No current dependents (verified by bunx turbo ls --filter=...@suaveplan/storage-mux returning the package itself).
Forward-declared consumers: @suaveplan/storage-plugin-cache (shield mode), @suaveplan/drive-plugin-chunks (sticky-by-key), application code composing storage-cdn-bunnycdn + storage-cdn-rackspace (failover).

Affected upstreams

@suaveplan/storage — consumed via AbstractStorage + StorageLike + runStorageConformance. No augmentation. [x] no action needed.
@suaveplan/retry — consumed via RetryStrategy + runWithRetry. No augmentation. [x] no action needed.
@suaveplan/error, @suaveplan/events, @suaveplan/types — standard foundation deps. [x] no action needed.

8.6 KiB Raw Blame History Unescape Escape