suaveplan/openspec/changes/complete-storage-mux/proposal.md

Complete @suaveplan/storage-mux — feature-complete multi-backend storage facade

Package: @suaveplan/storage-mux Status: PROPOSED Type: new-package (genesis-original)

• Design: ./design.md
• Spec delta: ./specs/storage-mux/spec.md

Why

@suaveplan/storage-cdn (archived 2026-05-16) explicitly deferred origin shielding and multi-origin failover as out-of-scope, naming storage-mux as the future home (openspec/archive/2026-05-16-add-storage-cdn/proposal.md "Out of scope": "Origin shielding / multi-origin failover. A failover/shielding layer is a separate concern composed via storage-mux (future) or application logic."). The deferral was correct — failover semantics are orthogonal to CDN-specific surface (purge/signing/edgeUrl) and belong in a composable layer over the bare AbstractStorage contract.

The first draft of this change (openspec/changes/_superseded-narrow-deferrals-2026-05-17/add-storage-mux/) scoped four policies (failover, round-robin, sticky-by-key, shield) and was archived as part of a deferral sweep. This rewrite widens the surface to the full set required by real consumers — weighted distribution, mirror-mode (write-fan-out + read-first), runtime mutation, drain, bulkhead timeout, stats API, recursive composition, and a per-attempt OTel span hierarchy — so the package ships once as a feature-complete primitive instead of accreting modes over six follow-up changes.

storage-mux is a universal-tier composition layer: it accepts a declarative policy plus an ordered list of StorageLike backends and exposes a single StorageLike to consumers. Backends can be any storage adapter — storage-cdn-*, storage-s3, storage-http, storage-file, in-memory, or another StorageMux — making the mux the canonical primitive for CDN failover, origin shielding, mirror reads, shard distribution, and tiered cache hierarchies.

≥2-consumer justification (per openspec/AGENTS.md §0 step 6)

Forward-declared consumers that share the mux primitive:

1. CDN failover for user-uploaded content — primary storage-cdn-bunnycdn with storage-cdn-rackspace fallback (failover mode).
2. Cache mirroring for storage-plugin-cache — local storage-file cache in front of remote storage-s3 origin (shield mode).
3. Drive chunk shielding — drive-plugin-chunks reads chunks from a regional mirror set (sticky-by-key + failover).
4. Multi-region write durability — application code writes media to three S3 buckets in parallel (mirror mode), surviving any single-region outage.
5. Weighted canary — gradually shift 10% / 25% / 50% of write traffic to a new backend before cutover (weighted mode + runtime weight mutation + drain on rollback).

All five scenarios share the same primitive (backend list + mode + health + retry); without a shared mux each consumer reinvents the same state machine and the STORAGE_MUX.* registry fragments.

What changes

• ADDED: StorageMux class extending AbstractStorage from @suaveplan/storage. Wraps the configured backend list and mode and exposes the full StorageLike surface (get, set, setPartial, has, delete, list, stat, getStream, setStream, capabilities).
• ADDED: MuxMode discriminated string union — "failover" | "round-robin" | "weighted" | "sticky-by-key" | "shield" | "mirror". Each mode is implemented as a BackendSelector / BackendExecutor strategy class behind a common interface.
• ADDED: BackendEntry interface — { name: string; storage: StorageLike; weight?: number; healthcheck?: (signal: AbortSignal) => Promise<void>; tags?: readonly string[] }.
• ADDED: StorageMuxOptions — declarative policy bundle: { backends, mode, retry?, clock?, events?, logger?, tracer?, cooldownMs?, maxCooldownMs?, healthcheckIntervalMs?, backendTimeoutMs?, hashFn?, throwOnFanoutPartial?, stats? }.
• ADDED: Six execution modes (full design in design.md §3) including weighted (Walker alias-method O(1) selection over weight field) and mirror (write-fan-out via Promise.allSettled; read-first-success via cascading race).
• ADDED: BackendHealth tracker with passive degrade (mark down on classified-transient error class, exponential cooldown capped at maxCooldownMs) and active polling (configurable healthcheck fn invoked on injected Clock interval).
• ADDED: Error classification table (design.md §4): transient → retry+fall-through; semantic → surface as-is; abort → surface as BACKEND_ABORTED. Pure function classify(error: unknown) => "transient" | "semantic" | "abort".
• ADDED: Retry composition via @suaveplan/retry (runWithRetry + injected strategy). The mux never reimplements backoff, jitter, or attempt counting.
• ADDED: Bulkhead — every backend call wrapped in Promise.race against a Clock-driven backendTimeoutMs so a hung primary cannot block siblings or starve fan-out. Underlying promise is allowed to settle in the background; the mux never awaits it.
• ADDED: AbortSignal.any composition linking user signal + per-attempt timeout signal + drain signal; the combined signal is forwarded to every backend call.
• ADDED: Backend isolation — failure, hang, or throw inside one backend MUST NOT corrupt the mux's internal state (health table, round-robin counter, weighted-mode alias table, stats counters). Per-backend state is mutated only through atomic helpers exercised by stress tests.
• ADDED: Recursive composition — a StorageMux is itself a StorageLike and SHALL be acceptable as a BackendEntry.storage. The conformance harness exercises a two-level mux tree.
• ADDED: Runtime mutation API — addBackend(entry), removeBackend(name), setWeight(name, weight), drain(name, { signal? }). Mutation is atomic relative to in-flight operations; a drained backend completes its outstanding calls before being removed.
• ADDED: Drain semantics — drain(name) marks the backend as draining, removes it from new-selection eligibility, awaits all in-flight ops (with caller-visible promise), then transitions to down and removes from the active set.
• ADDED: Stats API — mux.stats() returns per-backend counters (hits, misses, errors, timeouts, lastLatencyMs, p50LatencyMs, p99LatencyMs, consecutiveFailures, lastTransitionAt). Counters are sampled, not synchronous-locked.
• ADDED: Observability:
  • OTel per-backend-attempt span as a child of the operation span, attributes mux.backend.name, mux.attempt.index, mux.mode, mux.outcome, mux.latency.ms.
  • EventBusLike emissions on storage-mux.backend.down, .recovered, .failover, .fanout-partial, .drain.started, .drain.completed, .backend.added, .backend.removed, .weight.changed.
• ADDED: STORAGE_MUX_CODES via defineCodes("STORAGE_MUX", [...]) — comprehensive registry (ALL_BACKENDS_DOWN, NO_BACKENDS_CONFIGURED, HEALTHCHECK_FAILED, BACKEND_TIMEOUT, INVALID_MODE, INVALID_WEIGHT, WRITE_FANOUT_PARTIAL, BACKEND_ABORTED, BACKEND_NOT_FOUND, BACKEND_NAME_CONFLICT, BACKEND_DRAINING, WEIGHTED_REQUIRES_WEIGHTS, MIRROR_ORIGIN_REQUIRED, RECURSION_DEPTH_EXCEEDED).
• ADDED: StorageMuxError extends SuaveplanError<StorageMuxErrorCode> with backendName, attemptIndex, attemptCount, mode discriminants.
• ADDED: runStorageConformance (from @suaveplan/storage) executed against StorageMux with N MemoryStorage backends under every mode, plus a recursive StorageMux(failover, [StorageMux(mirror, [...]), StorageMux(mirror, [...])]) configuration.

Out of scope

• Backend implementations. Each backend stays its own package (storage-cdn-*, storage-s3, storage-http, storage-file, etc.). The mux composes StorageLike values handed in by the caller; it does not instantiate or authenticate any backend.

Source mapping

Genesis-original; no upstream in ../suaveplan/. Heavily reuses openspec/changes/_superseded-narrow-deferrals-2026-05-17/add-storage-mux/ as a starting point (same family conventions, same error registry namespace, same conformance approach) — this draft widens that surface; it does not narrow it. Industry references: HAProxy / Envoy passive+active health checks; HashiCorp Consul service-mesh failover; AWS Route 53 health-check failover routing; NGINX upstream max_fails / fail_timeout; Netflix Hystrix bulkhead.

Verification approach

Every requirement in specs/storage-mux/spec.md carries a **Verified by:** line naming planned test files. Coverage gate: 100/100/100/100. The conformance harness runs runStorageConformance once per mode against a three-backend MemoryStorage fleet, plus once against a two-level recursive mux to prove recursion does not break invariants.

Impact

Affected dependents

Run bunx turbo ls --filter=...@suaveplan/storage-mux to enumerate. The package is new — no current dependents.

• No current dependents (verified by bunx turbo ls --filter=...@suaveplan/storage-mux returning the package itself).
• Forward-declared consumers listed under the consumer justification above; each will land its own OpenSpec change to wire the mux in.

Affected upstreams

• @suaveplan/storage — consumed via AbstractStorage + StorageLike + runStorageConformance. No augmentation. [x] no action needed.
• @suaveplan/retry — consumed via RetryStrategy + runWithRetry. No augmentation. [x] no action needed.
• @suaveplan/error — defineCodes + SuaveplanError. [x] no action needed.
• @suaveplan/events — EventBusLike consumer only. [x] no action needed.
• @suaveplan/types — Clock, LoggerLike, TracerLike, EventBusLike. [x] no action needed.