# Complete @suaveplan/storage-mux — feature-complete multi-backend storage facade

**Package**: `@suaveplan/storage-mux`
**Status**: PROPOSED
**Type**: new-package (genesis-original)

- **Design**: ./design.md
- **Spec delta**: ./specs/storage-mux/spec.md

## Why

`@suaveplan/storage-cdn` (archived 2026-05-16) explicitly deferred origin shielding and multi-origin failover as out-of-scope, naming `storage-mux` as the future home (`openspec/archive/2026-05-16-add-storage-cdn/proposal.md` "Out of scope": *"Origin shielding / multi-origin failover. A failover/shielding layer is a separate concern composed via `storage-mux` (future) or application logic."*). The deferral was correct — failover semantics are orthogonal to CDN-specific surface (purge/signing/edgeUrl) and belong in a composable layer over the bare `AbstractStorage` contract.

The first draft of this change (`openspec/changes/_superseded-narrow-deferrals-2026-05-17/add-storage-mux/`) scoped four policies (failover, round-robin, sticky-by-key, shield) and was archived as part of a deferral sweep. This rewrite **widens** the surface to the full set required by real consumers — weighted distribution, mirror-mode (write-fan-out + read-first), runtime mutation, drain, bulkhead timeout, stats API, recursive composition, and a per-attempt OTel span hierarchy — so the package ships once as a feature-complete primitive instead of accreting modes over six follow-up changes.

`storage-mux` is a **universal-tier composition layer**: it accepts a declarative policy plus an ordered list of `StorageLike` backends and exposes a single `StorageLike` to consumers. Backends can be any storage adapter — `storage-cdn-*`, `storage-s3`, `storage-http`, `storage-file`, in-memory, **or another `StorageMux`** — making the mux the canonical primitive for CDN failover, origin shielding, mirror reads, shard distribution, and tiered cache hierarchies.

### ≥2-consumer justification (per `openspec/AGENTS.md` §0 step 6)

Forward-declared consumers that share the mux primitive:

1. **CDN failover for user-uploaded content** — primary `storage-cdn-bunnycdn` with `storage-cdn-rackspace` fallback (`failover` mode).
2. **Cache mirroring for `storage-plugin-cache`** — local `storage-file` cache in front of remote `storage-s3` origin (`shield` mode).
3. **Drive chunk shielding** — `drive-plugin-chunks` reads chunks from a regional mirror set (`sticky-by-key` + `failover`).
4. **Multi-region write durability** — application code writes media to three S3 buckets in parallel (`mirror` mode), surviving any single-region outage.
5. **Weighted canary** — gradually shift 10% / 25% / 50% of write traffic to a new backend before cutover (`weighted` mode + runtime weight mutation + drain on rollback).

All five scenarios share the same primitive (backend list + mode + health + retry); without a shared mux each consumer reinvents the same state machine and the `STORAGE_MUX.*` registry fragments.

## What changes

- ADDED: `StorageMux` class extending `AbstractStorage` from `@suaveplan/storage`. Wraps the configured backend list and mode and exposes the full `StorageLike` surface (`get`, `set`, `setPartial`, `has`, `delete`, `list`, `stat`, `getStream`, `setStream`, `capabilities`).
- ADDED: `MuxMode` discriminated string union — `"failover" | "round-robin" | "weighted" | "sticky-by-key" | "shield" | "mirror"`. Each mode is implemented as a `BackendSelector` / `BackendExecutor` strategy class behind a common interface.
- ADDED: `BackendEntry` interface — `{ name: string; storage: StorageLike; weight?: number; healthcheck?: (signal: AbortSignal) => Promise<void>; tags?: readonly string[] }`.
- ADDED: `StorageMuxOptions` — declarative policy bundle: `{ backends, mode, retry?, clock?, events?, logger?, tracer?, cooldownMs?, maxCooldownMs?, healthcheckIntervalMs?, backendTimeoutMs?, hashFn?, throwOnFanoutPartial?, stats? }`.
- ADDED: Six execution modes (full design in `design.md` §3) including `weighted` (Walker alias-method O(1) selection over `weight` field) and `mirror` (write-fan-out via `Promise.allSettled`; read-first-success via cascading race).
- ADDED: `BackendHealth` tracker with **passive degrade** (mark down on classified-transient error class, exponential cooldown capped at `maxCooldownMs`) **and active polling** (configurable `healthcheck` fn invoked on injected `Clock` interval).
- ADDED: Error classification table (`design.md` §4): `transient` → retry+fall-through; `semantic` → surface as-is; `abort` → surface as `BACKEND_ABORTED`. Pure function `classify(error: unknown) => "transient" | "semantic" | "abort"`.
- ADDED: Retry composition via `@suaveplan/retry` (`runWithRetry` + injected strategy). The mux never reimplements backoff, jitter, or attempt counting.
- ADDED: **Bulkhead** — every backend call wrapped in `Promise.race` against a `Clock`-driven `backendTimeoutMs` so a hung primary cannot block siblings or starve fan-out. Underlying promise is allowed to settle in the background; the mux never awaits it.
- ADDED: **AbortSignal.any composition** linking user signal + per-attempt timeout signal + drain signal; the combined signal is forwarded to every backend call.
- ADDED: **Backend isolation** — failure, hang, or throw inside one backend MUST NOT corrupt the mux's internal state (health table, round-robin counter, weighted-mode alias table, stats counters). Per-backend state is mutated only through atomic helpers exercised by stress tests.
- ADDED: **Recursive composition** — a `StorageMux` is itself a `StorageLike` and SHALL be acceptable as a `BackendEntry.storage`. The conformance harness exercises a two-level mux tree.
- ADDED: **Runtime mutation API** — `addBackend(entry)`, `removeBackend(name)`, `setWeight(name, weight)`, `drain(name, { signal? })`. Mutation is atomic relative to in-flight operations; a drained backend completes its outstanding calls before being removed.
- ADDED: **Drain semantics** — `drain(name)` marks the backend as `draining`, removes it from new-selection eligibility, awaits all in-flight ops (with caller-visible promise), then transitions to `down` and removes from the active set.
- ADDED: **Stats API** — `mux.stats()` returns per-backend counters (`hits`, `misses`, `errors`, `timeouts`, `lastLatencyMs`, `p50LatencyMs`, `p99LatencyMs`, `consecutiveFailures`, `lastTransitionAt`). Counters are sampled, not synchronous-locked.
- ADDED: **Observability**:
  - OTel **per-backend-attempt span** as a child of the operation span, attributes `mux.backend.name`, `mux.attempt.index`, `mux.mode`, `mux.outcome`, `mux.latency.ms`.
  - `EventBusLike` emissions on `storage-mux.backend.down`, `.recovered`, `.failover`, `.fanout-partial`, `.drain.started`, `.drain.completed`, `.backend.added`, `.backend.removed`, `.weight.changed`.
- ADDED: `STORAGE_MUX_CODES` via `defineCodes("STORAGE_MUX", [...])` — comprehensive registry (`ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_MODE`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`, `BACKEND_NOT_FOUND`, `BACKEND_NAME_CONFLICT`, `BACKEND_DRAINING`, `WEIGHTED_REQUIRES_WEIGHTS`, `MIRROR_ORIGIN_REQUIRED`, `RECURSION_DEPTH_EXCEEDED`).
- ADDED: `StorageMuxError extends SuaveplanError<StorageMuxErrorCode>` with `backendName`, `attemptIndex`, `attemptCount`, `mode` discriminants.
- ADDED: `runStorageConformance` (from `@suaveplan/storage`) executed against `StorageMux` with N `MemoryStorage` backends under **every** mode, plus a recursive `StorageMux(failover, [StorageMux(mirror, [...]), StorageMux(mirror, [...])])` configuration.

## Out of scope

- **Backend implementations.** Each backend stays its own package (`storage-cdn-*`, `storage-s3`, `storage-http`, `storage-file`, etc.). The mux composes `StorageLike` values handed in by the caller; it does not instantiate or authenticate any backend.

## Source mapping

Genesis-original; no upstream in `../suaveplan/`. Heavily reuses `openspec/changes/_superseded-narrow-deferrals-2026-05-17/add-storage-mux/` as a starting point (same family conventions, same error registry namespace, same conformance approach) — this draft **widens** that surface; it does not narrow it. Industry references: HAProxy / Envoy passive+active health checks; HashiCorp Consul service-mesh failover; AWS Route 53 health-check failover routing; NGINX `upstream` `max_fails` / `fail_timeout`; Netflix Hystrix bulkhead.

## Verification approach

Every requirement in `specs/storage-mux/spec.md` carries a `**Verified by:**` line naming planned test files. Coverage gate: 100/100/100/100. The conformance harness runs `runStorageConformance` once per mode against a three-backend `MemoryStorage` fleet, plus once against a two-level recursive mux to prove recursion does not break invariants.

## Impact

### Affected dependents

Run `bunx turbo ls --filter=...@suaveplan/storage-mux` to enumerate. The package is new — no current dependents.

- No current dependents (verified by `bunx turbo ls --filter=...@suaveplan/storage-mux` returning the package itself).
- Forward-declared consumers listed under the consumer justification above; each will land its own OpenSpec change to wire the mux in.

### Affected upstreams

- `@suaveplan/storage` — consumed via `AbstractStorage` + `StorageLike` + `runStorageConformance`. No augmentation. [x] no action needed.
- `@suaveplan/retry` — consumed via `RetryStrategy` + `runWithRetry`. No augmentation. [x] no action needed.
- `@suaveplan/error` — `defineCodes` + `SuaveplanError`. [x] no action needed.
- `@suaveplan/events` — `EventBusLike` consumer only. [x] no action needed.
- `@suaveplan/types` — `Clock`, `LoggerLike`, `TracerLike`, `EventBusLike`. [x] no action needed.