# Design — @suaveplan/storage-mux (complete)

Non-normative companion to `proposal.md` and `specs/storage-mux/spec.md`.

## 1. Role in the family

```text
@suaveplan/storage                 AbstractStorage core (key/value, in-memory)
  ├─ @suaveplan/storage-cdn        CdnAdapter contract
  │    ├─ @suaveplan/storage-cdn-bunnycdn
  │    └─ @suaveplan/storage-cdn-rackspace
  ├─ @suaveplan/storage-s3
  ├─ @suaveplan/storage-http
  ├─ @suaveplan/storage-file
  └─ @suaveplan/storage-mux        Multi-backend composition (THIS PACKAGE)
                                    └─ wraps N StorageLike (incl. other Mux), exposes 1 StorageLike
```

`storage-mux` is a **composition** package, not a transport. A `StorageMux` is itself a `StorageLike`, so a sharded-then-failed-over deployment is `Mux(round-robin, [Mux(failover, regionA), Mux(failover, regionB)])`. Recursion depth is capped at 8 (`RECURSION_DEPTH_EXCEEDED`) so a configuration cycle cannot DoS the process.

## 2. File layout

```text
src/
  index.ts                              # explicit named re-exports
  error-codes/
    error-codes.ts                      # defineCodes("STORAGE_MUX", [...]) + StorageMuxError
    error-codes.md
    error-codes.test.ts                 # reachability sweep — every code triggered
  types/
    backend-entry.types.ts              # BackendEntry interface
    mux-options.types.ts                # StorageMuxOptions, MuxMode union, MuxStats
    mux-options.schemas.ts              # Zod boundary schemas
  classify/
    classify.ts                         # error → "transient" | "semantic" | "abort"
    classify.md
    classify.test.ts
  health/
    health.ts                           # BackendHealth tracker + cooldown + active probe scheduler
    health.md
    health.test.ts
  bulkhead/
    bulkhead.ts                         # Promise.race timeout + AbortSignal.any composition
    bulkhead.md
    bulkhead.test.ts
  stats/
    stats.ts                            # per-backend counters + reservoir-sampled latency
    stats.md
    stats.test.ts
  modes/
    selector.types.ts                   # BackendSelector + BackendExecutor interfaces
    failover/
      failover.ts
      failover.md
      failover.test.ts
    round-robin/
      round-robin.ts
      round-robin.md
      round-robin.test.ts
    weighted/
      weighted.ts                       # Walker alias-method
      weighted.md
      weighted.test.ts
    sticky-by-key/
      sticky-by-key.ts                  # FNV-1a → backend index
      sticky-by-key.md
      sticky-by-key.test.ts
    shield/
      shield.ts                         # origin-first reads + best-effort mirror writes
      shield.md
      shield.test.ts
    mirror/
      mirror.ts                         # parallel writes (Promise.allSettled) + read-first-success
      mirror.md
      mirror.test.ts
  mutate/
    mutate.ts                           # addBackend / removeBackend / setWeight / drain
    mutate.md
    mutate.test.ts
  observability/
    events.ts                           # event topic constants + emit helpers
    events.md
    events.test.ts
    otel.ts                             # per-attempt span hierarchy + attributes
    otel.md
    otel.test.ts
  mux/
    mux.ts                              # StorageMux extends AbstractStorage
    mux.md
    mux.test.ts
    mux.spec-conformance.test.ts        # runStorageConformance × 6 modes + 1 recursive
```

## 3. Mode state machines

### 3.1 `failover`

Sequential fall-through. Per-attempt: `runWithRetry(retryStrategy, () => bulkhead(backend[idx].op(...)))`. Transient errors advance `idx`; semantic errors surface; abort surfaces `BACKEND_ABORTED`. Exhausting the list throws `ALL_BACKENDS_DOWN` with the prior error as `cause` and `attemptCount = backends.length`.

### 3.2 `round-robin`

Atomic counter modulo `healthy.length`. Down/draining backends skipped. Zero healthy → `ALL_BACKENDS_DOWN`.

### 3.3 `weighted`

Walker alias-method table rebuilt on `addBackend` / `removeBackend` / `setWeight` / health transition. Selection is O(1) — one random byte for the slot, one for the alias coin. `weight < 0` throws `INVALID_WEIGHT`. Construction without weights on any backend throws `WEIGHTED_REQUIRES_WEIGHTS`. Fall-through on transient: re-roll up to `healthy.length` times, then `ALL_BACKENDS_DOWN`.

### 3.4 `sticky-by-key`

`index = fnv1a(utf8(key)) % healthy.length`. Same key → same backend. Backend removal redistributes via the same formula recomputed against the shrunken set (acceptable cache-warmth trade-off). Operations without a key (e.g. `list`) round-robin internally.

### 3.5 `shield`

`backends[0]` = origin, `backends[1..n]` = mirrors.

- **read**: origin first; on transient/timeout, cascade to mirrors in order; first success wins.
- **write**: `Promise.allSettled` over **all** backends. Origin rejection throws (cause = origin error, mirror outcomes discarded). Origin success + any mirror rejection emits `storage-mux.fanout-partial` and returns successfully — unless `throwOnFanoutPartial: true`, then throws `WRITE_FANOUT_PARTIAL`.

### 3.6 `mirror`

Symmetric peers — no designated origin.

- **read**: `Promise.any` race across healthy backends; first success wins; all-reject → `ALL_BACKENDS_DOWN`.
- **write**: `Promise.allSettled` across **all** healthy backends; any rejection emits `storage-mux.fanout-partial`; **all** rejections → `ALL_BACKENDS_DOWN`. `MIRROR_ORIGIN_REQUIRED` is **not** thrown in this mode — every backend is equal.

The difference vs `shield`: `shield` privileges `backends[0]` (writes must succeed there); `mirror` treats every backend identically. Both fan out writes; only `mirror` races reads in parallel.

## 4. Error classification

| Error shape | Class | Triggers fall-through? | Code surfaced |
|---|---|---|---|
| `AbortError` / `signal.aborted` from caller | abort | NO | `STORAGE_MUX.BACKEND_ABORTED` |
| `STORAGE.KEY_NOT_FOUND` | semantic | NO | rethrow as-is |
| `STORAGE.INVALID_KEY` | semantic | NO | rethrow as-is |
| `STORAGE_CDN.CREDENTIAL_EXPIRED` | semantic | NO | rethrow (caller must rotate) |
| HTTP 4xx (except 408 / 429) | semantic | NO | rethrow |
| `backendTimeoutMs` race wins | transient | YES | wrapped, last in cause chain (`BACKEND_TIMEOUT` if exhausted) |
| `ECONNRESET` / `ETIMEDOUT` / fetch network error | transient | YES | wrapped |
| HTTP 5xx | transient | YES | wrapped |
| HTTP 408 / 429 | transient | YES | wrapped |
| Unknown thrown value (non-Error) | transient | YES | wrapped |
| `BACKEND_DRAINING` re-selection signal (internal) | transient | YES | re-select next backend |

`classify` is a pure function exported from `src/classify/classify.ts` and tested independently with property tests over the boundary shape.

## 5. Backend isolation guarantees

A slow / hung / throwing backend MUST NOT corrupt mux-wide state.

1. **Per-backend timeout** — every backend call wrapped in `Promise.race([call, timeoutPromise(backendTimeoutMs)])`. Default 30 s. Underlying promise is allowed to settle in the background.
2. **AbortSignal.any composition** — per-attempt signal = `AbortSignal.any([userSignal, timeoutSignal, drainSignal])`. Forwarded to backend; well-behaved backends abort promptly.
3. **State writes are atomic per backend** — health table, weighted alias table, round-robin counter, and stats counters mutate through dedicated helpers that take the backend name as a key. A throw inside backend N can never leave backend M in a partially-updated state.
4. **`mirror` / `shield` writes use `Promise.allSettled`** — never `Promise.all`; one slow mirror cannot reject the operation early.
5. **Recursion safety** — when a backend is itself a mux, the parent passes its composite `AbortSignal`. The child mux is a black box; its internal state is its own.

## 6. Health tracking

`BackendState`:

```ts
type Status = "up" | "down" | "draining";
interface BackendState {
  status: Status;
  consecutiveFailures: number;
  cooldownUntil: number;          // Clock.now() millis
  lastError?: unknown;
  lastTransitionAt: number;
  inFlightCount: number;          // for drain accounting
}
```

Transitions:

- **up → down**: any classified-transient error. `cooldownUntil = now + min(cooldownMs * 2 ** min(consecutiveFailures, 6), maxCooldownMs)`. Emits `storage-mux.backend.down`.
- **down → up**: `now >= cooldownUntil` AND next selection probes AND probe succeeds. Reset `consecutiveFailures = 0`. Emits `storage-mux.backend.recovered`.
- **\* → draining**: `mux.drain(name)`. Removed from new selection. When `inFlightCount === 0`, transitions to `down` and removes from active set. Emits `storage-mux.drain.started` and `.drain.completed`.
- **Active healthcheck**: if `BackendEntry.healthcheck` is provided AND `StorageMuxOptions.healthcheckIntervalMs` is set, the mux schedules `Clock.setInterval` to invoke it with a derived `AbortSignal`. A rejecting healthcheck marks `down` with cause `HEALTHCHECK_FAILED`.

`Clock` is injected (`@suaveplan/types#Clock`); tests use `FakeClock`.

## 7. Stats API

```ts
interface BackendStats {
  name: string;
  status: "up" | "down" | "draining";
  hits: number;
  misses: number;
  errors: number;
  timeouts: number;
  lastLatencyMs: number;
  p50LatencyMs: number;
  p99LatencyMs: number;
  consecutiveFailures: number;
  lastTransitionAt: number;
  inFlightCount: number;
}
interface MuxStats {
  mode: MuxMode;
  totalCalls: number;
  totalFailovers: number;
  totalFanoutPartials: number;
  backends: readonly BackendStats[];
}
```

Latencies use a 256-slot reservoir sample (Vitter Algorithm R) so memory is bounded regardless of call volume. Counters use plain `number` increments — JS is single-threaded; no atomics needed.

## 8. Runtime mutation

| API | Effect |
|---|---|
| `addBackend(entry)` | Append to backend list; rebuild weighted alias table if mode is `weighted`; emit `backend.added`. Conflict on name → `BACKEND_NAME_CONFLICT`. |
| `removeBackend(name)` | Synonym for `drain(name)` then immediate purge from list. |
| `setWeight(name, weight)` | Mutate `BackendEntry.weight`; rebuild alias table; emit `weight.changed`. `weight < 0` → `INVALID_WEIGHT`. |
| `drain(name, { signal? })` | Mark `draining`; await `inFlightCount === 0` (with optional caller abort); transition to removed. Returns `Promise<void>`. |

Mutation is atomic relative to in-flight ops: in-flight requests targeting a draining backend complete normally; new requests skip the backend.

## 9. Observability

### 9.1 OTel span hierarchy

```text
storage-mux.<operation>          (parent span, attrs: mux.mode, mux.key, mux.policy)
  └─ storage-mux.attempt         (child span per backend attempt)
        attrs: mux.backend.name, mux.attempt.index, mux.mode,
               mux.outcome ∈ {ok, transient, semantic, abort, timeout},
               mux.latency.ms
```

No-op when no `TracerLike` is provided.

### 9.2 Events (via `EventBusLike` from `@suaveplan/types`)

| Topic | Payload |
|---|---|
| `storage-mux.backend.down` | `{ name, cause, code }` |
| `storage-mux.backend.recovered` | `{ name, downForMs }` |
| `storage-mux.failover` | `{ from, to, attemptIndex, cause }` |
| `storage-mux.fanout-partial` | `{ mode, failed: readonly string[], causes: readonly unknown[] }` |
| `storage-mux.drain.started` | `{ name, inFlightCount }` |
| `storage-mux.drain.completed` | `{ name, drainedDurationMs }` |
| `storage-mux.backend.added` | `{ name, weight? }` |
| `storage-mux.backend.removed` | `{ name }` |
| `storage-mux.weight.changed` | `{ name, from, to }` |

No-op when no bus is provided.

## 10. Tier purity

Universal — no `node:*`, no `bun:*`, no DOM. Hashing is pure-JS FNV-1a. Randomness for `weighted` uses an injectable `Rng` from `@suaveplan/types` (defaults to `Math.random`). Timeouts use injected `Clock`. `AbortSignal.any` is standard Web API (ES2024); a tiny shim is included for runtimes that lack it.

## 11. Trade-offs

- **Six modes vs. four.** Adding `weighted` and `mirror` covers the canary and multi-region-write use cases without forcing callers to wrap `failover` in custom logic. Each mode is ≤ 250 lines; the cost is six test files instead of four.
- **Walker alias-method for `weighted`.** O(1) selection beats roulette-wheel O(n) under runtime mutation. The cost is O(n) rebuild on weight changes — acceptable for backend counts ≤ 100.
- **Recursive composition.** Allowed because `StorageMux` is itself `StorageLike`. Depth capped at 8 to bound debugging surface and prevent configuration loops. Span hierarchy makes nested traces legible.
- **Stats use plain counters.** JS single-threaded execution makes atomics unnecessary; the reservoir sample bounds memory at 256 floats per backend.
- **Drain returns a promise.** Callers can `await mux.drain("old-backend")` before tearing down credentials — no race between drain completion and credential rotation.