88 lines
8.6 KiB
Markdown
88 lines
8.6 KiB
Markdown
# Add @suaveplan/storage-mux — multi-backend storage failover / shielding mux
|
||
|
||
**Package**: `@suaveplan/storage-mux`
|
||
**Status**: PROPOSED
|
||
**Type**: new-package (genesis-original)
|
||
|
||
- **Design**: ./design.md
|
||
- **Spec delta**: ./specs/storage-mux/spec.md
|
||
|
||
## Why
|
||
|
||
`@suaveplan/storage-cdn` (archived 2026-05-16) explicitly deferred origin shielding and multi-origin failover as out-of-scope, naming `storage-mux` as the future home (`openspec/archive/2026-05-16-add-storage-cdn/proposal.md` "Out of scope": *"Origin shielding / multi-origin failover. A failover/shielding layer is a separate concern composed via `storage-mux` (future) or application logic."*). The deferral was correct — failover semantics are orthogonal to CDN-specific surface (purge/signing/edgeUrl) and belong in a composable layer over the bare `AbstractStorage` contract.
|
||
|
||
`storage-mux` is a **universal-tier composition layer**: it accepts an ordered list of `StorageLike` backends and a policy (`failover` / `round-robin` / `sticky-by-key` / `shield`) and exposes a single `StorageLike` to consumers. Backends can be any storage adapter — `storage-cdn-*`, `storage-s3`, `storage-http`, `storage-file`, in-memory — making the mux the canonical primitive for CDN failover, origin shielding, mirror reads, and shard distribution.
|
||
|
||
### ≥2-consumer justification (per `openspec/AGENTS.md` §0 step 6)
|
||
|
||
Three forward-declared scenarios consume `storage-mux` directly:
|
||
|
||
1. **CDN failover for user-uploaded content** — primary `storage-cdn-bunnycdn` with `storage-cdn-rackspace` fallback. When BunnyCDN edge degrades, reads transparently fall through. The `failover` policy.
|
||
2. **Cache mirroring for `storage-plugin-cache`** — local `storage-file` cache layered in front of remote `storage-s3` origin via the `shield` policy. Hits the fast tier first, falls through to origin on miss, writes to both.
|
||
3. **Drive chunk shielding** — `drive-plugin-chunks` reads chunks from a regional mirror set with `sticky-by-key` so the same chunk always hits the same backend for cache warmth, plus failover when a mirror is down.
|
||
|
||
All three scenarios share the same primitive (ordered backend list + policy + health), differing only in policy. Without a shared mux, each consumer reinvents the failover state machine and the registry of `STORAGE_MUX.*` codes would be scattered.
|
||
|
||
## What changes
|
||
|
||
- ADDED: `StorageMux` class extending `AbstractStorage` from `@suaveplan/storage`. Wraps an ordered backend list and a policy and exposes the full `StorageLike` surface (`get`, `set`, `setPartial`, `has`, `delete`, `list`, `stat`, `getStream`, `setStream`).
|
||
- ADDED: `MuxPolicy` discriminated union — `"failover" | "round-robin" | "sticky-by-key" | "shield"`. Each policy is implemented as a strategy class behind a common `BackendSelector` interface.
|
||
- ADDED: `BackendEntry` interface — `{ name: string; storage: StorageLike; weight?: number; healthcheck?: (signal: AbortSignal) => Promise<void> }`.
|
||
- ADDED: `BackendHealth` tracker — passive degrade (mark down on transient error, exponential cooldown via injected `Clock`) + optional active polling via `healthcheck` callbacks scheduled on `Clock`.
|
||
- ADDED: `StorageMuxOptions` — `{ backends: BackendEntry[]; policy: MuxPolicy; retry?: RetryStrategy; clock?: Clock; events?: EventBusLike; logger?: LoggerLike; tracer?: TracerLike; cooldownMs?: number; healthcheckIntervalMs?: number }`.
|
||
- ADDED: `STORAGE_MUX_CODES` via `defineCodes("STORAGE_MUX", [...])` — 8 codes: `ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_POLICY`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`.
|
||
- ADDED: `StorageMuxError extends SuaveplanError<StorageMuxErrorCode>` with `backendName`, `attemptIndex`, `attemptCount` discriminants.
|
||
- ADDED: Retry composition — every backend call routed through `@suaveplan/retry` (`runWithRetry` + injected strategy). The mux never reimplements backoff; it composes.
|
||
- ADDED: AbortSignal threaded through every backend call. A caller-aborted operation must surface `BACKEND_ABORTED`, not `ALL_BACKENDS_DOWN`.
|
||
- ADDED: Observability — `EventBusLike` emissions on backend down/recovery/failover; OTel span attributes per call (`mux.backend.name`, `mux.attempt.index`, `mux.policy`).
|
||
- ADDED: Conformance harness integration — `runStorageConformance` (from `@suaveplan/storage`) executed against a `StorageMux` configured with N `MemoryStorage` backends and each policy.
|
||
|
||
## Out of scope
|
||
|
||
- **Provider-specific backends.** The mux composes `StorageLike` values handed in by the caller; it does not instantiate CDN clients or know provider authentication.
|
||
- **Cross-backend replication / consistency.** Writes under `shield` are best-effort fan-out with `WRITE_FANOUT_PARTIAL` surfacing partial failures — there is no two-phase commit, no quorum, no vector clocks. Stronger consistency is `@suaveplan/vault-sync` territory.
|
||
- **Geo-aware routing.** Backend selection is by ordered list + policy. Geographic affinity belongs in a higher-level router that constructs the backend list per-request.
|
||
- **Adaptive policy switching.** The policy is fixed at construction; the mux never silently switches `failover` → `round-robin` based on load.
|
||
|
||
## Source mapping
|
||
|
||
Genesis-original. No upstream in `../suaveplan/`. The design draws from:
|
||
|
||
- `openspec/archive/2026-05-16-add-storage-cdn/proposal.md` — the explicit deferral naming `storage-mux`.
|
||
- Industry conventions: HAProxy / Envoy passive+active health checks; HashiCorp Consul service-mesh failover; AWS Route 53 health-check failover routing; NGINX `upstream` `max_fails` / `fail_timeout`. The four policies (`failover`, `round-robin`, `sticky-by-key`, `shield`) are the union of what those tools offer in the storage-IO dimension.
|
||
|
||
### Requirements with no source line
|
||
|
||
| Requirement | Justification |
|
||
|---|---|
|
||
| Pluggable `RetryStrategy` via `@suaveplan/retry` | Reimplementing retry in the mux would duplicate `retry`'s state machine and diverge from the rest of the monorepo. Composition keeps the contract uniform. |
|
||
| Backend isolation (slow primary cannot block fallback) | A timeout-aware race or per-backend abort is required because a hung primary that never rejects starves the failover chain. The design uses `Promise.race` against a clock-driven timeout, not a parallel call — parallel-call shielding (`shield` writes) is opt-in. |
|
||
| `WRITE_FANOUT_PARTIAL` distinct from `ALL_BACKENDS_DOWN` | `shield` writes intentionally fan out to mirrors; one mirror down does not fail the operation, but the caller deserves a signal. A separate code lets callers choose to surface or swallow it. |
|
||
| `sticky-by-key` uses a stable hash, not a Map cache | Cache memory grows unbounded under high-cardinality keys. Hashing keeps the policy O(1) memory regardless of key population. |
|
||
|
||
## Verification approach
|
||
|
||
- `src/mux/mux.test.ts` — construction validates non-empty backend list, policy whitelist, weights.
|
||
- `src/policies/failover/failover.test.ts` — fall-through on transient error; surface non-transient; respect AbortSignal; `ALL_BACKENDS_DOWN` after exhausting list.
|
||
- `src/policies/round-robin/round-robin.test.ts` — even distribution under N calls; skip down backends.
|
||
- `src/policies/sticky-by-key/sticky-by-key.test.ts` — same key → same backend across calls; redistribute when backend down.
|
||
- `src/policies/shield/shield.test.ts` — reads hit primary first then mirrors; writes fan out; `WRITE_FANOUT_PARTIAL` on partial mirror failure.
|
||
- `src/health/health.test.ts` — passive degrade + exponential cooldown; active polling via injected `Clock`; recovery emits event.
|
||
- `src/mux/mux.spec-conformance.test.ts` — `runStorageConformance` passes with 3× `MemoryStorage` backends under every policy.
|
||
- `src/error-codes/error-codes.test.ts` — every `STORAGE_MUX_*` code reachable from a real call site.
|
||
|
||
## Impact
|
||
|
||
### Affected dependents
|
||
|
||
Run `bunx turbo ls --filter=...@suaveplan/storage-mux` to enumerate. The package is new — no current dependents.
|
||
|
||
- No current dependents (verified by `bunx turbo ls --filter=...@suaveplan/storage-mux` returning the package itself).
|
||
- Forward-declared consumers: `@suaveplan/storage-plugin-cache` (shield mode), `@suaveplan/drive-plugin-chunks` (sticky-by-key), application code composing `storage-cdn-bunnycdn` + `storage-cdn-rackspace` (failover).
|
||
|
||
### Affected upstreams
|
||
|
||
- `@suaveplan/storage` — consumed via `AbstractStorage` + `StorageLike` + `runStorageConformance`. No augmentation. [x] no action needed.
|
||
- `@suaveplan/retry` — consumed via `RetryStrategy` + `runWithRetry`. No augmentation. [x] no action needed.
|
||
- `@suaveplan/error`, `@suaveplan/events`, `@suaveplan/types` — standard foundation deps. [x] no action needed.
|