261 lines
13 KiB
Markdown
261 lines
13 KiB
Markdown
# Design — @suaveplan/storage-mux (complete)
|
||
|
||
Non-normative companion to `proposal.md` and `specs/storage-mux/spec.md`.
|
||
|
||
## 1. Role in the family
|
||
|
||
```text
|
||
@suaveplan/storage AbstractStorage core (key/value, in-memory)
|
||
├─ @suaveplan/storage-cdn CdnAdapter contract
|
||
│ ├─ @suaveplan/storage-cdn-bunnycdn
|
||
│ └─ @suaveplan/storage-cdn-rackspace
|
||
├─ @suaveplan/storage-s3
|
||
├─ @suaveplan/storage-http
|
||
├─ @suaveplan/storage-file
|
||
└─ @suaveplan/storage-mux Multi-backend composition (THIS PACKAGE)
|
||
└─ wraps N StorageLike (incl. other Mux), exposes 1 StorageLike
|
||
```
|
||
|
||
`storage-mux` is a **composition** package, not a transport. A `StorageMux` is itself a `StorageLike`, so a sharded-then-failed-over deployment is `Mux(round-robin, [Mux(failover, regionA), Mux(failover, regionB)])`. Recursion depth is capped at 8 (`RECURSION_DEPTH_EXCEEDED`) so a configuration cycle cannot DoS the process.
|
||
|
||
## 2. File layout
|
||
|
||
```text
|
||
src/
|
||
index.ts # explicit named re-exports
|
||
error-codes/
|
||
error-codes.ts # defineCodes("STORAGE_MUX", [...]) + StorageMuxError
|
||
error-codes.md
|
||
error-codes.test.ts # reachability sweep — every code triggered
|
||
types/
|
||
backend-entry.types.ts # BackendEntry interface
|
||
mux-options.types.ts # StorageMuxOptions, MuxMode union, MuxStats
|
||
mux-options.schemas.ts # Zod boundary schemas
|
||
classify/
|
||
classify.ts # error → "transient" | "semantic" | "abort"
|
||
classify.md
|
||
classify.test.ts
|
||
health/
|
||
health.ts # BackendHealth tracker + cooldown + active probe scheduler
|
||
health.md
|
||
health.test.ts
|
||
bulkhead/
|
||
bulkhead.ts # Promise.race timeout + AbortSignal.any composition
|
||
bulkhead.md
|
||
bulkhead.test.ts
|
||
stats/
|
||
stats.ts # per-backend counters + reservoir-sampled latency
|
||
stats.md
|
||
stats.test.ts
|
||
modes/
|
||
selector.types.ts # BackendSelector + BackendExecutor interfaces
|
||
failover/
|
||
failover.ts
|
||
failover.md
|
||
failover.test.ts
|
||
round-robin/
|
||
round-robin.ts
|
||
round-robin.md
|
||
round-robin.test.ts
|
||
weighted/
|
||
weighted.ts # Walker alias-method
|
||
weighted.md
|
||
weighted.test.ts
|
||
sticky-by-key/
|
||
sticky-by-key.ts # FNV-1a → backend index
|
||
sticky-by-key.md
|
||
sticky-by-key.test.ts
|
||
shield/
|
||
shield.ts # origin-first reads + best-effort mirror writes
|
||
shield.md
|
||
shield.test.ts
|
||
mirror/
|
||
mirror.ts # parallel writes (Promise.allSettled) + read-first-success
|
||
mirror.md
|
||
mirror.test.ts
|
||
mutate/
|
||
mutate.ts # addBackend / removeBackend / setWeight / drain
|
||
mutate.md
|
||
mutate.test.ts
|
||
observability/
|
||
events.ts # event topic constants + emit helpers
|
||
events.md
|
||
events.test.ts
|
||
otel.ts # per-attempt span hierarchy + attributes
|
||
otel.md
|
||
otel.test.ts
|
||
mux/
|
||
mux.ts # StorageMux extends AbstractStorage
|
||
mux.md
|
||
mux.test.ts
|
||
mux.spec-conformance.test.ts # runStorageConformance × 6 modes + 1 recursive
|
||
```
|
||
|
||
## 3. Mode state machines
|
||
|
||
### 3.1 `failover`
|
||
|
||
Sequential fall-through. Per-attempt: `runWithRetry(retryStrategy, () => bulkhead(backend[idx].op(...)))`. Transient errors advance `idx`; semantic errors surface; abort surfaces `BACKEND_ABORTED`. Exhausting the list throws `ALL_BACKENDS_DOWN` with the prior error as `cause` and `attemptCount = backends.length`.
|
||
|
||
### 3.2 `round-robin`
|
||
|
||
Atomic counter modulo `healthy.length`. Down/draining backends skipped. Zero healthy → `ALL_BACKENDS_DOWN`.
|
||
|
||
### 3.3 `weighted`
|
||
|
||
Walker alias-method table rebuilt on `addBackend` / `removeBackend` / `setWeight` / health transition. Selection is O(1) — one random byte for the slot, one for the alias coin. `weight < 0` throws `INVALID_WEIGHT`. Construction without weights on any backend throws `WEIGHTED_REQUIRES_WEIGHTS`. Fall-through on transient: re-roll up to `healthy.length` times, then `ALL_BACKENDS_DOWN`.
|
||
|
||
### 3.4 `sticky-by-key`
|
||
|
||
`index = fnv1a(utf8(key)) % healthy.length`. Same key → same backend. Backend removal redistributes via the same formula recomputed against the shrunken set (acceptable cache-warmth trade-off). Operations without a key (e.g. `list`) round-robin internally.
|
||
|
||
### 3.5 `shield`
|
||
|
||
`backends[0]` = origin, `backends[1..n]` = mirrors.
|
||
|
||
- **read**: origin first; on transient/timeout, cascade to mirrors in order; first success wins.
|
||
- **write**: `Promise.allSettled` over **all** backends. Origin rejection throws (cause = origin error, mirror outcomes discarded). Origin success + any mirror rejection emits `storage-mux.fanout-partial` and returns successfully — unless `throwOnFanoutPartial: true`, then throws `WRITE_FANOUT_PARTIAL`.
|
||
|
||
### 3.6 `mirror`
|
||
|
||
Symmetric peers — no designated origin.
|
||
|
||
- **read**: `Promise.any` race across healthy backends; first success wins; all-reject → `ALL_BACKENDS_DOWN`.
|
||
- **write**: `Promise.allSettled` across **all** healthy backends; any rejection emits `storage-mux.fanout-partial`; **all** rejections → `ALL_BACKENDS_DOWN`. `MIRROR_ORIGIN_REQUIRED` is **not** thrown in this mode — every backend is equal.
|
||
|
||
The difference vs `shield`: `shield` privileges `backends[0]` (writes must succeed there); `mirror` treats every backend identically. Both fan out writes; only `mirror` races reads in parallel.
|
||
|
||
## 4. Error classification
|
||
|
||
| Error shape | Class | Triggers fall-through? | Code surfaced |
|
||
|---|---|---|---|
|
||
| `AbortError` / `signal.aborted` from caller | abort | NO | `STORAGE_MUX.BACKEND_ABORTED` |
|
||
| `STORAGE.KEY_NOT_FOUND` | semantic | NO | rethrow as-is |
|
||
| `STORAGE.INVALID_KEY` | semantic | NO | rethrow as-is |
|
||
| `STORAGE_CDN.CREDENTIAL_EXPIRED` | semantic | NO | rethrow (caller must rotate) |
|
||
| HTTP 4xx (except 408 / 429) | semantic | NO | rethrow |
|
||
| `backendTimeoutMs` race wins | transient | YES | wrapped, last in cause chain (`BACKEND_TIMEOUT` if exhausted) |
|
||
| `ECONNRESET` / `ETIMEDOUT` / fetch network error | transient | YES | wrapped |
|
||
| HTTP 5xx | transient | YES | wrapped |
|
||
| HTTP 408 / 429 | transient | YES | wrapped |
|
||
| Unknown thrown value (non-Error) | transient | YES | wrapped |
|
||
| `BACKEND_DRAINING` re-selection signal (internal) | transient | YES | re-select next backend |
|
||
|
||
`classify` is a pure function exported from `src/classify/classify.ts` and tested independently with property tests over the boundary shape.
|
||
|
||
## 5. Backend isolation guarantees
|
||
|
||
A slow / hung / throwing backend MUST NOT corrupt mux-wide state.
|
||
|
||
1. **Per-backend timeout** — every backend call wrapped in `Promise.race([call, timeoutPromise(backendTimeoutMs)])`. Default 30 s. Underlying promise is allowed to settle in the background.
|
||
2. **AbortSignal.any composition** — per-attempt signal = `AbortSignal.any([userSignal, timeoutSignal, drainSignal])`. Forwarded to backend; well-behaved backends abort promptly.
|
||
3. **State writes are atomic per backend** — health table, weighted alias table, round-robin counter, and stats counters mutate through dedicated helpers that take the backend name as a key. A throw inside backend N can never leave backend M in a partially-updated state.
|
||
4. **`mirror` / `shield` writes use `Promise.allSettled`** — never `Promise.all`; one slow mirror cannot reject the operation early.
|
||
5. **Recursion safety** — when a backend is itself a mux, the parent passes its composite `AbortSignal`. The child mux is a black box; its internal state is its own.
|
||
|
||
## 6. Health tracking
|
||
|
||
`BackendState`:
|
||
|
||
```ts
|
||
type Status = "up" | "down" | "draining";
|
||
interface BackendState {
|
||
status: Status;
|
||
consecutiveFailures: number;
|
||
cooldownUntil: number; // Clock.now() millis
|
||
lastError?: unknown;
|
||
lastTransitionAt: number;
|
||
inFlightCount: number; // for drain accounting
|
||
}
|
||
```
|
||
|
||
Transitions:
|
||
|
||
- **up → down**: any classified-transient error. `cooldownUntil = now + min(cooldownMs * 2 ** min(consecutiveFailures, 6), maxCooldownMs)`. Emits `storage-mux.backend.down`.
|
||
- **down → up**: `now >= cooldownUntil` AND next selection probes AND probe succeeds. Reset `consecutiveFailures = 0`. Emits `storage-mux.backend.recovered`.
|
||
- **\* → draining**: `mux.drain(name)`. Removed from new selection. When `inFlightCount === 0`, transitions to `down` and removes from active set. Emits `storage-mux.drain.started` and `.drain.completed`.
|
||
- **Active healthcheck**: if `BackendEntry.healthcheck` is provided AND `StorageMuxOptions.healthcheckIntervalMs` is set, the mux schedules `Clock.setInterval` to invoke it with a derived `AbortSignal`. A rejecting healthcheck marks `down` with cause `HEALTHCHECK_FAILED`.
|
||
|
||
`Clock` is injected (`@suaveplan/types#Clock`); tests use `FakeClock`.
|
||
|
||
## 7. Stats API
|
||
|
||
```ts
|
||
interface BackendStats {
|
||
name: string;
|
||
status: "up" | "down" | "draining";
|
||
hits: number;
|
||
misses: number;
|
||
errors: number;
|
||
timeouts: number;
|
||
lastLatencyMs: number;
|
||
p50LatencyMs: number;
|
||
p99LatencyMs: number;
|
||
consecutiveFailures: number;
|
||
lastTransitionAt: number;
|
||
inFlightCount: number;
|
||
}
|
||
interface MuxStats {
|
||
mode: MuxMode;
|
||
totalCalls: number;
|
||
totalFailovers: number;
|
||
totalFanoutPartials: number;
|
||
backends: readonly BackendStats[];
|
||
}
|
||
```
|
||
|
||
Latencies use a 256-slot reservoir sample (Vitter Algorithm R) so memory is bounded regardless of call volume. Counters use plain `number` increments — JS is single-threaded; no atomics needed.
|
||
|
||
## 8. Runtime mutation
|
||
|
||
| API | Effect |
|
||
|---|---|
|
||
| `addBackend(entry)` | Append to backend list; rebuild weighted alias table if mode is `weighted`; emit `backend.added`. Conflict on name → `BACKEND_NAME_CONFLICT`. |
|
||
| `removeBackend(name)` | Synonym for `drain(name)` then immediate purge from list. |
|
||
| `setWeight(name, weight)` | Mutate `BackendEntry.weight`; rebuild alias table; emit `weight.changed`. `weight < 0` → `INVALID_WEIGHT`. |
|
||
| `drain(name, { signal? })` | Mark `draining`; await `inFlightCount === 0` (with optional caller abort); transition to removed. Returns `Promise<void>`. |
|
||
|
||
Mutation is atomic relative to in-flight ops: in-flight requests targeting a draining backend complete normally; new requests skip the backend.
|
||
|
||
## 9. Observability
|
||
|
||
### 9.1 OTel span hierarchy
|
||
|
||
```text
|
||
storage-mux.<operation> (parent span, attrs: mux.mode, mux.key, mux.policy)
|
||
└─ storage-mux.attempt (child span per backend attempt)
|
||
attrs: mux.backend.name, mux.attempt.index, mux.mode,
|
||
mux.outcome ∈ {ok, transient, semantic, abort, timeout},
|
||
mux.latency.ms
|
||
```
|
||
|
||
No-op when no `TracerLike` is provided.
|
||
|
||
### 9.2 Events (via `EventBusLike` from `@suaveplan/types`)
|
||
|
||
| Topic | Payload |
|
||
|---|---|
|
||
| `storage-mux.backend.down` | `{ name, cause, code }` |
|
||
| `storage-mux.backend.recovered` | `{ name, downForMs }` |
|
||
| `storage-mux.failover` | `{ from, to, attemptIndex, cause }` |
|
||
| `storage-mux.fanout-partial` | `{ mode, failed: readonly string[], causes: readonly unknown[] }` |
|
||
| `storage-mux.drain.started` | `{ name, inFlightCount }` |
|
||
| `storage-mux.drain.completed` | `{ name, drainedDurationMs }` |
|
||
| `storage-mux.backend.added` | `{ name, weight? }` |
|
||
| `storage-mux.backend.removed` | `{ name }` |
|
||
| `storage-mux.weight.changed` | `{ name, from, to }` |
|
||
|
||
No-op when no bus is provided.
|
||
|
||
## 10. Tier purity
|
||
|
||
Universal — no `node:*`, no `bun:*`, no DOM. Hashing is pure-JS FNV-1a. Randomness for `weighted` uses an injectable `Rng` from `@suaveplan/types` (defaults to `Math.random`). Timeouts use injected `Clock`. `AbortSignal.any` is standard Web API (ES2024); a tiny shim is included for runtimes that lack it.
|
||
|
||
## 11. Trade-offs
|
||
|
||
- **Six modes vs. four.** Adding `weighted` and `mirror` covers the canary and multi-region-write use cases without forcing callers to wrap `failover` in custom logic. Each mode is ≤ 250 lines; the cost is six test files instead of four.
|
||
- **Walker alias-method for `weighted`.** O(1) selection beats roulette-wheel O(n) under runtime mutation. The cost is O(n) rebuild on weight changes — acceptable for backend counts ≤ 100.
|
||
- **Recursive composition.** Allowed because `StorageMux` is itself `StorageLike`. Depth capped at 8 to bound debugging surface and prevent configuration loops. Span hierarchy makes nested traces legible.
|
||
- **Stats use plain counters.** JS single-threaded execution makes atomics unnecessary; the reservoir sample bounds memory at 256 floats per backend.
|
||
- **Drain returns a promise.** Callers can `await mux.drain("old-backend")` before tearing down credentials — no race between drain completion and credential rotation.
|