Initial Commit
This commit is contained in:
@@ -0,0 +1,173 @@
|
||||
# Design — @suaveplan/storage-mux
|
||||
|
||||
Non-normative companion to `proposal.md` and `specs/storage-mux/spec.md`.
|
||||
|
||||
## 1. Role in the family
|
||||
|
||||
```text
|
||||
@suaveplan/storage AbstractStorage core (key/value, in-memory)
|
||||
├─ @suaveplan/storage-cdn CdnAdapter contract
|
||||
│ ├─ @suaveplan/storage-cdn-bunnycdn
|
||||
│ └─ @suaveplan/storage-cdn-rackspace
|
||||
├─ @suaveplan/storage-s3
|
||||
├─ @suaveplan/storage-http
|
||||
├─ @suaveplan/storage-file
|
||||
└─ @suaveplan/storage-mux Multi-backend composition (THIS PACKAGE)
|
||||
└─ wraps N StorageLike, exposes 1 StorageLike
|
||||
```
|
||||
|
||||
`storage-mux` is a **composition** package, not a transport. It accepts any list of `StorageLike` (cdn / s3 / http / file / memory / other mux instances) and produces a single `StorageLike`. Recursion is intentional — a sharded-then-failed-over deployment is `Mux(round-robin, [Mux(failover, shard1), Mux(failover, shard2)])`.
|
||||
|
||||
## 2. File layout
|
||||
|
||||
```text
|
||||
src/
|
||||
index.ts # explicit named re-exports
|
||||
error-codes.ts # defineCodes("STORAGE_MUX", [...]) + StorageMuxError
|
||||
types/
|
||||
backend-entry.types.ts # BackendEntry interface
|
||||
mux-options.types.ts # StorageMuxOptions, MuxPolicy union
|
||||
mux-options.schemas.ts # Zod boundary schemas
|
||||
mux/
|
||||
mux.ts # StorageMux extends AbstractStorage
|
||||
mux.md
|
||||
mux.test.ts
|
||||
mux.spec-conformance.test.ts # runStorageConformance × 4 policies
|
||||
policies/
|
||||
selector.types.ts # BackendSelector interface
|
||||
failover/
|
||||
failover.ts
|
||||
failover.md
|
||||
failover.test.ts
|
||||
round-robin/
|
||||
round-robin.ts
|
||||
round-robin.md
|
||||
round-robin.test.ts
|
||||
sticky-by-key/
|
||||
sticky-by-key.ts # stable hash → backend index
|
||||
sticky-by-key.md
|
||||
sticky-by-key.test.ts
|
||||
shield/
|
||||
shield.ts # reads cascade, writes fan-out
|
||||
shield.md
|
||||
shield.test.ts
|
||||
health/
|
||||
health.ts # BackendHealth tracker + cooldown
|
||||
health.md
|
||||
health.test.ts
|
||||
observability/
|
||||
events.ts # event topic constants + emit helpers
|
||||
events.md
|
||||
events.test.ts
|
||||
otel.ts # span attribute helpers
|
||||
otel.md
|
||||
otel.test.ts
|
||||
error-codes/
|
||||
error-codes.test.ts # reachability sweep
|
||||
```
|
||||
|
||||
## 3. Policy state machines
|
||||
|
||||
### 3.1 `failover`
|
||||
|
||||
```text
|
||||
┌──────────────────────────────────────────┐
|
||||
▼ │
|
||||
[idx=0] ─ call backend[idx] ── ok ──► return │
|
||||
│ │
|
||||
├── transient error ─────► idx++ ──────────┘
|
||||
│
|
||||
├── non-transient error ──► throw (surface as-is)
|
||||
│
|
||||
└── idx == backends.length ─► throw STORAGE_MUX.ALL_BACKENDS_DOWN
|
||||
(cause = last transient error)
|
||||
```
|
||||
|
||||
Per-attempt: `runWithRetry(retryStrategy, () => backend[idx].op(...))` so transient errors get the configured retry inside a single backend before falling through.
|
||||
|
||||
### 3.2 `round-robin`
|
||||
|
||||
Atomic counter modulo `healthyBackends.length`. On `ALL_BACKENDS_DOWN`, returns the failover error. Down backends skipped via `BackendHealth.isUp(name)`.
|
||||
|
||||
### 3.3 `sticky-by-key`
|
||||
|
||||
```ts
|
||||
const h = stableHash(key) % healthyBackends.length;
|
||||
return healthyBackends[h];
|
||||
```
|
||||
|
||||
Hash is FNV-1a over UTF-8 bytes (universal, no crypto dep). When a backend goes down, the modulo redistributes — same key may move to a different backend; this is acceptable for cache workloads and documented.
|
||||
|
||||
### 3.4 `shield`
|
||||
|
||||
```text
|
||||
read: backend[0] (origin) ?? backend[1..n] (edge mirrors) ── first success wins
|
||||
write: Promise.allSettled(backend[0..n].set(...))
|
||||
├── all ok ─► return
|
||||
├── primary ok, mirror failed ─► return + emit WRITE_FANOUT_PARTIAL event
|
||||
└── primary failed ─► throw (cause = primary error)
|
||||
```
|
||||
|
||||
Distinct from `failover` because `shield` reads **must** hit origin first (origin is canonical), and writes **must** fan out (mirrors only serve what was written to them). `failover` is symmetric — any backend can serve any operation.
|
||||
|
||||
## 4. Error classification
|
||||
|
||||
Which errors trigger failover vs surface directly:
|
||||
|
||||
| Error shape | Classification | Triggers fall-through? | Code surfaced |
|
||||
|---|---|---|---|
|
||||
| `AbortError` / `signal.aborted` | non-transient | NO | `STORAGE_MUX.BACKEND_ABORTED` |
|
||||
| `STORAGE.KEY_NOT_FOUND` (from `get`/`stat`) | semantic | NO | rethrow as-is |
|
||||
| `STORAGE.INVALID_KEY` | semantic | NO | rethrow as-is |
|
||||
| Network timeout (configurable `backendTimeoutMs`) | transient | YES | wrapped, last in cause chain |
|
||||
| `ECONNRESET` / `ETIMEDOUT` / fetch network error | transient | YES | wrapped |
|
||||
| HTTP 5xx from backend | transient | YES | wrapped |
|
||||
| HTTP 4xx (except 408/429) | non-transient | NO | rethrow |
|
||||
| HTTP 408 / 429 | transient | YES | wrapped |
|
||||
| `STORAGE_CDN.CREDENTIAL_EXPIRED` | non-transient | NO | rethrow (caller must rotate) |
|
||||
| Unknown thrown value (non-Error) | transient | YES | wrapped |
|
||||
|
||||
Classification lives in `src/policies/classify.ts`, pure function over `(error: unknown) => "transient" | "semantic" | "abort"`. Tested independently.
|
||||
|
||||
## 5. Backend-isolation guarantees
|
||||
|
||||
A slow primary MUST NOT block a fallback. Mechanism:
|
||||
|
||||
1. **Per-backend timeout** — every backend call wrapped in `Promise.race([call, timeoutPromise(backendTimeoutMs)])`. Default 30 s, configurable per-mux.
|
||||
2. **AbortSignal forwarding** — the per-attempt signal is linked (`AbortSignal.any([userSignal, timeoutSignal])`) and passed to the backend. A well-behaved backend respects it; a misbehaved backend will eventually reject via the timeout race.
|
||||
3. **`shield` reads cascade sequentially with timeout** — origin gets `backendTimeoutMs`; on timeout/error, the mux moves to mirror[0], etc. The next backend's call starts before the previous backend's underlying promise has settled (it is left to its own cleanup).
|
||||
4. **`shield` writes are `Promise.allSettled`** — one slow mirror cannot starve the others. Caller sees the result when all settle OR `backendTimeoutMs * 2` elapses (guard against a permanently-hung mirror).
|
||||
|
||||
The mux NEVER waits for a hung promise indefinitely — `Promise.race` with a clock-driven timeout is the contract.
|
||||
|
||||
## 6. Health tracking
|
||||
|
||||
`BackendHealth` per-backend state:
|
||||
|
||||
```ts
|
||||
interface BackendState {
|
||||
status: "up" | "down";
|
||||
consecutiveFailures: number;
|
||||
cooldownUntil: number; // Clock.now() millis
|
||||
lastError?: unknown;
|
||||
}
|
||||
```
|
||||
|
||||
Transitions:
|
||||
|
||||
- **up → down**: triggered by any classified-transient error. `cooldownUntil = now + cooldownMs * 2 ** min(consecutiveFailures, 6)` (exponential capped at 64× base). Emits `storage-mux.backend.down`.
|
||||
- **down → up**: triggered when `now >= cooldownUntil` AND the next selection call probes the backend AND the probe succeeds. Resets `consecutiveFailures`. Emits `storage-mux.backend.recovered`.
|
||||
- **active healthcheck**: if `BackendEntry.healthcheck` is provided AND `healthcheckIntervalMs` is set, the mux schedules `Clock.setTimeout` to invoke it; failure marks the backend down without waiting for a real request.
|
||||
|
||||
`Clock` is injected (`@suaveplan/types#Clock`), so tests use `FakeClock` for deterministic cooldown / healthcheck advancement.
|
||||
|
||||
## 7. Tier purity
|
||||
|
||||
Universal — no `node:*`, no `bun:*`, no DOM. Hash uses pure-JS FNV-1a. Timeouts use injected `Clock`. AbortSignal is standard Web API. Conformance harness runs under Node, Bun, browsers, Workers.
|
||||
|
||||
## 8. Trade-offs
|
||||
|
||||
- **Strategy classes vs. switch statement.** Chose strategy classes (`BackendSelector` interface). Each policy is a separate file ≤ 250 lines, individually tested, and new policies (e.g. `weighted-random` later) plug in without editing the core `StorageMux`.
|
||||
- **FNV-1a vs. xxhash for `sticky-by-key`.** FNV-1a is 20 lines of pure JS with no dep; xxhash needs a wasm or noble dep. For backend selection (low-cardinality output), FNV's distribution is sufficient. If a future workload needs better distribution, the hash function is injectable via options.
|
||||
- **`shield` writes fan out vs. primary-only.** Fan-out is the right default because mirrors that don't see writes can't serve reads. Callers who want primary-only writes should use `failover` instead — the policy choice is the API.
|
||||
- **Recursive Mux.** Allowed because `StorageMux` is itself a `StorageLike`. The conformance harness verifies recursion does not break invariants. The cost — debugging a deeply-nested mux tree — is mitigated by per-call OTel span attributes naming every backend hop.
|
||||
Reference in New Issue
Block a user