Initial Commit
This commit is contained in:
@@ -0,0 +1,173 @@
|
||||
# Design — @suaveplan/storage-mux
|
||||
|
||||
Non-normative companion to `proposal.md` and `specs/storage-mux/spec.md`.
|
||||
|
||||
## 1. Role in the family
|
||||
|
||||
```text
|
||||
@suaveplan/storage AbstractStorage core (key/value, in-memory)
|
||||
├─ @suaveplan/storage-cdn CdnAdapter contract
|
||||
│ ├─ @suaveplan/storage-cdn-bunnycdn
|
||||
│ └─ @suaveplan/storage-cdn-rackspace
|
||||
├─ @suaveplan/storage-s3
|
||||
├─ @suaveplan/storage-http
|
||||
├─ @suaveplan/storage-file
|
||||
└─ @suaveplan/storage-mux Multi-backend composition (THIS PACKAGE)
|
||||
└─ wraps N StorageLike, exposes 1 StorageLike
|
||||
```
|
||||
|
||||
`storage-mux` is a **composition** package, not a transport. It accepts any list of `StorageLike` (cdn / s3 / http / file / memory / other mux instances) and produces a single `StorageLike`. Recursion is intentional — a sharded-then-failed-over deployment is `Mux(round-robin, [Mux(failover, shard1), Mux(failover, shard2)])`.
|
||||
|
||||
## 2. File layout
|
||||
|
||||
```text
|
||||
src/
|
||||
index.ts # explicit named re-exports
|
||||
error-codes.ts # defineCodes("STORAGE_MUX", [...]) + StorageMuxError
|
||||
types/
|
||||
backend-entry.types.ts # BackendEntry interface
|
||||
mux-options.types.ts # StorageMuxOptions, MuxPolicy union
|
||||
mux-options.schemas.ts # Zod boundary schemas
|
||||
mux/
|
||||
mux.ts # StorageMux extends AbstractStorage
|
||||
mux.md
|
||||
mux.test.ts
|
||||
mux.spec-conformance.test.ts # runStorageConformance × 4 policies
|
||||
policies/
|
||||
selector.types.ts # BackendSelector interface
|
||||
failover/
|
||||
failover.ts
|
||||
failover.md
|
||||
failover.test.ts
|
||||
round-robin/
|
||||
round-robin.ts
|
||||
round-robin.md
|
||||
round-robin.test.ts
|
||||
sticky-by-key/
|
||||
sticky-by-key.ts # stable hash → backend index
|
||||
sticky-by-key.md
|
||||
sticky-by-key.test.ts
|
||||
shield/
|
||||
shield.ts # reads cascade, writes fan-out
|
||||
shield.md
|
||||
shield.test.ts
|
||||
health/
|
||||
health.ts # BackendHealth tracker + cooldown
|
||||
health.md
|
||||
health.test.ts
|
||||
observability/
|
||||
events.ts # event topic constants + emit helpers
|
||||
events.md
|
||||
events.test.ts
|
||||
otel.ts # span attribute helpers
|
||||
otel.md
|
||||
otel.test.ts
|
||||
error-codes/
|
||||
error-codes.test.ts # reachability sweep
|
||||
```
|
||||
|
||||
## 3. Policy state machines
|
||||
|
||||
### 3.1 `failover`
|
||||
|
||||
```text
|
||||
┌──────────────────────────────────────────┐
|
||||
▼ │
|
||||
[idx=0] ─ call backend[idx] ── ok ──► return │
|
||||
│ │
|
||||
├── transient error ─────► idx++ ──────────┘
|
||||
│
|
||||
├── non-transient error ──► throw (surface as-is)
|
||||
│
|
||||
└── idx == backends.length ─► throw STORAGE_MUX.ALL_BACKENDS_DOWN
|
||||
(cause = last transient error)
|
||||
```
|
||||
|
||||
Per-attempt: `runWithRetry(retryStrategy, () => backend[idx].op(...))` so transient errors get the configured retry inside a single backend before falling through.
|
||||
|
||||
### 3.2 `round-robin`
|
||||
|
||||
Atomic counter modulo `healthyBackends.length`. On `ALL_BACKENDS_DOWN`, returns the failover error. Down backends skipped via `BackendHealth.isUp(name)`.
|
||||
|
||||
### 3.3 `sticky-by-key`
|
||||
|
||||
```ts
|
||||
const h = stableHash(key) % healthyBackends.length;
|
||||
return healthyBackends[h];
|
||||
```
|
||||
|
||||
Hash is FNV-1a over UTF-8 bytes (universal, no crypto dep). When a backend goes down, the modulo redistributes — same key may move to a different backend; this is acceptable for cache workloads and documented.
|
||||
|
||||
### 3.4 `shield`
|
||||
|
||||
```text
|
||||
read: backend[0] (origin) ?? backend[1..n] (edge mirrors) ── first success wins
|
||||
write: Promise.allSettled(backend[0..n].set(...))
|
||||
├── all ok ─► return
|
||||
├── primary ok, mirror failed ─► return + emit WRITE_FANOUT_PARTIAL event
|
||||
└── primary failed ─► throw (cause = primary error)
|
||||
```
|
||||
|
||||
Distinct from `failover` because `shield` reads **must** hit origin first (origin is canonical), and writes **must** fan out (mirrors only serve what was written to them). `failover` is symmetric — any backend can serve any operation.
|
||||
|
||||
## 4. Error classification
|
||||
|
||||
Which errors trigger failover vs surface directly:
|
||||
|
||||
| Error shape | Classification | Triggers fall-through? | Code surfaced |
|
||||
|---|---|---|---|
|
||||
| `AbortError` / `signal.aborted` | non-transient | NO | `STORAGE_MUX.BACKEND_ABORTED` |
|
||||
| `STORAGE.KEY_NOT_FOUND` (from `get`/`stat`) | semantic | NO | rethrow as-is |
|
||||
| `STORAGE.INVALID_KEY` | semantic | NO | rethrow as-is |
|
||||
| Network timeout (configurable `backendTimeoutMs`) | transient | YES | wrapped, last in cause chain |
|
||||
| `ECONNRESET` / `ETIMEDOUT` / fetch network error | transient | YES | wrapped |
|
||||
| HTTP 5xx from backend | transient | YES | wrapped |
|
||||
| HTTP 4xx (except 408/429) | non-transient | NO | rethrow |
|
||||
| HTTP 408 / 429 | transient | YES | wrapped |
|
||||
| `STORAGE_CDN.CREDENTIAL_EXPIRED` | non-transient | NO | rethrow (caller must rotate) |
|
||||
| Unknown thrown value (non-Error) | transient | YES | wrapped |
|
||||
|
||||
Classification lives in `src/policies/classify.ts`, pure function over `(error: unknown) => "transient" | "semantic" | "abort"`. Tested independently.
|
||||
|
||||
## 5. Backend-isolation guarantees
|
||||
|
||||
A slow primary MUST NOT block a fallback. Mechanism:
|
||||
|
||||
1. **Per-backend timeout** — every backend call wrapped in `Promise.race([call, timeoutPromise(backendTimeoutMs)])`. Default 30 s, configurable per-mux.
|
||||
2. **AbortSignal forwarding** — the per-attempt signal is linked (`AbortSignal.any([userSignal, timeoutSignal])`) and passed to the backend. A well-behaved backend respects it; a misbehaved backend will eventually reject via the timeout race.
|
||||
3. **`shield` reads cascade sequentially with timeout** — origin gets `backendTimeoutMs`; on timeout/error, the mux moves to mirror[0], etc. The next backend's call starts before the previous backend's underlying promise has settled (it is left to its own cleanup).
|
||||
4. **`shield` writes are `Promise.allSettled`** — one slow mirror cannot starve the others. Caller sees the result when all settle OR `backendTimeoutMs * 2` elapses (guard against a permanently-hung mirror).
|
||||
|
||||
The mux NEVER waits for a hung promise indefinitely — `Promise.race` with a clock-driven timeout is the contract.
|
||||
|
||||
## 6. Health tracking
|
||||
|
||||
`BackendHealth` per-backend state:
|
||||
|
||||
```ts
|
||||
interface BackendState {
|
||||
status: "up" | "down";
|
||||
consecutiveFailures: number;
|
||||
cooldownUntil: number; // Clock.now() millis
|
||||
lastError?: unknown;
|
||||
}
|
||||
```
|
||||
|
||||
Transitions:
|
||||
|
||||
- **up → down**: triggered by any classified-transient error. `cooldownUntil = now + cooldownMs * 2 ** min(consecutiveFailures, 6)` (exponential capped at 64× base). Emits `storage-mux.backend.down`.
|
||||
- **down → up**: triggered when `now >= cooldownUntil` AND the next selection call probes the backend AND the probe succeeds. Resets `consecutiveFailures`. Emits `storage-mux.backend.recovered`.
|
||||
- **active healthcheck**: if `BackendEntry.healthcheck` is provided AND `healthcheckIntervalMs` is set, the mux schedules `Clock.setTimeout` to invoke it; failure marks the backend down without waiting for a real request.
|
||||
|
||||
`Clock` is injected (`@suaveplan/types#Clock`), so tests use `FakeClock` for deterministic cooldown / healthcheck advancement.
|
||||
|
||||
## 7. Tier purity
|
||||
|
||||
Universal — no `node:*`, no `bun:*`, no DOM. Hash uses pure-JS FNV-1a. Timeouts use injected `Clock`. AbortSignal is standard Web API. Conformance harness runs under Node, Bun, browsers, Workers.
|
||||
|
||||
## 8. Trade-offs
|
||||
|
||||
- **Strategy classes vs. switch statement.** Chose strategy classes (`BackendSelector` interface). Each policy is a separate file ≤ 250 lines, individually tested, and new policies (e.g. `weighted-random` later) plug in without editing the core `StorageMux`.
|
||||
- **FNV-1a vs. xxhash for `sticky-by-key`.** FNV-1a is 20 lines of pure JS with no dep; xxhash needs a wasm or noble dep. For backend selection (low-cardinality output), FNV's distribution is sufficient. If a future workload needs better distribution, the hash function is injectable via options.
|
||||
- **`shield` writes fan out vs. primary-only.** Fan-out is the right default because mirrors that don't see writes can't serve reads. Callers who want primary-only writes should use `failover` instead — the policy choice is the API.
|
||||
- **Recursive Mux.** Allowed because `StorageMux` is itself a `StorageLike`. The conformance harness verifies recursion does not break invariants. The cost — debugging a deeply-nested mux tree — is mitigated by per-call OTel span attributes naming every backend hop.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Add @suaveplan/storage-mux — multi-backend storage failover / shielding mux
|
||||
|
||||
**Package**: `@suaveplan/storage-mux`
|
||||
**Status**: PROPOSED
|
||||
**Type**: new-package (genesis-original)
|
||||
|
||||
- **Design**: ./design.md
|
||||
- **Spec delta**: ./specs/storage-mux/spec.md
|
||||
|
||||
## Why
|
||||
|
||||
`@suaveplan/storage-cdn` (archived 2026-05-16) explicitly deferred origin shielding and multi-origin failover as out-of-scope, naming `storage-mux` as the future home (`openspec/archive/2026-05-16-add-storage-cdn/proposal.md` "Out of scope": *"Origin shielding / multi-origin failover. A failover/shielding layer is a separate concern composed via `storage-mux` (future) or application logic."*). The deferral was correct — failover semantics are orthogonal to CDN-specific surface (purge/signing/edgeUrl) and belong in a composable layer over the bare `AbstractStorage` contract.
|
||||
|
||||
`storage-mux` is a **universal-tier composition layer**: it accepts an ordered list of `StorageLike` backends and a policy (`failover` / `round-robin` / `sticky-by-key` / `shield`) and exposes a single `StorageLike` to consumers. Backends can be any storage adapter — `storage-cdn-*`, `storage-s3`, `storage-http`, `storage-file`, in-memory — making the mux the canonical primitive for CDN failover, origin shielding, mirror reads, and shard distribution.
|
||||
|
||||
### ≥2-consumer justification (per `openspec/AGENTS.md` §0 step 6)
|
||||
|
||||
Three forward-declared scenarios consume `storage-mux` directly:
|
||||
|
||||
1. **CDN failover for user-uploaded content** — primary `storage-cdn-bunnycdn` with `storage-cdn-rackspace` fallback. When BunnyCDN edge degrades, reads transparently fall through. The `failover` policy.
|
||||
2. **Cache mirroring for `storage-plugin-cache`** — local `storage-file` cache layered in front of remote `storage-s3` origin via the `shield` policy. Hits the fast tier first, falls through to origin on miss, writes to both.
|
||||
3. **Drive chunk shielding** — `drive-plugin-chunks` reads chunks from a regional mirror set with `sticky-by-key` so the same chunk always hits the same backend for cache warmth, plus failover when a mirror is down.
|
||||
|
||||
All three scenarios share the same primitive (ordered backend list + policy + health), differing only in policy. Without a shared mux, each consumer reinvents the failover state machine and the registry of `STORAGE_MUX.*` codes would be scattered.
|
||||
|
||||
## What changes
|
||||
|
||||
- ADDED: `StorageMux` class extending `AbstractStorage` from `@suaveplan/storage`. Wraps an ordered backend list and a policy and exposes the full `StorageLike` surface (`get`, `set`, `setPartial`, `has`, `delete`, `list`, `stat`, `getStream`, `setStream`).
|
||||
- ADDED: `MuxPolicy` discriminated union — `"failover" | "round-robin" | "sticky-by-key" | "shield"`. Each policy is implemented as a strategy class behind a common `BackendSelector` interface.
|
||||
- ADDED: `BackendEntry` interface — `{ name: string; storage: StorageLike; weight?: number; healthcheck?: (signal: AbortSignal) => Promise<void> }`.
|
||||
- ADDED: `BackendHealth` tracker — passive degrade (mark down on transient error, exponential cooldown via injected `Clock`) + optional active polling via `healthcheck` callbacks scheduled on `Clock`.
|
||||
- ADDED: `StorageMuxOptions` — `{ backends: BackendEntry[]; policy: MuxPolicy; retry?: RetryStrategy; clock?: Clock; events?: EventBusLike; logger?: LoggerLike; tracer?: TracerLike; cooldownMs?: number; healthcheckIntervalMs?: number }`.
|
||||
- ADDED: `STORAGE_MUX_CODES` via `defineCodes("STORAGE_MUX", [...])` — 8 codes: `ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_POLICY`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`.
|
||||
- ADDED: `StorageMuxError extends SuaveplanError<StorageMuxErrorCode>` with `backendName`, `attemptIndex`, `attemptCount` discriminants.
|
||||
- ADDED: Retry composition — every backend call routed through `@suaveplan/retry` (`runWithRetry` + injected strategy). The mux never reimplements backoff; it composes.
|
||||
- ADDED: AbortSignal threaded through every backend call. A caller-aborted operation must surface `BACKEND_ABORTED`, not `ALL_BACKENDS_DOWN`.
|
||||
- ADDED: Observability — `EventBusLike` emissions on backend down/recovery/failover; OTel span attributes per call (`mux.backend.name`, `mux.attempt.index`, `mux.policy`).
|
||||
- ADDED: Conformance harness integration — `runStorageConformance` (from `@suaveplan/storage`) executed against a `StorageMux` configured with N `MemoryStorage` backends and each policy.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Provider-specific backends.** The mux composes `StorageLike` values handed in by the caller; it does not instantiate CDN clients or know provider authentication.
|
||||
- **Cross-backend replication / consistency.** Writes under `shield` are best-effort fan-out with `WRITE_FANOUT_PARTIAL` surfacing partial failures — there is no two-phase commit, no quorum, no vector clocks. Stronger consistency is `@suaveplan/vault-sync` territory.
|
||||
- **Geo-aware routing.** Backend selection is by ordered list + policy. Geographic affinity belongs in a higher-level router that constructs the backend list per-request.
|
||||
- **Adaptive policy switching.** The policy is fixed at construction; the mux never silently switches `failover` → `round-robin` based on load.
|
||||
|
||||
## Source mapping
|
||||
|
||||
Genesis-original. No upstream in `../suaveplan/`. The design draws from:
|
||||
|
||||
- `openspec/archive/2026-05-16-add-storage-cdn/proposal.md` — the explicit deferral naming `storage-mux`.
|
||||
- Industry conventions: HAProxy / Envoy passive+active health checks; HashiCorp Consul service-mesh failover; AWS Route 53 health-check failover routing; NGINX `upstream` `max_fails` / `fail_timeout`. The four policies (`failover`, `round-robin`, `sticky-by-key`, `shield`) are the union of what those tools offer in the storage-IO dimension.
|
||||
|
||||
### Requirements with no source line
|
||||
|
||||
| Requirement | Justification |
|
||||
|---|---|
|
||||
| Pluggable `RetryStrategy` via `@suaveplan/retry` | Reimplementing retry in the mux would duplicate `retry`'s state machine and diverge from the rest of the monorepo. Composition keeps the contract uniform. |
|
||||
| Backend isolation (slow primary cannot block fallback) | A timeout-aware race or per-backend abort is required because a hung primary that never rejects starves the failover chain. The design uses `Promise.race` against a clock-driven timeout, not a parallel call — parallel-call shielding (`shield` writes) is opt-in. |
|
||||
| `WRITE_FANOUT_PARTIAL` distinct from `ALL_BACKENDS_DOWN` | `shield` writes intentionally fan out to mirrors; one mirror down does not fail the operation, but the caller deserves a signal. A separate code lets callers choose to surface or swallow it. |
|
||||
| `sticky-by-key` uses a stable hash, not a Map cache | Cache memory grows unbounded under high-cardinality keys. Hashing keeps the policy O(1) memory regardless of key population. |
|
||||
|
||||
## Verification approach
|
||||
|
||||
- `src/mux/mux.test.ts` — construction validates non-empty backend list, policy whitelist, weights.
|
||||
- `src/policies/failover/failover.test.ts` — fall-through on transient error; surface non-transient; respect AbortSignal; `ALL_BACKENDS_DOWN` after exhausting list.
|
||||
- `src/policies/round-robin/round-robin.test.ts` — even distribution under N calls; skip down backends.
|
||||
- `src/policies/sticky-by-key/sticky-by-key.test.ts` — same key → same backend across calls; redistribute when backend down.
|
||||
- `src/policies/shield/shield.test.ts` — reads hit primary first then mirrors; writes fan out; `WRITE_FANOUT_PARTIAL` on partial mirror failure.
|
||||
- `src/health/health.test.ts` — passive degrade + exponential cooldown; active polling via injected `Clock`; recovery emits event.
|
||||
- `src/mux/mux.spec-conformance.test.ts` — `runStorageConformance` passes with 3× `MemoryStorage` backends under every policy.
|
||||
- `src/error-codes/error-codes.test.ts` — every `STORAGE_MUX_*` code reachable from a real call site.
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
Run `bunx turbo ls --filter=...@suaveplan/storage-mux` to enumerate. The package is new — no current dependents.
|
||||
|
||||
- No current dependents (verified by `bunx turbo ls --filter=...@suaveplan/storage-mux` returning the package itself).
|
||||
- Forward-declared consumers: `@suaveplan/storage-plugin-cache` (shield mode), `@suaveplan/drive-plugin-chunks` (sticky-by-key), application code composing `storage-cdn-bunnycdn` + `storage-cdn-rackspace` (failover).
|
||||
|
||||
### Affected upstreams
|
||||
|
||||
- `@suaveplan/storage` — consumed via `AbstractStorage` + `StorageLike` + `runStorageConformance`. No augmentation. [x] no action needed.
|
||||
- `@suaveplan/retry` — consumed via `RetryStrategy` + `runWithRetry`. No augmentation. [x] no action needed.
|
||||
- `@suaveplan/error`, `@suaveplan/events`, `@suaveplan/types` — standard foundation deps. [x] no action needed.
|
||||
@@ -0,0 +1,256 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: storage-mux (storage-mux)
|
||||
|
||||
**Package**: `@suaveplan/storage-mux`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. No exceptions.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: StorageMux extends AbstractStorage
|
||||
|
||||
The package SHALL provide a `StorageMux` class extending `AbstractStorage` from `@suaveplan/storage`. The class SHALL accept `StorageMuxOptions` containing an ordered non-empty `backends: BackendEntry[]` and a `policy: MuxPolicy`. The class SHALL implement every method of the `StorageLike` surface (`get`, `set`, `setPartial`, `has`, `delete`, `list`, `stat`, `getStream`, `setStream`) by delegating to a `BackendSelector` chosen from the configured policy.
|
||||
|
||||
#### Scenario: subclass is an instance of AbstractStorage
|
||||
|
||||
- **WHEN** a `StorageMux` is constructed with at least one `BackendEntry`
|
||||
- **THEN** the instance is `instanceof AbstractStorage` AND exposes every `StorageLike` method
|
||||
|
||||
#### Scenario: empty backend list is rejected at construction
|
||||
|
||||
- **WHEN** a caller invokes `new StorageMux({ backends: [], policy: "failover" })`
|
||||
- **THEN** the constructor throws `StorageMuxError` with `code === "STORAGE_MUX.NO_BACKENDS_CONFIGURED"`
|
||||
|
||||
**Verified by:** `src/mux/mux.test.ts::"is instance of AbstractStorage and exposes StorageLike surface"` AND `src/mux/mux.test.ts::"rejects empty backend list with NO_BACKENDS_CONFIGURED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: MuxPolicy discriminated union with four policies
|
||||
|
||||
The package SHALL declare `MuxPolicy` as the string union `"failover" | "round-robin" | "sticky-by-key" | "shield"`. Each policy SHALL be implemented as a `BackendSelector` strategy class in its own subdirectory under `src/policies/`. The `StorageMuxOptions.policy` field SHALL be validated by a Zod schema that rejects any value outside this whitelist with `STORAGE_MUX.INVALID_POLICY`.
|
||||
|
||||
#### Scenario: unknown policy string is rejected
|
||||
|
||||
- **WHEN** a caller invokes `new StorageMux({ backends, policy: "magic" as MuxPolicy })`
|
||||
- **THEN** the constructor throws `StorageMuxError` with `code === "STORAGE_MUX.INVALID_POLICY"`
|
||||
|
||||
**Verified by:** `src/mux/mux.test.ts::"rejects unknown policy with INVALID_POLICY"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: failover policy falls through on transient errors and surfaces semantic errors
|
||||
|
||||
Under the `failover` policy the mux SHALL invoke `backends[0]` first. On a transient classified error (per the error-classification table in `design.md` §4) the mux SHALL advance to `backends[1]`, then `backends[2]`, and so on. On a semantic error (`STORAGE.KEY_NOT_FOUND`, `STORAGE.INVALID_KEY`, 4xx other than 408/429, `CREDENTIAL_EXPIRED`) the mux SHALL surface the error directly without advancing. When every backend has been tried and the last attempt was transient, the mux SHALL throw `StorageMuxError` with `code === "STORAGE_MUX.ALL_BACKENDS_DOWN"`, the prior error as `cause`, and `attemptCount` equal to the number of backends tried.
|
||||
|
||||
#### Scenario: transient error advances to next backend
|
||||
|
||||
- **WHEN** `backends[0].get(key)` rejects with a 503 and `backends[1].get(key)` resolves with bytes
|
||||
- **THEN** the mux returns `backends[1]`'s bytes AND emits a `storage-mux.failover` event naming both backends
|
||||
|
||||
#### Scenario: semantic KEY_NOT_FOUND is not retried across backends
|
||||
|
||||
- **WHEN** `backends[0].get(key)` throws `STORAGE.KEY_NOT_FOUND`
|
||||
- **THEN** the mux rethrows `KEY_NOT_FOUND` without invoking `backends[1]`
|
||||
|
||||
#### Scenario: exhausting all backends throws ALL_BACKENDS_DOWN with cause chain
|
||||
|
||||
- **WHEN** every backend rejects with a transient error
|
||||
- **THEN** the mux throws `StorageMuxError` with `code === "STORAGE_MUX.ALL_BACKENDS_DOWN"` AND `cause` is the last backend's error AND `attemptCount === backends.length`
|
||||
|
||||
**Verified by:** `src/policies/failover/failover.test.ts::"falls through to next backend on transient 503"` AND `src/policies/failover/failover.test.ts::"surfaces semantic KEY_NOT_FOUND without fall-through"` AND `src/policies/failover/failover.test.ts::"throws ALL_BACKENDS_DOWN after exhausting list"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: round-robin policy distributes across healthy backends
|
||||
|
||||
Under the `round-robin` policy the mux SHALL maintain a monotonic counter and select `healthyBackends[counter mod healthyBackends.length]` for each call. Backends marked down by `BackendHealth` SHALL be skipped. When zero backends are healthy the mux SHALL throw `STORAGE_MUX.ALL_BACKENDS_DOWN`.
|
||||
|
||||
#### Scenario: ten calls across three healthy backends distribute evenly
|
||||
|
||||
- **WHEN** the caller issues 9 `get` calls against a mux with 3 healthy backends
|
||||
- **THEN** each backend receives exactly 3 calls
|
||||
|
||||
#### Scenario: down backend is skipped
|
||||
|
||||
- **WHEN** `backends[1]` is marked down and the caller issues 4 calls
|
||||
- **THEN** `backends[0]` and `backends[2]` each receive 2 calls AND `backends[1]` receives 0
|
||||
|
||||
**Verified by:** `src/policies/round-robin/round-robin.test.ts::"distributes nine calls evenly across three backends"` AND `src/policies/round-robin/round-robin.test.ts::"skips backends marked down"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: sticky-by-key policy is deterministic and skips down backends
|
||||
|
||||
Under the `sticky-by-key` policy the mux SHALL compute `index = fnv1a(utf8(key)) mod healthyBackends.length` and route every operation for that key to the same backend. The hash SHALL be pure-JS FNV-1a (no native or crypto dep). When a backend goes down its keys SHALL be redistributed across the remaining healthy backends via the same formula recomputed against the shrunken set.
|
||||
|
||||
#### Scenario: same key resolves to same backend across 100 calls
|
||||
|
||||
- **WHEN** the caller issues 100 `get("alpha")` calls against a mux with 3 healthy backends
|
||||
- **THEN** exactly one backend receives all 100 calls
|
||||
|
||||
#### Scenario: removed backend triggers redistribution
|
||||
|
||||
- **WHEN** the backend currently serving `"alpha"` is marked down
|
||||
- **THEN** subsequent `get("alpha")` calls land on one of the remaining healthy backends AND the choice is stable until the health set changes again
|
||||
|
||||
**Verified by:** `src/policies/sticky-by-key/sticky-by-key.test.ts::"routes the same key to the same backend across 100 calls"` AND `src/policies/sticky-by-key/sticky-by-key.test.ts::"redistributes keys when a backend goes down"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: shield policy cascades reads and fans out writes
|
||||
|
||||
Under the `shield` policy the mux SHALL treat `backends[0]` as the origin and `backends[1..n]` as mirrors. For read operations (`get`, `has`, `stat`, `getStream`) the mux SHALL invoke `backends[0]` first and on transient error or timeout fall through to mirrors in order. For write operations (`set`, `setPartial`, `delete`, `setStream`) the mux SHALL invoke `backends[0]` AND every mirror via `Promise.allSettled`. If the origin write rejects the mux SHALL throw with the origin error as `cause`. If the origin write succeeds but one or more mirror writes reject the mux SHALL emit `storage-mux.fanout-partial` carrying every failed mirror name AND return successfully (the operation is considered durable once the origin accepts it). The mux SHALL additionally surface a `StorageMuxError` with `code === "STORAGE_MUX.WRITE_FANOUT_PARTIAL"` when the caller opts in via `StorageMuxOptions.throwOnFanoutPartial === true`.
|
||||
|
||||
#### Scenario: read falls through to mirror after origin timeout
|
||||
|
||||
- **WHEN** `backends[0].get(key)` hangs past `backendTimeoutMs` and `backends[1].get(key)` resolves with bytes
|
||||
- **THEN** the mux returns `backends[1]`'s bytes
|
||||
|
||||
#### Scenario: write succeeds when origin accepts and mirror rejects
|
||||
|
||||
- **WHEN** `backends[0].set` resolves and `backends[1].set` rejects with a transient error
|
||||
- **THEN** the mux returns successfully AND emits `storage-mux.fanout-partial` naming `backends[1]`
|
||||
|
||||
#### Scenario: write fails when origin rejects
|
||||
|
||||
- **WHEN** `backends[0].set` rejects with any error
|
||||
- **THEN** the mux throws AND `cause` is the origin error AND mirror outcomes are not consulted
|
||||
|
||||
**Verified by:** `src/policies/shield/shield.test.ts::"reads fall through to mirror after origin timeout"` AND `src/policies/shield/shield.test.ts::"write succeeds when origin accepts and one mirror rejects"` AND `src/policies/shield/shield.test.ts::"write fails when origin rejects regardless of mirror outcomes"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: backend isolation — slow primary cannot block fallback
|
||||
|
||||
The mux SHALL wrap every backend call in a `Promise.race` against a `Clock`-driven timeout of `backendTimeoutMs` (default 30000, configurable per-mux). When the race rejects due to timeout the mux SHALL classify the timeout as transient AND proceed to the next backend per policy. The underlying backend promise MAY remain pending; the mux SHALL NOT await it. The mux SHALL link the user `AbortSignal` and the timeout signal via `AbortSignal.any` and pass the combined signal to every backend call so well-behaved backends abort promptly.
|
||||
|
||||
#### Scenario: hung primary does not delay fallback beyond backendTimeoutMs
|
||||
|
||||
- **WHEN** `backends[0].get` returns a never-resolving promise and `backends[1].get` resolves in 10ms
|
||||
- **THEN** the caller receives `backends[1]`'s value in less than `backendTimeoutMs + 50ms` measured against the injected `FakeClock`
|
||||
|
||||
**Verified by:** `src/mux/mux.test.ts::"hung primary does not block fallback past backendTimeoutMs"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: AbortSignal propagation surfaces BACKEND_ABORTED
|
||||
|
||||
When the caller's `AbortSignal` aborts during a mux operation, the mux SHALL surface a `StorageMuxError` with `code === "STORAGE_MUX.BACKEND_ABORTED"` rather than falling through to subsequent backends. The mux SHALL NOT classify a caller-driven abort as a transient backend failure.
|
||||
|
||||
#### Scenario: caller abort surfaces BACKEND_ABORTED and does not advance
|
||||
|
||||
- **WHEN** the caller passes a pre-aborted signal to `mux.get(key, signal)`
|
||||
- **THEN** the mux throws `StorageMuxError` with `code === "STORAGE_MUX.BACKEND_ABORTED"` AND `backends[1].get` is NEVER invoked
|
||||
|
||||
**Verified by:** `src/mux/mux.test.ts::"caller abort surfaces BACKEND_ABORTED without falling through"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: BackendHealth — passive degrade with exponential cooldown
|
||||
|
||||
The package SHALL maintain a `BackendHealth` tracker per backend. A backend SHALL transition from `up` to `down` on the first classified-transient error. The cooldown SHALL be `cooldownMs * 2 ** min(consecutiveFailures, 6)` (exponential, capped at 64× base, default `cooldownMs` 1000). A `down` backend SHALL be skipped by every selector. Once `Clock.now() >= cooldownUntil` the next operation SHALL probe the backend; success transitions it back to `up` and resets `consecutiveFailures`.
|
||||
|
||||
#### Scenario: failing backend cools down for 2000ms after second failure
|
||||
|
||||
- **WHEN** a backend rejects twice in succession (default `cooldownMs` 1000)
|
||||
- **THEN** the backend is `down` AND remains skipped until `FakeClock` advances by 2000ms
|
||||
|
||||
#### Scenario: successful probe after cooldown restores backend and emits recovered event
|
||||
|
||||
- **WHEN** `FakeClock` advances past `cooldownUntil` AND the next probe succeeds
|
||||
- **THEN** the backend is `up` AND a `storage-mux.backend.recovered` event is emitted AND `consecutiveFailures` is reset to 0
|
||||
|
||||
**Verified by:** `src/health/health.test.ts::"exponential cooldown after consecutive failures"` AND `src/health/health.test.ts::"successful probe after cooldown restores backend and emits recovered"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Optional active healthchecks via injected Clock
|
||||
|
||||
When `BackendEntry.healthcheck` is provided AND `StorageMuxOptions.healthcheckIntervalMs` is set, the mux SHALL schedule the healthcheck via the injected `Clock.setTimeout` (or `Clock.setInterval`) and invoke it with an `AbortSignal`. A rejecting healthcheck SHALL mark the backend `down` AND emit `storage-mux.backend.down` carrying `STORAGE_MUX.HEALTHCHECK_FAILED` as the cause. The mux SHALL stop scheduling when the mux is disposed.
|
||||
|
||||
#### Scenario: rejecting active healthcheck marks backend down
|
||||
|
||||
- **WHEN** an active healthcheck rejects with a network error
|
||||
- **THEN** the backend transitions to `down` AND a `storage-mux.backend.down` event is emitted AND the failure cause is `STORAGE_MUX.HEALTHCHECK_FAILED`
|
||||
|
||||
**Verified by:** `src/health/health.test.ts::"rejecting active healthcheck marks backend down with HEALTHCHECK_FAILED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Retry composition via @suaveplan/retry
|
||||
|
||||
Every per-backend invocation SHALL be wrapped by `runWithRetry(retryStrategy, ...)` from `@suaveplan/retry` using the strategy supplied in `StorageMuxOptions.retry` (defaulting to `exponentialBackoff()` from `@suaveplan/retry/strategies/exponential/exponential`). The mux SHALL NOT reimplement backoff, jitter, or attempt counting; all retry behavior MUST be expressed by composing the retry strategy.
|
||||
|
||||
#### Scenario: backend that fails twice then succeeds is retried in place before fall-through
|
||||
|
||||
- **WHEN** `backends[0].get` rejects twice then resolves on the third attempt under a 3-attempt retry strategy
|
||||
- **THEN** the mux returns `backends[0]`'s value AND `backends[1].get` is NEVER invoked
|
||||
|
||||
**Verified by:** `src/policies/failover/failover.test.ts::"retries in place before falling through to next backend"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: STORAGE_MUX error registry — 8 codes reachable
|
||||
|
||||
The package SHALL declare `STORAGE_MUX_CODES` via `defineCodes("STORAGE_MUX", [...])` with exactly these 8 codes: `ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_POLICY`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`. The package SHALL declare `StorageMuxError extends SuaveplanError<StorageMuxErrorCode>` with optional `backendName`, `attemptIndex`, and `attemptCount` discriminants. Every code SHALL be reachable from a real call site exercised by tests in this package.
|
||||
|
||||
#### Scenario: every declared code is reachable from a real call site
|
||||
|
||||
- **WHEN** the test suite is run
|
||||
- **THEN** every code in `STORAGE_MUX_CODES` has at least one test that exercises the call site that throws or carries it
|
||||
|
||||
**Verified by:** `src/error-codes/error-codes.test.ts::"every STORAGE_MUX code is reachable from a real call site"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Observability — EventBusLike events on lifecycle transitions
|
||||
|
||||
The mux SHALL emit events through an injected `EventBusLike` (from `@suaveplan/types`) on the following lifecycle transitions: `storage-mux.backend.down` (passive or active failure), `storage-mux.backend.recovered` (probe success after cooldown), `storage-mux.failover` (failover policy advances to a new backend), `storage-mux.fanout-partial` (shield-policy mirror write rejected while origin succeeded). When no `EventBusLike` is provided the mux SHALL operate normally without emitting (silent fallback).
|
||||
|
||||
#### Scenario: failover emits storage-mux.failover with both backend names
|
||||
|
||||
- **WHEN** the failover policy advances from `backends[0]` to `backends[1]` due to a 503
|
||||
- **THEN** exactly one `storage-mux.failover` event is emitted carrying both backend names AND the underlying error
|
||||
|
||||
**Verified by:** `src/observability/events.test.ts::"failover emits storage-mux.failover with from and to backend names"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Observability — OTel span attributes per backend call
|
||||
|
||||
Every backend call SHALL be wrapped in an OTel span carrying the attributes `mux.backend.name` (string), `mux.attempt.index` (number, 0-based), and `mux.policy` (string). When no tracer is provided (`TracerLike` from `@suaveplan/types`) the mux SHALL operate normally without recording spans.
|
||||
|
||||
#### Scenario: failover attempt records two spans with attempt indices 0 and 1
|
||||
|
||||
- **WHEN** the failover policy invokes `backends[0]` (which rejects) then `backends[1]` (which succeeds)
|
||||
- **THEN** the test tracer records two spans with `mux.attempt.index` equal to `0` and `1` respectively AND matching `mux.backend.name`
|
||||
|
||||
**Verified by:** `src/observability/otel.test.ts::"records one span per backend attempt with attempt index and backend name"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: StorageLike conformance under every policy
|
||||
|
||||
The mux SHALL pass `runStorageConformance` (from `@suaveplan/storage`) when configured with three `MemoryStorage` backends under each of the four policies (`failover`, `round-robin`, `sticky-by-key`, `shield`). The conformance suite SHALL include `setPartial` sparse-extension and overwrite semantics, capability-honest reporting, and AbortSignal propagation.
|
||||
|
||||
#### Scenario: conformance suite passes under failover, round-robin, sticky-by-key, and shield
|
||||
|
||||
- **WHEN** `runStorageConformance(mux)` is invoked four times — once per policy — against the same three-MemoryStorage backend set
|
||||
- **THEN** every conformance assertion passes for every policy
|
||||
|
||||
**Verified by:** `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under failover"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under round-robin"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under sticky-by-key"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under shield"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Tier purity — universal, no Node or DOM imports
|
||||
|
||||
The package SHALL NOT import from `node:*`, `bun:*`, or any DOM-only API. Hashing SHALL be pure-JS (FNV-1a). Timeouts SHALL use the injected `Clock`. The package SHALL run unmodified under Node 18+, Bun, Deno, modern browsers, and Cloudflare Workers.
|
||||
|
||||
#### Scenario: tier-purity grep finds no Node or DOM imports
|
||||
|
||||
- **WHEN** `grep -rnE "from ['\"](node:|bun:)" packages/storage/storage-mux/src/` is run
|
||||
- **THEN** the output is empty
|
||||
|
||||
**Verified by:** `src/mux/mux.spec-conformance.test.ts::"src tree contains no node: or bun: imports"`
|
||||
@@ -0,0 +1,92 @@
|
||||
# Tasks — storage-mux
|
||||
|
||||
> Every checkbox MUST reflect work actually done in this change.
|
||||
|
||||
## 1. Spec authoring
|
||||
|
||||
- [ ] 1.1 Read `openspec/specs/storage/spec.md` and confirm `AbstractStorage` surface
|
||||
- [ ] 1.2 Read `openspec/AGENTS.md` §0c, §13, §14
|
||||
- [ ] 1.3 Read archived `openspec/archive/2026-05-16-add-storage-cdn/` for the deferral and family conventions
|
||||
- [ ] 1.4 Read `packages/flow/retry/src/retry/retry.ts` to confirm `runWithRetry` + `RetryStrategy` surface
|
||||
- [ ] 1.5 Draft `proposal.md`, `design.md`, `specs/storage-mux/spec.md`, `tasks.md`
|
||||
- [ ] 1.6 Numbered Requirement/Scenario checklist per §0c step 6
|
||||
|
||||
## 2. Scaffolding
|
||||
|
||||
- [ ] 2.1 Create `packages/storage/storage-mux/` mirroring `packages/storage/storage-cdn/` conventions
|
||||
- [ ] 2.2 `package.json` v0.0.0, `"sideEffects": false`, subpath exports per feature
|
||||
- [ ] 2.3 Deps: `@suaveplan/error`, `@suaveplan/events`, `@suaveplan/retry`, `@suaveplan/storage`, `@suaveplan/types`, `@opentelemetry/api`, `zod`, `tslib`
|
||||
- [ ] 2.4 DevDeps: `@suaveplan/testing`, `@vitest/coverage-istanbul`, `fast-check`, `mitata`, `typescript`, `vite`, `vite-plugin-dts`, `vitest`
|
||||
- [ ] 2.5 `tsconfig`, `vite.config.ts`, `vitest.config.ts` via `createTestConfig`
|
||||
- [ ] 2.6 `README.md` root with Modules section linking every `.md`
|
||||
|
||||
## 3. Error registry
|
||||
|
||||
- [ ] 3.1 `src/error-codes.ts` — `defineCodes("STORAGE_MUX", [...])` with 8 codes (`ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_POLICY`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`)
|
||||
- [ ] 3.2 `src/error-codes.ts` — `StorageMuxError extends SuaveplanError<StorageMuxErrorCode>` with `backendName`, `attemptIndex`, `attemptCount` discriminants
|
||||
|
||||
## 4. Types & schemas
|
||||
|
||||
- [ ] 4.1 `src/types/backend-entry.types.ts` — `BackendEntry` interface (`name`, `storage`, `weight?`, `healthcheck?`)
|
||||
- [ ] 4.2 `src/types/mux-options.types.ts` — `MuxPolicy` discriminated string union, `StorageMuxOptions`
|
||||
- [ ] 4.3 `src/types/mux-options.schemas.ts` — Zod schemas validating non-empty backends, policy whitelist, weight ≥ 0
|
||||
|
||||
## 5. Selectors / policies
|
||||
|
||||
- [ ] 5.1 `src/policies/selector.types.ts` — `BackendSelector` interface (`select(op, key?, healthy): BackendEntry`)
|
||||
- [ ] 5.2 `src/policies/classify.ts` + `.md` + `.test.ts` — pure error classification per design §4 table
|
||||
- [ ] 5.3 `src/policies/failover/failover.ts` + `.md` + `.test.ts` — fall-through + retry composition
|
||||
- [ ] 5.4 `src/policies/round-robin/round-robin.ts` + `.md` + `.test.ts` — atomic counter, skip down
|
||||
- [ ] 5.5 `src/policies/sticky-by-key/sticky-by-key.ts` + `.md` + `.test.ts` — FNV-1a hash, deterministic
|
||||
- [ ] 5.6 `src/policies/shield/shield.ts` + `.md` + `.test.ts` — read cascade, write fan-out, partial signal
|
||||
|
||||
## 6. Health
|
||||
|
||||
- [ ] 6.1 `src/health/health.ts` + `.md` + `.test.ts` — `BackendHealth` tracker, exponential cooldown, recovery probe, active healthcheck via injected `Clock`
|
||||
|
||||
## 7. Mux core
|
||||
|
||||
- [ ] 7.1 `src/mux/mux.ts` + `.md` — `StorageMux extends AbstractStorage`, per-op routing through `BackendSelector` + `runWithRetry`, per-backend timeout race, AbortSignal linking
|
||||
- [ ] 7.2 `src/mux/mux.test.ts` — construction validation, policy dispatch, abort surfacing
|
||||
- [ ] 7.3 `src/mux/mux.spec-conformance.test.ts` — `runStorageConformance` × 4 policies × 3 `MemoryStorage` backends
|
||||
|
||||
## 8. Observability
|
||||
|
||||
- [ ] 8.1 `src/observability/events.ts` + `.md` + `.test.ts` — event topic constants (`storage-mux.backend.down`, `.recovered`, `.failover`, `.fanout-partial`), `emit*` helpers consuming `EventBusLike`
|
||||
- [ ] 8.2 `src/observability/otel.ts` + `.md` + `.test.ts` — span attribute helpers (`mux.backend.name`, `mux.attempt.index`, `mux.policy`)
|
||||
|
||||
## 9. Barrel + exports
|
||||
|
||||
- [ ] 9.1 `src/index.ts` — explicit named re-exports per §6a (NO `export *`)
|
||||
- [ ] 9.2 `package.json` subpath exports match every feature directory
|
||||
|
||||
## 10. Tests
|
||||
|
||||
- [ ] 10.1 `src/error-codes/error-codes.test.ts` — every `STORAGE_MUX_*` code reachable from a real call site
|
||||
- [ ] 10.2 All test imports via `@suaveplan/testing/runner` (§7)
|
||||
- [ ] 10.3 Every `it(...)` carries `{ timeout: 5000 }` (or 30000 for async health)
|
||||
- [ ] 10.4 Backend isolation test: hung-promise primary does NOT block fallback within `backendTimeoutMs`
|
||||
|
||||
## 11. Documentation
|
||||
|
||||
- [ ] 11.1 Sibling `.md` for every implementation `.ts` (≥ 200 words, all required sections)
|
||||
- [ ] 11.2 `README.md` documents each policy with a usage example
|
||||
|
||||
## 12. Gates
|
||||
|
||||
- [ ] 12.1 `bun run check`
|
||||
- [ ] 12.2 `bunx turbo lint typecheck test --filter=@suaveplan/storage-mux`
|
||||
- [ ] 12.3 `bunx turbo test:coverage --filter=@suaveplan/storage-mux` (100/100/100/100)
|
||||
- [ ] 12.4 `bun scripts/lint/check-test-runner-imports.ts`
|
||||
- [ ] 12.5 Tier-purity grep `grep -rnE "from ['\"](node:|bun:)" packages/storage/storage-mux/src/` returns empty
|
||||
- [ ] 12.6 Banned-pattern scan returns empty
|
||||
- [ ] 12.7 `bun scripts/openspec-conformance.ts @suaveplan/storage-mux`
|
||||
|
||||
## 13. Manifest & archival
|
||||
|
||||
- [ ] 13.1 `bun scripts/folder-reorg/generate-manifest.ts`
|
||||
- [ ] 13.2 `bun scripts/generate-missing.ts`
|
||||
- [ ] 13.3 Update `STATUS.md` and `SUMMARY.md`
|
||||
- [ ] 13.4 `bunx markdownlint-cli2 --fix`
|
||||
- [ ] 13.5 Fold `specs/storage-mux/spec.md` → `openspec/specs/storage-mux/spec.md`
|
||||
- [ ] 13.6 Move change folder → `openspec/archive/<yyyy-mm-dd>-add-storage-mux/`
|
||||
Reference in New Issue
Block a user