Initial Commit
This commit is contained in:
260
openspec/changes/complete-storage-mux/design.md
Normal file
260
openspec/changes/complete-storage-mux/design.md
Normal file
@@ -0,0 +1,260 @@
|
||||
# Design — @suaveplan/storage-mux (complete)
|
||||
|
||||
Non-normative companion to `proposal.md` and `specs/storage-mux/spec.md`.
|
||||
|
||||
## 1. Role in the family
|
||||
|
||||
```text
|
||||
@suaveplan/storage AbstractStorage core (key/value, in-memory)
|
||||
├─ @suaveplan/storage-cdn CdnAdapter contract
|
||||
│ ├─ @suaveplan/storage-cdn-bunnycdn
|
||||
│ └─ @suaveplan/storage-cdn-rackspace
|
||||
├─ @suaveplan/storage-s3
|
||||
├─ @suaveplan/storage-http
|
||||
├─ @suaveplan/storage-file
|
||||
└─ @suaveplan/storage-mux Multi-backend composition (THIS PACKAGE)
|
||||
└─ wraps N StorageLike (incl. other Mux), exposes 1 StorageLike
|
||||
```
|
||||
|
||||
`storage-mux` is a **composition** package, not a transport. A `StorageMux` is itself a `StorageLike`, so a sharded-then-failed-over deployment is `Mux(round-robin, [Mux(failover, regionA), Mux(failover, regionB)])`. Recursion depth is capped at 8 (`RECURSION_DEPTH_EXCEEDED`) so a configuration cycle cannot DoS the process.
|
||||
|
||||
## 2. File layout
|
||||
|
||||
```text
|
||||
src/
|
||||
index.ts # explicit named re-exports
|
||||
error-codes/
|
||||
error-codes.ts # defineCodes("STORAGE_MUX", [...]) + StorageMuxError
|
||||
error-codes.md
|
||||
error-codes.test.ts # reachability sweep — every code triggered
|
||||
types/
|
||||
backend-entry.types.ts # BackendEntry interface
|
||||
mux-options.types.ts # StorageMuxOptions, MuxMode union, MuxStats
|
||||
mux-options.schemas.ts # Zod boundary schemas
|
||||
classify/
|
||||
classify.ts # error → "transient" | "semantic" | "abort"
|
||||
classify.md
|
||||
classify.test.ts
|
||||
health/
|
||||
health.ts # BackendHealth tracker + cooldown + active probe scheduler
|
||||
health.md
|
||||
health.test.ts
|
||||
bulkhead/
|
||||
bulkhead.ts # Promise.race timeout + AbortSignal.any composition
|
||||
bulkhead.md
|
||||
bulkhead.test.ts
|
||||
stats/
|
||||
stats.ts # per-backend counters + reservoir-sampled latency
|
||||
stats.md
|
||||
stats.test.ts
|
||||
modes/
|
||||
selector.types.ts # BackendSelector + BackendExecutor interfaces
|
||||
failover/
|
||||
failover.ts
|
||||
failover.md
|
||||
failover.test.ts
|
||||
round-robin/
|
||||
round-robin.ts
|
||||
round-robin.md
|
||||
round-robin.test.ts
|
||||
weighted/
|
||||
weighted.ts # Walker alias-method
|
||||
weighted.md
|
||||
weighted.test.ts
|
||||
sticky-by-key/
|
||||
sticky-by-key.ts # FNV-1a → backend index
|
||||
sticky-by-key.md
|
||||
sticky-by-key.test.ts
|
||||
shield/
|
||||
shield.ts # origin-first reads + best-effort mirror writes
|
||||
shield.md
|
||||
shield.test.ts
|
||||
mirror/
|
||||
mirror.ts # parallel writes (Promise.allSettled) + read-first-success
|
||||
mirror.md
|
||||
mirror.test.ts
|
||||
mutate/
|
||||
mutate.ts # addBackend / removeBackend / setWeight / drain
|
||||
mutate.md
|
||||
mutate.test.ts
|
||||
observability/
|
||||
events.ts # event topic constants + emit helpers
|
||||
events.md
|
||||
events.test.ts
|
||||
otel.ts # per-attempt span hierarchy + attributes
|
||||
otel.md
|
||||
otel.test.ts
|
||||
mux/
|
||||
mux.ts # StorageMux extends AbstractStorage
|
||||
mux.md
|
||||
mux.test.ts
|
||||
mux.spec-conformance.test.ts # runStorageConformance × 6 modes + 1 recursive
|
||||
```
|
||||
|
||||
## 3. Mode state machines
|
||||
|
||||
### 3.1 `failover`
|
||||
|
||||
Sequential fall-through. Per-attempt: `runWithRetry(retryStrategy, () => bulkhead(backend[idx].op(...)))`. Transient errors advance `idx`; semantic errors surface; abort surfaces `BACKEND_ABORTED`. Exhausting the list throws `ALL_BACKENDS_DOWN` with the prior error as `cause` and `attemptCount = backends.length`.
|
||||
|
||||
### 3.2 `round-robin`
|
||||
|
||||
Atomic counter modulo `healthy.length`. Down/draining backends skipped. Zero healthy → `ALL_BACKENDS_DOWN`.
|
||||
|
||||
### 3.3 `weighted`
|
||||
|
||||
Walker alias-method table rebuilt on `addBackend` / `removeBackend` / `setWeight` / health transition. Selection is O(1) — one random byte for the slot, one for the alias coin. `weight < 0` throws `INVALID_WEIGHT`. Construction without weights on any backend throws `WEIGHTED_REQUIRES_WEIGHTS`. Fall-through on transient: re-roll up to `healthy.length` times, then `ALL_BACKENDS_DOWN`.
|
||||
|
||||
### 3.4 `sticky-by-key`
|
||||
|
||||
`index = fnv1a(utf8(key)) % healthy.length`. Same key → same backend. Backend removal redistributes via the same formula recomputed against the shrunken set (acceptable cache-warmth trade-off). Operations without a key (e.g. `list`) round-robin internally.
|
||||
|
||||
### 3.5 `shield`
|
||||
|
||||
`backends[0]` = origin, `backends[1..n]` = mirrors.
|
||||
|
||||
- **read**: origin first; on transient/timeout, cascade to mirrors in order; first success wins.
|
||||
- **write**: `Promise.allSettled` over **all** backends. Origin rejection throws (cause = origin error, mirror outcomes discarded). Origin success + any mirror rejection emits `storage-mux.fanout-partial` and returns successfully — unless `throwOnFanoutPartial: true`, then throws `WRITE_FANOUT_PARTIAL`.
|
||||
|
||||
### 3.6 `mirror`
|
||||
|
||||
Symmetric peers — no designated origin.
|
||||
|
||||
- **read**: `Promise.any` race across healthy backends; first success wins; all-reject → `ALL_BACKENDS_DOWN`.
|
||||
- **write**: `Promise.allSettled` across **all** healthy backends; any rejection emits `storage-mux.fanout-partial`; **all** rejections → `ALL_BACKENDS_DOWN`. `MIRROR_ORIGIN_REQUIRED` is **not** thrown in this mode — every backend is equal.
|
||||
|
||||
The difference vs `shield`: `shield` privileges `backends[0]` (writes must succeed there); `mirror` treats every backend identically. Both fan out writes; only `mirror` races reads in parallel.
|
||||
|
||||
## 4. Error classification
|
||||
|
||||
| Error shape | Class | Triggers fall-through? | Code surfaced |
|
||||
|---|---|---|---|
|
||||
| `AbortError` / `signal.aborted` from caller | abort | NO | `STORAGE_MUX.BACKEND_ABORTED` |
|
||||
| `STORAGE.KEY_NOT_FOUND` | semantic | NO | rethrow as-is |
|
||||
| `STORAGE.INVALID_KEY` | semantic | NO | rethrow as-is |
|
||||
| `STORAGE_CDN.CREDENTIAL_EXPIRED` | semantic | NO | rethrow (caller must rotate) |
|
||||
| HTTP 4xx (except 408 / 429) | semantic | NO | rethrow |
|
||||
| `backendTimeoutMs` race wins | transient | YES | wrapped, last in cause chain (`BACKEND_TIMEOUT` if exhausted) |
|
||||
| `ECONNRESET` / `ETIMEDOUT` / fetch network error | transient | YES | wrapped |
|
||||
| HTTP 5xx | transient | YES | wrapped |
|
||||
| HTTP 408 / 429 | transient | YES | wrapped |
|
||||
| Unknown thrown value (non-Error) | transient | YES | wrapped |
|
||||
| `BACKEND_DRAINING` re-selection signal (internal) | transient | YES | re-select next backend |
|
||||
|
||||
`classify` is a pure function exported from `src/classify/classify.ts` and tested independently with property tests over the boundary shape.
|
||||
|
||||
## 5. Backend isolation guarantees
|
||||
|
||||
A slow / hung / throwing backend MUST NOT corrupt mux-wide state.
|
||||
|
||||
1. **Per-backend timeout** — every backend call wrapped in `Promise.race([call, timeoutPromise(backendTimeoutMs)])`. Default 30 s. Underlying promise is allowed to settle in the background.
|
||||
2. **AbortSignal.any composition** — per-attempt signal = `AbortSignal.any([userSignal, timeoutSignal, drainSignal])`. Forwarded to backend; well-behaved backends abort promptly.
|
||||
3. **State writes are atomic per backend** — health table, weighted alias table, round-robin counter, and stats counters mutate through dedicated helpers that take the backend name as a key. A throw inside backend N can never leave backend M in a partially-updated state.
|
||||
4. **`mirror` / `shield` writes use `Promise.allSettled`** — never `Promise.all`; one slow mirror cannot reject the operation early.
|
||||
5. **Recursion safety** — when a backend is itself a mux, the parent passes its composite `AbortSignal`. The child mux is a black box; its internal state is its own.
|
||||
|
||||
## 6. Health tracking
|
||||
|
||||
`BackendState`:
|
||||
|
||||
```ts
|
||||
type Status = "up" | "down" | "draining";
|
||||
interface BackendState {
|
||||
status: Status;
|
||||
consecutiveFailures: number;
|
||||
cooldownUntil: number; // Clock.now() millis
|
||||
lastError?: unknown;
|
||||
lastTransitionAt: number;
|
||||
inFlightCount: number; // for drain accounting
|
||||
}
|
||||
```
|
||||
|
||||
Transitions:
|
||||
|
||||
- **up → down**: any classified-transient error. `cooldownUntil = now + min(cooldownMs * 2 ** min(consecutiveFailures, 6), maxCooldownMs)`. Emits `storage-mux.backend.down`.
|
||||
- **down → up**: `now >= cooldownUntil` AND next selection probes AND probe succeeds. Reset `consecutiveFailures = 0`. Emits `storage-mux.backend.recovered`.
|
||||
- **\* → draining**: `mux.drain(name)`. Removed from new selection. When `inFlightCount === 0`, transitions to `down` and removes from active set. Emits `storage-mux.drain.started` and `.drain.completed`.
|
||||
- **Active healthcheck**: if `BackendEntry.healthcheck` is provided AND `StorageMuxOptions.healthcheckIntervalMs` is set, the mux schedules `Clock.setInterval` to invoke it with a derived `AbortSignal`. A rejecting healthcheck marks `down` with cause `HEALTHCHECK_FAILED`.
|
||||
|
||||
`Clock` is injected (`@suaveplan/types#Clock`); tests use `FakeClock`.
|
||||
|
||||
## 7. Stats API
|
||||
|
||||
```ts
|
||||
interface BackendStats {
|
||||
name: string;
|
||||
status: "up" | "down" | "draining";
|
||||
hits: number;
|
||||
misses: number;
|
||||
errors: number;
|
||||
timeouts: number;
|
||||
lastLatencyMs: number;
|
||||
p50LatencyMs: number;
|
||||
p99LatencyMs: number;
|
||||
consecutiveFailures: number;
|
||||
lastTransitionAt: number;
|
||||
inFlightCount: number;
|
||||
}
|
||||
interface MuxStats {
|
||||
mode: MuxMode;
|
||||
totalCalls: number;
|
||||
totalFailovers: number;
|
||||
totalFanoutPartials: number;
|
||||
backends: readonly BackendStats[];
|
||||
}
|
||||
```
|
||||
|
||||
Latencies use a 256-slot reservoir sample (Vitter Algorithm R) so memory is bounded regardless of call volume. Counters use plain `number` increments — JS is single-threaded; no atomics needed.
|
||||
|
||||
## 8. Runtime mutation
|
||||
|
||||
| API | Effect |
|
||||
|---|---|
|
||||
| `addBackend(entry)` | Append to backend list; rebuild weighted alias table if mode is `weighted`; emit `backend.added`. Conflict on name → `BACKEND_NAME_CONFLICT`. |
|
||||
| `removeBackend(name)` | Synonym for `drain(name)` then immediate purge from list. |
|
||||
| `setWeight(name, weight)` | Mutate `BackendEntry.weight`; rebuild alias table; emit `weight.changed`. `weight < 0` → `INVALID_WEIGHT`. |
|
||||
| `drain(name, { signal? })` | Mark `draining`; await `inFlightCount === 0` (with optional caller abort); transition to removed. Returns `Promise<void>`. |
|
||||
|
||||
Mutation is atomic relative to in-flight ops: in-flight requests targeting a draining backend complete normally; new requests skip the backend.
|
||||
|
||||
## 9. Observability
|
||||
|
||||
### 9.1 OTel span hierarchy
|
||||
|
||||
```text
|
||||
storage-mux.<operation> (parent span, attrs: mux.mode, mux.key, mux.policy)
|
||||
└─ storage-mux.attempt (child span per backend attempt)
|
||||
attrs: mux.backend.name, mux.attempt.index, mux.mode,
|
||||
mux.outcome ∈ {ok, transient, semantic, abort, timeout},
|
||||
mux.latency.ms
|
||||
```
|
||||
|
||||
No-op when no `TracerLike` is provided.
|
||||
|
||||
### 9.2 Events (via `EventBusLike` from `@suaveplan/types`)
|
||||
|
||||
| Topic | Payload |
|
||||
|---|---|
|
||||
| `storage-mux.backend.down` | `{ name, cause, code }` |
|
||||
| `storage-mux.backend.recovered` | `{ name, downForMs }` |
|
||||
| `storage-mux.failover` | `{ from, to, attemptIndex, cause }` |
|
||||
| `storage-mux.fanout-partial` | `{ mode, failed: readonly string[], causes: readonly unknown[] }` |
|
||||
| `storage-mux.drain.started` | `{ name, inFlightCount }` |
|
||||
| `storage-mux.drain.completed` | `{ name, drainedDurationMs }` |
|
||||
| `storage-mux.backend.added` | `{ name, weight? }` |
|
||||
| `storage-mux.backend.removed` | `{ name }` |
|
||||
| `storage-mux.weight.changed` | `{ name, from, to }` |
|
||||
|
||||
No-op when no bus is provided.
|
||||
|
||||
## 10. Tier purity
|
||||
|
||||
Universal — no `node:*`, no `bun:*`, no DOM. Hashing is pure-JS FNV-1a. Randomness for `weighted` uses an injectable `Rng` from `@suaveplan/types` (defaults to `Math.random`). Timeouts use injected `Clock`. `AbortSignal.any` is standard Web API (ES2024); a tiny shim is included for runtimes that lack it.
|
||||
|
||||
## 11. Trade-offs
|
||||
|
||||
- **Six modes vs. four.** Adding `weighted` and `mirror` covers the canary and multi-region-write use cases without forcing callers to wrap `failover` in custom logic. Each mode is ≤ 250 lines; the cost is six test files instead of four.
|
||||
- **Walker alias-method for `weighted`.** O(1) selection beats roulette-wheel O(n) under runtime mutation. The cost is O(n) rebuild on weight changes — acceptable for backend counts ≤ 100.
|
||||
- **Recursive composition.** Allowed because `StorageMux` is itself `StorageLike`. Depth capped at 8 to bound debugging surface and prevent configuration loops. Span hierarchy makes nested traces legible.
|
||||
- **Stats use plain counters.** JS single-threaded execution makes atomics unnecessary; the reservoir sample bounds memory at 256 floats per backend.
|
||||
- **Drain returns a promise.** Callers can `await mux.drain("old-backend")` before tearing down credentials — no race between drain completion and credential rotation.
|
||||
81
openspec/changes/complete-storage-mux/proposal.md
Normal file
81
openspec/changes/complete-storage-mux/proposal.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Complete @suaveplan/storage-mux — feature-complete multi-backend storage facade
|
||||
|
||||
**Package**: `@suaveplan/storage-mux`
|
||||
**Status**: PROPOSED
|
||||
**Type**: new-package (genesis-original)
|
||||
|
||||
- **Design**: ./design.md
|
||||
- **Spec delta**: ./specs/storage-mux/spec.md
|
||||
|
||||
## Why
|
||||
|
||||
`@suaveplan/storage-cdn` (archived 2026-05-16) explicitly deferred origin shielding and multi-origin failover as out-of-scope, naming `storage-mux` as the future home (`openspec/archive/2026-05-16-add-storage-cdn/proposal.md` "Out of scope": *"Origin shielding / multi-origin failover. A failover/shielding layer is a separate concern composed via `storage-mux` (future) or application logic."*). The deferral was correct — failover semantics are orthogonal to CDN-specific surface (purge/signing/edgeUrl) and belong in a composable layer over the bare `AbstractStorage` contract.
|
||||
|
||||
The first draft of this change (`openspec/changes/_superseded-narrow-deferrals-2026-05-17/add-storage-mux/`) scoped four policies (failover, round-robin, sticky-by-key, shield) and was archived as part of a deferral sweep. This rewrite **widens** the surface to the full set required by real consumers — weighted distribution, mirror-mode (write-fan-out + read-first), runtime mutation, drain, bulkhead timeout, stats API, recursive composition, and a per-attempt OTel span hierarchy — so the package ships once as a feature-complete primitive instead of accreting modes over six follow-up changes.
|
||||
|
||||
`storage-mux` is a **universal-tier composition layer**: it accepts a declarative policy plus an ordered list of `StorageLike` backends and exposes a single `StorageLike` to consumers. Backends can be any storage adapter — `storage-cdn-*`, `storage-s3`, `storage-http`, `storage-file`, in-memory, **or another `StorageMux`** — making the mux the canonical primitive for CDN failover, origin shielding, mirror reads, shard distribution, and tiered cache hierarchies.
|
||||
|
||||
### ≥2-consumer justification (per `openspec/AGENTS.md` §0 step 6)
|
||||
|
||||
Forward-declared consumers that share the mux primitive:
|
||||
|
||||
1. **CDN failover for user-uploaded content** — primary `storage-cdn-bunnycdn` with `storage-cdn-rackspace` fallback (`failover` mode).
|
||||
2. **Cache mirroring for `storage-plugin-cache`** — local `storage-file` cache in front of remote `storage-s3` origin (`shield` mode).
|
||||
3. **Drive chunk shielding** — `drive-plugin-chunks` reads chunks from a regional mirror set (`sticky-by-key` + `failover`).
|
||||
4. **Multi-region write durability** — application code writes media to three S3 buckets in parallel (`mirror` mode), surviving any single-region outage.
|
||||
5. **Weighted canary** — gradually shift 10% / 25% / 50% of write traffic to a new backend before cutover (`weighted` mode + runtime weight mutation + drain on rollback).
|
||||
|
||||
All five scenarios share the same primitive (backend list + mode + health + retry); without a shared mux each consumer reinvents the same state machine and the `STORAGE_MUX.*` registry fragments.
|
||||
|
||||
## What changes
|
||||
|
||||
- ADDED: `StorageMux` class extending `AbstractStorage` from `@suaveplan/storage`. Wraps the configured backend list and mode and exposes the full `StorageLike` surface (`get`, `set`, `setPartial`, `has`, `delete`, `list`, `stat`, `getStream`, `setStream`, `capabilities`).
|
||||
- ADDED: `MuxMode` discriminated string union — `"failover" | "round-robin" | "weighted" | "sticky-by-key" | "shield" | "mirror"`. Each mode is implemented as a `BackendSelector` / `BackendExecutor` strategy class behind a common interface.
|
||||
- ADDED: `BackendEntry` interface — `{ name: string; storage: StorageLike; weight?: number; healthcheck?: (signal: AbortSignal) => Promise<void>; tags?: readonly string[] }`.
|
||||
- ADDED: `StorageMuxOptions` — declarative policy bundle: `{ backends, mode, retry?, clock?, events?, logger?, tracer?, cooldownMs?, maxCooldownMs?, healthcheckIntervalMs?, backendTimeoutMs?, hashFn?, throwOnFanoutPartial?, stats? }`.
|
||||
- ADDED: Six execution modes (full design in `design.md` §3) including `weighted` (Walker alias-method O(1) selection over `weight` field) and `mirror` (write-fan-out via `Promise.allSettled`; read-first-success via cascading race).
|
||||
- ADDED: `BackendHealth` tracker with **passive degrade** (mark down on classified-transient error class, exponential cooldown capped at `maxCooldownMs`) **and active polling** (configurable `healthcheck` fn invoked on injected `Clock` interval).
|
||||
- ADDED: Error classification table (`design.md` §4): `transient` → retry+fall-through; `semantic` → surface as-is; `abort` → surface as `BACKEND_ABORTED`. Pure function `classify(error: unknown) => "transient" | "semantic" | "abort"`.
|
||||
- ADDED: Retry composition via `@suaveplan/retry` (`runWithRetry` + injected strategy). The mux never reimplements backoff, jitter, or attempt counting.
|
||||
- ADDED: **Bulkhead** — every backend call wrapped in `Promise.race` against a `Clock`-driven `backendTimeoutMs` so a hung primary cannot block siblings or starve fan-out. Underlying promise is allowed to settle in the background; the mux never awaits it.
|
||||
- ADDED: **AbortSignal.any composition** linking user signal + per-attempt timeout signal + drain signal; the combined signal is forwarded to every backend call.
|
||||
- ADDED: **Backend isolation** — failure, hang, or throw inside one backend MUST NOT corrupt the mux's internal state (health table, round-robin counter, weighted-mode alias table, stats counters). Per-backend state is mutated only through atomic helpers exercised by stress tests.
|
||||
- ADDED: **Recursive composition** — a `StorageMux` is itself a `StorageLike` and SHALL be acceptable as a `BackendEntry.storage`. The conformance harness exercises a two-level mux tree.
|
||||
- ADDED: **Runtime mutation API** — `addBackend(entry)`, `removeBackend(name)`, `setWeight(name, weight)`, `drain(name, { signal? })`. Mutation is atomic relative to in-flight operations; a drained backend completes its outstanding calls before being removed.
|
||||
- ADDED: **Drain semantics** — `drain(name)` marks the backend as `draining`, removes it from new-selection eligibility, awaits all in-flight ops (with caller-visible promise), then transitions to `down` and removes from the active set.
|
||||
- ADDED: **Stats API** — `mux.stats()` returns per-backend counters (`hits`, `misses`, `errors`, `timeouts`, `lastLatencyMs`, `p50LatencyMs`, `p99LatencyMs`, `consecutiveFailures`, `lastTransitionAt`). Counters are sampled, not synchronous-locked.
|
||||
- ADDED: **Observability**:
|
||||
- OTel **per-backend-attempt span** as a child of the operation span, attributes `mux.backend.name`, `mux.attempt.index`, `mux.mode`, `mux.outcome`, `mux.latency.ms`.
|
||||
- `EventBusLike` emissions on `storage-mux.backend.down`, `.recovered`, `.failover`, `.fanout-partial`, `.drain.started`, `.drain.completed`, `.backend.added`, `.backend.removed`, `.weight.changed`.
|
||||
- ADDED: `STORAGE_MUX_CODES` via `defineCodes("STORAGE_MUX", [...])` — comprehensive registry (`ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_MODE`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`, `BACKEND_NOT_FOUND`, `BACKEND_NAME_CONFLICT`, `BACKEND_DRAINING`, `WEIGHTED_REQUIRES_WEIGHTS`, `MIRROR_ORIGIN_REQUIRED`, `RECURSION_DEPTH_EXCEEDED`).
|
||||
- ADDED: `StorageMuxError extends SuaveplanError<StorageMuxErrorCode>` with `backendName`, `attemptIndex`, `attemptCount`, `mode` discriminants.
|
||||
- ADDED: `runStorageConformance` (from `@suaveplan/storage`) executed against `StorageMux` with N `MemoryStorage` backends under **every** mode, plus a recursive `StorageMux(failover, [StorageMux(mirror, [...]), StorageMux(mirror, [...])])` configuration.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Backend implementations.** Each backend stays its own package (`storage-cdn-*`, `storage-s3`, `storage-http`, `storage-file`, etc.). The mux composes `StorageLike` values handed in by the caller; it does not instantiate or authenticate any backend.
|
||||
|
||||
## Source mapping
|
||||
|
||||
Genesis-original; no upstream in `../suaveplan/`. Heavily reuses `openspec/changes/_superseded-narrow-deferrals-2026-05-17/add-storage-mux/` as a starting point (same family conventions, same error registry namespace, same conformance approach) — this draft **widens** that surface; it does not narrow it. Industry references: HAProxy / Envoy passive+active health checks; HashiCorp Consul service-mesh failover; AWS Route 53 health-check failover routing; NGINX `upstream` `max_fails` / `fail_timeout`; Netflix Hystrix bulkhead.
|
||||
|
||||
## Verification approach
|
||||
|
||||
Every requirement in `specs/storage-mux/spec.md` carries a `**Verified by:**` line naming planned test files. Coverage gate: 100/100/100/100. The conformance harness runs `runStorageConformance` once per mode against a three-backend `MemoryStorage` fleet, plus once against a two-level recursive mux to prove recursion does not break invariants.
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
Run `bunx turbo ls --filter=...@suaveplan/storage-mux` to enumerate. The package is new — no current dependents.
|
||||
|
||||
- No current dependents (verified by `bunx turbo ls --filter=...@suaveplan/storage-mux` returning the package itself).
|
||||
- Forward-declared consumers listed under the consumer justification above; each will land its own OpenSpec change to wire the mux in.
|
||||
|
||||
### Affected upstreams
|
||||
|
||||
- `@suaveplan/storage` — consumed via `AbstractStorage` + `StorageLike` + `runStorageConformance`. No augmentation. [x] no action needed.
|
||||
- `@suaveplan/retry` — consumed via `RetryStrategy` + `runWithRetry`. No augmentation. [x] no action needed.
|
||||
- `@suaveplan/error` — `defineCodes` + `SuaveplanError`. [x] no action needed.
|
||||
- `@suaveplan/events` — `EventBusLike` consumer only. [x] no action needed.
|
||||
- `@suaveplan/types` — `Clock`, `LoggerLike`, `TracerLike`, `EventBusLike`. [x] no action needed.
|
||||
442
openspec/changes/complete-storage-mux/specs/storage-mux/spec.md
Normal file
442
openspec/changes/complete-storage-mux/specs/storage-mux/spec.md
Normal file
@@ -0,0 +1,442 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: storage-mux (storage-mux)
|
||||
|
||||
**Package**: `@suaveplan/storage-mux`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. No exceptions.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: StorageMux extends AbstractStorage
|
||||
|
||||
The package SHALL provide a `StorageMux` class extending `AbstractStorage` from `@suaveplan/storage`. The constructor SHALL accept `StorageMuxOptions` containing an ordered non-empty `backends: BackendEntry[]` and a `mode: MuxMode`. The class SHALL implement every method of the `StorageLike` surface (`get`, `set`, `setPartial`, `has`, `delete`, `list`, `stat`, `getStream`, `setStream`) by delegating to a `BackendSelector` chosen from the configured mode. The class SHALL expose `capabilities` reflecting the intersection of backend capabilities.
|
||||
|
||||
#### Scenario: subclass is an instance of AbstractStorage
|
||||
|
||||
- **WHEN** a `StorageMux` is constructed with at least one `BackendEntry`
|
||||
- **THEN** the instance is `instanceof AbstractStorage` AND exposes every `StorageLike` method
|
||||
|
||||
#### Scenario: empty backend list is rejected at construction
|
||||
|
||||
- **WHEN** a caller invokes `new StorageMux({ backends: [], mode: "failover" })`
|
||||
- **THEN** the constructor throws `StorageMuxError` with `code === "STORAGE_MUX.NO_BACKENDS_CONFIGURED"`
|
||||
|
||||
#### Scenario: duplicate backend names rejected
|
||||
|
||||
- **WHEN** two `BackendEntry` values share the same `name`
|
||||
- **THEN** the constructor throws `StorageMuxError` with `code === "STORAGE_MUX.BACKEND_NAME_CONFLICT"`
|
||||
|
||||
**Verified by:** `src/mux/mux.test.ts::"is instance of AbstractStorage and exposes StorageLike surface"` AND `src/mux/mux.test.ts::"rejects empty backend list with NO_BACKENDS_CONFIGURED"` AND `src/mux/mux.test.ts::"rejects duplicate backend names with BACKEND_NAME_CONFLICT"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: MuxMode discriminated union with six modes
|
||||
|
||||
The package SHALL declare `MuxMode` as the string union `"failover" | "round-robin" | "weighted" | "sticky-by-key" | "shield" | "mirror"`. Each mode SHALL be implemented as a `BackendSelector` strategy class in its own subdirectory under `src/modes/`. The `StorageMuxOptions.mode` field SHALL be validated by a Zod schema that rejects any value outside this whitelist with `STORAGE_MUX.INVALID_MODE`.
|
||||
|
||||
#### Scenario: unknown mode string is rejected
|
||||
|
||||
- **WHEN** a caller invokes `new StorageMux({ backends, mode: "magic" as MuxMode })`
|
||||
- **THEN** the constructor throws `StorageMuxError` with `code === "STORAGE_MUX.INVALID_MODE"`
|
||||
|
||||
**Verified by:** `src/mux/mux.test.ts::"rejects unknown mode with INVALID_MODE"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: failover mode falls through on transient errors and surfaces semantic errors
|
||||
|
||||
Under `failover` the mux SHALL invoke `backends[0]` first. On a transient classified error (per `design.md` §4) the mux SHALL advance to `backends[1]`, then `backends[2]`, and so on. On a semantic error (`STORAGE.KEY_NOT_FOUND`, `STORAGE.INVALID_KEY`, 4xx other than 408/429, `CREDENTIAL_EXPIRED`) the mux SHALL surface the error directly without advancing. When every backend has been tried and the last attempt was transient, the mux SHALL throw `StorageMuxError` with `code === "STORAGE_MUX.ALL_BACKENDS_DOWN"`, the prior error as `cause`, and `attemptCount` equal to the number of backends tried.
|
||||
|
||||
#### Scenario: transient error advances to next backend
|
||||
|
||||
- **WHEN** `backends[0].get(key)` rejects with a 503 and `backends[1].get(key)` resolves with bytes
|
||||
- **THEN** the mux returns `backends[1]`'s bytes AND emits a `storage-mux.failover` event naming both backends
|
||||
|
||||
#### Scenario: semantic KEY_NOT_FOUND is not retried across backends
|
||||
|
||||
- **WHEN** `backends[0].get(key)` throws `STORAGE.KEY_NOT_FOUND`
|
||||
- **THEN** the mux rethrows `KEY_NOT_FOUND` without invoking `backends[1]`
|
||||
|
||||
#### Scenario: exhausting all backends throws ALL_BACKENDS_DOWN with cause chain
|
||||
|
||||
- **WHEN** every backend rejects with a transient error
|
||||
- **THEN** the mux throws `StorageMuxError` with `code === "STORAGE_MUX.ALL_BACKENDS_DOWN"` AND `cause` is the last backend's error AND `attemptCount === backends.length`
|
||||
|
||||
**Verified by:** `src/modes/failover/failover.test.ts::"falls through to next backend on transient 503"` AND `src/modes/failover/failover.test.ts::"surfaces semantic KEY_NOT_FOUND without fall-through"` AND `src/modes/failover/failover.test.ts::"throws ALL_BACKENDS_DOWN after exhausting list"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: round-robin mode distributes across healthy backends
|
||||
|
||||
Under `round-robin` the mux SHALL maintain a monotonic counter and select `healthyBackends[counter mod healthyBackends.length]` for each call. Backends marked `down` or `draining` by `BackendHealth` SHALL be skipped. When zero backends are healthy the mux SHALL throw `STORAGE_MUX.ALL_BACKENDS_DOWN`.
|
||||
|
||||
#### Scenario: nine calls across three healthy backends distribute evenly
|
||||
|
||||
- **WHEN** the caller issues 9 `get` calls against a mux with 3 healthy backends
|
||||
- **THEN** each backend receives exactly 3 calls
|
||||
|
||||
#### Scenario: down backend is skipped
|
||||
|
||||
- **WHEN** `backends[1]` is marked down and the caller issues 4 calls
|
||||
- **THEN** `backends[0]` and `backends[2]` each receive 2 calls AND `backends[1]` receives 0
|
||||
|
||||
**Verified by:** `src/modes/round-robin/round-robin.test.ts::"distributes nine calls evenly across three backends"` AND `src/modes/round-robin/round-robin.test.ts::"skips backends marked down"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: weighted mode selects via Walker alias-method
|
||||
|
||||
Under `weighted` the mux SHALL build a Walker alias-method table over `BackendEntry.weight` values and select a backend in O(1) per call via two random draws (slot + alias coin). The mux SHALL rebuild the alias table on `addBackend`, `removeBackend`, `setWeight`, and any health transition. Construction SHALL throw `STORAGE_MUX.WEIGHTED_REQUIRES_WEIGHTS` when `mode === "weighted"` AND any `BackendEntry.weight` is `undefined`. The mux SHALL throw `STORAGE_MUX.INVALID_WEIGHT` when any `weight < 0` or non-finite. On a transient error the mux SHALL re-roll up to `healthy.length` times before throwing `ALL_BACKENDS_DOWN`.
|
||||
|
||||
#### Scenario: weights 1/3/6 distribute roughly 10/30/60 percent over 1000 calls
|
||||
|
||||
- **WHEN** 1000 calls are issued under a `Rng` seeded for determinism with weights `[1, 3, 6]`
|
||||
- **THEN** each backend's call count is within ±5% of its expected share (100/300/600)
|
||||
|
||||
#### Scenario: weighted without weights rejects at construction
|
||||
|
||||
- **WHEN** a caller invokes `new StorageMux({ backends: [{ name: "a", storage }, { name: "b", storage }], mode: "weighted" })`
|
||||
- **THEN** the constructor throws `STORAGE_MUX.WEIGHTED_REQUIRES_WEIGHTS`
|
||||
|
||||
#### Scenario: negative weight rejected
|
||||
|
||||
- **WHEN** any backend's weight is `-1`
|
||||
- **THEN** the constructor throws `STORAGE_MUX.INVALID_WEIGHT`
|
||||
|
||||
**Verified by:** `src/modes/weighted/weighted.test.ts::"distributes calls roughly per weight under seeded rng"` AND `src/modes/weighted/weighted.test.ts::"rejects mode=weighted with any undefined weight as WEIGHTED_REQUIRES_WEIGHTS"` AND `src/modes/weighted/weighted.test.ts::"rejects negative weight with INVALID_WEIGHT"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: sticky-by-key mode is deterministic and skips down backends
|
||||
|
||||
Under `sticky-by-key` the mux SHALL compute `index = fnv1a(utf8(key)) mod healthyBackends.length` and route every operation for that key to the same backend. The hash SHALL be pure-JS FNV-1a (no native or crypto dep). When a backend goes down its keys SHALL be redistributed across the remaining healthy backends via the same formula recomputed against the shrunken set. Operations without a key (e.g. `list`) SHALL fall back to round-robin selection.
|
||||
|
||||
#### Scenario: same key resolves to same backend across 100 calls
|
||||
|
||||
- **WHEN** the caller issues 100 `get("alpha")` calls against a mux with 3 healthy backends
|
||||
- **THEN** exactly one backend receives all 100 calls
|
||||
|
||||
#### Scenario: removed backend triggers redistribution
|
||||
|
||||
- **WHEN** the backend currently serving `"alpha"` is marked down
|
||||
- **THEN** subsequent `get("alpha")` calls land on one of the remaining healthy backends AND the choice is stable until the health set changes again
|
||||
|
||||
**Verified by:** `src/modes/sticky-by-key/sticky-by-key.test.ts::"routes the same key to the same backend across 100 calls"` AND `src/modes/sticky-by-key/sticky-by-key.test.ts::"redistributes keys when a backend goes down"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: shield mode cascades reads and fans out writes with origin priority
|
||||
|
||||
Under `shield` the mux SHALL treat `backends[0]` as the origin and `backends[1..n]` as mirrors. Construction SHALL throw `STORAGE_MUX.MIRROR_ORIGIN_REQUIRED` when fewer than two backends are configured. For read operations (`get`, `has`, `stat`, `getStream`) the mux SHALL invoke `backends[0]` first and on transient error or timeout fall through to mirrors in order. For write operations (`set`, `setPartial`, `delete`, `setStream`) the mux SHALL invoke `backends[0]` AND every mirror via `Promise.allSettled`. If the origin write rejects the mux SHALL throw with the origin error as `cause`. If the origin write succeeds but one or more mirror writes reject the mux SHALL emit `storage-mux.fanout-partial` carrying every failed mirror name AND return successfully — unless `StorageMuxOptions.throwOnFanoutPartial === true`, in which case the mux SHALL additionally throw `StorageMuxError` with `code === "STORAGE_MUX.WRITE_FANOUT_PARTIAL"`.
|
||||
|
||||
#### Scenario: shield with single backend rejected at construction
|
||||
|
||||
- **WHEN** `new StorageMux({ backends: [origin], mode: "shield" })`
|
||||
- **THEN** the constructor throws `STORAGE_MUX.MIRROR_ORIGIN_REQUIRED`
|
||||
|
||||
#### Scenario: read falls through to mirror after origin timeout
|
||||
|
||||
- **WHEN** `backends[0].get(key)` hangs past `backendTimeoutMs` and `backends[1].get(key)` resolves with bytes
|
||||
- **THEN** the mux returns `backends[1]`'s bytes
|
||||
|
||||
#### Scenario: write succeeds when origin accepts and one mirror rejects (default)
|
||||
|
||||
- **WHEN** `backends[0].set` resolves and `backends[1].set` rejects with a transient error
|
||||
- **THEN** the mux returns successfully AND emits `storage-mux.fanout-partial` naming `backends[1]`
|
||||
|
||||
#### Scenario: write throws WRITE_FANOUT_PARTIAL when caller opts in
|
||||
|
||||
- **WHEN** `throwOnFanoutPartial: true` AND origin succeeds while a mirror rejects
|
||||
- **THEN** the mux throws `StorageMuxError` with `code === "STORAGE_MUX.WRITE_FANOUT_PARTIAL"`
|
||||
|
||||
#### Scenario: write fails when origin rejects regardless of mirror outcomes
|
||||
|
||||
- **WHEN** `backends[0].set` rejects with any error
|
||||
- **THEN** the mux throws AND `cause` is the origin error AND mirror outcomes are not consulted
|
||||
|
||||
**Verified by:** `src/modes/shield/shield.test.ts::"rejects single-backend shield with MIRROR_ORIGIN_REQUIRED"` AND `src/modes/shield/shield.test.ts::"reads fall through to mirror after origin timeout"` AND `src/modes/shield/shield.test.ts::"write succeeds when origin accepts and one mirror rejects"` AND `src/modes/shield/shield.test.ts::"write throws WRITE_FANOUT_PARTIAL when opt-in flag is set"` AND `src/modes/shield/shield.test.ts::"write fails when origin rejects regardless of mirror outcomes"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: mirror mode races reads and fans out writes across symmetric peers
|
||||
|
||||
Under `mirror` every backend is a symmetric peer — no designated origin. Read operations SHALL race healthy backends via `Promise.any`; the first successful result wins; all rejections SHALL produce `STORAGE_MUX.ALL_BACKENDS_DOWN` (cause = `AggregateError` of underlying errors). Write operations SHALL invoke every healthy backend via `Promise.allSettled`; any single rejection SHALL emit `storage-mux.fanout-partial`; all rejections SHALL throw `STORAGE_MUX.ALL_BACKENDS_DOWN`. The `MIRROR_ORIGIN_REQUIRED` code SHALL NOT be thrown in this mode.
|
||||
|
||||
#### Scenario: read returns first successful peer
|
||||
|
||||
- **WHEN** `backends[0].get` resolves at 30ms and `backends[1].get` resolves at 5ms
|
||||
- **THEN** the mux returns `backends[1]`'s value (no wait for `backends[0]`)
|
||||
|
||||
#### Scenario: write fans out to all peers and emits partial when one rejects
|
||||
|
||||
- **WHEN** writes to backends 0 and 1 succeed but backend 2 rejects
|
||||
- **THEN** the mux returns successfully AND emits `storage-mux.fanout-partial` naming `backends[2]`
|
||||
|
||||
#### Scenario: all peers reject a read throws ALL_BACKENDS_DOWN
|
||||
|
||||
- **WHEN** every peer rejects a `get`
|
||||
- **THEN** the mux throws `STORAGE_MUX.ALL_BACKENDS_DOWN` AND `cause instanceof AggregateError`
|
||||
|
||||
**Verified by:** `src/modes/mirror/mirror.test.ts::"read returns first successful peer"` AND `src/modes/mirror/mirror.test.ts::"write fans out and emits partial on single rejection"` AND `src/modes/mirror/mirror.test.ts::"all peers rejecting read throws ALL_BACKENDS_DOWN with AggregateError cause"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: bulkhead — per-backend timeout via Promise.race
|
||||
|
||||
Every backend invocation SHALL be wrapped in `Promise.race` against a `Clock`-driven timeout of `backendTimeoutMs` (default 30000, configurable per-mux). When the race rejects due to timeout the mux SHALL classify the timeout as transient AND proceed per mode policy. The underlying backend promise MAY remain pending; the mux SHALL NOT await it. A backend whose timeout fires consecutively without recovery SHALL surface `STORAGE_MUX.BACKEND_TIMEOUT` on exhaustion.
|
||||
|
||||
#### Scenario: hung primary does not delay fallback beyond backendTimeoutMs
|
||||
|
||||
- **WHEN** `backends[0].get` returns a never-resolving promise and `backends[1].get` resolves in 10ms
|
||||
- **THEN** the caller receives `backends[1]`'s value within `backendTimeoutMs + 50ms` measured against the injected `FakeClock`
|
||||
|
||||
#### Scenario: exhausted timeouts surface BACKEND_TIMEOUT
|
||||
|
||||
- **WHEN** every backend hangs past `backendTimeoutMs`
|
||||
- **THEN** the mux throws `STORAGE_MUX.ALL_BACKENDS_DOWN` AND `cause.code === "STORAGE_MUX.BACKEND_TIMEOUT"`
|
||||
|
||||
**Verified by:** `src/bulkhead/bulkhead.test.ts::"hung primary does not block fallback past backendTimeoutMs"` AND `src/bulkhead/bulkhead.test.ts::"exhausted timeouts surface BACKEND_TIMEOUT in cause chain"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: AbortSignal.any composition surfaces BACKEND_ABORTED
|
||||
|
||||
The per-attempt signal forwarded to each backend SHALL be composed via `AbortSignal.any([userSignal, timeoutSignal, drainSignal])`. When the caller's `AbortSignal` aborts during a mux operation, the mux SHALL surface a `StorageMuxError` with `code === "STORAGE_MUX.BACKEND_ABORTED"` rather than falling through to subsequent backends. The mux SHALL NOT classify a caller-driven abort as a transient backend failure.
|
||||
|
||||
#### Scenario: caller abort surfaces BACKEND_ABORTED and does not advance
|
||||
|
||||
- **WHEN** the caller passes a pre-aborted signal to `mux.get(key, signal)`
|
||||
- **THEN** the mux throws `StorageMuxError` with `code === "STORAGE_MUX.BACKEND_ABORTED"` AND `backends[1].get` is NEVER invoked
|
||||
|
||||
#### Scenario: composed signal aborts backend mid-flight on user cancellation
|
||||
|
||||
- **WHEN** the user aborts `signal` 5ms after `backends[0].get` begins
|
||||
- **THEN** `backends[0].get`'s received `AbortSignal` reports `aborted === true` within 1 tick
|
||||
|
||||
**Verified by:** `src/mux/mux.test.ts::"caller abort surfaces BACKEND_ABORTED without falling through"` AND `src/mux/mux.test.ts::"composed AbortSignal propagates user cancellation to in-flight backend"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: BackendHealth — passive degrade with capped exponential cooldown
|
||||
|
||||
The package SHALL maintain a `BackendHealth` tracker per backend. A backend SHALL transition from `up` to `down` on the first classified-transient error. The cooldown SHALL be `min(cooldownMs * 2 ** min(consecutiveFailures, 6), maxCooldownMs)` (default `cooldownMs` 1000, default `maxCooldownMs` 64000). A `down` backend SHALL be skipped by every selector. Once `Clock.now() >= cooldownUntil` the next operation SHALL probe the backend; success transitions it back to `up` and resets `consecutiveFailures`.
|
||||
|
||||
#### Scenario: failing backend cools down for 2000ms after second failure
|
||||
|
||||
- **WHEN** a backend rejects twice in succession (default `cooldownMs` 1000)
|
||||
- **THEN** the backend is `down` AND remains skipped until `FakeClock` advances by 2000ms
|
||||
|
||||
#### Scenario: cooldown is capped at maxCooldownMs
|
||||
|
||||
- **WHEN** a backend rejects 20 consecutive times with `cooldownMs: 1000, maxCooldownMs: 5000`
|
||||
- **THEN** the cooldown never exceeds 5000ms
|
||||
|
||||
#### Scenario: successful probe after cooldown restores backend and emits recovered event
|
||||
|
||||
- **WHEN** `FakeClock` advances past `cooldownUntil` AND the next probe succeeds
|
||||
- **THEN** the backend is `up` AND a `storage-mux.backend.recovered` event is emitted AND `consecutiveFailures` is reset to 0
|
||||
|
||||
**Verified by:** `src/health/health.test.ts::"exponential cooldown after consecutive failures"` AND `src/health/health.test.ts::"cooldown capped at maxCooldownMs"` AND `src/health/health.test.ts::"successful probe after cooldown restores backend and emits recovered"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Active healthchecks via injected Clock
|
||||
|
||||
When `BackendEntry.healthcheck` is provided AND `StorageMuxOptions.healthcheckIntervalMs` is set, the mux SHALL schedule the healthcheck via the injected `Clock.setInterval` and invoke it with a derived `AbortSignal`. A rejecting healthcheck SHALL mark the backend `down` AND emit `storage-mux.backend.down` carrying `STORAGE_MUX.HEALTHCHECK_FAILED` as the cause. The mux SHALL clear all schedules when disposed.
|
||||
|
||||
#### Scenario: rejecting active healthcheck marks backend down
|
||||
|
||||
- **WHEN** an active healthcheck rejects with a network error
|
||||
- **THEN** the backend transitions to `down` AND a `storage-mux.backend.down` event is emitted AND the failure cause is `STORAGE_MUX.HEALTHCHECK_FAILED`
|
||||
|
||||
#### Scenario: dispose cancels scheduled healthchecks
|
||||
|
||||
- **WHEN** `mux[Symbol.asyncDispose]()` is invoked
|
||||
- **THEN** `FakeClock` advancing past the next interval does NOT invoke the healthcheck
|
||||
|
||||
**Verified by:** `src/health/health.test.ts::"rejecting active healthcheck marks backend down with HEALTHCHECK_FAILED"` AND `src/health/health.test.ts::"dispose cancels scheduled healthchecks"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Retry composition via @suaveplan/retry — no inline retry
|
||||
|
||||
Every per-backend invocation SHALL be wrapped by `runWithRetry(retryStrategy, ...)` from `@suaveplan/retry` using the strategy supplied in `StorageMuxOptions.retry` (defaulting to `exponentialBackoff()` from `@suaveplan/retry`). The mux SHALL NOT reimplement backoff, jitter, or attempt counting; all in-backend retry behavior MUST be expressed by composing the retry strategy. Cross-backend fall-through is policy logic; in-backend retry is `runWithRetry`.
|
||||
|
||||
#### Scenario: backend that fails twice then succeeds is retried in place before fall-through
|
||||
|
||||
- **WHEN** `backends[0].get` rejects twice then resolves on the third attempt under a 3-attempt retry strategy
|
||||
- **THEN** the mux returns `backends[0]`'s value AND `backends[1].get` is NEVER invoked
|
||||
|
||||
#### Scenario: no inline retry implementation in package
|
||||
|
||||
- **WHEN** the package source is grepped for `backoff`, `setTimeout`, or `retryCount` outside `src/health/`
|
||||
- **THEN** the only references are imports from `@suaveplan/retry`
|
||||
|
||||
**Verified by:** `src/modes/failover/failover.test.ts::"retries in place before falling through to next backend"` AND `src/modes/failover/failover.test.ts::"package contains no inline retry implementation outside health"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Backend isolation — one backend's failure cannot corrupt mux state
|
||||
|
||||
A throw, hang, or rejection inside one backend SHALL NOT leave shared mux state (`BackendHealth` table, round-robin counter, weighted alias table, stats counters) in a partially-mutated state observable to any other backend. State updates SHALL be made through dedicated keyed helpers; no helper SHALL mutate state for backend B while handling an error from backend A.
|
||||
|
||||
#### Scenario: throw from backend A leaves backend B health unchanged
|
||||
|
||||
- **WHEN** `backends[0].get` throws a transient error and `backends[1]` has never been called
|
||||
- **THEN** `mux.stats().backends[1]` shows `status === "up"`, `consecutiveFailures === 0`, `errors === 0`
|
||||
|
||||
#### Scenario: concurrent in-flight ops on different backends do not interleave state writes
|
||||
|
||||
- **WHEN** 100 concurrent `get` calls fan out over 4 round-robin backends with one of them throwing
|
||||
- **THEN** `mux.stats().backends` totals exactly equal the issued call count (no lost or double-counted attempts)
|
||||
|
||||
**Verified by:** `src/mux/mux.test.ts::"throw from one backend leaves siblings health untouched"` AND `src/mux/mux.test.ts::"concurrent calls across backends produce consistent stats totals"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Recursive composition — StorageMux acceptable as a backend
|
||||
|
||||
A `StorageMux` SHALL itself satisfy `StorageLike` AND be acceptable as a `BackendEntry.storage`. Configurations like `Mux(failover, [Mux(mirror, [...]), Mux(mirror, [...])])` SHALL function correctly. Recursion depth SHALL be capped at 8; constructing a mux whose backend tree exceeds the cap SHALL throw `STORAGE_MUX.RECURSION_DEPTH_EXCEEDED`.
|
||||
|
||||
#### Scenario: two-level recursive mux passes basic round-trip
|
||||
|
||||
- **WHEN** a `Mux(failover, [Mux(mirror, [a, b]), Mux(mirror, [c, d])])` is constructed and a value is set then read
|
||||
- **THEN** the round-trip succeeds AND OTel spans show a parent-child hierarchy with both mux levels named
|
||||
|
||||
#### Scenario: deeply nested mux beyond depth 8 throws RECURSION_DEPTH_EXCEEDED
|
||||
|
||||
- **WHEN** a caller constructs a mux tree 9 levels deep
|
||||
- **THEN** the outermost constructor throws `STORAGE_MUX.RECURSION_DEPTH_EXCEEDED`
|
||||
|
||||
**Verified by:** `src/mux/mux.spec-conformance.test.ts::"two-level recursive mux passes round-trip and emits nested spans"` AND `src/mux/mux.test.ts::"nine-level recursive construction throws RECURSION_DEPTH_EXCEEDED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Runtime mutation API — add / remove / setWeight / drain
|
||||
|
||||
The mux SHALL expose `addBackend(entry)`, `removeBackend(name)`, `setWeight(name, weight)`, and `drain(name, options?)` methods. Mutations SHALL be atomic relative to in-flight operations: in-flight requests targeting a draining backend complete normally; new requests skip it. `addBackend` SHALL throw `BACKEND_NAME_CONFLICT` on duplicate names. `setWeight` and `removeBackend` SHALL throw `BACKEND_NOT_FOUND` for unknown names. `setWeight` SHALL reject `weight < 0` with `INVALID_WEIGHT`. Each mutation SHALL emit a corresponding event (`backend.added`, `backend.removed`, `weight.changed`).
|
||||
|
||||
#### Scenario: addBackend with duplicate name rejected
|
||||
|
||||
- **WHEN** `mux.addBackend({ name: "existing", storage })` is called for a name already present
|
||||
- **THEN** the call throws `STORAGE_MUX.BACKEND_NAME_CONFLICT` AND no event is emitted
|
||||
|
||||
#### Scenario: setWeight rebuilds alias table under weighted mode
|
||||
|
||||
- **WHEN** the mode is `weighted` AND `mux.setWeight("b", 10)` doubles backend b's weight
|
||||
- **THEN** subsequent 1000 calls under the same seeded `Rng` show b's share roughly doubled
|
||||
|
||||
**Verified by:** `src/mutate/mutate.test.ts::"addBackend rejects duplicate name with BACKEND_NAME_CONFLICT"` AND `src/mutate/mutate.test.ts::"setWeight rebuilds weighted alias and shifts distribution"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Drain — graceful backend removal
|
||||
|
||||
`mux.drain(name, { signal? }): Promise<void>` SHALL mark the named backend as `draining`, remove it from new-selection eligibility, await all in-flight operations to complete, then transition the backend to `down` and remove from the active set. The returned promise SHALL resolve when in-flight count reaches zero (or rejects when the caller-supplied signal aborts). `storage-mux.drain.started` SHALL be emitted on transition; `.drain.completed` on resolution. A backend in `draining` status SHALL surface `BACKEND_DRAINING` to selectors as a re-selection signal (classified transient).
|
||||
|
||||
#### Scenario: drain awaits in-flight calls before completing
|
||||
|
||||
- **WHEN** `mux.drain("primary")` is called while one `get` is in flight on `primary`
|
||||
- **THEN** the drain promise does NOT resolve until that `get` settles AND the `storage-mux.drain.completed` event carries a non-zero `drainedDurationMs`
|
||||
|
||||
#### Scenario: draining backend is skipped by new selections
|
||||
|
||||
- **WHEN** `mux.drain("primary")` is in progress under `round-robin`
|
||||
- **THEN** new calls bypass `primary` AND distribute across remaining healthy backends
|
||||
|
||||
**Verified by:** `src/mutate/mutate.test.ts::"drain awaits in-flight operations before completing"` AND `src/mutate/mutate.test.ts::"draining backend is skipped by new round-robin selections"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Stats API — per-backend counters and sampled latencies
|
||||
|
||||
The mux SHALL expose `mux.stats(): MuxStats` returning a snapshot containing the configured `mode`, aggregate totals (`totalCalls`, `totalFailovers`, `totalFanoutPartials`), and per-backend records with `name`, `status`, `hits`, `misses`, `errors`, `timeouts`, `lastLatencyMs`, `p50LatencyMs`, `p99LatencyMs`, `consecutiveFailures`, `lastTransitionAt`, `inFlightCount`. Latency percentiles SHALL be computed from a 256-slot Vitter Algorithm R reservoir sample to bound memory.
|
||||
|
||||
#### Scenario: stats reflect a successful call and a failed call
|
||||
|
||||
- **WHEN** one successful `get` and one failing `get` are issued against a single-backend mux
|
||||
- **THEN** `mux.stats().backends[0].hits === 1` AND `errors === 1` AND `lastLatencyMs > 0`
|
||||
|
||||
#### Scenario: reservoir bounds memory under high call volume
|
||||
|
||||
- **WHEN** 100000 calls are issued
|
||||
- **THEN** internal latency-sample arrays never exceed 256 entries per backend
|
||||
|
||||
**Verified by:** `src/stats/stats.test.ts::"reflects one success and one failure in counters"` AND `src/stats/stats.test.ts::"latency reservoir stays bounded at 256 samples under 100k calls"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: STORAGE_MUX error registry — all 14 codes reachable
|
||||
|
||||
The package SHALL declare `STORAGE_MUX_CODES` via `defineCodes("STORAGE_MUX", [...])` with exactly these 14 codes: `ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_MODE`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`, `BACKEND_NOT_FOUND`, `BACKEND_NAME_CONFLICT`, `BACKEND_DRAINING`, `WEIGHTED_REQUIRES_WEIGHTS`, `MIRROR_ORIGIN_REQUIRED`, `RECURSION_DEPTH_EXCEEDED`. The package SHALL declare `StorageMuxError extends SuaveplanError<StorageMuxErrorCode>` with optional `backendName`, `attemptIndex`, `attemptCount`, and `mode` discriminants. Every code SHALL be reachable from a real call site exercised by tests in this package.
|
||||
|
||||
#### Scenario: every declared code is reachable from a real call site
|
||||
|
||||
- **WHEN** the test suite is run
|
||||
- **THEN** every code in `STORAGE_MUX_CODES` has at least one test that triggers the call site that throws or surfaces it
|
||||
|
||||
**Verified by:** `src/error-codes/error-codes.test.ts::"every STORAGE_MUX code is reachable from a real call site"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Observability — EventBusLike events on lifecycle transitions
|
||||
|
||||
The mux SHALL emit events through an injected `EventBusLike` (from `@suaveplan/types`) on the following lifecycle transitions: `storage-mux.backend.down`, `storage-mux.backend.recovered`, `storage-mux.failover`, `storage-mux.fanout-partial`, `storage-mux.drain.started`, `storage-mux.drain.completed`, `storage-mux.backend.added`, `storage-mux.backend.removed`, `storage-mux.weight.changed`. When no `EventBusLike` is provided the mux SHALL operate normally without emitting (silent fallback).
|
||||
|
||||
#### Scenario: failover emits storage-mux.failover with both backend names
|
||||
|
||||
- **WHEN** the failover mode advances from `backends[0]` to `backends[1]` due to a 503
|
||||
- **THEN** exactly one `storage-mux.failover` event is emitted carrying both backend names AND the underlying error
|
||||
|
||||
#### Scenario: no bus configured does not throw
|
||||
|
||||
- **WHEN** the mux is constructed without `events` and a failover occurs
|
||||
- **THEN** the operation completes successfully AND no error is thrown for the missing bus
|
||||
|
||||
**Verified by:** `src/observability/events.test.ts::"failover emits storage-mux.failover with from and to backend names"` AND `src/observability/events.test.ts::"absent EventBus does not throw on lifecycle transitions"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Observability — OTel parent + per-attempt child spans
|
||||
|
||||
Every public operation SHALL open a parent OTel span named `storage-mux.<op>` with attributes `mux.mode`, `mux.key` (when present). Every backend attempt SHALL open a child span named `storage-mux.attempt` carrying `mux.backend.name`, `mux.attempt.index` (0-based), `mux.mode`, `mux.outcome` ∈ `{ok, transient, semantic, abort, timeout}`, and `mux.latency.ms`. When no tracer (`TracerLike` from `@suaveplan/types`) is provided the mux SHALL operate normally without recording spans.
|
||||
|
||||
#### Scenario: failover attempt records two child spans under one parent
|
||||
|
||||
- **WHEN** the failover mode invokes `backends[0]` (rejects) then `backends[1]` (succeeds)
|
||||
- **THEN** the test tracer records one parent `storage-mux.get` span AND two child `storage-mux.attempt` spans with `mux.attempt.index` `0` and `1` AND `mux.outcome` `transient` and `ok`
|
||||
|
||||
**Verified by:** `src/observability/otel.test.ts::"records parent op span and one child span per backend attempt with outcome"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: StorageLike conformance under every mode and recursive composition
|
||||
|
||||
The mux SHALL pass `runStorageConformance` (from `@suaveplan/storage`) when configured with three `MemoryStorage` backends under each of the six modes (`failover`, `round-robin`, `weighted`, `sticky-by-key`, `shield`, `mirror`). The mux SHALL additionally pass conformance when configured recursively as `Mux(failover, [Mux(mirror, mems), Mux(mirror, mems)])`. The conformance suite SHALL include `setPartial` sparse-extension and overwrite semantics, capability-honest reporting, and AbortSignal propagation.
|
||||
|
||||
#### Scenario: conformance passes under all six modes
|
||||
|
||||
- **WHEN** `runStorageConformance(mux)` runs six times — once per mode — against three MemoryStorage backends
|
||||
- **THEN** every conformance assertion passes for every mode
|
||||
|
||||
#### Scenario: conformance passes under recursive composition
|
||||
|
||||
- **WHEN** `runStorageConformance(recursiveMux)` runs against the two-level configuration
|
||||
- **THEN** every conformance assertion passes
|
||||
|
||||
**Verified by:** `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under failover"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under round-robin"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under weighted"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under sticky-by-key"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under shield"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under mirror"` AND `src/mux/mux.spec-conformance.test.ts::"passes runStorageConformance under recursive failover-of-mirrors"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Tier purity — universal, no Node or DOM imports
|
||||
|
||||
The package SHALL NOT import from `node:*`, `bun:*`, or any DOM-only API. Hashing SHALL be pure-JS (FNV-1a). Randomness SHALL come from an injectable `Rng` (`@suaveplan/types`). Timeouts SHALL use the injected `Clock`. The package SHALL run unmodified under Node 18+, Bun, Deno, modern browsers, and Cloudflare Workers.
|
||||
|
||||
#### Scenario: tier-purity grep finds no Node or DOM imports
|
||||
|
||||
- **WHEN** `grep -rnE "from ['\"](node:|bun:)" packages/storage/storage-mux/src/` is run
|
||||
- **THEN** the output is empty
|
||||
|
||||
**Verified by:** `src/mux/mux.spec-conformance.test.ts::"src tree contains no node: or bun: imports"`
|
||||
110
openspec/changes/complete-storage-mux/tasks.md
Normal file
110
openspec/changes/complete-storage-mux/tasks.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Tasks — complete-storage-mux
|
||||
|
||||
> Every checkbox MUST reflect work actually done in this change.
|
||||
|
||||
## 0. Pre-flight (non-negotiable)
|
||||
|
||||
- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
|
||||
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
|
||||
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
|
||||
|
||||
## 1. Spec authoring
|
||||
|
||||
- [ ] 1.1 Read `openspec/specs/storage/spec.md` and confirm `AbstractStorage` surface
|
||||
- [ ] 1.2 Read `openspec/AGENTS.md` §0c, §13, §14
|
||||
- [ ] 1.3 Read archived `openspec/archive/2026-05-16-add-storage-cdn/` for the deferral and family conventions
|
||||
- [ ] 1.4 Read `packages/flow/retry/src/` to confirm `runWithRetry` + `RetryStrategy` surface
|
||||
- [ ] 1.5 Read `packages/events/events/src/` to confirm `EventBusLike` shape
|
||||
- [ ] 1.6 Draft `proposal.md`, `design.md`, `specs/storage-mux/spec.md`, `tasks.md`
|
||||
- [ ] 1.7 Numbered Requirement/Scenario checklist per §0c step 6
|
||||
|
||||
## 2. Scaffolding
|
||||
|
||||
- [ ] 2.1 Create `packages/storage/storage-mux/` mirroring `packages/storage/storage-cdn/` conventions
|
||||
- [ ] 2.2 `package.json` v0.0.0, `"sideEffects": false`, subpath exports per feature directory
|
||||
- [ ] 2.3 Deps: `@suaveplan/error`, `@suaveplan/events`, `@suaveplan/retry`, `@suaveplan/storage`, `@suaveplan/types`, `@opentelemetry/api`, `zod`, `tslib`
|
||||
- [ ] 2.4 DevDeps: `@suaveplan/testing`, `@vitest/coverage-istanbul`, `fast-check`, `mitata`, `typescript`, `vite`, `vite-plugin-dts`, `vitest`
|
||||
- [ ] 2.5 `tsconfig.json`, `vite.config.ts`, `vitest.config.ts` via `createTestConfig` with `thresholds: 100/100/100/100`
|
||||
- [ ] 2.6 Root `README.md` with Modules section linking every co-located `.md`
|
||||
|
||||
## 3. Error registry
|
||||
|
||||
- [ ] 3.1 `src/error-codes/error-codes.ts` — `defineCodes("STORAGE_MUX", [...])` with all 14 codes: `ALL_BACKENDS_DOWN`, `NO_BACKENDS_CONFIGURED`, `HEALTHCHECK_FAILED`, `BACKEND_TIMEOUT`, `INVALID_MODE`, `INVALID_WEIGHT`, `WRITE_FANOUT_PARTIAL`, `BACKEND_ABORTED`, `BACKEND_NOT_FOUND`, `BACKEND_NAME_CONFLICT`, `BACKEND_DRAINING`, `WEIGHTED_REQUIRES_WEIGHTS`, `MIRROR_ORIGIN_REQUIRED`, `RECURSION_DEPTH_EXCEEDED`
|
||||
- [ ] 3.2 `StorageMuxError extends SuaveplanError<StorageMuxErrorCode>` with `backendName`, `attemptIndex`, `attemptCount`, `mode` discriminants
|
||||
- [ ] 3.3 `error-codes.md` + `error-codes.test.ts` — reachability sweep proving every code is thrown from a real call site
|
||||
|
||||
## 4. Types & schemas
|
||||
|
||||
- [ ] 4.1 `src/types/backend-entry.types.ts` — `BackendEntry` (`name`, `storage`, `weight?`, `healthcheck?`, `tags?`)
|
||||
- [ ] 4.2 `src/types/mux-options.types.ts` — `MuxMode` union, `StorageMuxOptions`, `BackendStats`, `MuxStats`
|
||||
- [ ] 4.3 `src/types/mux-options.schemas.ts` — Zod schemas: non-empty backends, mode whitelist, `weight >= 0`, `cooldownMs >= 0`, unique names
|
||||
|
||||
## 5. Classification
|
||||
|
||||
- [ ] 5.1 `src/classify/classify.ts` + `.md` + `.test.ts` — pure `classify(err) => "transient" | "semantic" | "abort"` per design §4
|
||||
|
||||
## 6. Bulkhead
|
||||
|
||||
- [ ] 6.1 `src/bulkhead/bulkhead.ts` + `.md` + `.test.ts` — `Promise.race` timeout, `AbortSignal.any` composition, hung-promise non-blocking
|
||||
|
||||
## 7. Health
|
||||
|
||||
- [ ] 7.1 `src/health/health.ts` + `.md` + `.test.ts` — `BackendHealth` tracker, exponential cooldown capped at `maxCooldownMs`, recovery probe, active healthcheck scheduling via injected `Clock`, draining accounting
|
||||
|
||||
## 8. Stats
|
||||
|
||||
- [ ] 8.1 `src/stats/stats.ts` + `.md` + `.test.ts` — per-backend counters, Vitter Algorithm R reservoir-sampled latencies (256 slots), `mux.stats()` snapshot API
|
||||
|
||||
## 9. Modes (six)
|
||||
|
||||
- [ ] 9.1 `src/modes/selector.types.ts` — `BackendSelector` + `BackendExecutor` interfaces
|
||||
- [ ] 9.2 `src/modes/failover/failover.ts` + `.md` + `.test.ts` — fall-through + retry composition + `ALL_BACKENDS_DOWN`
|
||||
- [ ] 9.3 `src/modes/round-robin/round-robin.ts` + `.md` + `.test.ts` — atomic counter, skip down/draining
|
||||
- [ ] 9.4 `src/modes/weighted/weighted.ts` + `.md` + `.test.ts` — Walker alias-method, rebuild on mutation, weight validation
|
||||
- [ ] 9.5 `src/modes/sticky-by-key/sticky-by-key.ts` + `.md` + `.test.ts` — FNV-1a, deterministic, redistribution on backend removal
|
||||
- [ ] 9.6 `src/modes/shield/shield.ts` + `.md` + `.test.ts` — origin-first reads, fan-out writes, `WRITE_FANOUT_PARTIAL` opt-in
|
||||
- [ ] 9.7 `src/modes/mirror/mirror.ts` + `.md` + `.test.ts` — `Promise.any` reads, `Promise.allSettled` writes, symmetric peers
|
||||
|
||||
## 10. Runtime mutation
|
||||
|
||||
- [ ] 10.1 `src/mutate/mutate.ts` + `.md` + `.test.ts` — `addBackend`, `removeBackend`, `setWeight`, `drain` with in-flight accounting
|
||||
|
||||
## 11. Observability
|
||||
|
||||
- [ ] 11.1 `src/observability/events.ts` + `.md` + `.test.ts` — event topic constants and emit helpers consuming `EventBusLike`
|
||||
- [ ] 11.2 `src/observability/otel.ts` + `.md` + `.test.ts` — per-attempt child-span helper with attribute set
|
||||
|
||||
## 12. Mux core
|
||||
|
||||
- [ ] 12.1 `src/mux/mux.ts` + `.md` — `StorageMux extends AbstractStorage`, per-op routing through selector + bulkhead + `runWithRetry`, AbortSignal.any composition, recursion depth guard
|
||||
- [ ] 12.2 `src/mux/mux.test.ts` — construction validation, mode dispatch, abort surfacing, recursion cap
|
||||
- [ ] 12.3 `src/mux/mux.spec-conformance.test.ts` — `runStorageConformance` × 6 modes × 3-backend MemoryStorage fleet, plus one two-level recursive mux
|
||||
|
||||
## 13. Barrel + exports
|
||||
|
||||
- [ ] 13.1 `src/index.ts` — explicit named re-exports per §6a (NO `export *`)
|
||||
- [ ] 13.2 `package.json` subpath exports — one entry per feature directory; verify count matches
|
||||
|
||||
## 14. Documentation
|
||||
|
||||
- [ ] 14.1 Sibling `.md` for every implementation `.ts` (≥ 200 words, all required sections, real-import examples)
|
||||
- [ ] 14.2 `README.md` documents each of the six modes with a usage example + a section on runtime mutation + stats
|
||||
|
||||
## 15. Gates
|
||||
|
||||
- [ ] 15.1 `bun run check`
|
||||
- [ ] 15.2 `bunx turbo lint typecheck test --filter=@suaveplan/storage-mux`
|
||||
- [ ] 15.3 `bunx turbo test:coverage --filter=@suaveplan/storage-mux` (100/100/100/100)
|
||||
- [ ] 15.4 `bun scripts/lint/check-test-runner-imports.ts` → zero output
|
||||
- [ ] 15.5 Tier-purity grep `grep -rnE "from ['\"](node:|bun:)" packages/storage/storage-mux/src/` → empty
|
||||
- [ ] 15.6 Banned-pattern scan → empty
|
||||
- [ ] 15.7 `bun scripts/openspec-conformance.ts @suaveplan/storage-mux`
|
||||
|
||||
## 16. Manifest & archival
|
||||
|
||||
- [ ] 16.1 `bun scripts/folder-reorg/generate-manifest.ts`
|
||||
- [ ] 16.2 `bun scripts/generate-missing.ts`
|
||||
- [ ] 16.3 Update `STATUS.md` and `SUMMARY.md`
|
||||
- [ ] 16.4 `bunx markdownlint-cli2 --fix`
|
||||
- [ ] 16.5 Fold `specs/storage-mux/spec.md` → `openspec/specs/storage-mux/spec.md`
|
||||
- [ ] 16.6 Move change folder → `openspec/archive/<yyyy-mm-dd>-complete-storage-mux/`
|
||||
Reference in New Issue
Block a user