254 lines
17 KiB
Markdown
254 lines
17 KiB
Markdown
# Design — @suaveplan/storage-uri-plugin-fallback
|
||
|
||
Non-normative companion to `proposal.md` and `specs/storage-uri-plugin-fallback/spec.md`.
|
||
|
||
## 1. URI grammar — decision matrix
|
||
|
||
The `fallback://` scheme must encode an **ordered list of embedded URIs**, each of which is itself a complete URI (so contains `://` and arbitrary reserved characters). Four candidate grammars were evaluated:
|
||
|
||
| # | Form | Example | Pros | Cons |
|
||
|---|---|---|---|---|
|
||
| 1 | Percent-encoded comma-separated authority | `fallback://https%3A%2F%2Fcdn.example.com%2Ffoo,bundle%3A%2F%2Fpkg%2Ffoo` | Pure URI shape. | Authority section is not designed for `,` separators per RFC 3986 §3.2; hard to read; hard to author; awkward when embedded URIs carry their own query strings. |
|
||
| 2 | Opaque body after `fallback:` (no `//`), pipe-delimited | `fallback:https://cdn.example.com/foo\|bundle://pkg/foo` | Readable. | Not a hierarchical URI shape; `\|` is not URI-safe; needs a custom parser; pipes inside any candidate path break the parse. |
|
||
| 3 | **Query-param form with empty authority/path** | `fallback://?u=https%3A%2F%2Fcdn.example.com%2Ffoo&u=drive%3A%2F%2Fprimary%2Ffoo` | Natural URL handling via `new URL` / `URLSearchParams`; encoding done by stdlib; any number of candidates; each candidate is a single query value with stdlib-validated escaping; round-trips through any URL parser unchanged. | URLs can be long (rare in practice; cap at 8 candidates × 4096 chars). |
|
||
| 4 | List object passed via factory + symbolic short URI | `fallback://my-list-id`, list registered out-of-band | Shortest URI. | Requires out-of-band registration; loses the property that the URI is self-describing; not portable between processes. |
|
||
|
||
**Decision: Option 3 (query-param form).**
|
||
|
||
Rationale:
|
||
|
||
- The Web URL standard (`URL` constructor, `URLSearchParams`) parses and serialises it correctly with **zero** custom code; every JavaScript runtime (browser, Bun, Node, Workers) handles encoding identically.
|
||
- Each candidate is one `?u=` value; `URLSearchParams.getAll("u")` returns the ordered list in O(n) without ambiguity.
|
||
- Order is preserved because `URLSearchParams` is ordered.
|
||
- Encoding rules are RFC 3986-compliant out of the box; no edge case for `?`, `&`, `#`, `/`, `:` inside candidates.
|
||
- The URI is self-describing — no out-of-band registry; the URI string IS the manifest.
|
||
- Length is bounded but ample: `policy.maxCandidates` (default 16, hard ceiling 64) × `INVALID_URI` threshold from `@suaveplan/storage-uri` (4096 chars per candidate) = ~64 KiB worst case, comfortably within any real-world URI handler.
|
||
|
||
The grammar formally:
|
||
|
||
```text
|
||
fallback-uri = "fallback:" "//" authority "/" "?" candidates [ "#" fragment ]
|
||
authority = "" ; always empty
|
||
candidates = "u=" pct-encoded-uri *( "&" "u=" pct-encoded-uri )
|
||
; ordered; at least 1, at most maxCandidates
|
||
```
|
||
|
||
Canonical examples:
|
||
|
||
```text
|
||
fallback://?u=https%3A%2F%2Fcdn.example.com%2Ffoo&u=drive%3A%2F%2Fprimary%2Ffoo
|
||
fallback://?u=file%3A%2F%2F%2Ftmp%2Ffixtures%2Fa.json&u=https%3A%2F%2Ffixtures.test%2Fa.json
|
||
fallback://?u=cache%3A%2F%2Flocal%2Favatars%2Falice.jpg&u=cdn%3A%2F%2Fzone-eu%2Favatars%2Falice.jpg&u=drive%3A%2F%2Fprimary%2Favatars%2Falice.jpg
|
||
```
|
||
|
||
A helper `buildFallbackUri(uris: readonly string[]): string` is exported so authors never construct the URI by hand; it uses `URLSearchParams` and rejects empty / oversized inputs at the boundary.
|
||
|
||
## 2. File layout
|
||
|
||
```text
|
||
src/
|
||
index.ts # explicit named re-exports (rule 10)
|
||
error-codes/
|
||
error-codes.ts # defineCodes("FALLBACK_URI", [...]) + FallbackUriError
|
||
error-codes.md
|
||
error-codes.test.ts # reachability sweep — every code triggered
|
||
types/
|
||
policy.types.ts # FallbackPolicy, FallbackPluginOptions
|
||
policy.types.md
|
||
policy.schemas.ts # Zod boundary schemas
|
||
candidate.types.ts # CandidateError, CandidateAttempt
|
||
candidate.types.md
|
||
parse-uri/
|
||
parse-fallback-uri.ts # parseFallbackUri + buildFallbackUri
|
||
parse-fallback-uri.md
|
||
parse-fallback-uri.test.ts
|
||
classify/
|
||
classify.ts # error → "transient" | "semantic" | "abort"
|
||
classify.md
|
||
classify.test.ts
|
||
walk/
|
||
sequential.ts # ordered fall-through walk
|
||
sequential.md
|
||
sequential.test.ts
|
||
parallel.ts # Promise.any race; first-to-resolve wins
|
||
parallel.md
|
||
parallel.test.ts
|
||
recursion/
|
||
depth-context.ts # AsyncLocalStorage-equivalent depth + visited-set
|
||
depth-context.md
|
||
depth-context.test.ts
|
||
ops/
|
||
read.ts # get / createReadStream / range
|
||
read.md
|
||
read.test.ts
|
||
write.ts # set / createWriteStream / mirror fan-out
|
||
write.md
|
||
write.test.ts
|
||
metadata.ts # exists / getMetadata / stat
|
||
metadata.md
|
||
metadata.test.ts
|
||
delete.ts # mirror delete
|
||
delete.md
|
||
delete.test.ts
|
||
list.ts # union-with-dedup list
|
||
list.md
|
||
list.test.ts
|
||
observability/
|
||
events.ts # EventBusLike emit helpers
|
||
events.md
|
||
events.test.ts
|
||
otel.ts # counter / histogram / span hierarchy
|
||
otel.md
|
||
otel.test.ts
|
||
plugin/
|
||
plugin.ts # fallbackUriPlugin factory
|
||
plugin.md
|
||
plugin.test.ts
|
||
plugin.spec-conformance.test.ts # runUriPluginConformance pass
|
||
conformance/
|
||
fallback-conformance.ts # runFallbackConformance harness
|
||
fallback-conformance.md
|
||
fallback-conformance.test.ts
|
||
```
|
||
|
||
Every implementation file is ≤ 300 lines (250 soft); every test file ≤ 500 lines. The `walk/`, `ops/`, and `recursion/` directories are the natural responsibility boundaries — splitting `walk` from `ops` keeps the candidate-iteration algorithm separate from the per-operation surface so each can be tested in isolation.
|
||
|
||
## 3. Recursion-prevention algorithm
|
||
|
||
`fallback://` URIs can be nested (a candidate of a `fallback://` may itself be a `fallback://`). Two invariants protect the process:
|
||
|
||
1. **Depth cap** — each plugin call increments a depth counter carried on a context token (`DepthContext`). Construction injects an `AsyncLocalStorage`-equivalent abstraction (`DepthContext.run(state, fn)`); under the hood the universal-tier default is a closure-passed state object (the plugin does not depend on `node:async_hooks`). Exceeding `policy.maxDepth` (default 4, hard ceiling 8) throws `FALLBACK_URI.MAX_DEPTH_EXCEEDED`.
|
||
2. **Visited set** — every recursion frame appends the **original input URI** (not the parsed candidate) to a `visited: Set<string>`. When a candidate URI normalises to a value already in `visited`, the walk skips that candidate and emits `storage-uri.fallback.cycle-detected`; if **every** candidate cycles, the walk throws `FALLBACK_URI.CIRCULAR_REFERENCE` with `cycleChain: readonly string[]` populated.
|
||
|
||
Normalisation: the visited-set key is the URI string after `parseFallbackUri` reserialisation (canonical query-param order, lowercased scheme, no fragment). This catches `?u=A&u=B` vs `?u=B&u=A` re-orderings of the same set.
|
||
|
||
Pseudocode:
|
||
|
||
```ts
|
||
function resolveCandidate(uri: string, ctx: DepthContext) {
|
||
if (ctx.depth >= ctx.maxDepth) {
|
||
throw new FallbackUriError("MAX_DEPTH_EXCEEDED", { uri, depth: ctx.depth });
|
||
}
|
||
const canonical = canonicalise(uri);
|
||
if (ctx.visited.has(canonical)) {
|
||
ctx.events?.emit("storage-uri.fallback.cycle-detected", { uri: canonical });
|
||
return { kind: "cycle" } as const;
|
||
}
|
||
return ctx.resolver.dispatch(
|
||
uri,
|
||
{ ...ctx, depth: ctx.depth + 1, visited: new Set([...ctx.visited, canonical]) }
|
||
);
|
||
}
|
||
```
|
||
|
||
## 4. Write-mirror semantics
|
||
|
||
By default, writes mirror to **all** candidates so subsequent reads see consistent content regardless of which candidate wins the race. Trade-offs documented:
|
||
|
||
| `writeMode` | Behaviour | Use case |
|
||
|---|---|---|
|
||
| `"all"` (default) | `Promise.allSettled` over every candidate; any rejection throws `FALLBACK_URI.WRITE_MIRROR_FAILED` with `mirrorOutcomes: readonly { uri, status, error? }[]`. | CDN + origin pairing where every mirror must stay current. |
|
||
| `"first-success"` | Walk in order; first success returns; pending writes are allowed to settle in the background (logged via events but not awaited). | Caching tiers where the upstream origin is authoritative and lower tiers are best-effort populated. |
|
||
|
||
Per-candidate write failures preserve the underlying `SuaveplanError` via `mirrorOutcomes[i].error`; non-`SuaveplanError` throwables are wrapped as `FALLBACK_URI.ALL_CANDIDATES_FAILED`-style sub-errors via the resolver's normal `STORAGE_URI.ADAPTER_ERROR` path (the plugin trusts the resolver's wrap rules; it never re-wraps a namespaced error).
|
||
|
||
When `writeMode === "all"` and **all** candidates fail, the plugin throws `FALLBACK_URI.WRITE_MIRROR_FAILED` (not `ALL_CANDIDATES_FAILED`) so callers can pattern-match on intent — `WRITE_MIRROR_FAILED` is "the operation was a mirror; some leg failed"; `ALL_CANDIDATES_FAILED` is "the operation was a walk; every leg failed."
|
||
|
||
`delete` follows the same matrix with one variation: success is defined as "at least one candidate deleted" by default (matches `StorageLike.delete`'s boolean return semantics — any consistent mirror has been removed). Opt in to strict deletion with `policy.deleteMode === "all"` for parity with writes.
|
||
|
||
## 5. List-union dedup
|
||
|
||
`list()` collects results from every candidate, then deduplicates by canonical URI string:
|
||
|
||
```ts
|
||
const seen = new Set<string>();
|
||
const merged: string[] = [];
|
||
for (const result of perCandidateResults) {
|
||
for (const uri of result.uris) {
|
||
if (seen.has(uri)) continue;
|
||
seen.add(uri);
|
||
merged.push(uri);
|
||
}
|
||
}
|
||
```
|
||
|
||
Order is **stable across candidates**: first-occurrence wins. `hasMore` is true if **any** candidate reports `hasMore: true`. `continuationToken` is intentionally NOT propagated by default — the plugin returns `continuationToken: undefined` because tokens are opaque per-adapter and cannot be merged. Callers needing paginated listing across mirrors must drive each candidate's `list` themselves; the plugin documents this in `ops/list.md`.
|
||
|
||
Under `listMode: "concat"` (opt-in), the plugin returns the raw concatenation without dedup — useful when callers want to count per-candidate hits.
|
||
|
||
When per-candidate `list` calls partially fail, the plugin throws `FALLBACK_URI.LIST_UNION_FAILED` carrying the per-candidate outcomes; the policy may relax this to "emit warning event, return partial union" via `policy.listFailureMode: "throw" | "partial"`.
|
||
|
||
## 6. Complementarity with `@suaveplan/storage-mux`
|
||
|
||
`storage-mux` and `storage-uri-plugin-fallback` solve adjacent problems at different layers:
|
||
|
||
| Concern | `storage-mux` | `storage-uri-plugin-fallback` |
|
||
|---|---|---|
|
||
| Layer | `AbstractStorage` (adapter) | URI scheme |
|
||
| Input | Pre-constructed `StorageLike` instances + policy | URI string with embedded candidate URIs |
|
||
| Backend identity | `BackendEntry.name` | URI string |
|
||
| Cross-scheme support | All backends must satisfy `StorageLike`; the mux does not know about schemes | Every candidate may use a different scheme (`https`, `drive`, `bundle`, `cache`, …) |
|
||
| Composability | Recursive (a mux is a `StorageLike`) | Recursive (a `fallback://` candidate may itself be `fallback://`) |
|
||
| Policy surface | Six modes (failover, round-robin, weighted, sticky, shield, mirror) | One mode (sequential walk with optional race), simpler |
|
||
| State | Per-backend health, weights, drain | Stateless per call |
|
||
| Observability | Per-backend span hierarchy + stats | Per-candidate span hierarchy + per-candidate counter |
|
||
|
||
**When to choose which:**
|
||
|
||
- If the calling code holds `StorageLike` instances directly (the storage layer is a private composition concern), use `storage-mux`.
|
||
- If the calling code receives **URIs from manifests / config / wire formats** and needs failover without changing its interface, use `fallback://`.
|
||
- They can be combined: a `fallback://` candidate may be a URI that ultimately resolves through a `storage-mux`-backed plugin, and vice versa.
|
||
|
||
The two packages share a **classification table** (`design.md` §7) so transient/semantic/abort semantics are uniform across the stack — copy-paste sync between `storage-mux/src/classify` and `storage-uri-plugin-fallback/src/classify` is enforced by a per-PR check in CI.
|
||
|
||
## 7. Error classification table
|
||
|
||
Mirrors `storage-mux` design §4 for cross-package consistency:
|
||
|
||
| Error shape | Class | Triggers fall-through? | Code surfaced |
|
||
|---|---|---|---|
|
||
| `AbortError` / `signal.aborted` from caller | abort | NO (stop walk) | `FALLBACK_URI.ABORTED` |
|
||
| `STORAGE_URI.NOT_FOUND` | semantic-but-fall-through (policy) | YES (configurable; default YES) | wrapped, last in cause chain on exhaustion |
|
||
| `STORAGE_URI.INVALID_URI` / `STORAGE_URI.UNKNOWN_SCHEME` | semantic | NO (rethrow — caller bug) | rethrow |
|
||
| `STORAGE_URI.WRITE_UNSUPPORTED` | semantic | NO (rethrow) | rethrow |
|
||
| Per-candidate `STORAGE_URI.ADAPTER_ERROR` | transient | YES | last in cause chain |
|
||
| Per-candidate `STORAGE_URI.ABORTED` from inner timeout | transient | YES (if `stopOnAbort: false`) | wrapped |
|
||
| Unknown thrown value (non-`Error`) | transient | YES | wrapped (resolver normalises first) |
|
||
| `FALLBACK_URI.MAX_DEPTH_EXCEEDED` | semantic | NO | rethrow (programmer bug) |
|
||
| `FALLBACK_URI.CIRCULAR_REFERENCE` | semantic | NO | rethrow |
|
||
|
||
`policy.errorClass: "any"` re-classifies all semantic errors as fall-through-eligible; the default (`"transient"` + NOT_FOUND) is the safer choice because it preserves the resolver's caller-bug surfaces (`INVALID_URI`, `UNKNOWN_SCHEME`) without forcing every consumer to opt out.
|
||
|
||
## 8. Error-code matrix
|
||
|
||
| Code | Thrown from | Discriminants populated |
|
||
|---|---|---|
|
||
| `FALLBACK_URI.NO_CANDIDATES` | `parseFallbackUri` when `?u=` count is 0 | `uri` |
|
||
| `FALLBACK_URI.INVALID_URI` | `parseFallbackUri` on malformed URI (missing scheme, non-empty authority, > 4096 chars total) | `uri` |
|
||
| `FALLBACK_URI.ALL_CANDIDATES_FAILED` | `walk/sequential.ts`, `walk/parallel.ts` on exhaustion (reads) | `uri`, `candidateErrors`, `cause` (last error) |
|
||
| `FALLBACK_URI.WRITE_MIRROR_FAILED` | `ops/write.ts` under `writeMode: "all"` when any candidate rejects | `uri`, `mirrorOutcomes` |
|
||
| `FALLBACK_URI.LIST_UNION_FAILED` | `ops/list.ts` under `listFailureMode: "throw"` when any candidate rejects | `uri`, `candidateErrors` |
|
||
| `FALLBACK_URI.ABORTED` | walk entry when `signal.aborted === true`, or mid-walk when `stopOnAbort: true` | `uri` |
|
||
| `FALLBACK_URI.MAX_DEPTH_EXCEEDED` | `recursion/depth-context.ts` when `ctx.depth >= ctx.maxDepth` | `uri`, `depth` |
|
||
| `FALLBACK_URI.CIRCULAR_REFERENCE` | `walk/*.ts` when every candidate has already been visited | `uri`, `cycleChain` |
|
||
|
||
Every code is reachable from a real call site exercised by `src/error-codes/error-codes.test.ts` (the reachability sweep).
|
||
|
||
## 9. Tier purity
|
||
|
||
Universal — no `node:*`, no `bun:*`, no DOM. Specifically:
|
||
|
||
- `URL` and `URLSearchParams` are Web standards (available in every runtime).
|
||
- `AbortSignal.any` is ES2024; a tiny polyfill is provided under `src/parse-uri/abort-any.ts` for runtimes that lack it (same approach as `storage-mux`).
|
||
- Depth context uses closure passing (`DepthContext` value threaded through call options) rather than `node:async_hooks`. This is sufficient because the plugin's recursion is all within a single call chain — there is no cross-`await`-boundary state to track that the caller doesn't already own.
|
||
- `Clock` is injected via `@suaveplan/types#Clock` for any time-keeping (latency histograms only); the plugin has no scheduled work of its own.
|
||
|
||
## 10. Trade-offs
|
||
|
||
- **Query-param URI grammar vs. shorter forms.** Option 3 trades a few characters of length for zero custom parsing and standards alignment. Worth it; the URIs are not user-facing.
|
||
- **Mirror writes by default.** Cheaper to mirror up-front than to discover a stale mirror on the next read. Callers that explicitly want first-success-wins opt in via `writeMode: "first-success"`.
|
||
- **List dedup by canonical URI.** Cannot detect content drift between mirrors; callers needing byte-equality must use a content-addressed scheme. Acceptable because content-verification is out of scope (proposal §"Out of scope").
|
||
- **Depth cap 4 default, 8 hard ceiling.** Most use cases are 2–3 levels (cache → CDN → origin). Eight covers pathological compositions without enabling DoS via runaway recursion.
|
||
- **Stateless plugin.** Unlike `storage-mux`, there is no per-candidate health tracking. The fallback walk is one attempt per candidate per call; if you need backoff/health, wrap each candidate's underlying plugin in `@suaveplan/retry` or stack a `storage-mux` underneath. Keeps the plugin's surface small and the semantics auditable.
|
||
- **Cycle detection by URI string.** Cheap and correct for the common case; cannot catch a cycle that goes through a non-`fallback://` indirection (e.g. an HTTP redirect chain). That's not the plugin's problem to solve.
|