222 lines
9.3 KiB
Markdown
222 lines
9.3 KiB
Markdown
# Design — @suaveplan/storage-uri
|
|
|
|
## Tier and dependency posture
|
|
|
|
Universal tier. Zero `node:`/`bun:` imports. Runtime deps:
|
|
|
|
- `@suaveplan/error` — `defineCodes`, `SuaveplanError`
|
|
- `@suaveplan/types` — `StorageLike` structural contract
|
|
- `@opentelemetry/api` — meter + tracer
|
|
- `tslib` — runtime helpers
|
|
|
|
`@suaveplan/storage` is **not** a dep. The resolver types against `StorageLike` from `@suaveplan/types` so an in-memory test stub satisfies the contract without pulling in the `AbstractStorage` runtime.
|
|
|
|
## File layout
|
|
|
|
```
|
|
packages/storage/storage-uri/
|
|
package.json
|
|
README.md
|
|
src/
|
|
index.ts # explicit named re-exports (rule 10)
|
|
error-codes/
|
|
error-codes.ts
|
|
error-codes.test.ts
|
|
error-codes.md
|
|
index.ts
|
|
parse-uri/
|
|
parse-uri.ts # RFC 3986 split — scheme + authority + path + query
|
|
parse-uri.test.ts
|
|
parse-uri.md
|
|
index.ts
|
|
types/
|
|
plugin.types.ts # UriPlugin, ParsedUri
|
|
plugin.types.md
|
|
resolver.types.ts # ResolverOptions, StatResult
|
|
resolver.types.md
|
|
index.ts
|
|
resolver/
|
|
resolver.ts # StorageUriResolver, dispatch + delegation
|
|
resolver.test.ts
|
|
resolver.streams.ts # open() / createReadStream fallback
|
|
resolver.streams.test.ts
|
|
resolver.writes.ts # write() / createWriteStream drain
|
|
resolver.writes.test.ts
|
|
resolver.list.ts # list() prefix → URIs re-serialisation
|
|
resolver.list.test.ts
|
|
resolver.spec-conformance.test.ts
|
|
resolver.md
|
|
index.ts
|
|
conformance/
|
|
conformance.ts # runUriPluginConformance(plugin, fixtures)
|
|
conformance.test.ts
|
|
conformance.md
|
|
index.ts
|
|
otel/
|
|
otel.ts
|
|
otel.test.ts
|
|
otel.md
|
|
index.ts
|
|
tests/
|
|
setup/polyfills.ts
|
|
integration/cross-scheme.test.ts # storage-file + storage-http composition
|
|
tsconfig.json
|
|
tsconfig.lib.json
|
|
vite.config.ts
|
|
vitest.config.ts
|
|
```
|
|
|
|
Every `.ts` outside the listed exclusions (`.test.ts`, `.types.ts` inside a feature dir, `index.ts` barrels) has a sibling `.md`.
|
|
|
|
## Resolver dispatch
|
|
|
|
```ts
|
|
class StorageUriResolver {
|
|
readonly #plugins: ReadonlyMap<string, UriPlugin>;
|
|
|
|
constructor(plugins: readonly UriPlugin[]) {
|
|
const map = new Map<string, UriPlugin>();
|
|
for (const plugin of plugins) {
|
|
const existing = map.get(plugin.scheme);
|
|
if (existing && existing !== plugin) {
|
|
throw new StorageUriError("SCHEME_CONFLICT", { scheme: plugin.scheme });
|
|
}
|
|
map.set(plugin.scheme, plugin);
|
|
}
|
|
this.#plugins = map;
|
|
}
|
|
|
|
#dispatch(uri: string): { plugin: UriPlugin; key: string; meta?: Record<string, string> } {
|
|
const parsed = parseUri(uri); // throws INVALID_URI
|
|
const plugin = this.#plugins.get(parsed.scheme); // throws UNKNOWN_SCHEME
|
|
if (!plugin) throw new StorageUriError("UNKNOWN_SCHEME", { uri, scheme: parsed.scheme });
|
|
return { plugin, ...plugin.parse(uri) };
|
|
}
|
|
}
|
|
```
|
|
|
|
Dispatch is O(1) in plugin count. The resolver is frozen after construction; mutation is impossible.
|
|
|
|
## Stream fallback strategy
|
|
|
|
```ts
|
|
async open(uri: string, signal?: AbortSignal): Promise<ReadableStream<Uint8Array>> {
|
|
const { plugin, key } = this.#dispatch(uri);
|
|
const adapter = plugin.adapter as StorageLike & { createReadStream?(k: string, s?: AbortSignal): ReadableStream<Uint8Array> };
|
|
|
|
if (typeof adapter.createReadStream === "function") {
|
|
return adapter.createReadStream(key, signal);
|
|
}
|
|
|
|
// Fallback: one-chunk stream from get()
|
|
this.#otel.streamFallback(plugin.scheme);
|
|
const bytes = await adapter.get(key);
|
|
if (!bytes) throw new StorageUriError("NOT_FOUND", { uri });
|
|
return new ReadableStream({
|
|
start(ctrl) { ctrl.enqueue(bytes); ctrl.close(); },
|
|
});
|
|
}
|
|
```
|
|
|
|
The fallback path materialises the whole object in memory. The `storage_uri.stream_fallback_total{scheme}` counter is the operator's alert signal that a misuse is occurring (large object behind a non-streaming adapter).
|
|
|
|
## Write buffering strategy
|
|
|
|
Symmetric: `write(uri, body)` accepts `Uint8Array` or `ReadableStream<Uint8Array>`. Streams are drained via `Response(body).arrayBuffer()` into a single `Uint8Array`, then handed to `adapter.set(key, bytes)`. Counter: `storage_uri.buffered_write_total{scheme}`.
|
|
|
|
Adapters with native `createWriteStream` SHALL be used directly when present — the fallback runs only when the adapter has no streaming write surface.
|
|
|
|
## Per-scheme parse contract
|
|
|
|
Each adapter package owns its scheme parser. Examples:
|
|
|
|
| Scheme | Plugin owner | `parse(uri)` returns |
|
|
|---|---|---|
|
|
| `file:` | `storage-file` | `{ key: <abs path resolved against rootDir>, meta: {} }` |
|
|
| `http:` / `https:` | `storage-http` | `{ key: <pathname>, meta: { query: search } }` |
|
|
| `rackspace:` | `storage-cdn-rackspace` | `{ key: <pathname>, meta: { container: <authority> } }` |
|
|
| `bunny:` | `storage-cdn-bunnycdn` | `{ key: <pathname>, meta: { zone: <authority> } }` |
|
|
| `s3:` | `storage-s3` | `{ key: <pathname>, meta: { bucket: <authority> } }` |
|
|
|
|
Adapters that need scheme-specific addressing (bucket, container, zone) populate `meta` and are responsible for forwarding those values to their pre-bound adapter instance. The resolver never inspects `meta`.
|
|
|
|
## Error-code map
|
|
|
|
| Code | When | HTTP-ish analogue |
|
|
|---|---|---|
|
|
| `STORAGE_URI.UNKNOWN_SCHEME` | URI scheme has no plugin registered | 404 (route miss) |
|
|
| `STORAGE_URI.INVALID_URI` | `parseUri` rejects the input | 400 |
|
|
| `STORAGE_URI.SCHEME_CONFLICT` | Two distinct plugins claim the same scheme | configuration error |
|
|
| `STORAGE_URI.NOT_FOUND` | Adapter returned `undefined` for `get`/`open` | 404 |
|
|
| `STORAGE_URI.LIST_UNSUPPORTED` | Plugin has no `build()` | 501 |
|
|
| `STORAGE_URI.WRITE_UNSUPPORTED` | Adapter throws on `set` (e.g. read-only HTTP) | 405 |
|
|
| `STORAGE_URI.ABORTED` | `AbortSignal` fired | 499 |
|
|
| `STORAGE_URI.ADAPTER_ERROR` | Adapter threw something else | 502 (upstream) |
|
|
|
|
`ADAPTER_ERROR` preserves the original via `cause` — operators trace back to the underlying `STORAGE_HTTP.*`/`STORAGE_FILE.*`/`STORAGE_S3.*` code.
|
|
|
|
## Conformance harness
|
|
|
|
```ts
|
|
export function runUriPluginConformance(plugin: UriPlugin, fx: {
|
|
validUris: string[];
|
|
invalidUris: string[];
|
|
buildRoundTrip?: Array<{ key: string; meta?: Record<string, string>; uri: string }>;
|
|
}): void;
|
|
```
|
|
|
|
Called from each adapter package's test suite (e.g. `storage-file/src/uri-plugin/uri-plugin.conformance.test.ts`). Asserts:
|
|
|
|
1. Every `validUris[i]` parses without throwing, and `plugin.parse(uri).key` is a non-empty string.
|
|
2. Every `invalidUris[i]` throws `STORAGE_URI.INVALID_URI` (or a wrapped equivalent).
|
|
3. If `build` is implemented, `build(parse(uri)) === uri` for the round-trip fixtures.
|
|
4. The plugin's scheme matches the URI's scheme prefix.
|
|
|
|
This is the bridge between the resolver's spec and each adapter's contribution — the spec stays in `storage-uri`, the test fixtures live with the adapter.
|
|
|
|
## OTel surface
|
|
|
|
Tracer: `@suaveplan/storage-uri`. Meter: same.
|
|
|
|
Spans: one per public op (`storage.uri.read`, `storage.uri.write`, `storage.uri.stat`, `storage.uri.exists`, `storage.uri.delete`, `storage.uri.list`, `storage.uri.open`). Attributes:
|
|
|
|
- `storage.uri` — the full input URI (redacted to authority + path; query string dropped to avoid signed-URL leaks).
|
|
- `storage.uri.scheme` — extracted scheme (no colon).
|
|
- `storage.uri.key` — post-parse key (subject to adapter-side redaction).
|
|
|
|
Histograms:
|
|
|
|
- `storage_uri.op_duration_ms{op, scheme, outcome}` — `outcome` ∈ `success` / `not_found` / `error` / `aborted`.
|
|
|
|
Counters:
|
|
|
|
- `storage_uri.reads_total{scheme, outcome}`
|
|
- `storage_uri.writes_total{scheme, outcome}`
|
|
- `storage_uri.deletes_total{scheme, outcome}`
|
|
- `storage_uri.errors_total{scheme, code}` — `code` is the `STORAGE_URI.*` enum (`ADAPTER_ERROR` covers all downstream)
|
|
- `storage_uri.stream_fallback_total{scheme}`
|
|
- `storage_uri.buffered_write_total{scheme}`
|
|
|
|
## Composition recipes (in README)
|
|
|
|
```ts
|
|
// Single resolver for a media pipeline.
|
|
const resolver = createStorageUriResolver([
|
|
fileUriPlugin({ rootDir: "/var/media" }),
|
|
httpsUriPlugin(httpStorage),
|
|
rackspaceUriPlugin(rackspace, "media-prod"),
|
|
bunnyUriPlugin(bunny, "cdn-prod"),
|
|
]);
|
|
|
|
const stream = await resolver.open("rackspace://media-prod/raw/clip.mov");
|
|
await resolver.write("file:///cache/thumbs/clip.jpg", thumbBytes);
|
|
const meta = await resolver.stat("https://cdn.example.com/clip/master.m3u8");
|
|
```
|
|
|
|
## Trade-offs accepted
|
|
|
|
- **No global registry.** Every consumer constructs its own resolver. Cost: a little boot-time wiring. Benefit: tree-shaking works, tests can build isolated resolvers, multi-tenant configurations are explicit.
|
|
- **No copy/move primitives.** Reason given in proposal Out of Scope. A future `storage-pipeline` package can offer streaming copy with progress, retries, and integrity checks.
|
|
- **`meta` is a `Record<string, string>`.** Adapter-specific addressing (bucket, container, zone) is stringly-typed at the plugin interface; type safety is the plugin's job (each adapter exports its own typed `parseXxxUri`).
|
|
- **`stat()` returns `undefined` for adapters that can't answer cheaply.** Avoids forcing a `HEAD` request inside `storage-file` or a full-object read inside CDN adapters that don't support `HEAD`. Consumers that need metadata SHOULD compose with `storage-http`'s ETag cache directly.
|