Initial Commit
This commit is contained in:
161
openspec/archive/2026-05-16-add-storage-http/design.md
Normal file
161
openspec/archive/2026-05-16-add-storage-http/design.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Design — @suaveplan/storage-http
|
||||
|
||||
Non-normative companion to `proposal.md` and `specs/storage-http/spec.md`. Architecture, algorithms, and trade-offs only — no testable requirements live here.
|
||||
|
||||
## 1. Mental model
|
||||
|
||||
`HttpStorage` is an `AbstractStorage` view over a URL prefix. The mapping is:
|
||||
|
||||
```text
|
||||
key "a/b/test.doc"
|
||||
baseUrl "https://cdn.example.com/start/here/"
|
||||
URL "https://cdn.example.com/start/here/a/b/test.doc"
|
||||
op →
|
||||
get GET <URL>
|
||||
exists HEAD <URL>
|
||||
getMetadata HEAD <URL> (or cached from previous GET)
|
||||
getPartial GET <URL>, Range: bytes=<o>-<o+l-1>
|
||||
set uploader.put(key, body, headers, signal) [if uploader wired]
|
||||
delete uploader.delete(key, signal) [if uploader supports it]
|
||||
list throw STORAGE_HTTP.READ_ONLY (always — see proposal §"Out of scope")
|
||||
```
|
||||
|
||||
Read-side is always available; write-side is gated by the presence of an `HttpUploader`. The split is captured in `capabilities`:
|
||||
|
||||
```ts
|
||||
{
|
||||
partialReads: true,
|
||||
partialWrites: false,
|
||||
nativeStreams: true,
|
||||
atomicWrites: false, // request body is committed in one call; not torn, but not the POSIX rename guarantee either
|
||||
contentAddressable: false,
|
||||
}
|
||||
```
|
||||
|
||||
## 2. File layout
|
||||
|
||||
```text
|
||||
src/
|
||||
index.ts # explicit named re-exports
|
||||
error-codes.ts # defineCodes("STORAGE_HTTP", [...]) + StorageHttpError
|
||||
types/
|
||||
http-storage.types.ts # HttpStorageOptions, HttpUploader, FetchLike
|
||||
http-storage.schemas.ts # HttpStorageOptionsSchema (Zod)
|
||||
fetch-adapter/
|
||||
fetch-adapter.ts # defaultFetch() wrapping globalThis.fetch
|
||||
fetch-adapter.md
|
||||
fetch-adapter.test.ts
|
||||
key-sanitizer/
|
||||
key-sanitizer.ts # sanitizeKey + per-segment encoder
|
||||
key-sanitizer.md
|
||||
key-sanitizer.test.ts
|
||||
range/
|
||||
range.ts # Range header builder + 206/200 disambiguation
|
||||
range.md
|
||||
range.test.ts
|
||||
etag-cache/
|
||||
etag-cache.ts # in-memory LRU; If-None-Match wiring
|
||||
etag-cache.md
|
||||
etag-cache.test.ts
|
||||
uploader/
|
||||
uploader.ts # HttpUploader contract, presignedPutUploader helper
|
||||
uploader.md
|
||||
uploader.test.ts
|
||||
retry-helper/
|
||||
retry-helper.ts # transient-status classifier + Retry-After honour
|
||||
retry-helper.md
|
||||
retry-helper.test.ts
|
||||
otel/
|
||||
otel.ts # meter/tracer wiring; recordOp(op, outcome, status_class, ms, bytes?)
|
||||
otel.md
|
||||
otel.test.ts
|
||||
http-storage/
|
||||
http-storage.ts # HttpStorage extends AbstractStorage
|
||||
http-storage.md
|
||||
http-storage.test.ts
|
||||
http-storage.spec-conformance.test.ts
|
||||
registry/
|
||||
registry.ts # declare module "@suaveplan/storage" + registerHttpProvider
|
||||
registry.md
|
||||
registry.test.ts
|
||||
```
|
||||
|
||||
Every implementation file ≤300 lines (soft 250); every `.test.ts` ≤500.
|
||||
|
||||
## 3. Algorithms / decisions
|
||||
|
||||
### 3a. Key → URL resolution
|
||||
|
||||
`new URL(encodedKey, baseUrl)` is the only safe primitive. Pitfalls:
|
||||
|
||||
- `baseUrl` MUST end with `/`; the Zod schema rejects non-`/` suffixes. Without the trailing slash, `new URL("a", "https://x/b")` resolves to `https://x/a` (the last path segment is treated as a file).
|
||||
- `encodedKey` is built by splitting the sanitised key on `/`, applying `keyEncoder` to each segment, then joining. This preserves the hierarchy (`a/b/test.doc` → `a/b/test.doc`) while encoding spaces, `?`, `#`, etc. (`hello world.txt` → `hello%20world.txt`).
|
||||
- Keys with a leading `/` are rejected by `sanitizeKey` because `new URL("/abs", "https://x/b/")` resolves to `https://x/abs` — losing the prefix.
|
||||
|
||||
### 3b. Status code → error code map
|
||||
|
||||
| HTTP | Error code | Retryable? |
|
||||
|---|---|---|
|
||||
| `200`, `206` | (success) | n/a |
|
||||
| `304` | (success — cached body returned) | n/a |
|
||||
| `401` | `UNAUTHORIZED` | no |
|
||||
| `403` | `FORBIDDEN` | no |
|
||||
| `404`, `410` | `NOT_FOUND` | no |
|
||||
| `408`, `425`, `429` | `UPSTREAM_ERROR` (then `NETWORK_ERROR` after retry exhaustion) | yes |
|
||||
| `5xx` | `UPSTREAM_ERROR` (then exhaustion → still `UPSTREAM_ERROR` with `retries_exhausted` attr) | yes |
|
||||
| `4xx` (other) | `UPSTREAM_ERROR` | no |
|
||||
| `fetch` reject (DNS, TLS, abort-without-signal) | `NETWORK_ERROR` | yes (network class) |
|
||||
| Pre-aborted signal | `ABORTED` (no request issued) | n/a |
|
||||
|
||||
`Retry-After` (seconds or HTTP-date) is honoured when present; otherwise the `@suaveplan/retry` exponential backoff is used. Jitter is delegated to `@suaveplan/retry`.
|
||||
|
||||
### 3c. Range reads
|
||||
|
||||
`getPartial(offset, length)` issues `Range: bytes=<offset>-<offset+length-1>`. Three response shapes are handled:
|
||||
|
||||
1. **`206 Partial Content`** — happy path. `Content-Range: bytes <o>-<o+l-1>/<total>` is parsed for size accounting; body is returned verbatim.
|
||||
2. **`200 OK`** — origin does not support Range. Full body is consumed and sliced; `storage_http.range_fallback_total` is incremented. Useful diagnostic for misconfigured origins.
|
||||
3. **`416 Range Not Satisfiable`** — caller asked beyond EOF. Mapped to `STORAGE_HTTP.NOT_FOUND` when the resource itself exists with shorter length (HEAD probe distinguishes "object missing" from "range past EOF" only on first encounter).
|
||||
|
||||
### 3d. ETag cache
|
||||
|
||||
Optional in-memory `Map`-backed LRU (default capacity: 256 entries). Keyed by sanitised key, valued by `{ etag, body, lastModified, expiresAt }`. On `get`:
|
||||
|
||||
1. If `cache.has(key)`, attach `If-None-Match: <etag>`.
|
||||
2. On `304`, return the cached body; refresh `lastModified` if present.
|
||||
3. On `200`, replace the cache entry with the new body (subject to LRU eviction).
|
||||
|
||||
The cache is **per-instance**, never shared across `HttpStorage` instances. Cross-process caching is composed by the caller (typically `storage-file` underneath).
|
||||
|
||||
### 3e. Write strategy
|
||||
|
||||
`HttpUploader` is the entire write surface. Its contract:
|
||||
|
||||
```ts
|
||||
interface HttpUploader {
|
||||
put(key: string, body: Uint8Array | ReadableStream<Uint8Array>, headers?: Record<string, string>, signal?: AbortSignal): Promise<void>;
|
||||
delete?(key: string, signal?: AbortSignal): Promise<void>; // optional
|
||||
}
|
||||
```
|
||||
|
||||
`HttpStorage.set` calls `uploader.put`; `delete` calls `uploader.delete` (or throws `READ_ONLY` if the uploader does not implement it). `createWriteStream` collects to memory and dispatches `uploader.put` on `close` — request-bodied origins do not support true chunked uploads without provider-specific multipart logic (deliberately out of scope).
|
||||
|
||||
The shipped `presignedPutUploader(presigner: (key) => Promise<{ url, headers? }>)` helper composes signing with `defaultFetch`. It is provider-agnostic — S3, R2, GCS, Azure Blob, or a custom endpoint all work by plugging in a presigner.
|
||||
|
||||
### 3f. Tier purity
|
||||
|
||||
Universal tier: `fetch`, `URL`, `AbortController`, `Uint8Array`, `ReadableStream` are all in the universal subset (Node 18+, modern browsers, Bun, Deno, Workers). No Node-only or DOM-only imports. The package is published with `"browser"` field omitted (universal) and no `engines` Node-version gate beyond the workspace baseline.
|
||||
|
||||
## 4. Trade-offs considered
|
||||
|
||||
- **Listing.** Rejected. WebDAV `PROPFIND` is the only HTTP-native directory protocol and is not what CDNs serve. Forcing a "best-effort" listing via index pages would lock callers into a specific origin's HTML layout. Shipping no `list` is honest.
|
||||
- **Built-in disk cache.** Rejected for the v0. Cache invalidation is the consumer's policy choice (TTL vs. ETag vs. revalidate-always). Composing `storage-http` behind a `storage-file` `NamespacedStorage` covers the disk-cache case without baking opinions into the adapter.
|
||||
- **Provider-specific helpers in this package.** Rejected. The moment we ship `s3PresignedPutUploader`, we drag `@aws-sdk/*` into the dep graph. The shipped `presignedPutUploader(presigner)` is the seam — provider packages live elsewhere.
|
||||
- **Atomic writes.** HTTP `PUT` is idempotent but not POSIX-rename-atomic. We declare `capabilities.atomicWrites: false` honestly. Callers that need atomicity (cache-coherent reads during writes) should layer their own staging logic.
|
||||
- **Conditional writes (`If-Match`).** Considered. Not v0 — would expand `HttpUploader` contract and the matching error code (`PRECONDITION_FAILED`). Easy follow-up if a consumer asks.
|
||||
|
||||
## 5. References
|
||||
|
||||
- `openspec/specs/storage/spec.md` — `AbstractStorage` contract this adapter conforms to.
|
||||
- `openspec/archive/2026-05-12-storage-file/` — closest precedent for adapter shape (capabilities, error registry, retry wiring).
|
||||
- `openspec/archive/2026-04-24-enterprise-storage-s3/` — network-backed adapter precedent (OTel attribute set, retry classification).
|
||||
Reference in New Issue
Block a user