suaveplan/openspec/archive/2026-05-16-add-storage-http/design.md

Design — @suaveplan/storage-http

Non-normative companion to proposal.md and specs/storage-http/spec.md. Architecture, algorithms, and trade-offs only — no testable requirements live here.

1. Mental model

HttpStorage is an AbstractStorage view over a URL prefix. The mapping is:

key       "a/b/test.doc"
baseUrl   "https://cdn.example.com/start/here/"
URL       "https://cdn.example.com/start/here/a/b/test.doc"
op        →
  get        GET <URL>
  exists     HEAD <URL>
  getMetadata HEAD <URL>          (or cached from previous GET)
  getPartial GET <URL>, Range: bytes=<o>-<o+l-1>
  set        uploader.put(key, body, headers, signal)     [if uploader wired]
  delete     uploader.delete(key, signal)                 [if uploader supports it]
  list       throw STORAGE_HTTP.READ_ONLY                 (always — see proposal §"Out of scope")

Read-side is always available; write-side is gated by the presence of an HttpUploader. The split is captured in capabilities:

{
  partialReads: true,
  partialWrites: false,
  nativeStreams: true,
  atomicWrites: false,           // request body is committed in one call; not torn, but not the POSIX rename guarantee either
  contentAddressable: false,
}

2. File layout

src/
  index.ts                                 # explicit named re-exports
  error-codes.ts                           # defineCodes("STORAGE_HTTP", [...]) + StorageHttpError
  types/
    http-storage.types.ts                  # HttpStorageOptions, HttpUploader, FetchLike
    http-storage.schemas.ts                # HttpStorageOptionsSchema (Zod)
  fetch-adapter/
    fetch-adapter.ts                       # defaultFetch() wrapping globalThis.fetch
    fetch-adapter.md
    fetch-adapter.test.ts
  key-sanitizer/
    key-sanitizer.ts                       # sanitizeKey + per-segment encoder
    key-sanitizer.md
    key-sanitizer.test.ts
  range/
    range.ts                               # Range header builder + 206/200 disambiguation
    range.md
    range.test.ts
  etag-cache/
    etag-cache.ts                          # in-memory LRU; If-None-Match wiring
    etag-cache.md
    etag-cache.test.ts
  uploader/
    uploader.ts                            # HttpUploader contract, presignedPutUploader helper
    uploader.md
    uploader.test.ts
  retry-helper/
    retry-helper.ts                        # transient-status classifier + Retry-After honour
    retry-helper.md
    retry-helper.test.ts
  otel/
    otel.ts                                # meter/tracer wiring; recordOp(op, outcome, status_class, ms, bytes?)
    otel.md
    otel.test.ts
  http-storage/
    http-storage.ts                        # HttpStorage extends AbstractStorage
    http-storage.md
    http-storage.test.ts
    http-storage.spec-conformance.test.ts
  registry/
    registry.ts                            # declare module "@suaveplan/storage" + registerHttpProvider
    registry.md
    registry.test.ts

Every implementation file ≤300 lines (soft 250); every .test.ts ≤500.

3. Algorithms / decisions

3a. Key → URL resolution

new URL(encodedKey, baseUrl) is the only safe primitive. Pitfalls:

• baseUrl MUST end with /; the Zod schema rejects non-/ suffixes. Without the trailing slash, new URL("a", "https://x/b") resolves to https://x/a (the last path segment is treated as a file).
• encodedKey is built by splitting the sanitised key on /, applying keyEncoder to each segment, then joining. This preserves the hierarchy (a/b/test.doc → a/b/test.doc) while encoding spaces, ?, #, etc. (hello world.txt → hello%20world.txt).
• Keys with a leading / are rejected by sanitizeKey because new URL("/abs", "https://x/b/") resolves to https://x/abs — losing the prefix.

3b. Status code → error code map

HTTP	Error code	Retryable?
200, 206	(success)	n/a
304	(success — cached body returned)	n/a
401	UNAUTHORIZED	no
403	FORBIDDEN	no
404, 410	NOT_FOUND	no
408, 425, 429	UPSTREAM_ERROR (then NETWORK_ERROR after retry exhaustion)	yes
5xx	UPSTREAM_ERROR (then exhaustion → still UPSTREAM_ERROR with retries_exhausted attr)	yes
4xx (other)	UPSTREAM_ERROR	no
fetch reject (DNS, TLS, abort-without-signal)	NETWORK_ERROR	yes (network class)
Pre-aborted signal	ABORTED (no request issued)	n/a

Retry-After (seconds or HTTP-date) is honoured when present; otherwise the @suaveplan/retry exponential backoff is used. Jitter is delegated to @suaveplan/retry.

3c. Range reads

getPartial(offset, length) issues Range: bytes=<offset>-<offset+length-1>. Three response shapes are handled:

1. 206 Partial Content — happy path. Content-Range: bytes <o>-<o+l-1>/<total> is parsed for size accounting; body is returned verbatim.
2. 200 OK — origin does not support Range. Full body is consumed and sliced; storage_http.range_fallback_total is incremented. Useful diagnostic for misconfigured origins.
3. 416 Range Not Satisfiable — caller asked beyond EOF. Mapped to STORAGE_HTTP.NOT_FOUND when the resource itself exists with shorter length (HEAD probe distinguishes "object missing" from "range past EOF" only on first encounter).

3d. ETag cache

Optional in-memory Map-backed LRU (default capacity: 256 entries). Keyed by sanitised key, valued by { etag, body, lastModified, expiresAt }. On get:

1. If cache.has(key), attach If-None-Match: <etag>.
2. On 304, return the cached body; refresh lastModified if present.
3. On 200, replace the cache entry with the new body (subject to LRU eviction).

The cache is per-instance, never shared across HttpStorage instances. Cross-process caching is composed by the caller (typically storage-file underneath).

3e. Write strategy

HttpUploader is the entire write surface. Its contract:

interface HttpUploader {
  put(key: string, body: Uint8Array | ReadableStream<Uint8Array>, headers?: Record<string, string>, signal?: AbortSignal): Promise<void>;
  delete?(key: string, signal?: AbortSignal): Promise<void>;  // optional
}

HttpStorage.set calls uploader.put; delete calls uploader.delete (or throws READ_ONLY if the uploader does not implement it). createWriteStream collects to memory and dispatches uploader.put on close — request-bodied origins do not support true chunked uploads without provider-specific multipart logic (deliberately out of scope).

The shipped presignedPutUploader(presigner: (key) => Promise<{ url, headers? }>) helper composes signing with defaultFetch. It is provider-agnostic — S3, R2, GCS, Azure Blob, or a custom endpoint all work by plugging in a presigner.

3f. Tier purity

Universal tier: fetch, URL, AbortController, Uint8Array, ReadableStream are all in the universal subset (Node 18+, modern browsers, Bun, Deno, Workers). No Node-only or DOM-only imports. The package is published with "browser" field omitted (universal) and no engines Node-version gate beyond the workspace baseline.

4. Trade-offs considered

• Listing. Rejected. WebDAV PROPFIND is the only HTTP-native directory protocol and is not what CDNs serve. Forcing a "best-effort" listing via index pages would lock callers into a specific origin's HTML layout. Shipping no list is honest.
• Built-in disk cache. Rejected for the v0. Cache invalidation is the consumer's policy choice (TTL vs. ETag vs. revalidate-always). Composing storage-http behind a storage-file NamespacedStorage covers the disk-cache case without baking opinions into the adapter.
• Provider-specific helpers in this package. Rejected. The moment we ship s3PresignedPutUploader, we drag @aws-sdk/* into the dep graph. The shipped presignedPutUploader(presigner) is the seam — provider packages live elsewhere.
• Atomic writes. HTTP PUT is idempotent but not POSIX-rename-atomic. We declare capabilities.atomicWrites: false honestly. Callers that need atomicity (cache-coherent reads during writes) should layer their own staging logic.
• Conditional writes (If-Match). Considered. Not v0 — would expand HttpUploader contract and the matching error code (PRECONDITION_FAILED). Easy follow-up if a consumer asks.

5. References

• openspec/specs/storage/spec.md — AbstractStorage contract this adapter conforms to.
• openspec/archive/2026-05-12-storage-file/ — closest precedent for adapter shape (capabilities, error registry, retry wiring).
• openspec/archive/2026-04-24-enterprise-storage-s3/ — network-backed adapter precedent (OTel attribute set, retry classification).