119 lines
12 KiB
Markdown
119 lines
12 KiB
Markdown
# Add byte-range reads to Drive.createReadStream and `drive*://` URI plugins
|
|
|
|
**Packages**: `@suaveplan/drive`, `@suaveplan/drive-uri`
|
|
**Status**: PROPOSED
|
|
**Type**: capability-extension
|
|
**Defers from**: `openspec/archive/2026-05-16-add-drive-uri/proposal.md` § "Out of scope → Range reads on `drive+mesh://`"
|
|
|
|
- **Design**: ./design.md
|
|
- **Spec deltas**:
|
|
- ./specs/drive/spec.md
|
|
- ./specs/drive-uri/spec.md
|
|
|
|
## Why
|
|
|
|
`add-drive-uri` shipped `resolver.read("drive+mesh://swarm-x/clip.mov")` as a whole-file fetch only, explicitly carving out byte-range reads:
|
|
|
|
> **Range reads on `drive+mesh://`.** `drive.createReadStream` already streams chunk-by-chunk via the chunks plugin; byte-range slicing on top requires a future `Drive.createReadStream({ offset, length })` extension — tracked separately.
|
|
|
|
Range reads are the single most-asked-for missing capability for the drive surface — every consumer that wants HTTP `Range:` header support (audio/video seek, virtual scrolling over chunked logs, partial-image decode, resume-on-reconnect, S3-style multipart staging) is currently forced to fetch the full file and slice client-side. Over `drive+mesh://` this is catastrophic: a 4 GiB media file pulled to seek the last 1 MiB downloads the entire swarm payload. The chunked layout in `drive-plugin-chunks` already addresses every chunk by `(fileId, chunkId)` with a known fixed `chunkSize` per manifest, so range → chunk math is deterministic; the missing piece is plumbing the offset/length pair from `Drive.createReadStream` through the chunks-plugin middleware so only the chunks intersecting the range are fetched, decrypted, and emitted.
|
|
|
|
This change adds the `{ offset, length }` options to `Drive.createReadStream`, extends the chunks plugin's `#reassemble` to honour them (computing first/last chunk indices, slicing the boundary chunks), threads range hints through the `drive-uri` adapters, and — over `drive+mesh://` — emits a `ChunkRequest.specific` (or `bitset` for large contiguous ranges) listing only the in-range chunkIds rather than `ChunkRequest.whole`. The local-storage and template paths get the same offset/length math for free because they go through the same chunks-plugin middleware.
|
|
|
|
## What changes
|
|
|
|
### `@suaveplan/drive`
|
|
|
|
- MODIFIED: `Drive.createReadStream(path, signal?)` → `Drive.createReadStream(path, options?: RangeReadOptions | AbortSignal)`. The new positional-options overload accepts `{ offset?: number; length?: number; signal?: AbortSignal }`. The bare-`AbortSignal` form is preserved via overload so all existing call sites compile unchanged.
|
|
- ADDED: `RangeReadOptions` interface in `src/types/drive.types.ts` — `offset` defaults to `0`, `length` defaults to "to EOF". Both MUST be non-negative integers when present; violations raise `DRIVE.RANGE_INVALID` synchronously (before stream construction).
|
|
- ADDED: `DRIVE.RANGE_INVALID` and `DRIVE.RANGE_OUT_OF_BOUNDS` codes in `DRIVE_CODES`. `RANGE_INVALID` covers negative / non-integer / NaN inputs. `RANGE_OUT_OF_BOUNDS` covers `offset > size` and `offset + length > size` where `size` is known at stream-start (e.g. via the chunks-plugin manifest); when size is unknown at start-time, the stream end-event MUST surface `RANGE_OUT_OF_BOUNDS` if the underlying source closed before `length` bytes were emitted.
|
|
- MODIFIED: `defaultCreateReadStream` and the storage-capability passthrough in `src/file-operations/streams.ts` honour `RangeReadOptions`. The default buffered path slices the loaded `Uint8Array`; the native-streams path forwards `{ offset, length }` to `storage.createReadStream` when `storage.capabilities?.nativeRange === true`, otherwise wraps the native stream in a slicing transform that drops bytes before `offset` and stops after `length`.
|
|
- ADDED: `DriveStorage` capability flag `capabilities.nativeRange?: boolean` and an extended `createReadStream(path, options?: { offset?: number; length?: number; signal?: AbortSignal })` signature on the storage layer. Storages that already accept a range (e.g. file with `fs.createReadStream({ start, end })`) opt in by setting the flag; existing storages are unaffected.
|
|
- ADDED: Middleware context carries `range?: { offset: number; length?: number }`. The chunks-plugin middleware reads it and short-circuits its `#reassemble` to the intersecting chunk window.
|
|
|
|
### `@suaveplan/drive-plugin-chunks`
|
|
|
|
- MODIFIED: `ChunksPluginHandle.#reassemble(fileId, range?)` accepts an optional range and emits only the chunks intersecting `[offset, offset + length)`. First and last chunks are sliced; interior chunks are emitted whole. Algorithm in `design.md` § "Range → chunk math".
|
|
- MODIFIED: The `read` middleware forwards `ctx.range` (when present) to `#reassemble` and to a sibling `#reassembleAsStream` used by `createReadStream`. The middleware's `read(path)` buffered call buffers only the in-range bytes.
|
|
- ADDED: `CHUNKS.RANGE_OUT_OF_BOUNDS` code — raised when the requested range exceeds `manifest.fileSize`. Surfaced on stream `error`.
|
|
|
|
### `@suaveplan/drive-mesh-protocol`
|
|
|
|
- MODIFIED: `drive-mesh`'s pull path (consumer of this package; no protocol wire change) constructs `ChunkRequest.specific { chunkIds }` for ranges spanning ≤ 64 chunks and `ChunkRequest.bitset { bytes }` for larger contiguous spans. No new wire fields — both forms already exist in `ChunkRequestPayload`.
|
|
- No `DMP_*` registry changes.
|
|
|
|
### `@suaveplan/drive-uri`
|
|
|
|
- MODIFIED: `DriveStorageLikeAdapter.createReadStream(key, options?)` accepts and forwards `{ offset?, length?, signal? }`. The three plugins (`driveUriPlugin`, `driveMeshUriPlugin`, `driveTemplateUriPlugin`) advertise `capabilities.nativeRange = true` on their adapter so `StorageUriResolver` issues a single range-aware `createReadStream` call instead of a whole-file `get` + client-side slice.
|
|
- ADDED: `DRIVE_URI.RANGE_UNSUPPORTED` is **not** introduced — every `drive*://` URI now supports ranges because the underlying chunks plugin does. The resolver surfaces upstream `DRIVE.RANGE_INVALID` / `DRIVE.RANGE_OUT_OF_BOUNDS` directly under the existing pass-through rule.
|
|
- ADDED: OTel attribute `drive.uri.range = "{offset}-{end}"` on `createReadStream` spans (omitted when no range supplied). Histogram `drive_uri.range_chunks_fetched` tags `{scheme}` and records the count of chunks actually pulled — measurable savings vs. whole-file fetch.
|
|
|
|
## Out of scope
|
|
|
|
- **Multi-range reads** (RFC 7233 `multipart/byteranges`). One range per call. A caller wanting N disjoint ranges issues N `createReadStream` calls.
|
|
- **`Drive.read(path, { offset, length })`** buffered range read. Achievable today via `createReadStream({ offset, length })` + buffer-collect; adding it to the buffered path would duplicate the chunk-math without changing observable behaviour. Hoist if a real consumer needs it.
|
|
- **Range writes / partial uploads.** Writes remain whole-file. Chunk-level patch semantics belong in a future `add-drive-patch` change.
|
|
- **Range hints on `drive`-tier `read()`** for non-chunked storages without `nativeRange`. Falls back to the slicing transform; performance characteristics are documented but no new code path is added.
|
|
- **Backpressure-aware peer scheduling** for large bitset requests. The mesh plugin's existing `ParallelPuller` window applies unchanged; range simply narrows the chunk set it pulls from.
|
|
|
|
## Source mapping
|
|
|
|
This is a **genesis-original** extension; deferral source is `openspec/archive/2026-05-16-add-drive-uri/proposal.md`. Surface anchors:
|
|
|
|
- `Drive.createReadStream` — `packages/drive/drive/src/drive/drive.ts:222-230`, `packages/drive/drive/src/file-operations/streams.ts`.
|
|
- Chunks plugin `#reassemble` — `packages/drive/drive-plugin-chunks/src/plugin.ts:218-251`; chunkSize lives on the manifest (`packages/drive/drive-plugin-chunks/src/manifest/build.ts:11,30,74`), default 65536 (`packages/drive/drive-plugin-chunks/src/plugin.ts:78`).
|
|
- `ChunkRequest` union — `packages/drive/drive-mesh-protocol/src/types/protocol.types.ts:38-42`; both `specific` and `bitset` already encodable.
|
|
- `drive-uri` adapter — `packages/drive/drive-uri/src/plugins/drive-mesh/drive-mesh.ts` extends `InstrumentedDriveAdapter`.
|
|
|
|
### Requirements with no source line
|
|
|
|
| Requirement | Justification |
|
|
|---|---|
|
|
| Overload `createReadStream(path, signal)` AND `createReadStream(path, options)` rather than breaking the single-signal form | Every existing consumer in the monorepo (chunks plugin, drive-uri adapters, drive integration tests) passes the bare `AbortSignal`. A breaking signature change for an additive feature is unjustified when overloads cost nothing. |
|
|
| Switch `ChunkRequest.specific` ↔ `ChunkRequest.bitset` at 64 chunks | `MAX_CHUNKS_DEFAULT = 64` in `drive-mesh-protocol`; beyond that, the JSON `chunkIds` array out-weighs a bitset that addresses every chunk in two bytes per 16. The 64-chunk pivot matches the wire-cap and minimises bytes-on-wire for the common case (4 MiB range over 64 KiB chunks). |
|
|
| `RANGE_OUT_OF_BOUNDS` may fire at start OR at stream-end | When `size` is known (chunks-plugin manifest loaded, storage `stat` populated), reject synchronously. When unknown (raw `defaultCreateReadStream` over a non-stat-able storage), accept the range, emit what's available, then surface the code on `close` if shortfall is detected. Two paths because the size-known/size-unknown split is a real storage capability variance, not a defect. |
|
|
| Zero-length range is a valid request | `length: 0` returns an immediately-closed empty stream. Useful as a probe-for-existence pattern; rejecting it would surprise consumers who compute ranges from upstream API responses. |
|
|
| `nativeRange` storage capability flag rather than always-slicing in drive | `bun:sqlite`-backed and `fs`-backed storages can deliver true range reads with no read-amplification. Always wrapping in a slicing transform would discard that win for the storages that have it. The flag is opt-in so adding it is purely additive on every existing storage. |
|
|
|
|
## Migration
|
|
|
|
This is a purely additive change for every existing call site:
|
|
|
|
- **Old**: `drive.createReadStream("file.bin", abortSignal)` → still compiles, still returns whole-file stream.
|
|
- **New**: `drive.createReadStream("file.bin", { offset: 1024, length: 4096, signal: abortSignal })` → emits exactly bytes [1024, 5120). Over `drive+mesh://`, requests at most ⌈(4096 + 1024 % chunkSize) / chunkSize⌉ chunks from peers.
|
|
|
|
No consumer must change. Consumers that want range behaviour migrate at their leisure. Search command for opt-in adoption:
|
|
|
|
```bash
|
|
grep -rn "createReadStream" packages/ | grep -v "/dist/\|/coverage/"
|
|
```
|
|
|
|
Recommended replacement when range-aware: pass `{ offset, length, signal }` instead of the bare `signal`.
|
|
|
|
## Impact
|
|
|
|
### Affected dependents
|
|
|
|
Run `bunx turbo ls --filter=...@suaveplan/drive` to enumerate. The blast radius is "every package that calls `Drive.createReadStream`," which today is:
|
|
|
|
- `@suaveplan/drive-uri` — [x] tests run / [x] docs reviewed / [ ] no action needed (adapter `createReadStream` extended to forward range; new requirement added to spec delta)
|
|
- `@suaveplan/drive-plugin-chunks` — [x] tests run / [x] docs reviewed / [ ] no action needed (middleware honours range; `#reassemble` extended)
|
|
- `@suaveplan/drive-mesh` — [x] tests run / [x] docs reviewed / [ ] no action needed (pull path constructs `ChunkRequest.specific` / `bitset` from range)
|
|
- `@suaveplan/drive-transport-https` — [ ] tests run / [x] docs reviewed / [x] no action needed (transport carries chunk envelopes opaquely; no surface change)
|
|
- `@suaveplan/drive-transport-webrtc` — [ ] tests run / [x] docs reviewed / [x] no action needed (same as above)
|
|
- `@suaveplan/drive-template` — [ ] tests run / [x] docs reviewed / [x] no action needed (hydration unchanged; range applies post-hydration)
|
|
- `@suaveplan/drive-sharing` — [ ] tests run / [x] docs reviewed / [x] no action needed (calls `Drive.read`, not `createReadStream`)
|
|
- `@suaveplan/drive-webrtc` — [ ] tests run / [x] docs reviewed / [x] no action needed (browser-tier transport, unaffected)
|
|
|
|
### Affected upstreams (none modified)
|
|
|
|
| Package | Action | Breaking? |
|
|
|---|---|---|
|
|
| `@suaveplan/storage` | Optional: extend `AbstractStorage.createReadStream` with `{ offset, length }` options. Existing storages opt in via `capabilities.nativeRange`. | no |
|
|
| `@suaveplan/storage-uri` | None — already routes `createReadStream` through `UriPlugin.adapter`. | no |
|
|
| `@suaveplan/types` | None — `RangeReadOptions` lives in `@suaveplan/drive` until a second consumer emerges. | no |
|
|
|
|
### Ordering
|
|
|
|
Unblocked. All packages exist on genesis today; the change is additive.
|