165 lines
10 KiB
Markdown
165 lines
10 KiB
Markdown
# Design — add-drive-range-reads
|
||
|
||
> Non-normative. Normative requirements live in `./specs/drive/spec.md` and `./specs/drive-uri/spec.md`.
|
||
|
||
## Goals
|
||
|
||
1. Extend `Drive.createReadStream` with `{ offset, length }` without breaking the bare-`AbortSignal` signature used at every existing call site.
|
||
2. Push the range hint through the plugin middleware so `drive-plugin-chunks` only fetches/decrypts/emits chunks intersecting the range.
|
||
3. Over `drive+mesh://`, request only the in-range chunkIds from peers — `ChunkRequest.specific` for small windows, `ChunkRequest.bitset` for large contiguous spans.
|
||
4. Surface range failures via two new `DRIVE_*` codes; reuse existing wire format and pass-through semantics elsewhere.
|
||
|
||
## API surface evolution
|
||
|
||
### `Drive.createReadStream` — overloaded
|
||
|
||
```ts
|
||
interface RangeReadOptions {
|
||
offset?: number; // default 0; integer ≥ 0
|
||
length?: number; // default → EOF; integer ≥ 0
|
||
signal?: AbortSignal;
|
||
}
|
||
|
||
class Drive {
|
||
// Existing (preserved)
|
||
createReadStream(path: string, signal?: AbortSignal): ReadableStream<Uint8Array>;
|
||
// Added
|
||
createReadStream(path: string, options: RangeReadOptions): ReadableStream<Uint8Array>;
|
||
// Implementation
|
||
createReadStream(
|
||
path: string,
|
||
signalOrOptions?: AbortSignal | RangeReadOptions,
|
||
): ReadableStream<Uint8Array> { … }
|
||
}
|
||
```
|
||
|
||
Detection: `signalOrOptions instanceof AbortSignal ? { signal: signalOrOptions } : (signalOrOptions ?? {})`. The runtime `instanceof` check is reliable in every supported runtime (Bun, Node 22+, Chromium 102+, Safari 16+) and avoids a brittle duck-type sniff.
|
||
|
||
Validation: synchronous — when `offset` or `length` is present, it MUST be a finite non-negative integer (`Number.isInteger(x) && x >= 0`). On failure, throw `DriveError(DRIVE.RANGE_INVALID, "offset must be a non-negative integer")` BEFORE constructing the stream.
|
||
|
||
### `StorageUriResolver` contract
|
||
|
||
`@suaveplan/storage-uri`'s `UriPlugin.adapter.createReadStream(key, options?)` already accepts an options bag; we extend the contract to include `{ offset?, length? }` per the `StorageLike` evolution path in `add-signal-to-storage-like` (archived). The resolver passes the bag through unchanged. `drive-uri`'s three plugins each set `adapter.capabilities.nativeRange = true` so the resolver issues one range-aware `createReadStream` call instead of buffering and slicing.
|
||
|
||
## Range → chunk math
|
||
|
||
Given:
|
||
|
||
- `chunkSize` (uint32, from `manifest.chunkSize` for chunks-plugin; from `storage.chunkSize` otherwise; conventionally 65536)
|
||
- `fileSize` (bigint, from `manifest.fileSize`)
|
||
- requested `offset` (default 0) and `length` (default `fileSize - offset`)
|
||
|
||
Compute:
|
||
|
||
- `end = min(offset + length, fileSize)` // exclusive
|
||
- `firstChunk = floor(offset / chunkSize)`
|
||
- `lastChunk = floor((end - 1) / chunkSize)` // inclusive; only when `end > offset`
|
||
- `firstChunkSkip = offset - firstChunk * chunkSize` // bytes to drop from the front of `firstChunk`
|
||
- `lastChunkKeep = end - lastChunk * chunkSize` // bytes to retain from the front of `lastChunk`
|
||
|
||
Emit:
|
||
|
||
- For `firstChunk`: decrypt, slice `[firstChunkSkip, min(chunkSize, lastChunkKeep))` if `firstChunk === lastChunk`, else `[firstChunkSkip, payloadLen)`
|
||
- For interior chunks `(firstChunk, lastChunk)`: emit whole `chunk.slice(0, payloadLen)`
|
||
- For `lastChunk` (when distinct from `firstChunk`): decrypt, slice `[0, lastChunkKeep)`
|
||
|
||
Zero-length: when `length === 0`, do not enter the loop — emit an empty `Uint8Array` and close.
|
||
|
||
### Worked examples (chunkSize = 65 536; last chunk holds `payloadLen` bytes only)
|
||
|
||
| Scenario | offset | length | end | firstChunk | lastChunk | firstSkip | lastKeep | Chunks pulled |
|
||
|--------------------------------|---------|---------|-----------|------------|-----------|-----------|----------|---------------|
|
||
| Range fully inside one chunk | 1 024 | 2 048 | 3 072 | 0 | 0 | 1 024 | 3 072 | {0} (1) |
|
||
| Spans one boundary | 60 000 | 10 000 | 70 000 | 0 | 1 | 60 000 | 4 464 | {0, 1} (2) |
|
||
| Aligned multi-chunk | 65 536 | 196 608 | 262 144 | 1 | 3 | 0 | 65 536 | {1, 2, 3} (3) |
|
||
| Tail to EOF (fileSize=200 000) | 150 000 | — | 200 000 | 2 | 3 | 18 928 | 3 392 | {2, 3} (2) |
|
||
| Zero-length | 12 345 | 0 | 12 345 | n/a | n/a | n/a | n/a | ∅ (0) |
|
||
| Whole file | 0 | — | fileSize | 0 | last | 0 | tailLen | {0..last} (N) |
|
||
|
||
## Mesh-request optimisation
|
||
|
||
The mesh pull path today issues `ChunkRequest.whole`. With a range, build:
|
||
|
||
```ts
|
||
const span = lastChunk - firstChunk + 1;
|
||
const req: ChunkRequest =
|
||
span <= 64
|
||
? { kind: "specific", chunkIds: rangeArray(firstChunk, lastChunk) }
|
||
: { kind: "bitset", bytes: bitsetForRange(firstChunk, lastChunk, totalChunks) };
|
||
```
|
||
|
||
Pivot rationale:
|
||
|
||
- `MAX_CHUNKS_DEFAULT = 64` (`drive-mesh-protocol/src/schemas/envelope.schema.ts`). JSON `chunkIds` for 64 numbers ~ 200 B; a bitset for the same window is ~ 8 B header + ⌈64/8⌉ = 16 B = under 30 B framed. At 64+ chunks, the bitset is uniformly cheaper.
|
||
- The bitset is constructed via `encodeBitset` from `drive-mesh-protocol`; the receiver decode path is unchanged.
|
||
- `ParallelPuller`'s window applies to whichever chunkIds the request enumerated, so range simply narrows the parallelism domain.
|
||
|
||
## Abort handling
|
||
|
||
The `AbortSignal` (from `RangeReadOptions.signal` OR the bare-signal form) is threaded into:
|
||
|
||
1. `storage.createReadStream(path, { signal })` when the storage has `nativeStreams`.
|
||
2. The chunks-plugin `#reassembleAsStream` loop — each `layout.getChunk(fileId, i)` call is wrapped by `throwIfAborted(signal)` between iterations.
|
||
3. The mesh pull — `drive-mesh.ParallelPuller` already accepts a signal; the range path passes it through unchanged.
|
||
|
||
On abort: cancel in-flight chunk fetches, close the stream with `controller.error(new DOMException("aborted", "AbortError"))`, and (over mesh) emit a single `cd:error` envelope with `code = "ABORTED"` so peers can free reservation slots.
|
||
|
||
## Error codes table
|
||
|
||
| Code | Origin | When raised |
|
||
|-------------------------------|---------------------------------|--------------------------------------------------------------------------------------------------------------|
|
||
| `DRIVE.RANGE_INVALID` | `Drive.createReadStream` | `offset`/`length` is non-integer / negative / NaN / not finite. Thrown synchronously before stream creation. |
|
||
| `DRIVE.RANGE_OUT_OF_BOUNDS` | `Drive.createReadStream` OR stream `error` event | Synchronously: `offset > size` or `offset + length > size` when `size` is known at start. Asynchronously: short-fall on stream close when size was unknown at start. |
|
||
| `CHUNKS.RANGE_OUT_OF_BOUNDS` | `chunksPlugin` middleware | Manifest `fileSize` is loaded and the range exceeds it. Surfaced on stream `error`; the drive layer re-wraps as `DRIVE.RANGE_OUT_OF_BOUNDS` with `cause`. |
|
||
|
||
No new codes in `DRIVE_URI`, `DMP`, or `DTP`. The resolver pass-through rule already forwards `DRIVE.*` codes to the caller intact.
|
||
|
||
## File layout
|
||
|
||
```text
|
||
packages/drive/drive/src/
|
||
file-operations/
|
||
streams.ts # MODIFIED — overload signatures, range slicing transform
|
||
streams.test.ts # MODIFIED — new range cases
|
||
streams.range.test.ts # ADDED — boundary cases (table above)
|
||
streams.range.md # ADDED — docs for the new path
|
||
types/
|
||
drive.types.ts # MODIFIED — RangeReadOptions, DriveStorage.capabilities.nativeRange
|
||
error-codes.ts # MODIFIED — RANGE_INVALID, RANGE_OUT_OF_BOUNDS
|
||
|
||
packages/drive/drive-plugin-chunks/src/
|
||
pipeline/
|
||
range.ts # ADDED — pure range→chunk math (table above)
|
||
range.test.ts # ADDED — property tests over (offset, length, fileSize, chunkSize)
|
||
range.md # ADDED
|
||
plugin.ts # MODIFIED — #reassemble takes range?, #reassembleAsStream
|
||
error-codes.ts # MODIFIED — RANGE_OUT_OF_BOUNDS
|
||
|
||
packages/drive/drive-mesh/src/
|
||
pull/
|
||
range-request.ts # ADDED — specific-vs-bitset pivot
|
||
range-request.test.ts # ADDED
|
||
range-request.md # ADDED
|
||
parallel-puller.ts # MODIFIED — accept range, build request via range-request
|
||
|
||
packages/drive/drive-uri/src/
|
||
adapter/
|
||
drive-storage-like.ts # MODIFIED — forward { offset, length } from createReadStream options
|
||
drive-storage-like.test.ts # MODIFIED — range forwarding
|
||
plugins/
|
||
drive/drive.streams.ts # MODIFIED — InstrumentedDriveAdapter.createReadStream forwards range
|
||
drive/drive.streams.test.ts# MODIFIED — range + AbortSignal
|
||
otel/
|
||
otel.ts # MODIFIED — drive.uri.range attribute, range_chunks_fetched histogram
|
||
otel.test.ts # MODIFIED
|
||
```
|
||
|
||
Every new `.ts` carries a sibling `.md` per CLAUDE.md §4; every new test uses `@suaveplan/testing/runner` per rule 11.
|
||
|
||
## Trade-offs considered
|
||
|
||
- **`{ offset, length }` vs. `{ start, end }`.** Picked `offset`/`length` to match the HTTP `Range: bytes=offset-(offset+length-1)` mental model and avoid the inclusive-vs-exclusive `end` ambiguity that bites every range API at least once.
|
||
- **Always-bitset vs. always-specific request.** Always-bitset wastes ~ 20 bytes on a 3-chunk range; always-specific blows the `MAX_CHUNKS_DEFAULT` cap on multi-MiB ranges. Pivoting at the cap is honest about the wire's own constraint.
|
||
- **`Drive.read({ offset, length })` companion.** Tempting but redundant: `await new Response(stream).arrayBuffer()` reduces to the same thing. Defer to a hoist when a concrete consumer asks.
|
||
- **Throw vs. clamp on overflow.** Clamping (HTTP `Range: bytes=900-2000` on a 1 000-byte file silently returning bytes 900–999) hides upstream bugs. Throwing `RANGE_OUT_OF_BOUNDS` is louder and matches the rest of `DRIVE_*`'s explicit failure style.
|