152 lines
4.7 KiB
Markdown
152 lines
4.7 KiB
Markdown
# streaming-chunk-queue
|
|
|
|
## Purpose
|
|
|
|
`StreamingChunkQueue` is the public class that reconstitutes
|
|
out-of-order chunked file transfers into a sequential
|
|
`ReadableStream<Uint8Array>`. It composes the package's pluggable
|
|
validator, decryptor, persister, rate limiter, hooks, event emitter,
|
|
and OTel telemetry into a single coherent reception primitive — the
|
|
piece every chunked-file consumer (drive-mesh, drive-webrtc,
|
|
drive-transport-https) plugs into when chunks arrive via a jittery or
|
|
out-of-order transport.
|
|
|
|
## Features
|
|
|
|
- Out-of-order ingestion: chunks may arrive in any order; duplicates
|
|
are silently ignored.
|
|
- Sequential output: the readable stream emits chunks strictly in
|
|
ascending `chunkId` order.
|
|
- Auto-start: the first `addChunk` automatically materialises the
|
|
stream (configurable via `autoStart: false`).
|
|
- Pluggable validator + decryptor strategies with adapters for
|
|
`@suaveplan/packer` `ChunkVerifier`.
|
|
- Optional `TokenBucketRateLimiterAdapter` integration for bandwidth
|
|
throttling without dropping chunks.
|
|
- Optional `StorageChunkPersister` for resumable transfers.
|
|
- Eight lifecycle hooks, ten event types, and wildcard listener
|
|
support.
|
|
- Binary chunk-request protocol exposed through `requestChunks` and
|
|
`requestMissing`.
|
|
- Missing-chunk timeout that errors the readable stream after
|
|
`chunkTimeout` ms (default 30 000).
|
|
- Frozen status snapshot via `getStatus`.
|
|
- OTel counters and a histogram (`stream_duration_ms`).
|
|
|
|
## Usage: Basic
|
|
|
|
```ts
|
|
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
|
|
|
|
const queue = new StreamingChunkQueue({
|
|
fileId: 12345n,
|
|
totalChunks: 5n,
|
|
});
|
|
|
|
await queue.addChunk(0n, bytes0);
|
|
await queue.addChunk(4n, bytes4);
|
|
await queue.addChunk(1n, bytes1);
|
|
await queue.addChunk(2n, bytes2);
|
|
await queue.addChunk(3n, bytes3);
|
|
|
|
const reader = queue.getStream().getReader();
|
|
for (;;) {
|
|
const { done, value } = await reader.read();
|
|
if (done) break;
|
|
process(value);
|
|
}
|
|
```
|
|
|
|
## Usage: Advanced
|
|
|
|
```ts
|
|
import { ChunkVerifier } from "@suaveplan/packer";
|
|
import { TokenBucket } from "@suaveplan/rate-limiter";
|
|
import { MemoryStorage } from "@suaveplan/storage";
|
|
import {
|
|
ChunkVerifierValidator,
|
|
StorageChunkPersister,
|
|
StreamingChunkQueue,
|
|
TokenBucketRateLimiterAdapter,
|
|
} from "@suaveplan/streams-chunk-queue";
|
|
|
|
const verifier = new ChunkVerifier(serverPubKey);
|
|
const bucket = new TokenBucket({ capacity: 1_048_576, refillRate: 1024 });
|
|
const storage = new MemoryStorage();
|
|
|
|
const queue = new StreamingChunkQueue({
|
|
fileId: 12345n,
|
|
totalChunks: 1024n,
|
|
validator: new ChunkVerifierValidator(verifier),
|
|
rateLimiter: new TokenBucketRateLimiterAdapter(bucket),
|
|
persister: new StorageChunkPersister({ storage, fileId: 12345n }),
|
|
chunkTimeout: 60_000,
|
|
hooks: {
|
|
onStreamProgress: (pct) => statusBar.set(pct),
|
|
onError: (err) => toaster.error(err.message),
|
|
},
|
|
});
|
|
|
|
queue.addEventListener("*", (event) =>
|
|
telemetry.record("chunk_queue.event", { type: event.type }),
|
|
);
|
|
|
|
const stream = queue.getStream();
|
|
await stream.pipeTo(diskWriter);
|
|
```
|
|
|
|
## API Reference
|
|
|
|
### `new StreamingChunkQueue(options)`
|
|
|
|
`options` is `StreamingChunkQueueOptions`. The constructor throws
|
|
`StreamsChunkQueueError(INVALID_TOTAL_CHUNKS)` when `totalChunks <= 0n`.
|
|
|
|
### `addChunk(chunkId, data)` → `Promise<void>`
|
|
|
|
Persists, validates, decrypts, and enqueues the chunk. Throws
|
|
`INVALID_CHUNK_ID` on out-of-range ids, or
|
|
`VALIDATION_FAILED` / `DECRYPTION_FAILED` if the pipeline rejects.
|
|
|
|
### `getStream()` → `ReadableStream<Uint8Array>`
|
|
|
|
Idempotent: subsequent calls return the same stream.
|
|
|
|
### `requestChunks(ids?)` / `requestMissing()` → `Uint8Array`
|
|
|
|
Encode the binary chunk-request frame.
|
|
|
|
### `getMissingChunkIds()` → `bigint[]`
|
|
|
|
Ascending list of un-received chunk ids.
|
|
|
|
### `getStatus()` → `StreamingQueueStatus`
|
|
|
|
Frozen snapshot.
|
|
|
|
### `loadChunk(chunkId)` / `loadAllChunks()` → `Promise<void>`
|
|
|
|
Hydrate from the configured persister. Throw
|
|
`STORAGE_NOT_CONFIGURED` / `STORAGE_CHUNK_MISSING` on failure.
|
|
|
|
### `isAllChunksReceived()` / `isStreamingComplete()` → `boolean`
|
|
|
|
### `addEventListener(type, listener)` / `removeEventListener(type, listener)`
|
|
|
|
`type` is any `StreamEvent['type']` or `'*'`.
|
|
|
|
### `clear()` → `void`
|
|
|
|
Reset internal state for reuse.
|
|
|
|
## Implementation Notes
|
|
|
|
The validator and decryptor stages are extracted to `pipeline.ts` so
|
|
the queue file stays close to the per-file line budget. Storage guards
|
|
live in `storage-ops.ts` for the same reason. Rate-limiting denial is
|
|
non-destructive — the queue sleeps for the bucket's `retryAfter` (or
|
|
the adapter's fallback delay) and retries the same chunk; no bytes are
|
|
ever dropped. Missing-chunk timeout polls the in-memory map at a
|
|
configurable interval (default 100 ms) so deterministic tests can
|
|
inject a custom `clock` + `sleep`.
|