101 lines
7.3 KiB
Markdown
101 lines
7.3 KiB
Markdown
# @suaveplan/streams-chunk-queue - Universal out-of-order chunk reconstitution: pluggable validation, decryption, rate limiting, storage, hooks, events, binary protocol, ReadableStream output
|
||
|
||
Universal out-of-order chunk reconstitution. Accepts chunks for a
|
||
file in any order and exposes a sequential
|
||
`ReadableStream<Uint8Array>` consumer surface, composed with pluggable
|
||
validation, decryption, rate limiting, persistent storage, lifecycle
|
||
hooks, a wildcard event bus, an efficient binary chunk-request
|
||
protocol, missing-chunk timeouts, and OpenTelemetry instrumentation.
|
||
|
||
<!-- begin:auto-docs -->
|
||
## Documentation tree
|
||
|
||
- [binary-protocol](./src/binary-protocol/encoder.md) — Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`, `drive-transport-https`, and any peer that n…
|
||
- [encoder](./src/binary-protocol/encoder.md) — Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`, `drive-transport-https`, and any peer that n…
|
||
- [decryptor](./src/decryptor/decryptor.md) — Declares the `ChunkDecryptor` strategy interface and provides the `NoopChunkDecryptor` default. Decryption is the third…
|
||
- [decryptor](./src/decryptor/decryptor.md) — Declares the `ChunkDecryptor` strategy interface and provides the `NoopChunkDecryptor` default. Decryption is the third…
|
||
- [errors](./src/errors/streams-chunk-queue-error.md) — `StreamsChunkQueueError` is the single typed error class raised by every public surface of `@suaveplan/streams-chunk-qu…
|
||
- [streams-chunk-queue-error](./src/errors/streams-chunk-queue-error.md) — `StreamsChunkQueueError` is the single typed error class raised by every public surface of `@suaveplan/streams-chunk-qu…
|
||
- [events](./src/events/events.md) — Implements the lightweight event bus that powers the queue's `addEventListener` / `removeEventListener` surface. The em…
|
||
- [events](./src/events/events.md) — Implements the lightweight event bus that powers the queue's `addEventListener` / `removeEventListener` surface. The em…
|
||
- [hooks](./src/hooks/hooks.md) — Provides the helper functions that invoke the queue's eight lifecycle callbacks safely. The queue itself wires hook fir…
|
||
- [hooks](./src/hooks/hooks.md) — Provides the helper functions that invoke the queue's eight lifecycle callbacks safely. The queue itself wires hook fir…
|
||
- [pipeline](./src/queue/pipeline.md) — Houses the validation + decryption pipeline stages that the `StreamingChunkQueue` runs on every received chunk. Extract…
|
||
- [storage-ops](./src/queue/storage-ops.md) — Holds the storage-related guards used by `StreamingChunkQueue.loadChunk` and `loadAllChunks`. Both methods need the sam…
|
||
- [streaming-chunk-queue](./src/queue/streaming-chunk-queue.md) — `StreamingChunkQueue` is the public class that reconstitutes out-of-order chunked file transfers into a sequential `Rea…
|
||
- [rate-limiter](./src/rate-limiter/adapter.md) — Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto the `ChunkRateLimiter` interface used internall…
|
||
- [adapter](./src/rate-limiter/adapter.md) — Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto the `ChunkRateLimiter` interface used internall…
|
||
- [storage-adapter](./src/storage-adapter/persister.md) — Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's `ChunkPersister` contract. Persistence enables resuming…
|
||
- [persister](./src/storage-adapter/persister.md) — Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's `ChunkPersister` contract. Persistence enables resuming…
|
||
- [telemetry](./src/telemetry/telemetry.md) — Exposes the OpenTelemetry instruments that the `StreamingChunkQueue` records during its lifecycle. Centralising instrum…
|
||
- [telemetry](./src/telemetry/telemetry.md) — Exposes the OpenTelemetry instruments that the `StreamingChunkQueue` records during its lifecycle. Centralising instrum…
|
||
- [types](./src/types/types.md) — Declares the shared types consumed across every feature dir of `@suaveplan/streams-chunk-queue`. Keeping them in one ty…
|
||
- [types](./src/types/types.md) — Declares the shared types consumed across every feature dir of `@suaveplan/streams-chunk-queue`. Keeping them in one ty…
|
||
- [validator](./src/validator/validator.md) — Declares the `ChunkValidator` strategy interface and provides the `NoopChunkValidator` default. Validation is the secon…
|
||
- [chunk-verifier-validator](./src/validator/chunk-verifier-validator.md) — Bridges the structural `ChunkVerifier` shape from `@suaveplan/packer` into the canonical `ChunkValidator` strategy. The…
|
||
- [validator](./src/validator/validator.md) — Declares the `ChunkValidator` strategy interface and provides the `NoopChunkValidator` default. Validation is the secon…
|
||
<!-- end:auto-docs -->
|
||
|
||
## Install
|
||
|
||
```bash
|
||
bun add @suaveplan/streams-chunk-queue
|
||
```
|
||
|
||
## Quick start
|
||
|
||
```ts
|
||
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
|
||
|
||
const queue = new StreamingChunkQueue({
|
||
fileId: 12345n,
|
||
totalChunks: 5n,
|
||
});
|
||
|
||
await queue.addChunk(2n, chunk2);
|
||
await queue.addChunk(0n, chunk0);
|
||
await queue.addChunk(1n, chunk1);
|
||
await queue.addChunk(3n, chunk3);
|
||
await queue.addChunk(4n, chunk4);
|
||
|
||
await queue.getStream().pipeTo(destination);
|
||
```
|
||
|
||
## Modules
|
||
|
||
- [`src/queue/streaming-chunk-queue.md`](./src/queue/streaming-chunk-queue.md) — the public `StreamingChunkQueue` class.
|
||
- [`src/queue/pipeline.md`](./src/queue/pipeline.md) — validation + decryption stages.
|
||
- [`src/queue/storage-ops.md`](./src/queue/storage-ops.md) — storage-side guards.
|
||
- [`src/validator/validator.md`](./src/validator/validator.md) — `ChunkValidator` strategy + `NoopChunkValidator`.
|
||
- [`src/validator/chunk-verifier-validator.md`](./src/validator/chunk-verifier-validator.md) — adapter for `@suaveplan/packer` `ChunkVerifier`.
|
||
- [`src/decryptor/decryptor.md`](./src/decryptor/decryptor.md) — `ChunkDecryptor` strategy + `NoopChunkDecryptor`.
|
||
- [`src/rate-limiter/adapter.md`](./src/rate-limiter/adapter.md) — `TokenBucket` adapter onto the queue's `ChunkRateLimiter`.
|
||
- [`src/storage-adapter/persister.md`](./src/storage-adapter/persister.md) — `StorageChunkPersister` over `AbstractStorage`.
|
||
- [`src/binary-protocol/encoder.md`](./src/binary-protocol/encoder.md) — chunk-request wire format.
|
||
- [`src/hooks/hooks.md`](./src/hooks/hooks.md) — safe lifecycle-hook invocation.
|
||
- [`src/events/events.md`](./src/events/events.md) — wildcard-capable event emitter.
|
||
- [`src/telemetry/telemetry.md`](./src/telemetry/telemetry.md) — OTel counters + histogram.
|
||
- [`src/errors/streams-chunk-queue-error.md`](./src/errors/streams-chunk-queue-error.md) — error codes + class.
|
||
- [`src/types/types.md`](./src/types/types.md) — shared types.
|
||
|
||
## Features
|
||
|
||
- Out-of-order ingestion; duplicates silently ignored.
|
||
- Sequential `ReadableStream<Uint8Array>` output.
|
||
- Auto-start on first chunk.
|
||
- Pluggable validator + decryptor strategies with packer `ChunkVerifier` adapter.
|
||
- Non-destructive rate-limit backoff via `TokenBucket` adapter.
|
||
- Resumable transfers via `StorageChunkPersister` over `AbstractStorage`.
|
||
- Eight lifecycle hooks, wildcard event bus, 10 event types.
|
||
- Binary chunk-request protocol (12-byte header + bitmap).
|
||
- Configurable missing-chunk timeout (default 30 s).
|
||
- Progress events (0–100) + frozen `StreamingQueueStatus` snapshot.
|
||
- Bigint chunk identifiers (packer v2 u64).
|
||
- `StreamsChunkQueueError` with 8 stable codes.
|
||
- OTel: 5 counters + 1 histogram.
|
||
|
||
## License
|
||
|
||
MIT
|
||
|