# @suaveplan/streams-chunk-queue - Universal out-of-order chunk reconstitution: pluggable validation, decryption, rate limiting, storage, hooks, events, binary protocol, ReadableStream output Universal out-of-order chunk reconstitution. Accepts chunks for a file in any order and exposes a sequential `ReadableStream` consumer surface, composed with pluggable validation, decryption, rate limiting, persistent storage, lifecycle hooks, a wildcard event bus, an efficient binary chunk-request protocol, missing-chunk timeouts, and OpenTelemetry instrumentation. ## Documentation tree - [binary-protocol](./src/binary-protocol/encoder.md) — Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`, `drive-transport-https`, and any peer that n… - [encoder](./src/binary-protocol/encoder.md) — Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`, `drive-transport-https`, and any peer that n… - [decryptor](./src/decryptor/decryptor.md) — Declares the `ChunkDecryptor` strategy interface and provides the `NoopChunkDecryptor` default. Decryption is the third… - [decryptor](./src/decryptor/decryptor.md) — Declares the `ChunkDecryptor` strategy interface and provides the `NoopChunkDecryptor` default. Decryption is the third… - [errors](./src/errors/streams-chunk-queue-error.md) — `StreamsChunkQueueError` is the single typed error class raised by every public surface of `@suaveplan/streams-chunk-qu… - [streams-chunk-queue-error](./src/errors/streams-chunk-queue-error.md) — `StreamsChunkQueueError` is the single typed error class raised by every public surface of `@suaveplan/streams-chunk-qu… - [events](./src/events/events.md) — Implements the lightweight event bus that powers the queue's `addEventListener` / `removeEventListener` surface. The em… - [events](./src/events/events.md) — Implements the lightweight event bus that powers the queue's `addEventListener` / `removeEventListener` surface. The em… - [hooks](./src/hooks/hooks.md) — Provides the helper functions that invoke the queue's eight lifecycle callbacks safely. The queue itself wires hook fir… - [hooks](./src/hooks/hooks.md) — Provides the helper functions that invoke the queue's eight lifecycle callbacks safely. The queue itself wires hook fir… - [pipeline](./src/queue/pipeline.md) — Houses the validation + decryption pipeline stages that the `StreamingChunkQueue` runs on every received chunk. Extract… - [storage-ops](./src/queue/storage-ops.md) — Holds the storage-related guards used by `StreamingChunkQueue.loadChunk` and `loadAllChunks`. Both methods need the sam… - [streaming-chunk-queue](./src/queue/streaming-chunk-queue.md) — `StreamingChunkQueue` is the public class that reconstitutes out-of-order chunked file transfers into a sequential `Rea… - [rate-limiter](./src/rate-limiter/adapter.md) — Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto the `ChunkRateLimiter` interface used internall… - [adapter](./src/rate-limiter/adapter.md) — Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto the `ChunkRateLimiter` interface used internall… - [storage-adapter](./src/storage-adapter/persister.md) — Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's `ChunkPersister` contract. Persistence enables resuming… - [persister](./src/storage-adapter/persister.md) — Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's `ChunkPersister` contract. Persistence enables resuming… - [telemetry](./src/telemetry/telemetry.md) — Exposes the OpenTelemetry instruments that the `StreamingChunkQueue` records during its lifecycle. Centralising instrum… - [telemetry](./src/telemetry/telemetry.md) — Exposes the OpenTelemetry instruments that the `StreamingChunkQueue` records during its lifecycle. Centralising instrum… - [types](./src/types/types.md) — Declares the shared types consumed across every feature dir of `@suaveplan/streams-chunk-queue`. Keeping them in one ty… - [types](./src/types/types.md) — Declares the shared types consumed across every feature dir of `@suaveplan/streams-chunk-queue`. Keeping them in one ty… - [validator](./src/validator/validator.md) — Declares the `ChunkValidator` strategy interface and provides the `NoopChunkValidator` default. Validation is the secon… - [chunk-verifier-validator](./src/validator/chunk-verifier-validator.md) — Bridges the structural `ChunkVerifier` shape from `@suaveplan/packer` into the canonical `ChunkValidator` strategy. The… - [validator](./src/validator/validator.md) — Declares the `ChunkValidator` strategy interface and provides the `NoopChunkValidator` default. Validation is the secon… ## Install ```bash bun add @suaveplan/streams-chunk-queue ``` ## Quick start ```ts import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue"; const queue = new StreamingChunkQueue({ fileId: 12345n, totalChunks: 5n, }); await queue.addChunk(2n, chunk2); await queue.addChunk(0n, chunk0); await queue.addChunk(1n, chunk1); await queue.addChunk(3n, chunk3); await queue.addChunk(4n, chunk4); await queue.getStream().pipeTo(destination); ``` ## Modules - [`src/queue/streaming-chunk-queue.md`](./src/queue/streaming-chunk-queue.md) — the public `StreamingChunkQueue` class. - [`src/queue/pipeline.md`](./src/queue/pipeline.md) — validation + decryption stages. - [`src/queue/storage-ops.md`](./src/queue/storage-ops.md) — storage-side guards. - [`src/validator/validator.md`](./src/validator/validator.md) — `ChunkValidator` strategy + `NoopChunkValidator`. - [`src/validator/chunk-verifier-validator.md`](./src/validator/chunk-verifier-validator.md) — adapter for `@suaveplan/packer` `ChunkVerifier`. - [`src/decryptor/decryptor.md`](./src/decryptor/decryptor.md) — `ChunkDecryptor` strategy + `NoopChunkDecryptor`. - [`src/rate-limiter/adapter.md`](./src/rate-limiter/adapter.md) — `TokenBucket` adapter onto the queue's `ChunkRateLimiter`. - [`src/storage-adapter/persister.md`](./src/storage-adapter/persister.md) — `StorageChunkPersister` over `AbstractStorage`. - [`src/binary-protocol/encoder.md`](./src/binary-protocol/encoder.md) — chunk-request wire format. - [`src/hooks/hooks.md`](./src/hooks/hooks.md) — safe lifecycle-hook invocation. - [`src/events/events.md`](./src/events/events.md) — wildcard-capable event emitter. - [`src/telemetry/telemetry.md`](./src/telemetry/telemetry.md) — OTel counters + histogram. - [`src/errors/streams-chunk-queue-error.md`](./src/errors/streams-chunk-queue-error.md) — error codes + class. - [`src/types/types.md`](./src/types/types.md) — shared types. ## Features - Out-of-order ingestion; duplicates silently ignored. - Sequential `ReadableStream` output. - Auto-start on first chunk. - Pluggable validator + decryptor strategies with packer `ChunkVerifier` adapter. - Non-destructive rate-limit backoff via `TokenBucket` adapter. - Resumable transfers via `StorageChunkPersister` over `AbstractStorage`. - Eight lifecycle hooks, wildcard event bus, 10 event types. - Binary chunk-request protocol (12-byte header + bitmap). - Configurable missing-chunk timeout (default 30 s). - Progress events (0–100) + frozen `StreamingQueueStatus` snapshot. - Bigint chunk identifiers (packer v2 u64). - `StreamsChunkQueueError` with 8 stable codes. - OTel: 5 counters + 1 histogram. ## License MIT