first commit

This commit is contained in:
2026-05-29 08:20:56 +01:00
commit 949299f58d
6368 changed files with 527176 additions and 0 deletions

View File

@@ -0,0 +1,100 @@
# @suaveplan/streams-chunk-queue - Universal out-of-order chunk reconstitution: pluggable validation, decryption, rate limiting, storage, hooks, events, binary protocol, ReadableStream output
Universal out-of-order chunk reconstitution. Accepts chunks for a
file in any order and exposes a sequential
`ReadableStream<Uint8Array>` consumer surface, composed with pluggable
validation, decryption, rate limiting, persistent storage, lifecycle
hooks, a wildcard event bus, an efficient binary chunk-request
protocol, missing-chunk timeouts, and OpenTelemetry instrumentation.
<!-- begin:auto-docs -->
## Documentation tree
- [binary-protocol](./src/binary-protocol/encoder.md) — Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`, `drive-transport-https`, and any peer that n…
- [encoder](./src/binary-protocol/encoder.md) — Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`, `drive-transport-https`, and any peer that n…
- [decryptor](./src/decryptor/decryptor.md) — Declares the `ChunkDecryptor` strategy interface and provides the `NoopChunkDecryptor` default. Decryption is the third…
- [decryptor](./src/decryptor/decryptor.md) — Declares the `ChunkDecryptor` strategy interface and provides the `NoopChunkDecryptor` default. Decryption is the third…
- [errors](./src/errors/streams-chunk-queue-error.md) — `StreamsChunkQueueError` is the single typed error class raised by every public surface of `@suaveplan/streams-chunk-qu…
- [streams-chunk-queue-error](./src/errors/streams-chunk-queue-error.md) — `StreamsChunkQueueError` is the single typed error class raised by every public surface of `@suaveplan/streams-chunk-qu…
- [events](./src/events/events.md) — Implements the lightweight event bus that powers the queue's `addEventListener` / `removeEventListener` surface. The em…
- [events](./src/events/events.md) — Implements the lightweight event bus that powers the queue's `addEventListener` / `removeEventListener` surface. The em…
- [hooks](./src/hooks/hooks.md) — Provides the helper functions that invoke the queue's eight lifecycle callbacks safely. The queue itself wires hook fir…
- [hooks](./src/hooks/hooks.md) — Provides the helper functions that invoke the queue's eight lifecycle callbacks safely. The queue itself wires hook fir…
- [pipeline](./src/queue/pipeline.md) — Houses the validation + decryption pipeline stages that the `StreamingChunkQueue` runs on every received chunk. Extract…
- [storage-ops](./src/queue/storage-ops.md) — Holds the storage-related guards used by `StreamingChunkQueue.loadChunk` and `loadAllChunks`. Both methods need the sam…
- [streaming-chunk-queue](./src/queue/streaming-chunk-queue.md) — `StreamingChunkQueue` is the public class that reconstitutes out-of-order chunked file transfers into a sequential `Rea…
- [rate-limiter](./src/rate-limiter/adapter.md) — Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto the `ChunkRateLimiter` interface used internall…
- [adapter](./src/rate-limiter/adapter.md) — Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto the `ChunkRateLimiter` interface used internall…
- [storage-adapter](./src/storage-adapter/persister.md) — Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's `ChunkPersister` contract. Persistence enables resuming…
- [persister](./src/storage-adapter/persister.md) — Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's `ChunkPersister` contract. Persistence enables resuming…
- [telemetry](./src/telemetry/telemetry.md) — Exposes the OpenTelemetry instruments that the `StreamingChunkQueue` records during its lifecycle. Centralising instrum…
- [telemetry](./src/telemetry/telemetry.md) — Exposes the OpenTelemetry instruments that the `StreamingChunkQueue` records during its lifecycle. Centralising instrum…
- [types](./src/types/types.md) — Declares the shared types consumed across every feature dir of `@suaveplan/streams-chunk-queue`. Keeping them in one ty…
- [types](./src/types/types.md) — Declares the shared types consumed across every feature dir of `@suaveplan/streams-chunk-queue`. Keeping them in one ty…
- [validator](./src/validator/validator.md) — Declares the `ChunkValidator` strategy interface and provides the `NoopChunkValidator` default. Validation is the secon…
- [chunk-verifier-validator](./src/validator/chunk-verifier-validator.md) — Bridges the structural `ChunkVerifier` shape from `@suaveplan/packer` into the canonical `ChunkValidator` strategy. The…
- [validator](./src/validator/validator.md) — Declares the `ChunkValidator` strategy interface and provides the `NoopChunkValidator` default. Validation is the secon…
<!-- end:auto-docs -->
## Install
```bash
bun add @suaveplan/streams-chunk-queue
```
## Quick start
```ts
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
const queue = new StreamingChunkQueue({
fileId: 12345n,
totalChunks: 5n,
});
await queue.addChunk(2n, chunk2);
await queue.addChunk(0n, chunk0);
await queue.addChunk(1n, chunk1);
await queue.addChunk(3n, chunk3);
await queue.addChunk(4n, chunk4);
await queue.getStream().pipeTo(destination);
```
## Modules
- [`src/queue/streaming-chunk-queue.md`](./src/queue/streaming-chunk-queue.md) — the public `StreamingChunkQueue` class.
- [`src/queue/pipeline.md`](./src/queue/pipeline.md) — validation + decryption stages.
- [`src/queue/storage-ops.md`](./src/queue/storage-ops.md) — storage-side guards.
- [`src/validator/validator.md`](./src/validator/validator.md) — `ChunkValidator` strategy + `NoopChunkValidator`.
- [`src/validator/chunk-verifier-validator.md`](./src/validator/chunk-verifier-validator.md) — adapter for `@suaveplan/packer` `ChunkVerifier`.
- [`src/decryptor/decryptor.md`](./src/decryptor/decryptor.md) — `ChunkDecryptor` strategy + `NoopChunkDecryptor`.
- [`src/rate-limiter/adapter.md`](./src/rate-limiter/adapter.md) — `TokenBucket` adapter onto the queue's `ChunkRateLimiter`.
- [`src/storage-adapter/persister.md`](./src/storage-adapter/persister.md) — `StorageChunkPersister` over `AbstractStorage`.
- [`src/binary-protocol/encoder.md`](./src/binary-protocol/encoder.md) — chunk-request wire format.
- [`src/hooks/hooks.md`](./src/hooks/hooks.md) — safe lifecycle-hook invocation.
- [`src/events/events.md`](./src/events/events.md) — wildcard-capable event emitter.
- [`src/telemetry/telemetry.md`](./src/telemetry/telemetry.md) — OTel counters + histogram.
- [`src/errors/streams-chunk-queue-error.md`](./src/errors/streams-chunk-queue-error.md) — error codes + class.
- [`src/types/types.md`](./src/types/types.md) — shared types.
## Features
- Out-of-order ingestion; duplicates silently ignored.
- Sequential `ReadableStream<Uint8Array>` output.
- Auto-start on first chunk.
- Pluggable validator + decryptor strategies with packer `ChunkVerifier` adapter.
- Non-destructive rate-limit backoff via `TokenBucket` adapter.
- Resumable transfers via `StorageChunkPersister` over `AbstractStorage`.
- Eight lifecycle hooks, wildcard event bus, 10 event types.
- Binary chunk-request protocol (12-byte header + bitmap).
- Configurable missing-chunk timeout (default 30 s).
- Progress events (0–100) + frozen `StreamingQueueStatus` snapshot.
- Bigint chunk identifiers (packer v2 u64).
- `StreamsChunkQueueError` with 8 stable codes.
- OTel: 5 counters + 1 histogram.
## License
MIT

View File

@@ -0,0 +1,73 @@
# binary-protocol / encoder
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`,
`drive-transport-https`, and any peer that needs to tell the sender which
chunks of a file it is still missing. The frame is compact (12-byte
header + bitmap) so even a 100k-chunk request stays under 13 kB, and it
is fixed-layout so receivers can validate it without a schema.
## Features
- 8-byte big-endian `fileId` (u64).
- 4-byte big-endian `totalChunks` (u32).
- `⌈totalChunks / 8⌉` byte bitmap where bit `i` of byte `floor(i/8)` is
set iff chunk `i` is requested.
- Zero-allocation hot path: a single `Uint8Array` of the exact size and
one DataView for the header.
- Range checks raise `StreamsChunkQueueError` with the matching code
(`INVALID_CHUNK_ID` for fileId, `INVALID_TOTAL_CHUNKS` for
totalChunks).
- Chunk ids outside `[0, totalChunks)` are silently dropped — the
encoder's contract is "request these chunks, capped by the file
size".
## Usage: Basic
```ts
import { encodeChunkRequest } from "@suaveplan/streams-chunk-queue";
const frame = encodeChunkRequest({
fileId: 12345n,
totalChunks: 10n,
chunkIds: [0n, 2n, 5n],
});
```
## Usage: Advanced
```ts
const missing = queue.getMissingChunkIds();
const frame = encodeChunkRequest({
fileId: queue.getStatus().fileId,
totalChunks: queue.getStatus().totalChunks,
chunkIds: missing,
});
await webrtcDc.send(frame);
```
## API Reference
### `encodeChunkRequest({ fileId, totalChunks, chunkIds })` → `Uint8Array`
Returns the encoded frame. Throws `StreamsChunkQueueError` when
`fileId` is outside the u64 range or `totalChunks` is outside the u32
range.
### `CHUNK_REQUEST_HEADER_SIZE`
Constant `12`. Exposed for callers that decode the frame.
## Implementation Notes
The encoder iterates over `chunkIds` (typically the missing set,
bounded by realistic transfer sizes) instead of `[0, totalChunks)` so
the hot path stays O(|chunkIds|) regardless of file size. The 8-byte
big-endian `fileId` is written as two `setUint32` calls because
`DataView` does not have a `setBigUint64` on every supported runtime
target.

View File

@@ -0,0 +1,91 @@
# decryptor
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Declares the `ChunkDecryptor` strategy interface and provides the
`NoopChunkDecryptor` default. Decryption is the third stage of the
chunk pipeline (after validation, before streaming). Externalising the
strategy lets callers plug in AEAD primitives (XChaCha20-Poly1305,
AES-256-GCM), envelope decryption, or HSM-bound decryption without the
core queue knowing the cryptographic details.
## Features
- `ChunkDecryptor` interface with a single async `decrypt` method.
- `NoopChunkDecryptor` pass-through default — used for unencrypted
transports.
- `noopChunkDecryptor` singleton shorthand.
- Failures are caught by the queue and surfaced as
`StreamsChunkQueueError(DECRYPTION_FAILED)`, never an opaque rejection.
## Usage: Basic
```ts
import { noopChunkDecryptor } from "@suaveplan/streams-chunk-queue";
const queue = new StreamingChunkQueue({
fileId: 1n,
totalChunks: 5n,
decryptor: noopChunkDecryptor,
});
```
## Usage: Advanced
```ts
import type { ChunkDecryptor } from "@suaveplan/streams-chunk-queue";
import { chacha20poly1305 } from "@noble/ciphers/chacha";
class XChaChaDecryptor implements ChunkDecryptor {
constructor(private readonly key: Uint8Array) {}
async decrypt(_chunkId: bigint, data: Uint8Array) {
const nonce = data.slice(0, 24);
const ciphertext = data.slice(24);
const plaintext = chacha20poly1305(this.key, nonce).decrypt(ciphertext);
return { data: plaintext };
}
}
```
## API Reference
### `ChunkDecryptor`
```ts
interface ChunkDecryptor {
decrypt(chunkId: bigint, data: Uint8Array): Promise<DecryptionOutcome>;
}
```
### `DecryptionOutcome`
`{ data: Uint8Array }` — plaintext bytes that replace the ciphertext
in the queue's storage.
### `NoopChunkDecryptor`
Concrete class returning `{ data }` unchanged. Suitable for plaintext
transports or when decryption happens further upstream.
### `noopChunkDecryptor`
Singleton instance of `NoopChunkDecryptor`.
## Implementation Notes
The interface is async because real AEAD primitives invariably are.
Errors thrown by `decrypt` are caught by the queue and remapped to the
canonical `DECRYPTION_FAILED` code so downstream observability code can
filter on a stable code instead of a library-specific error class. The
adapter contract returns `{ data }` so a decryptor can also concatenate
multiple ciphertext fragments, strip a nonce prefix, or replace the
bytes wholesale — the queue replaces its in-memory slot with the
returned bytes, freeing the original ciphertext for garbage
collection. The noop default lets the queue avoid a conditional check
on every chunk; it returns the same `Uint8Array` reference unchanged,
so there is no allocation cost.

View File

@@ -0,0 +1,97 @@
# streams-chunk-queue-error
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`StreamsChunkQueueError` is the single typed error class raised by every
public surface of `@suaveplan/streams-chunk-queue`. It extends
`SuaveplanError` so callers get the full enterprise error contract: a
stable `code` string, a structured `context` map, a redactable wire form,
OpenTelemetry span-event mapping, and an unbroken cause chain. The
package registers every code through `defineCodes`, guaranteeing the
codes are immutable, frozen, and reachable from real call sites.
## Features
- One frozen registry — `STREAMS_CHUNK_QUEUE_CODES` — exposes the eight
canonical codes (`INVALID_TOTAL_CHUNKS`, `INVALID_CHUNK_ID`,
`STORAGE_NOT_CONFIGURED`, `STORAGE_CHUNK_MISSING`, `VALIDATION_FAILED`,
`DECRYPTION_FAILED`, `STREAM_TIMEOUT`, `RATE_LIMIT_EXHAUSTED`).
- One literal-typed error class — `StreamsChunkQueueError` — narrows
`code` through `instanceof`.
- Cause chaining via `SuaveplanError`'s `{ cause }` option, so the
underlying network or crypto failure is preserved.
- Structured `context` map (e.g. `{ chunkId, fileId, totalChunks }`) for
observability and i18n.
## Usage: Basic
```ts
import {
STREAMS_CHUNK_QUEUE_CODES,
StreamsChunkQueueError,
} from "@suaveplan/streams-chunk-queue";
throw new StreamsChunkQueueError(
STREAMS_CHUNK_QUEUE_CODES.INVALID_CHUNK_ID,
`chunk ${chunkId} out of range`,
{ context: { chunkId: chunkId.toString() } },
);
```
## Usage: Advanced
```ts
try {
await queue.addChunk(chunkId, bytes);
} catch (err) {
if (
err instanceof StreamsChunkQueueError &&
err.code === STREAMS_CHUNK_QUEUE_CODES.VALIDATION_FAILED
) {
observabilityBus.publish("chunk-fraud", {
chunkId: err.context?.chunkId,
cause: err.cause,
});
}
throw err;
}
```
## API Reference
### `STREAMS_CHUNK_QUEUE_CODES`
A frozen record built by `defineCodes("STREAMS_CHUNK_QUEUE", […])`. Each
key maps to the fully-qualified code string
`STREAMS_CHUNK_QUEUE.<reason>`.
### `StreamsChunkQueueErrorCode`
Discriminated union of every code in the registry.
### `StreamsChunkQueueError`
```ts
new StreamsChunkQueueError(
code: StreamsChunkQueueErrorCode,
message: string,
options?: { cause?: unknown; context?: Record<string, unknown> },
)
```
- `code` — one of the registered code strings.
- `message` — human-readable description (or i18n message key).
- `options.cause` — underlying error preserved for `walkCauses`.
- `options.context` — structured fields surfaced through the wire schema.
## Implementation Notes
`SuaveplanError` is `abstract`, so this module supplies the concrete
subclass. The constructor delegates to `super` with the cause and context
forwarded through `SuaveplanErrorOptions`. Codes are intentionally
namespaced with `STREAMS_CHUNK_QUEUE.` so multi-package log scans can
filter by prefix without false positives from sibling packages.

View File

@@ -0,0 +1,69 @@
# events
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Implements the lightweight event bus that powers the queue's
`addEventListener` / `removeEventListener` surface. The emitter keeps a
`Map<StreamEventType, Set<Listener>>` so registration and removal are
both O(1), and exposes a wildcard `"*"` subscription so debugging tools
can observe every event without enumerating types.
## Features
- Concrete-type subscriptions (`chunk:received`, `stream:progress`, …).
- Wildcard subscription via `"*"` that receives every emitted event.
- O(1) add, remove, and emit per listener.
- `clear()` for tear-down (used by `StreamingChunkQueue.clear()`).
## Usage: Basic
```ts
import { StreamEventEmitter } from "@suaveplan/streams-chunk-queue";
const emitter = new StreamEventEmitter();
emitter.addEventListener("chunk:received", (event) => {
console.info("got chunk", event.chunkId);
});
```
## Usage: Advanced
```ts
const wildcard = (event: StreamEvent) =>
telemetry.record("stream.event", { type: event.type });
emitter.addEventListener("*", wildcard);
emitter.removeEventListener("*", wildcard);
```
## API Reference
### `StreamEventEmitter`
- `addEventListener(type, listener)` — register `listener` under
`type` (either a concrete event type or `"*"`).
- `removeEventListener(type, listener)` — remove a previously
registered listener.
- `emit(event)` — dispatch to all matching subscribers, then to all
wildcard subscribers.
- `clear()` — drop every registered listener.
### `WILDCARD_EVENT_TYPE`
The constant `"*"`. Exported so callers don't sprinkle string literals.
## Implementation Notes
The wildcard dispatch happens after concrete dispatch, mirroring the
ordering observable consumers expect (a tracing tool sees the event
after the application code has reacted). Listeners are synchronous —
the bus does not await listener return values; callers that need
async work should `queueMicrotask` inside their listener. The emitter
is intentionally allocation-light: a `Map<type, Set<listener>>` means
add/remove are O(1) and dispatch is O(N) over only the matching
listeners. There is no priority ordering; concrete-type listeners are
dispatched in insertion order, then wildcard listeners in their own
insertion order.

View File

@@ -0,0 +1,69 @@
# hooks
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides the helper functions that invoke the queue's eight lifecycle
callbacks safely. The queue itself wires hook firing into its pipeline;
this module isolates the "safe invocation" logic — coercion of thrown
values to real `Error` instances, async-await of both sync and async
hooks, and re-routing of secondary failures through `onError`.
## Features
- `invokeHook` swallows hook errors after re-routing them via `onError`,
so a misbehaving listener cannot crash the queue's main loop.
- `toError` coerces any thrown value (string, object, primitive) into a
real `Error` so downstream consumers see a uniform shape.
- The helper avoids recursive re-routing when the failing hook IS the
`onError` hook itself.
## Usage: Basic
```ts
import { invokeHook } from "@suaveplan/streams-chunk-queue";
await invokeHook(hooks.onChunkReceived, [chunkId, metadata], hooks.onError, chunkId);
```
## Usage: Advanced
```ts
import { invokeHook, toError } from "@suaveplan/streams-chunk-queue";
const onError = (err: Error) => logger.error({ err: toError(err) });
await invokeHook(
async (id: bigint) => await reportToObservability(id),
[chunkId],
onError,
chunkId,
);
```
## API Reference
### `invokeHook(hook, args, onError?, chunkId?)` → `Promise<void>`
- `hook` — optional `LifecycleHook<Args>`. If `undefined`, the call is
a no-op.
- `args` — tuple of arguments to forward.
- `onError` — optional `(error, chunkId?) => void | Promise<void>`
re-route. Not invoked if it is the same reference as `hook`.
- `chunkId` — optional bigint forwarded to `onError`.
### `toError(value)` → `Error`
Coerces any value to an `Error`.
## Implementation Notes
The recursion guard `onError !== hook` is the difference between a
queue that survives a buggy `onError` callback and one that hangs in an
infinite loop. When the secondary `onError` invocation itself throws,
the helper swallows the failure — the primary path is preserved
because the queue independently surfaces the error via the event
emitter and (for streaming errors) the controller.

View File

@@ -0,0 +1,73 @@
# queue / pipeline
## Purpose
Houses the validation + decryption pipeline stages that the
`StreamingChunkQueue` runs on every received chunk. Extracting these
stages into pure helpers keeps the main queue class focused on
orchestration (stream control, status tracking, request building) and
lets the pipeline grow more stages — e.g. a future deduplication step —
without ballooning the queue file past the per-file line budget.
## Features
- `runValidate` and `runDecrypt` share the same retry / event / hook /
telemetry shape via the internal `runStage` helper.
- Failures are coerced to `StreamsChunkQueueError` with the
appropriate code (`VALIDATION_FAILED` / `DECRYPTION_FAILED`) and the
original error is preserved as `cause`.
- Successful stages mutate `metadata.validated` / `metadata.decrypted`
through a callback so the queue's snapshot reflects pipeline state.
- Telemetry counters (`validation_failures_total` /
`decryption_failures_total`) are incremented on failure exactly once.
## Usage: Basic
The pipeline functions are internal to the queue package; the queue
itself wires them.
```ts
await runValidate(stored, validator, deps);
await runDecrypt(stored, decryptor, deps);
```
## Usage: Advanced
```ts
await runValidate(stored, this.validator, this.pipelineDeps);
await this.runChecksum(stored);
await runDecrypt(stored, this.decryptor, this.pipelineDeps);
```
## API Reference
### `PipelineDeps`
```ts
interface PipelineDeps {
emitter: StreamEventEmitter;
hooks: StreamingChunkQueueHooks;
telemetry: StreamsChunkQueueTelemetry;
}
```
### `runValidate(stored, validator, deps)` → `Promise<void>`
Runs `validator.validate(chunkId, data)`. On success: replaces
`stored.data`, sets `metadata.validated = true`, emits `chunk:validated`,
and fires `onChunkValidated`. On failure: emits `validation:failed`,
fires `onError`, and re-throws a `StreamsChunkQueueError(VALIDATION_FAILED)`.
### `runDecrypt(stored, decryptor, deps)` → `Promise<void>`
Same shape, using `decryptor.decrypt` and the
`chunk:decrypted` / `decryption:failed` events and the
`DECRYPTION_FAILED` code.
## Implementation Notes
`runStage` re-throws an existing `StreamsChunkQueueError` untouched so a
downstream adapter (e.g. `ChunkVerifierValidator`) can attach its own
context and have that context survive the queue's catch handler. The
`onError` re-route is suppressed when the failing hook is the same
function — see `hooks.md` for the rationale.

View File

@@ -0,0 +1,57 @@
# queue / storage-ops
## Purpose
Holds the storage-related guards used by `StreamingChunkQueue.loadChunk`
and `loadAllChunks`. Both methods need the same "is the persister
configured?" check and the same "is the chunk actually in storage?"
check, so factoring them out keeps the queue file from duplicating the
guards and from spilling past the per-file line budget.
## Features
- `loadChunkBytes` throws `STORAGE_NOT_CONFIGURED` when no persister is
set, `STORAGE_CHUNK_MISSING` when the chunk is absent, and returns
the bytes otherwise.
- `listStoredChunkIds` throws `STORAGE_NOT_CONFIGURED` when no
persister is set, otherwise forwards to `persister.listChunkIds()`.
- Both helpers preserve a stringified `chunkId` in
`error.context` so logs are scannable.
## Usage: Basic
```ts
const bytes = await loadChunkBytes(this.persister, chunkId);
```
## Usage: Advanced
```ts
const ids = await listStoredChunkIds(this.persister);
for (const id of ids) await this.loadChunk(id);
```
## API Reference
### `loadChunkBytes(persister, chunkId)` → `Promise<Uint8Array>`
Throws `StreamsChunkQueueError(STORAGE_NOT_CONFIGURED)` if `persister`
is `undefined`, `StreamsChunkQueueError(STORAGE_CHUNK_MISSING)` if the
backend returns `undefined`, otherwise resolves with the bytes.
### `listStoredChunkIds(persister)` → `Promise<bigint[]>`
Throws `StreamsChunkQueueError(STORAGE_NOT_CONFIGURED)` if `persister`
is `undefined`, otherwise forwards to `persister.listChunkIds()`.
## Implementation Notes
These helpers are pure functions instead of methods so they can be
unit-tested without spinning up a queue. The queue's `loadChunk`
delegates to `loadChunkBytes` and then re-uses `addChunk` to push the
bytes into the in-memory map; this means storage-loaded chunks travel
through the same validator / decryptor / hook pipeline as
network-arrived chunks. The two helpers throw a `StreamsChunkQueueError`
rather than a `TypeError` because the missing-persister condition is
an operational fault (the caller forgot to wire storage) and should be
filterable by error code in dashboards.

View File

@@ -0,0 +1,151 @@
# streaming-chunk-queue
## Purpose
`StreamingChunkQueue` is the public class that reconstitutes
out-of-order chunked file transfers into a sequential
`ReadableStream<Uint8Array>`. It composes the package's pluggable
validator, decryptor, persister, rate limiter, hooks, event emitter,
and OTel telemetry into a single coherent reception primitive — the
piece every chunked-file consumer (drive-mesh, drive-webrtc,
drive-transport-https) plugs into when chunks arrive via a jittery or
out-of-order transport.
## Features
- Out-of-order ingestion: chunks may arrive in any order; duplicates
are silently ignored.
- Sequential output: the readable stream emits chunks strictly in
ascending `chunkId` order.
- Auto-start: the first `addChunk` automatically materialises the
stream (configurable via `autoStart: false`).
- Pluggable validator + decryptor strategies with adapters for
`@suaveplan/packer` `ChunkVerifier`.
- Optional `TokenBucketRateLimiterAdapter` integration for bandwidth
throttling without dropping chunks.
- Optional `StorageChunkPersister` for resumable transfers.
- Eight lifecycle hooks, ten event types, and wildcard listener
support.
- Binary chunk-request protocol exposed through `requestChunks` and
`requestMissing`.
- Missing-chunk timeout that errors the readable stream after
`chunkTimeout` ms (default 30 000).
- Frozen status snapshot via `getStatus`.
- OTel counters and a histogram (`stream_duration_ms`).
## Usage: Basic
```ts
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
const queue = new StreamingChunkQueue({
fileId: 12345n,
totalChunks: 5n,
});
await queue.addChunk(0n, bytes0);
await queue.addChunk(4n, bytes4);
await queue.addChunk(1n, bytes1);
await queue.addChunk(2n, bytes2);
await queue.addChunk(3n, bytes3);
const reader = queue.getStream().getReader();
for (;;) {
const { done, value } = await reader.read();
if (done) break;
process(value);
}
```
## Usage: Advanced
```ts
import { ChunkVerifier } from "@suaveplan/packer";
import { TokenBucket } from "@suaveplan/rate-limiter";
import { MemoryStorage } from "@suaveplan/storage";
import {
ChunkVerifierValidator,
StorageChunkPersister,
StreamingChunkQueue,
TokenBucketRateLimiterAdapter,
} from "@suaveplan/streams-chunk-queue";
const verifier = new ChunkVerifier(serverPubKey);
const bucket = new TokenBucket({ capacity: 1_048_576, refillRate: 1024 });
const storage = new MemoryStorage();
const queue = new StreamingChunkQueue({
fileId: 12345n,
totalChunks: 1024n,
validator: new ChunkVerifierValidator(verifier),
rateLimiter: new TokenBucketRateLimiterAdapter(bucket),
persister: new StorageChunkPersister({ storage, fileId: 12345n }),
chunkTimeout: 60_000,
hooks: {
onStreamProgress: (pct) => statusBar.set(pct),
onError: (err) => toaster.error(err.message),
},
});
queue.addEventListener("*", (event) =>
telemetry.record("chunk_queue.event", { type: event.type }),
);
const stream = queue.getStream();
await stream.pipeTo(diskWriter);
```
## API Reference
### `new StreamingChunkQueue(options)`
`options` is `StreamingChunkQueueOptions`. The constructor throws
`StreamsChunkQueueError(INVALID_TOTAL_CHUNKS)` when `totalChunks <= 0n`.
### `addChunk(chunkId, data)` → `Promise<void>`
Persists, validates, decrypts, and enqueues the chunk. Throws
`INVALID_CHUNK_ID` on out-of-range ids, or
`VALIDATION_FAILED` / `DECRYPTION_FAILED` if the pipeline rejects.
### `getStream()` → `ReadableStream<Uint8Array>`
Idempotent: subsequent calls return the same stream.
### `requestChunks(ids?)` / `requestMissing()` → `Uint8Array`
Encode the binary chunk-request frame.
### `getMissingChunkIds()` → `bigint[]`
Ascending list of un-received chunk ids.
### `getStatus()` → `StreamingQueueStatus`
Frozen snapshot.
### `loadChunk(chunkId)` / `loadAllChunks()` → `Promise<void>`
Hydrate from the configured persister. Throw
`STORAGE_NOT_CONFIGURED` / `STORAGE_CHUNK_MISSING` on failure.
### `isAllChunksReceived()` / `isStreamingComplete()` → `boolean`
### `addEventListener(type, listener)` / `removeEventListener(type, listener)`
`type` is any `StreamEvent['type']` or `'*'`.
### `clear()` → `void`
Reset internal state for reuse.
## Implementation Notes
The validator and decryptor stages are extracted to `pipeline.ts` so
the queue file stays close to the per-file line budget. Storage guards
live in `storage-ops.ts` for the same reason. Rate-limiting denial is
non-destructive — the queue sleeps for the bucket's `retryAfter` (or
the adapter's fallback delay) and retries the same chunk; no bytes are
ever dropped. Missing-chunk timeout polls the in-memory map at a
configurable interval (default 100 ms) so deterministic tests can
inject a custom `clock` + `sleep`.

View File

@@ -0,0 +1,86 @@
# rate-limiter / adapter
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto
the `ChunkRateLimiter` interface used internally by the streaming chunk
queue. The `TokenBucket` does NOT throw on denial — it returns
`{ allowed: boolean, retryAfter?: number }`. This adapter encodes the
denial branch into the canonical `{ allowed: false, delay }` shape the
queue understands, with a configurable fallback when `retryAfter` is
missing.
## Features
- Wraps any structurally compatible `TokenBucketLike`.
- Returns `{ allowed: true, delay: 0 }` on admission.
- Returns `{ allowed: false, delay }` on denial, using `retryAfter` when
positive or a configurable fallback (default 100 ms) otherwise.
- Stateless beyond the bucket reference — safe to share across queues.
## Usage: Basic
```ts
import { TokenBucket } from "@suaveplan/rate-limiter";
import { TokenBucketRateLimiterAdapter } from "@suaveplan/streams-chunk-queue";
const bucket = new TokenBucket({ capacity: 1024 * 1024, refillRate: 1024 });
const limiter = new TokenBucketRateLimiterAdapter(bucket);
```
## Usage: Advanced
```ts
// Aggressive fallback for tightly-bound real-time streams.
const limiter = new TokenBucketRateLimiterAdapter(bucket, 25);
```
## API Reference
### `ChunkRateLimiter`
```ts
interface ChunkRateLimiter {
consume(bytes: number): { allowed: boolean; delay: number };
}
```
### `TokenBucketLike`
Structural subset of the `TokenBucket`:
```ts
interface TokenBucketLike {
consume(tokens?: number): { allowed: boolean; retryAfter?: number };
}
```
### `TokenBucketRateLimiterAdapter`
```ts
new TokenBucketRateLimiterAdapter(
bucket: TokenBucketLike,
fallbackDelayMs?: number, // default 100
)
```
`consume(bytes)` calls `bucket.consume(bytes)` and normalises the
result. The fallback delay is used only when the bucket denies AND
omits / zeroes `retryAfter`.
### `DEFAULT_RATE_LIMIT_DELAY_MS`
The constant `100`. Exposed for callers who want to reuse the default
in their own diagnostics.
## Implementation Notes
The structural `TokenBucketLike` keeps `@suaveplan/rate-limiter` an
optional peer concern at the type boundary — tests inject a fake
bucket and never need to spin up the real one. The adapter does not
sleep itself; the queue is in charge of timing because it owns the
`onRateLimit` hook and the `rate:limited` event.

View File

@@ -0,0 +1,92 @@
# storage-adapter / persister
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's
`ChunkPersister` contract. Persistence enables resuming an interrupted
transfer: every received chunk is written through to a key-value
backend (IndexedDB in browsers, file-backed stores on the server)
keyed by `file:<fileId>:chunk:<chunkId>` so a fresh queue with the
same `fileId` can hydrate via `loadAllChunks`.
## Features
- Deterministic key layout (`chunkStorageKey`, `chunkStoragePrefix`).
- Persists chunk bytes alongside the metadata via the storage backend's
optional `metadata` slot.
- Loads a single chunk or lists every persisted chunk id for the
configured `fileId`, in ascending order.
- Structural `ChunkStorageLike` view so tests can substitute a Map.
## Usage: Basic
```ts
import { MemoryStorage } from "@suaveplan/storage";
import { StorageChunkPersister } from "@suaveplan/streams-chunk-queue";
const storage = new MemoryStorage();
const persister = new StorageChunkPersister({ storage, fileId: 12345n });
```
## Usage: Advanced
```ts
import { IndexedDBStorage } from "@suaveplan/storage";
const storage = await IndexedDBStorage.open("downloads", "chunks");
const persister = new StorageChunkPersister({
storage,
fileId: 12345n,
});
const queue = new StreamingChunkQueue({
fileId: 12345n,
totalChunks: 100n,
persister,
});
await queue.loadAllChunks();
```
## API Reference
### `chunkStorageKey(fileId, chunkId)` → `string`
`file:<fileId>:chunk:<chunkId>`.
### `chunkStoragePrefix(fileId)` → `string`
`file:<fileId>:chunk:`.
### `ChunkPersister`
```ts
interface ChunkPersister {
persist(chunk: StoredChunk): Promise<void>;
load(chunkId: bigint): Promise<Uint8Array | undefined>;
listChunkIds(): Promise<bigint[]>;
}
```
### `StorageChunkPersister`
```ts
new StorageChunkPersister({ storage, fileId });
```
Implements `ChunkPersister` over any `ChunkStorageLike`. `listChunkIds`
filters by prefix, parses the trailing segment as `bigint`, and sorts
ascending — the ordering means
`StreamingChunkQueue.loadAllChunks` ingests in the canonical chunk
order.
## Implementation Notes
`BigInt` is converted to `string` before being stored in storage
metadata so the JSON wire form is portable. The persister does NOT
attempt to delete entries — chunks are durable until the consumer
explicitly clears the underlying storage. Keys with an empty trailing
segment (`file:7:chunk:`) are silently dropped from `listChunkIds`
because they cannot be parsed as a bigint.

View File

@@ -0,0 +1,79 @@
# telemetry
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Exposes the OpenTelemetry instruments that the `StreamingChunkQueue`
records during its lifecycle. Centralising instrument creation in one
lazy singleton means the SDK can be configured after import without
losing observations, and tests can swap the underlying meter via
`resetTelemetryForTest`.
## Features
- One `meter` per process (`@suaveplan/streams-chunk-queue`).
- Five counters (`chunks_received_total`, `chunks_streamed_total`,
`validation_failures_total`, `decryption_failures_total`,
`rate_limited_total`).
- One histogram (`stream_duration_ms`) recorded when a transfer
completes.
- Lazy initialisation — instruments are created on first call to
`getTelemetry()`.
- `resetTelemetryForTest()` discards the cache so each test fixture
observes a fresh holder.
## Usage: Basic
```ts
import { getTelemetry } from "@suaveplan/streams-chunk-queue";
const t = getTelemetry();
t.chunksReceived.add(1);
```
## Usage: Advanced
```ts
import { getTelemetry, resetTelemetryForTest } from "@suaveplan/streams-chunk-queue";
beforeEach(() => resetTelemetryForTest());
```
## API Reference
### `getTelemetry()` → `StreamsChunkQueueTelemetry`
Returns the singleton holder of OTel instruments.
### `resetTelemetryForTest()` → `void`
Drops the cache so the next `getTelemetry()` call rebuilds a new
holder. Intended for test isolation only.
### `StreamsChunkQueueTelemetry`
```ts
interface StreamsChunkQueueTelemetry {
chunksReceived: Counter;
chunksStreamed: Counter;
validationFailures: Counter;
decryptionFailures: Counter;
rateLimited: Counter;
streamDuration: Histogram;
}
```
## Implementation Notes
OTel's `metrics.getMeter` returns a no-op meter when no SDK is
registered, which is the desired behaviour for unit tests and for
applications that have not yet configured observability. The lazy
cache ensures that a late `MeterProvider` registration is picked up the
first time a chunk is received. The counters were chosen so a single
dashboard tile (`chunks_received_total - chunks_streamed_total`)
exposes the in-flight backlog without requiring a separate gauge. The
histogram records on `stream:complete` only, so it never includes
partial transfers that errored on timeout.

View File

@@ -0,0 +1,95 @@
# types
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Declares the shared types consumed across every feature dir of
`@suaveplan/streams-chunk-queue`. Keeping them in one types-only file
prevents circular imports between the queue, validator, decryptor,
storage adapter, rate-limiter adapter, hooks, events, and telemetry
modules. Every type lives behind a named interface (no inline object
shapes) per suaveplan convention.
## Features
- `ChunkMetadata` — per-chunk metadata recorded on arrival.
- `StoredChunk` — bytes plus metadata, with `data` mutable so validator
and decryptor pipelines can rewrite it in-place.
- `StreamEvent` — discriminated union covering every emitted event.
- `StreamEventType` — concrete event names or the `*` wildcard.
- `StreamingChunkQueueHooks` — eight lifecycle callbacks.
- `StreamingQueueStatus` — frozen status snapshot.
- `ChunkRequest` — decoded chunk-request frame.
## Usage: Basic
```ts
import type { ChunkMetadata } from "@suaveplan/streams-chunk-queue";
function logArrival(meta: ChunkMetadata): void {
console.info(`got ${meta.size}B for chunk ${meta.chunkId}`);
}
```
## Usage: Advanced
```ts
import type {
StreamEvent,
StreamEventListener,
} from "@suaveplan/streams-chunk-queue";
const listener: StreamEventListener = (event: StreamEvent) => {
if (event.type === "rate:limited") {
retryAfterTelemetry.observe(event.delay);
}
};
```
## API Reference
### `ChunkMetadata`
Immutable identifiers (`fileId`, `chunkId`, `totalChunks`, `receivedAt`,
`size`) with mutable booleans `validated` and `decrypted` flipped by the
validator / decryptor pipeline.
### `StoredChunk`
`{ data: Uint8Array; metadata: ChunkMetadata }`. The mutability of
`data` is intentional — validator and decryptor adapters return new
bytes that supersede the inline ciphertext.
### `StreamEvent`
Discriminated by `type`. Variants cover `chunk:received`,
`chunk:validated`, `chunk:decrypted`, `chunk:streamed`,
`stream:progress`, `stream:complete`, `stream:error`,
`validation:failed`, `decryption:failed`, and `rate:limited`.
### `StreamingChunkQueueHooks`
Eight callbacks: `onChunkReceived`, `onChunkValidated`,
`onChunkDecrypted`, `onChunkStreamed`, `onStreamProgress`,
`onStreamComplete`, `onError`, `onRateLimit`. All may return `void` or
`Promise<void>`; the queue awaits both forms.
### `StreamingQueueStatus`
Frozen snapshot returned by `StreamingChunkQueue.getStatus()`.
### `ChunkRequest`
Decoded form of the chunk-request frame produced by the binary protocol
encoder.
## Implementation Notes
The mutable `validated` / `decrypted` booleans are an intentional
optimisation: it lets the queue keep a single map keyed by `chunkId`
without allocating a parallel "state" map. Consumers MUST NOT mutate
these fields directly — only the queue does so, exposing the result
through the immutable `StreamingQueueStatus` snapshot.

View File

@@ -0,0 +1,89 @@
# chunk-verifier-validator
## Purpose
Bridges the structural `ChunkVerifier` shape from `@suaveplan/packer`
into the canonical `ChunkValidator` strategy. The packer's verifier
returns a discriminated result (`{ ok: true, metadata }` or
`{ ok: false, code, message }`); this adapter coerces the failure
branch into the canonical `StreamsChunkQueueError(VALIDATION_FAILED)`
with the packer's diagnostic preserved in `context.verifierCode`.
## Features
- Accepts any structurally compatible verifier (`ChunkVerifierLike`),
so adapter tests don't need a real packer instance.
- Preserves the verifier's `code` + `message` inside the typed error's
`context` for downstream observability.
- Returns the input bytes unchanged on success — `ChunkVerifier`
validates the wire frame but does not modify it.
- Falls back to a sensible default `code`/`message` if the verifier
omits them, keeping the contract uniform.
## Usage: Basic
```ts
import { ChunkVerifier } from "@suaveplan/packer";
import { ChunkVerifierValidator } from "@suaveplan/streams-chunk-queue";
const verifier = new ChunkVerifier(serverPubKey);
const queue = new StreamingChunkQueue({
fileId: 1n,
totalChunks: 5n,
validator: new ChunkVerifierValidator(verifier),
});
```
## Usage: Advanced
```ts
import type { ChunkVerifierLike } from "@suaveplan/streams-chunk-queue";
const hsmVerifier: ChunkVerifierLike = {
verify(bytes) {
return hsmSignatureVerify(bytes)
? { ok: true }
: { ok: false, code: "HSM.REJECTED", message: "HSM rejected" };
},
};
const validator = new ChunkVerifierValidator(hsmVerifier);
```
## API Reference
### `ChunkVerifierLike`
```ts
interface ChunkVerifierLike {
verify(bytes: Uint8Array): {
ok: boolean;
code?: string;
message?: string;
};
}
```
### `ChunkVerifierValidator`
```ts
new ChunkVerifierValidator(verifier: ChunkVerifierLike)
```
Implements `ChunkValidator`. On `ok: true` returns
`{ data: <input bytes> }`. On `ok: false` rejects with
`StreamsChunkQueueError(VALIDATION_FAILED)` whose
`context.chunkId` is the offending chunk id and `context.verifierCode`
carries the verifier's own code.
## Implementation Notes
The adapter does not mutate `data` because `ChunkVerifier.verify` only
checks the bytes — it does not strip a header. If a future verifier
returns transformed bytes, the adapter contract can be extended without
breaking callers (the success branch already returns `{ data }`). The
adapter is intentionally synchronous on the verifier side (the wrapper
itself returns a `Promise` to satisfy the strategy interface), so a
fast in-process verifier does not allocate an extra microtask. The
structural typing of `ChunkVerifierLike` means hardware-backed
verifiers, mock test verifiers, and the real packer class are all
interchangeable.

View File

@@ -0,0 +1,93 @@
# validator
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Declares the `ChunkValidator` strategy interface and provides the
`NoopChunkValidator` default. Validation is the second stage of the
chunk pipeline (between ingestion and decryption). The strategy
interface lets callers plug in any verification primitive — Ed25519
signature checks, HMAC tags, custom integrity schemes — without the
core queue needing to know the cryptographic details.
## Features
- `ChunkValidator` interface with a single async `validate` method.
- `NoopChunkValidator` default that passes every chunk through
unchanged.
- `noopChunkValidator` singleton shorthand for callers that don't want
to instantiate.
- Adapters live in sibling modules (`chunk-verifier-validator.ts`).
## Usage: Basic
```ts
import { noopChunkValidator } from "@suaveplan/streams-chunk-queue";
const queue = new StreamingChunkQueue({
fileId: 1n,
totalChunks: 5n,
validator: noopChunkValidator,
});
```
## Usage: Advanced
```ts
import type { ChunkValidator } from "@suaveplan/streams-chunk-queue";
class HmacValidator implements ChunkValidator {
constructor(private readonly key: Uint8Array) {}
async validate(chunkId: bigint, data: Uint8Array) {
const tagLen = 32;
const tag = data.slice(0, tagLen);
const body = data.slice(tagLen);
if (!(await hmacVerify(this.key, body, tag))) {
throw new Error(`HMAC failed for chunk ${chunkId}`);
}
return { data: body };
}
}
```
## API Reference
### `ChunkValidator`
```ts
interface ChunkValidator {
validate(chunkId: bigint, data: Uint8Array): Promise<ValidationOutcome>;
}
```
Implementations MUST throw on failure. The thrown error is wrapped by
the queue into `StreamsChunkQueueError(VALIDATION_FAILED)`.
### `ValidationOutcome`
`{ data: Uint8Array }`. The returned bytes supersede the inline
ciphertext — useful when validation strips a header or signature.
### `NoopChunkValidator`
Concrete class implementing `ChunkValidator` as a pass-through.
### `noopChunkValidator`
Singleton instance of `NoopChunkValidator`.
## Implementation Notes
The interface is async even for synchronous validators because all
real-world validators end up wanting WebCrypto / native crypto, which
is async. Forcing the contract to be uniformly async means the queue
can `await` without per-implementation branches. The `data` return
value is intentionally writable — adapters may strip a signature
header before passing bytes downstream, so the queue stores the
adapter-returned bytes (not the original wire frame) in its in-memory
map. A noop validator is included so the queue can call
`validator.validate` unconditionally without checking for a
configured strategy.