first commit
This commit is contained in:
100
docs/foundation/flow/streams-chunk-queue/README.md
Normal file
100
docs/foundation/flow/streams-chunk-queue/README.md
Normal file
@@ -0,0 +1,100 @@
|
||||
# @suaveplan/streams-chunk-queue - Universal out-of-order chunk reconstitution: pluggable validation, decryption, rate limiting, storage, hooks, events, binary protocol, ReadableStream output
|
||||
|
||||
Universal out-of-order chunk reconstitution. Accepts chunks for a
|
||||
file in any order and exposes a sequential
|
||||
`ReadableStream<Uint8Array>` consumer surface, composed with pluggable
|
||||
validation, decryption, rate limiting, persistent storage, lifecycle
|
||||
hooks, a wildcard event bus, an efficient binary chunk-request
|
||||
protocol, missing-chunk timeouts, and OpenTelemetry instrumentation.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [binary-protocol](./src/binary-protocol/encoder.md) — Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`, `drive-transport-https`, and any peer that n…
|
||||
- [encoder](./src/binary-protocol/encoder.md) — Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`, `drive-transport-https`, and any peer that n…
|
||||
- [decryptor](./src/decryptor/decryptor.md) — Declares the `ChunkDecryptor` strategy interface and provides the `NoopChunkDecryptor` default. Decryption is the third…
|
||||
- [decryptor](./src/decryptor/decryptor.md) — Declares the `ChunkDecryptor` strategy interface and provides the `NoopChunkDecryptor` default. Decryption is the third…
|
||||
- [errors](./src/errors/streams-chunk-queue-error.md) — `StreamsChunkQueueError` is the single typed error class raised by every public surface of `@suaveplan/streams-chunk-qu…
|
||||
- [streams-chunk-queue-error](./src/errors/streams-chunk-queue-error.md) — `StreamsChunkQueueError` is the single typed error class raised by every public surface of `@suaveplan/streams-chunk-qu…
|
||||
- [events](./src/events/events.md) — Implements the lightweight event bus that powers the queue's `addEventListener` / `removeEventListener` surface. The em…
|
||||
- [events](./src/events/events.md) — Implements the lightweight event bus that powers the queue's `addEventListener` / `removeEventListener` surface. The em…
|
||||
- [hooks](./src/hooks/hooks.md) — Provides the helper functions that invoke the queue's eight lifecycle callbacks safely. The queue itself wires hook fir…
|
||||
- [hooks](./src/hooks/hooks.md) — Provides the helper functions that invoke the queue's eight lifecycle callbacks safely. The queue itself wires hook fir…
|
||||
- [pipeline](./src/queue/pipeline.md) — Houses the validation + decryption pipeline stages that the `StreamingChunkQueue` runs on every received chunk. Extract…
|
||||
- [storage-ops](./src/queue/storage-ops.md) — Holds the storage-related guards used by `StreamingChunkQueue.loadChunk` and `loadAllChunks`. Both methods need the sam…
|
||||
- [streaming-chunk-queue](./src/queue/streaming-chunk-queue.md) — `StreamingChunkQueue` is the public class that reconstitutes out-of-order chunked file transfers into a sequential `Rea…
|
||||
- [rate-limiter](./src/rate-limiter/adapter.md) — Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto the `ChunkRateLimiter` interface used internall…
|
||||
- [adapter](./src/rate-limiter/adapter.md) — Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto the `ChunkRateLimiter` interface used internall…
|
||||
- [storage-adapter](./src/storage-adapter/persister.md) — Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's `ChunkPersister` contract. Persistence enables resuming…
|
||||
- [persister](./src/storage-adapter/persister.md) — Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's `ChunkPersister` contract. Persistence enables resuming…
|
||||
- [telemetry](./src/telemetry/telemetry.md) — Exposes the OpenTelemetry instruments that the `StreamingChunkQueue` records during its lifecycle. Centralising instrum…
|
||||
- [telemetry](./src/telemetry/telemetry.md) — Exposes the OpenTelemetry instruments that the `StreamingChunkQueue` records during its lifecycle. Centralising instrum…
|
||||
- [types](./src/types/types.md) — Declares the shared types consumed across every feature dir of `@suaveplan/streams-chunk-queue`. Keeping them in one ty…
|
||||
- [types](./src/types/types.md) — Declares the shared types consumed across every feature dir of `@suaveplan/streams-chunk-queue`. Keeping them in one ty…
|
||||
- [validator](./src/validator/validator.md) — Declares the `ChunkValidator` strategy interface and provides the `NoopChunkValidator` default. Validation is the secon…
|
||||
- [chunk-verifier-validator](./src/validator/chunk-verifier-validator.md) — Bridges the structural `ChunkVerifier` shape from `@suaveplan/packer` into the canonical `ChunkValidator` strategy. The…
|
||||
- [validator](./src/validator/validator.md) — Declares the `ChunkValidator` strategy interface and provides the `NoopChunkValidator` default. Validation is the secon…
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
bun add @suaveplan/streams-chunk-queue
|
||||
```
|
||||
|
||||
## Quick start
|
||||
|
||||
```ts
|
||||
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 12345n,
|
||||
totalChunks: 5n,
|
||||
});
|
||||
|
||||
await queue.addChunk(2n, chunk2);
|
||||
await queue.addChunk(0n, chunk0);
|
||||
await queue.addChunk(1n, chunk1);
|
||||
await queue.addChunk(3n, chunk3);
|
||||
await queue.addChunk(4n, chunk4);
|
||||
|
||||
await queue.getStream().pipeTo(destination);
|
||||
```
|
||||
|
||||
## Modules
|
||||
|
||||
- [`src/queue/streaming-chunk-queue.md`](./src/queue/streaming-chunk-queue.md) — the public `StreamingChunkQueue` class.
|
||||
- [`src/queue/pipeline.md`](./src/queue/pipeline.md) — validation + decryption stages.
|
||||
- [`src/queue/storage-ops.md`](./src/queue/storage-ops.md) — storage-side guards.
|
||||
- [`src/validator/validator.md`](./src/validator/validator.md) — `ChunkValidator` strategy + `NoopChunkValidator`.
|
||||
- [`src/validator/chunk-verifier-validator.md`](./src/validator/chunk-verifier-validator.md) — adapter for `@suaveplan/packer` `ChunkVerifier`.
|
||||
- [`src/decryptor/decryptor.md`](./src/decryptor/decryptor.md) — `ChunkDecryptor` strategy + `NoopChunkDecryptor`.
|
||||
- [`src/rate-limiter/adapter.md`](./src/rate-limiter/adapter.md) — `TokenBucket` adapter onto the queue's `ChunkRateLimiter`.
|
||||
- [`src/storage-adapter/persister.md`](./src/storage-adapter/persister.md) — `StorageChunkPersister` over `AbstractStorage`.
|
||||
- [`src/binary-protocol/encoder.md`](./src/binary-protocol/encoder.md) — chunk-request wire format.
|
||||
- [`src/hooks/hooks.md`](./src/hooks/hooks.md) — safe lifecycle-hook invocation.
|
||||
- [`src/events/events.md`](./src/events/events.md) — wildcard-capable event emitter.
|
||||
- [`src/telemetry/telemetry.md`](./src/telemetry/telemetry.md) — OTel counters + histogram.
|
||||
- [`src/errors/streams-chunk-queue-error.md`](./src/errors/streams-chunk-queue-error.md) — error codes + class.
|
||||
- [`src/types/types.md`](./src/types/types.md) — shared types.
|
||||
|
||||
## Features
|
||||
|
||||
- Out-of-order ingestion; duplicates silently ignored.
|
||||
- Sequential `ReadableStream<Uint8Array>` output.
|
||||
- Auto-start on first chunk.
|
||||
- Pluggable validator + decryptor strategies with packer `ChunkVerifier` adapter.
|
||||
- Non-destructive rate-limit backoff via `TokenBucket` adapter.
|
||||
- Resumable transfers via `StorageChunkPersister` over `AbstractStorage`.
|
||||
- Eight lifecycle hooks, wildcard event bus, 10 event types.
|
||||
- Binary chunk-request protocol (12-byte header + bitmap).
|
||||
- Configurable missing-chunk timeout (default 30 s).
|
||||
- Progress events (0–100) + frozen `StreamingQueueStatus` snapshot.
|
||||
- Bigint chunk identifiers (packer v2 u64).
|
||||
- `StreamsChunkQueueError` with 8 stable codes.
|
||||
- OTel: 5 counters + 1 histogram.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# binary-protocol / encoder
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Produces the chunk-request wire frame consumed by `@suaveplan/drive-mesh`,
|
||||
`drive-transport-https`, and any peer that needs to tell the sender which
|
||||
chunks of a file it is still missing. The frame is compact (12-byte
|
||||
header + bitmap) so even a 100k-chunk request stays under 13 kB, and it
|
||||
is fixed-layout so receivers can validate it without a schema.
|
||||
|
||||
## Features
|
||||
|
||||
- 8-byte big-endian `fileId` (u64).
|
||||
- 4-byte big-endian `totalChunks` (u32).
|
||||
- `⌈totalChunks / 8⌉` byte bitmap where bit `i` of byte `floor(i/8)` is
|
||||
set iff chunk `i` is requested.
|
||||
- Zero-allocation hot path: a single `Uint8Array` of the exact size and
|
||||
one DataView for the header.
|
||||
- Range checks raise `StreamsChunkQueueError` with the matching code
|
||||
(`INVALID_CHUNK_ID` for fileId, `INVALID_TOTAL_CHUNKS` for
|
||||
totalChunks).
|
||||
- Chunk ids outside `[0, totalChunks)` are silently dropped — the
|
||||
encoder's contract is "request these chunks, capped by the file
|
||||
size".
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { encodeChunkRequest } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const frame = encodeChunkRequest({
|
||||
fileId: 12345n,
|
||||
totalChunks: 10n,
|
||||
chunkIds: [0n, 2n, 5n],
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
const missing = queue.getMissingChunkIds();
|
||||
const frame = encodeChunkRequest({
|
||||
fileId: queue.getStatus().fileId,
|
||||
totalChunks: queue.getStatus().totalChunks,
|
||||
chunkIds: missing,
|
||||
});
|
||||
await webrtcDc.send(frame);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `encodeChunkRequest({ fileId, totalChunks, chunkIds })` → `Uint8Array`
|
||||
|
||||
Returns the encoded frame. Throws `StreamsChunkQueueError` when
|
||||
`fileId` is outside the u64 range or `totalChunks` is outside the u32
|
||||
range.
|
||||
|
||||
### `CHUNK_REQUEST_HEADER_SIZE`
|
||||
|
||||
Constant `12`. Exposed for callers that decode the frame.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The encoder iterates over `chunkIds` (typically the missing set,
|
||||
bounded by realistic transfer sizes) instead of `[0, totalChunks)` so
|
||||
the hot path stays O(|chunkIds|) regardless of file size. The 8-byte
|
||||
big-endian `fileId` is written as two `setUint32` calls because
|
||||
`DataView` does not have a `setBigUint64` on every supported runtime
|
||||
target.
|
||||
@@ -0,0 +1,91 @@
|
||||
# decryptor
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Declares the `ChunkDecryptor` strategy interface and provides the
|
||||
`NoopChunkDecryptor` default. Decryption is the third stage of the
|
||||
chunk pipeline (after validation, before streaming). Externalising the
|
||||
strategy lets callers plug in AEAD primitives (XChaCha20-Poly1305,
|
||||
AES-256-GCM), envelope decryption, or HSM-bound decryption without the
|
||||
core queue knowing the cryptographic details.
|
||||
|
||||
## Features
|
||||
|
||||
- `ChunkDecryptor` interface with a single async `decrypt` method.
|
||||
- `NoopChunkDecryptor` pass-through default — used for unencrypted
|
||||
transports.
|
||||
- `noopChunkDecryptor` singleton shorthand.
|
||||
- Failures are caught by the queue and surfaced as
|
||||
`StreamsChunkQueueError(DECRYPTION_FAILED)`, never an opaque rejection.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { noopChunkDecryptor } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 1n,
|
||||
totalChunks: 5n,
|
||||
decryptor: noopChunkDecryptor,
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
import type { ChunkDecryptor } from "@suaveplan/streams-chunk-queue";
|
||||
import { chacha20poly1305 } from "@noble/ciphers/chacha";
|
||||
|
||||
class XChaChaDecryptor implements ChunkDecryptor {
|
||||
constructor(private readonly key: Uint8Array) {}
|
||||
|
||||
async decrypt(_chunkId: bigint, data: Uint8Array) {
|
||||
const nonce = data.slice(0, 24);
|
||||
const ciphertext = data.slice(24);
|
||||
const plaintext = chacha20poly1305(this.key, nonce).decrypt(ciphertext);
|
||||
return { data: plaintext };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `ChunkDecryptor`
|
||||
|
||||
```ts
|
||||
interface ChunkDecryptor {
|
||||
decrypt(chunkId: bigint, data: Uint8Array): Promise<DecryptionOutcome>;
|
||||
}
|
||||
```
|
||||
|
||||
### `DecryptionOutcome`
|
||||
|
||||
`{ data: Uint8Array }` — plaintext bytes that replace the ciphertext
|
||||
in the queue's storage.
|
||||
|
||||
### `NoopChunkDecryptor`
|
||||
|
||||
Concrete class returning `{ data }` unchanged. Suitable for plaintext
|
||||
transports or when decryption happens further upstream.
|
||||
|
||||
### `noopChunkDecryptor`
|
||||
|
||||
Singleton instance of `NoopChunkDecryptor`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The interface is async because real AEAD primitives invariably are.
|
||||
Errors thrown by `decrypt` are caught by the queue and remapped to the
|
||||
canonical `DECRYPTION_FAILED` code so downstream observability code can
|
||||
filter on a stable code instead of a library-specific error class. The
|
||||
adapter contract returns `{ data }` so a decryptor can also concatenate
|
||||
multiple ciphertext fragments, strip a nonce prefix, or replace the
|
||||
bytes wholesale — the queue replaces its in-memory slot with the
|
||||
returned bytes, freeing the original ciphertext for garbage
|
||||
collection. The noop default lets the queue avoid a conditional check
|
||||
on every chunk; it returns the same `Uint8Array` reference unchanged,
|
||||
so there is no allocation cost.
|
||||
@@ -0,0 +1,97 @@
|
||||
# streams-chunk-queue-error
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
`StreamsChunkQueueError` is the single typed error class raised by every
|
||||
public surface of `@suaveplan/streams-chunk-queue`. It extends
|
||||
`SuaveplanError` so callers get the full enterprise error contract: a
|
||||
stable `code` string, a structured `context` map, a redactable wire form,
|
||||
OpenTelemetry span-event mapping, and an unbroken cause chain. The
|
||||
package registers every code through `defineCodes`, guaranteeing the
|
||||
codes are immutable, frozen, and reachable from real call sites.
|
||||
|
||||
## Features
|
||||
|
||||
- One frozen registry — `STREAMS_CHUNK_QUEUE_CODES` — exposes the eight
|
||||
canonical codes (`INVALID_TOTAL_CHUNKS`, `INVALID_CHUNK_ID`,
|
||||
`STORAGE_NOT_CONFIGURED`, `STORAGE_CHUNK_MISSING`, `VALIDATION_FAILED`,
|
||||
`DECRYPTION_FAILED`, `STREAM_TIMEOUT`, `RATE_LIMIT_EXHAUSTED`).
|
||||
- One literal-typed error class — `StreamsChunkQueueError` — narrows
|
||||
`code` through `instanceof`.
|
||||
- Cause chaining via `SuaveplanError`'s `{ cause }` option, so the
|
||||
underlying network or crypto failure is preserved.
|
||||
- Structured `context` map (e.g. `{ chunkId, fileId, totalChunks }`) for
|
||||
observability and i18n.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import {
|
||||
STREAMS_CHUNK_QUEUE_CODES,
|
||||
StreamsChunkQueueError,
|
||||
} from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
throw new StreamsChunkQueueError(
|
||||
STREAMS_CHUNK_QUEUE_CODES.INVALID_CHUNK_ID,
|
||||
`chunk ${chunkId} out of range`,
|
||||
{ context: { chunkId: chunkId.toString() } },
|
||||
);
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
try {
|
||||
await queue.addChunk(chunkId, bytes);
|
||||
} catch (err) {
|
||||
if (
|
||||
err instanceof StreamsChunkQueueError &&
|
||||
err.code === STREAMS_CHUNK_QUEUE_CODES.VALIDATION_FAILED
|
||||
) {
|
||||
observabilityBus.publish("chunk-fraud", {
|
||||
chunkId: err.context?.chunkId,
|
||||
cause: err.cause,
|
||||
});
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `STREAMS_CHUNK_QUEUE_CODES`
|
||||
|
||||
A frozen record built by `defineCodes("STREAMS_CHUNK_QUEUE", […])`. Each
|
||||
key maps to the fully-qualified code string
|
||||
`STREAMS_CHUNK_QUEUE.<reason>`.
|
||||
|
||||
### `StreamsChunkQueueErrorCode`
|
||||
|
||||
Discriminated union of every code in the registry.
|
||||
|
||||
### `StreamsChunkQueueError`
|
||||
|
||||
```ts
|
||||
new StreamsChunkQueueError(
|
||||
code: StreamsChunkQueueErrorCode,
|
||||
message: string,
|
||||
options?: { cause?: unknown; context?: Record<string, unknown> },
|
||||
)
|
||||
```
|
||||
|
||||
- `code` — one of the registered code strings.
|
||||
- `message` — human-readable description (or i18n message key).
|
||||
- `options.cause` — underlying error preserved for `walkCauses`.
|
||||
- `options.context` — structured fields surfaced through the wire schema.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`SuaveplanError` is `abstract`, so this module supplies the concrete
|
||||
subclass. The constructor delegates to `super` with the cause and context
|
||||
forwarded through `SuaveplanErrorOptions`. Codes are intentionally
|
||||
namespaced with `STREAMS_CHUNK_QUEUE.` so multi-package log scans can
|
||||
filter by prefix without false positives from sibling packages.
|
||||
@@ -0,0 +1,69 @@
|
||||
# events
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Implements the lightweight event bus that powers the queue's
|
||||
`addEventListener` / `removeEventListener` surface. The emitter keeps a
|
||||
`Map<StreamEventType, Set<Listener>>` so registration and removal are
|
||||
both O(1), and exposes a wildcard `"*"` subscription so debugging tools
|
||||
can observe every event without enumerating types.
|
||||
|
||||
## Features
|
||||
|
||||
- Concrete-type subscriptions (`chunk:received`, `stream:progress`, …).
|
||||
- Wildcard subscription via `"*"` that receives every emitted event.
|
||||
- O(1) add, remove, and emit per listener.
|
||||
- `clear()` for tear-down (used by `StreamingChunkQueue.clear()`).
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { StreamEventEmitter } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const emitter = new StreamEventEmitter();
|
||||
emitter.addEventListener("chunk:received", (event) => {
|
||||
console.info("got chunk", event.chunkId);
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
const wildcard = (event: StreamEvent) =>
|
||||
telemetry.record("stream.event", { type: event.type });
|
||||
emitter.addEventListener("*", wildcard);
|
||||
emitter.removeEventListener("*", wildcard);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `StreamEventEmitter`
|
||||
|
||||
- `addEventListener(type, listener)` — register `listener` under
|
||||
`type` (either a concrete event type or `"*"`).
|
||||
- `removeEventListener(type, listener)` — remove a previously
|
||||
registered listener.
|
||||
- `emit(event)` — dispatch to all matching subscribers, then to all
|
||||
wildcard subscribers.
|
||||
- `clear()` — drop every registered listener.
|
||||
|
||||
### `WILDCARD_EVENT_TYPE`
|
||||
|
||||
The constant `"*"`. Exported so callers don't sprinkle string literals.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The wildcard dispatch happens after concrete dispatch, mirroring the
|
||||
ordering observable consumers expect (a tracing tool sees the event
|
||||
after the application code has reacted). Listeners are synchronous —
|
||||
the bus does not await listener return values; callers that need
|
||||
async work should `queueMicrotask` inside their listener. The emitter
|
||||
is intentionally allocation-light: a `Map<type, Set<listener>>` means
|
||||
add/remove are O(1) and dispatch is O(N) over only the matching
|
||||
listeners. There is no priority ordering; concrete-type listeners are
|
||||
dispatched in insertion order, then wildcard listeners in their own
|
||||
insertion order.
|
||||
69
docs/foundation/flow/streams-chunk-queue/src/hooks/hooks.md
Normal file
69
docs/foundation/flow/streams-chunk-queue/src/hooks/hooks.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# hooks
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Provides the helper functions that invoke the queue's eight lifecycle
|
||||
callbacks safely. The queue itself wires hook firing into its pipeline;
|
||||
this module isolates the "safe invocation" logic — coercion of thrown
|
||||
values to real `Error` instances, async-await of both sync and async
|
||||
hooks, and re-routing of secondary failures through `onError`.
|
||||
|
||||
## Features
|
||||
|
||||
- `invokeHook` swallows hook errors after re-routing them via `onError`,
|
||||
so a misbehaving listener cannot crash the queue's main loop.
|
||||
- `toError` coerces any thrown value (string, object, primitive) into a
|
||||
real `Error` so downstream consumers see a uniform shape.
|
||||
- The helper avoids recursive re-routing when the failing hook IS the
|
||||
`onError` hook itself.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { invokeHook } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
await invokeHook(hooks.onChunkReceived, [chunkId, metadata], hooks.onError, chunkId);
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
import { invokeHook, toError } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const onError = (err: Error) => logger.error({ err: toError(err) });
|
||||
|
||||
await invokeHook(
|
||||
async (id: bigint) => await reportToObservability(id),
|
||||
[chunkId],
|
||||
onError,
|
||||
chunkId,
|
||||
);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `invokeHook(hook, args, onError?, chunkId?)` → `Promise<void>`
|
||||
|
||||
- `hook` — optional `LifecycleHook<Args>`. If `undefined`, the call is
|
||||
a no-op.
|
||||
- `args` — tuple of arguments to forward.
|
||||
- `onError` — optional `(error, chunkId?) => void | Promise<void>`
|
||||
re-route. Not invoked if it is the same reference as `hook`.
|
||||
- `chunkId` — optional bigint forwarded to `onError`.
|
||||
|
||||
### `toError(value)` → `Error`
|
||||
|
||||
Coerces any value to an `Error`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The recursion guard `onError !== hook` is the difference between a
|
||||
queue that survives a buggy `onError` callback and one that hangs in an
|
||||
infinite loop. When the secondary `onError` invocation itself throws,
|
||||
the helper swallows the failure — the primary path is preserved
|
||||
because the queue independently surfaces the error via the event
|
||||
emitter and (for streaming errors) the controller.
|
||||
@@ -0,0 +1,73 @@
|
||||
# queue / pipeline
|
||||
|
||||
## Purpose
|
||||
|
||||
Houses the validation + decryption pipeline stages that the
|
||||
`StreamingChunkQueue` runs on every received chunk. Extracting these
|
||||
stages into pure helpers keeps the main queue class focused on
|
||||
orchestration (stream control, status tracking, request building) and
|
||||
lets the pipeline grow more stages — e.g. a future deduplication step —
|
||||
without ballooning the queue file past the per-file line budget.
|
||||
|
||||
## Features
|
||||
|
||||
- `runValidate` and `runDecrypt` share the same retry / event / hook /
|
||||
telemetry shape via the internal `runStage` helper.
|
||||
- Failures are coerced to `StreamsChunkQueueError` with the
|
||||
appropriate code (`VALIDATION_FAILED` / `DECRYPTION_FAILED`) and the
|
||||
original error is preserved as `cause`.
|
||||
- Successful stages mutate `metadata.validated` / `metadata.decrypted`
|
||||
through a callback so the queue's snapshot reflects pipeline state.
|
||||
- Telemetry counters (`validation_failures_total` /
|
||||
`decryption_failures_total`) are incremented on failure exactly once.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
The pipeline functions are internal to the queue package; the queue
|
||||
itself wires them.
|
||||
|
||||
```ts
|
||||
await runValidate(stored, validator, deps);
|
||||
await runDecrypt(stored, decryptor, deps);
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
await runValidate(stored, this.validator, this.pipelineDeps);
|
||||
await this.runChecksum(stored);
|
||||
await runDecrypt(stored, this.decryptor, this.pipelineDeps);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `PipelineDeps`
|
||||
|
||||
```ts
|
||||
interface PipelineDeps {
|
||||
emitter: StreamEventEmitter;
|
||||
hooks: StreamingChunkQueueHooks;
|
||||
telemetry: StreamsChunkQueueTelemetry;
|
||||
}
|
||||
```
|
||||
|
||||
### `runValidate(stored, validator, deps)` → `Promise<void>`
|
||||
|
||||
Runs `validator.validate(chunkId, data)`. On success: replaces
|
||||
`stored.data`, sets `metadata.validated = true`, emits `chunk:validated`,
|
||||
and fires `onChunkValidated`. On failure: emits `validation:failed`,
|
||||
fires `onError`, and re-throws a `StreamsChunkQueueError(VALIDATION_FAILED)`.
|
||||
|
||||
### `runDecrypt(stored, decryptor, deps)` → `Promise<void>`
|
||||
|
||||
Same shape, using `decryptor.decrypt` and the
|
||||
`chunk:decrypted` / `decryption:failed` events and the
|
||||
`DECRYPTION_FAILED` code.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`runStage` re-throws an existing `StreamsChunkQueueError` untouched so a
|
||||
downstream adapter (e.g. `ChunkVerifierValidator`) can attach its own
|
||||
context and have that context survive the queue's catch handler. The
|
||||
`onError` re-route is suppressed when the failing hook is the same
|
||||
function — see `hooks.md` for the rationale.
|
||||
@@ -0,0 +1,57 @@
|
||||
# queue / storage-ops
|
||||
|
||||
## Purpose
|
||||
|
||||
Holds the storage-related guards used by `StreamingChunkQueue.loadChunk`
|
||||
and `loadAllChunks`. Both methods need the same "is the persister
|
||||
configured?" check and the same "is the chunk actually in storage?"
|
||||
check, so factoring them out keeps the queue file from duplicating the
|
||||
guards and from spilling past the per-file line budget.
|
||||
|
||||
## Features
|
||||
|
||||
- `loadChunkBytes` throws `STORAGE_NOT_CONFIGURED` when no persister is
|
||||
set, `STORAGE_CHUNK_MISSING` when the chunk is absent, and returns
|
||||
the bytes otherwise.
|
||||
- `listStoredChunkIds` throws `STORAGE_NOT_CONFIGURED` when no
|
||||
persister is set, otherwise forwards to `persister.listChunkIds()`.
|
||||
- Both helpers preserve a stringified `chunkId` in
|
||||
`error.context` so logs are scannable.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
const bytes = await loadChunkBytes(this.persister, chunkId);
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
const ids = await listStoredChunkIds(this.persister);
|
||||
for (const id of ids) await this.loadChunk(id);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `loadChunkBytes(persister, chunkId)` → `Promise<Uint8Array>`
|
||||
|
||||
Throws `StreamsChunkQueueError(STORAGE_NOT_CONFIGURED)` if `persister`
|
||||
is `undefined`, `StreamsChunkQueueError(STORAGE_CHUNK_MISSING)` if the
|
||||
backend returns `undefined`, otherwise resolves with the bytes.
|
||||
|
||||
### `listStoredChunkIds(persister)` → `Promise<bigint[]>`
|
||||
|
||||
Throws `StreamsChunkQueueError(STORAGE_NOT_CONFIGURED)` if `persister`
|
||||
is `undefined`, otherwise forwards to `persister.listChunkIds()`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
These helpers are pure functions instead of methods so they can be
|
||||
unit-tested without spinning up a queue. The queue's `loadChunk`
|
||||
delegates to `loadChunkBytes` and then re-uses `addChunk` to push the
|
||||
bytes into the in-memory map; this means storage-loaded chunks travel
|
||||
through the same validator / decryptor / hook pipeline as
|
||||
network-arrived chunks. The two helpers throw a `StreamsChunkQueueError`
|
||||
rather than a `TypeError` because the missing-persister condition is
|
||||
an operational fault (the caller forgot to wire storage) and should be
|
||||
filterable by error code in dashboards.
|
||||
@@ -0,0 +1,151 @@
|
||||
# streaming-chunk-queue
|
||||
|
||||
## Purpose
|
||||
|
||||
`StreamingChunkQueue` is the public class that reconstitutes
|
||||
out-of-order chunked file transfers into a sequential
|
||||
`ReadableStream<Uint8Array>`. It composes the package's pluggable
|
||||
validator, decryptor, persister, rate limiter, hooks, event emitter,
|
||||
and OTel telemetry into a single coherent reception primitive — the
|
||||
piece every chunked-file consumer (drive-mesh, drive-webrtc,
|
||||
drive-transport-https) plugs into when chunks arrive via a jittery or
|
||||
out-of-order transport.
|
||||
|
||||
## Features
|
||||
|
||||
- Out-of-order ingestion: chunks may arrive in any order; duplicates
|
||||
are silently ignored.
|
||||
- Sequential output: the readable stream emits chunks strictly in
|
||||
ascending `chunkId` order.
|
||||
- Auto-start: the first `addChunk` automatically materialises the
|
||||
stream (configurable via `autoStart: false`).
|
||||
- Pluggable validator + decryptor strategies with adapters for
|
||||
`@suaveplan/packer` `ChunkVerifier`.
|
||||
- Optional `TokenBucketRateLimiterAdapter` integration for bandwidth
|
||||
throttling without dropping chunks.
|
||||
- Optional `StorageChunkPersister` for resumable transfers.
|
||||
- Eight lifecycle hooks, ten event types, and wildcard listener
|
||||
support.
|
||||
- Binary chunk-request protocol exposed through `requestChunks` and
|
||||
`requestMissing`.
|
||||
- Missing-chunk timeout that errors the readable stream after
|
||||
`chunkTimeout` ms (default 30 000).
|
||||
- Frozen status snapshot via `getStatus`.
|
||||
- OTel counters and a histogram (`stream_duration_ms`).
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 12345n,
|
||||
totalChunks: 5n,
|
||||
});
|
||||
|
||||
await queue.addChunk(0n, bytes0);
|
||||
await queue.addChunk(4n, bytes4);
|
||||
await queue.addChunk(1n, bytes1);
|
||||
await queue.addChunk(2n, bytes2);
|
||||
await queue.addChunk(3n, bytes3);
|
||||
|
||||
const reader = queue.getStream().getReader();
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
process(value);
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
import { ChunkVerifier } from "@suaveplan/packer";
|
||||
import { TokenBucket } from "@suaveplan/rate-limiter";
|
||||
import { MemoryStorage } from "@suaveplan/storage";
|
||||
import {
|
||||
ChunkVerifierValidator,
|
||||
StorageChunkPersister,
|
||||
StreamingChunkQueue,
|
||||
TokenBucketRateLimiterAdapter,
|
||||
} from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const verifier = new ChunkVerifier(serverPubKey);
|
||||
const bucket = new TokenBucket({ capacity: 1_048_576, refillRate: 1024 });
|
||||
const storage = new MemoryStorage();
|
||||
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 12345n,
|
||||
totalChunks: 1024n,
|
||||
validator: new ChunkVerifierValidator(verifier),
|
||||
rateLimiter: new TokenBucketRateLimiterAdapter(bucket),
|
||||
persister: new StorageChunkPersister({ storage, fileId: 12345n }),
|
||||
chunkTimeout: 60_000,
|
||||
hooks: {
|
||||
onStreamProgress: (pct) => statusBar.set(pct),
|
||||
onError: (err) => toaster.error(err.message),
|
||||
},
|
||||
});
|
||||
|
||||
queue.addEventListener("*", (event) =>
|
||||
telemetry.record("chunk_queue.event", { type: event.type }),
|
||||
);
|
||||
|
||||
const stream = queue.getStream();
|
||||
await stream.pipeTo(diskWriter);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `new StreamingChunkQueue(options)`
|
||||
|
||||
`options` is `StreamingChunkQueueOptions`. The constructor throws
|
||||
`StreamsChunkQueueError(INVALID_TOTAL_CHUNKS)` when `totalChunks <= 0n`.
|
||||
|
||||
### `addChunk(chunkId, data)` → `Promise<void>`
|
||||
|
||||
Persists, validates, decrypts, and enqueues the chunk. Throws
|
||||
`INVALID_CHUNK_ID` on out-of-range ids, or
|
||||
`VALIDATION_FAILED` / `DECRYPTION_FAILED` if the pipeline rejects.
|
||||
|
||||
### `getStream()` → `ReadableStream<Uint8Array>`
|
||||
|
||||
Idempotent: subsequent calls return the same stream.
|
||||
|
||||
### `requestChunks(ids?)` / `requestMissing()` → `Uint8Array`
|
||||
|
||||
Encode the binary chunk-request frame.
|
||||
|
||||
### `getMissingChunkIds()` → `bigint[]`
|
||||
|
||||
Ascending list of un-received chunk ids.
|
||||
|
||||
### `getStatus()` → `StreamingQueueStatus`
|
||||
|
||||
Frozen snapshot.
|
||||
|
||||
### `loadChunk(chunkId)` / `loadAllChunks()` → `Promise<void>`
|
||||
|
||||
Hydrate from the configured persister. Throw
|
||||
`STORAGE_NOT_CONFIGURED` / `STORAGE_CHUNK_MISSING` on failure.
|
||||
|
||||
### `isAllChunksReceived()` / `isStreamingComplete()` → `boolean`
|
||||
|
||||
### `addEventListener(type, listener)` / `removeEventListener(type, listener)`
|
||||
|
||||
`type` is any `StreamEvent['type']` or `'*'`.
|
||||
|
||||
### `clear()` → `void`
|
||||
|
||||
Reset internal state for reuse.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The validator and decryptor stages are extracted to `pipeline.ts` so
|
||||
the queue file stays close to the per-file line budget. Storage guards
|
||||
live in `storage-ops.ts` for the same reason. Rate-limiting denial is
|
||||
non-destructive — the queue sleeps for the bucket's `retryAfter` (or
|
||||
the adapter's fallback delay) and retries the same chunk; no bytes are
|
||||
ever dropped. Missing-chunk timeout polls the in-memory map at a
|
||||
configurable interval (default 100 ms) so deterministic tests can
|
||||
inject a custom `clock` + `sleep`.
|
||||
@@ -0,0 +1,86 @@
|
||||
# rate-limiter / adapter
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Normalises the `@suaveplan/rate-limiter` `TokenBucket` API onto
|
||||
the `ChunkRateLimiter` interface used internally by the streaming chunk
|
||||
queue. The `TokenBucket` does NOT throw on denial — it returns
|
||||
`{ allowed: boolean, retryAfter?: number }`. This adapter encodes the
|
||||
denial branch into the canonical `{ allowed: false, delay }` shape the
|
||||
queue understands, with a configurable fallback when `retryAfter` is
|
||||
missing.
|
||||
|
||||
## Features
|
||||
|
||||
- Wraps any structurally compatible `TokenBucketLike`.
|
||||
- Returns `{ allowed: true, delay: 0 }` on admission.
|
||||
- Returns `{ allowed: false, delay }` on denial, using `retryAfter` when
|
||||
positive or a configurable fallback (default 100 ms) otherwise.
|
||||
- Stateless beyond the bucket reference — safe to share across queues.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { TokenBucket } from "@suaveplan/rate-limiter";
|
||||
import { TokenBucketRateLimiterAdapter } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const bucket = new TokenBucket({ capacity: 1024 * 1024, refillRate: 1024 });
|
||||
const limiter = new TokenBucketRateLimiterAdapter(bucket);
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
// Aggressive fallback for tightly-bound real-time streams.
|
||||
const limiter = new TokenBucketRateLimiterAdapter(bucket, 25);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `ChunkRateLimiter`
|
||||
|
||||
```ts
|
||||
interface ChunkRateLimiter {
|
||||
consume(bytes: number): { allowed: boolean; delay: number };
|
||||
}
|
||||
```
|
||||
|
||||
### `TokenBucketLike`
|
||||
|
||||
Structural subset of the `TokenBucket`:
|
||||
|
||||
```ts
|
||||
interface TokenBucketLike {
|
||||
consume(tokens?: number): { allowed: boolean; retryAfter?: number };
|
||||
}
|
||||
```
|
||||
|
||||
### `TokenBucketRateLimiterAdapter`
|
||||
|
||||
```ts
|
||||
new TokenBucketRateLimiterAdapter(
|
||||
bucket: TokenBucketLike,
|
||||
fallbackDelayMs?: number, // default 100
|
||||
)
|
||||
```
|
||||
|
||||
`consume(bytes)` calls `bucket.consume(bytes)` and normalises the
|
||||
result. The fallback delay is used only when the bucket denies AND
|
||||
omits / zeroes `retryAfter`.
|
||||
|
||||
### `DEFAULT_RATE_LIMIT_DELAY_MS`
|
||||
|
||||
The constant `100`. Exposed for callers who want to reuse the default
|
||||
in their own diagnostics.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The structural `TokenBucketLike` keeps `@suaveplan/rate-limiter` an
|
||||
optional peer concern at the type boundary — tests inject a fake
|
||||
bucket and never need to spin up the real one. The adapter does not
|
||||
sleep itself; the queue is in charge of timing because it owns the
|
||||
`onRateLimit` hook and the `rate:limited` event.
|
||||
@@ -0,0 +1,92 @@
|
||||
# storage-adapter / persister
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Bridges `@suaveplan/storage` `AbstractStorage` onto the queue's
|
||||
`ChunkPersister` contract. Persistence enables resuming an interrupted
|
||||
transfer: every received chunk is written through to a key-value
|
||||
backend (IndexedDB in browsers, file-backed stores on the server)
|
||||
keyed by `file:<fileId>:chunk:<chunkId>` so a fresh queue with the
|
||||
same `fileId` can hydrate via `loadAllChunks`.
|
||||
|
||||
## Features
|
||||
|
||||
- Deterministic key layout (`chunkStorageKey`, `chunkStoragePrefix`).
|
||||
- Persists chunk bytes alongside the metadata via the storage backend's
|
||||
optional `metadata` slot.
|
||||
- Loads a single chunk or lists every persisted chunk id for the
|
||||
configured `fileId`, in ascending order.
|
||||
- Structural `ChunkStorageLike` view so tests can substitute a Map.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { MemoryStorage } from "@suaveplan/storage";
|
||||
import { StorageChunkPersister } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const storage = new MemoryStorage();
|
||||
const persister = new StorageChunkPersister({ storage, fileId: 12345n });
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
import { IndexedDBStorage } from "@suaveplan/storage";
|
||||
|
||||
const storage = await IndexedDBStorage.open("downloads", "chunks");
|
||||
const persister = new StorageChunkPersister({
|
||||
storage,
|
||||
fileId: 12345n,
|
||||
});
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 12345n,
|
||||
totalChunks: 100n,
|
||||
persister,
|
||||
});
|
||||
await queue.loadAllChunks();
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `chunkStorageKey(fileId, chunkId)` → `string`
|
||||
|
||||
`file:<fileId>:chunk:<chunkId>`.
|
||||
|
||||
### `chunkStoragePrefix(fileId)` → `string`
|
||||
|
||||
`file:<fileId>:chunk:`.
|
||||
|
||||
### `ChunkPersister`
|
||||
|
||||
```ts
|
||||
interface ChunkPersister {
|
||||
persist(chunk: StoredChunk): Promise<void>;
|
||||
load(chunkId: bigint): Promise<Uint8Array | undefined>;
|
||||
listChunkIds(): Promise<bigint[]>;
|
||||
}
|
||||
```
|
||||
|
||||
### `StorageChunkPersister`
|
||||
|
||||
```ts
|
||||
new StorageChunkPersister({ storage, fileId });
|
||||
```
|
||||
|
||||
Implements `ChunkPersister` over any `ChunkStorageLike`. `listChunkIds`
|
||||
filters by prefix, parses the trailing segment as `bigint`, and sorts
|
||||
ascending — the ordering means
|
||||
`StreamingChunkQueue.loadAllChunks` ingests in the canonical chunk
|
||||
order.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`BigInt` is converted to `string` before being stored in storage
|
||||
metadata so the JSON wire form is portable. The persister does NOT
|
||||
attempt to delete entries — chunks are durable until the consumer
|
||||
explicitly clears the underlying storage. Keys with an empty trailing
|
||||
segment (`file:7:chunk:`) are silently dropped from `listChunkIds`
|
||||
because they cannot be parsed as a bigint.
|
||||
@@ -0,0 +1,79 @@
|
||||
# telemetry
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Exposes the OpenTelemetry instruments that the `StreamingChunkQueue`
|
||||
records during its lifecycle. Centralising instrument creation in one
|
||||
lazy singleton means the SDK can be configured after import without
|
||||
losing observations, and tests can swap the underlying meter via
|
||||
`resetTelemetryForTest`.
|
||||
|
||||
## Features
|
||||
|
||||
- One `meter` per process (`@suaveplan/streams-chunk-queue`).
|
||||
- Five counters (`chunks_received_total`, `chunks_streamed_total`,
|
||||
`validation_failures_total`, `decryption_failures_total`,
|
||||
`rate_limited_total`).
|
||||
- One histogram (`stream_duration_ms`) recorded when a transfer
|
||||
completes.
|
||||
- Lazy initialisation — instruments are created on first call to
|
||||
`getTelemetry()`.
|
||||
- `resetTelemetryForTest()` discards the cache so each test fixture
|
||||
observes a fresh holder.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { getTelemetry } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const t = getTelemetry();
|
||||
t.chunksReceived.add(1);
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
import { getTelemetry, resetTelemetryForTest } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
beforeEach(() => resetTelemetryForTest());
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `getTelemetry()` → `StreamsChunkQueueTelemetry`
|
||||
|
||||
Returns the singleton holder of OTel instruments.
|
||||
|
||||
### `resetTelemetryForTest()` → `void`
|
||||
|
||||
Drops the cache so the next `getTelemetry()` call rebuilds a new
|
||||
holder. Intended for test isolation only.
|
||||
|
||||
### `StreamsChunkQueueTelemetry`
|
||||
|
||||
```ts
|
||||
interface StreamsChunkQueueTelemetry {
|
||||
chunksReceived: Counter;
|
||||
chunksStreamed: Counter;
|
||||
validationFailures: Counter;
|
||||
decryptionFailures: Counter;
|
||||
rateLimited: Counter;
|
||||
streamDuration: Histogram;
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
OTel's `metrics.getMeter` returns a no-op meter when no SDK is
|
||||
registered, which is the desired behaviour for unit tests and for
|
||||
applications that have not yet configured observability. The lazy
|
||||
cache ensures that a late `MeterProvider` registration is picked up the
|
||||
first time a chunk is received. The counters were chosen so a single
|
||||
dashboard tile (`chunks_received_total - chunks_streamed_total`)
|
||||
exposes the in-flight backlog without requiring a separate gauge. The
|
||||
histogram records on `stream:complete` only, so it never includes
|
||||
partial transfers that errored on timeout.
|
||||
95
docs/foundation/flow/streams-chunk-queue/src/types/types.md
Normal file
95
docs/foundation/flow/streams-chunk-queue/src/types/types.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# types
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Declares the shared types consumed across every feature dir of
|
||||
`@suaveplan/streams-chunk-queue`. Keeping them in one types-only file
|
||||
prevents circular imports between the queue, validator, decryptor,
|
||||
storage adapter, rate-limiter adapter, hooks, events, and telemetry
|
||||
modules. Every type lives behind a named interface (no inline object
|
||||
shapes) per suaveplan convention.
|
||||
|
||||
## Features
|
||||
|
||||
- `ChunkMetadata` — per-chunk metadata recorded on arrival.
|
||||
- `StoredChunk` — bytes plus metadata, with `data` mutable so validator
|
||||
and decryptor pipelines can rewrite it in-place.
|
||||
- `StreamEvent` — discriminated union covering every emitted event.
|
||||
- `StreamEventType` — concrete event names or the `*` wildcard.
|
||||
- `StreamingChunkQueueHooks` — eight lifecycle callbacks.
|
||||
- `StreamingQueueStatus` — frozen status snapshot.
|
||||
- `ChunkRequest` — decoded chunk-request frame.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import type { ChunkMetadata } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
function logArrival(meta: ChunkMetadata): void {
|
||||
console.info(`got ${meta.size}B for chunk ${meta.chunkId}`);
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
import type {
|
||||
StreamEvent,
|
||||
StreamEventListener,
|
||||
} from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const listener: StreamEventListener = (event: StreamEvent) => {
|
||||
if (event.type === "rate:limited") {
|
||||
retryAfterTelemetry.observe(event.delay);
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `ChunkMetadata`
|
||||
|
||||
Immutable identifiers (`fileId`, `chunkId`, `totalChunks`, `receivedAt`,
|
||||
`size`) with mutable booleans `validated` and `decrypted` flipped by the
|
||||
validator / decryptor pipeline.
|
||||
|
||||
### `StoredChunk`
|
||||
|
||||
`{ data: Uint8Array; metadata: ChunkMetadata }`. The mutability of
|
||||
`data` is intentional — validator and decryptor adapters return new
|
||||
bytes that supersede the inline ciphertext.
|
||||
|
||||
### `StreamEvent`
|
||||
|
||||
Discriminated by `type`. Variants cover `chunk:received`,
|
||||
`chunk:validated`, `chunk:decrypted`, `chunk:streamed`,
|
||||
`stream:progress`, `stream:complete`, `stream:error`,
|
||||
`validation:failed`, `decryption:failed`, and `rate:limited`.
|
||||
|
||||
### `StreamingChunkQueueHooks`
|
||||
|
||||
Eight callbacks: `onChunkReceived`, `onChunkValidated`,
|
||||
`onChunkDecrypted`, `onChunkStreamed`, `onStreamProgress`,
|
||||
`onStreamComplete`, `onError`, `onRateLimit`. All may return `void` or
|
||||
`Promise<void>`; the queue awaits both forms.
|
||||
|
||||
### `StreamingQueueStatus`
|
||||
|
||||
Frozen snapshot returned by `StreamingChunkQueue.getStatus()`.
|
||||
|
||||
### `ChunkRequest`
|
||||
|
||||
Decoded form of the chunk-request frame produced by the binary protocol
|
||||
encoder.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The mutable `validated` / `decrypted` booleans are an intentional
|
||||
optimisation: it lets the queue keep a single map keyed by `chunkId`
|
||||
without allocating a parallel "state" map. Consumers MUST NOT mutate
|
||||
these fields directly — only the queue does so, exposing the result
|
||||
through the immutable `StreamingQueueStatus` snapshot.
|
||||
@@ -0,0 +1,89 @@
|
||||
# chunk-verifier-validator
|
||||
|
||||
## Purpose
|
||||
|
||||
Bridges the structural `ChunkVerifier` shape from `@suaveplan/packer`
|
||||
into the canonical `ChunkValidator` strategy. The packer's verifier
|
||||
returns a discriminated result (`{ ok: true, metadata }` or
|
||||
`{ ok: false, code, message }`); this adapter coerces the failure
|
||||
branch into the canonical `StreamsChunkQueueError(VALIDATION_FAILED)`
|
||||
with the packer's diagnostic preserved in `context.verifierCode`.
|
||||
|
||||
## Features
|
||||
|
||||
- Accepts any structurally compatible verifier (`ChunkVerifierLike`),
|
||||
so adapter tests don't need a real packer instance.
|
||||
- Preserves the verifier's `code` + `message` inside the typed error's
|
||||
`context` for downstream observability.
|
||||
- Returns the input bytes unchanged on success — `ChunkVerifier`
|
||||
validates the wire frame but does not modify it.
|
||||
- Falls back to a sensible default `code`/`message` if the verifier
|
||||
omits them, keeping the contract uniform.
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { ChunkVerifier } from "@suaveplan/packer";
|
||||
import { ChunkVerifierValidator } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const verifier = new ChunkVerifier(serverPubKey);
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 1n,
|
||||
totalChunks: 5n,
|
||||
validator: new ChunkVerifierValidator(verifier),
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
import type { ChunkVerifierLike } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const hsmVerifier: ChunkVerifierLike = {
|
||||
verify(bytes) {
|
||||
return hsmSignatureVerify(bytes)
|
||||
? { ok: true }
|
||||
: { ok: false, code: "HSM.REJECTED", message: "HSM rejected" };
|
||||
},
|
||||
};
|
||||
const validator = new ChunkVerifierValidator(hsmVerifier);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `ChunkVerifierLike`
|
||||
|
||||
```ts
|
||||
interface ChunkVerifierLike {
|
||||
verify(bytes: Uint8Array): {
|
||||
ok: boolean;
|
||||
code?: string;
|
||||
message?: string;
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### `ChunkVerifierValidator`
|
||||
|
||||
```ts
|
||||
new ChunkVerifierValidator(verifier: ChunkVerifierLike)
|
||||
```
|
||||
|
||||
Implements `ChunkValidator`. On `ok: true` returns
|
||||
`{ data: <input bytes> }`. On `ok: false` rejects with
|
||||
`StreamsChunkQueueError(VALIDATION_FAILED)` whose
|
||||
`context.chunkId` is the offending chunk id and `context.verifierCode`
|
||||
carries the verifier's own code.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The adapter does not mutate `data` because `ChunkVerifier.verify` only
|
||||
checks the bytes — it does not strip a header. If a future verifier
|
||||
returns transformed bytes, the adapter contract can be extended without
|
||||
breaking callers (the success branch already returns `{ data }`). The
|
||||
adapter is intentionally synchronous on the verifier side (the wrapper
|
||||
itself returns a `Promise` to satisfy the strategy interface), so a
|
||||
fast in-process verifier does not allocate an extra microtask. The
|
||||
structural typing of `ChunkVerifierLike` means hardware-backed
|
||||
verifiers, mock test verifiers, and the real packer class are all
|
||||
interchangeable.
|
||||
@@ -0,0 +1,93 @@
|
||||
# validator
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Declares the `ChunkValidator` strategy interface and provides the
|
||||
`NoopChunkValidator` default. Validation is the second stage of the
|
||||
chunk pipeline (between ingestion and decryption). The strategy
|
||||
interface lets callers plug in any verification primitive — Ed25519
|
||||
signature checks, HMAC tags, custom integrity schemes — without the
|
||||
core queue needing to know the cryptographic details.
|
||||
|
||||
## Features
|
||||
|
||||
- `ChunkValidator` interface with a single async `validate` method.
|
||||
- `NoopChunkValidator` default that passes every chunk through
|
||||
unchanged.
|
||||
- `noopChunkValidator` singleton shorthand for callers that don't want
|
||||
to instantiate.
|
||||
- Adapters live in sibling modules (`chunk-verifier-validator.ts`).
|
||||
|
||||
## Usage: Basic
|
||||
|
||||
```ts
|
||||
import { noopChunkValidator } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 1n,
|
||||
totalChunks: 5n,
|
||||
validator: noopChunkValidator,
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced
|
||||
|
||||
```ts
|
||||
import type { ChunkValidator } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
class HmacValidator implements ChunkValidator {
|
||||
constructor(private readonly key: Uint8Array) {}
|
||||
async validate(chunkId: bigint, data: Uint8Array) {
|
||||
const tagLen = 32;
|
||||
const tag = data.slice(0, tagLen);
|
||||
const body = data.slice(tagLen);
|
||||
if (!(await hmacVerify(this.key, body, tag))) {
|
||||
throw new Error(`HMAC failed for chunk ${chunkId}`);
|
||||
}
|
||||
return { data: body };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `ChunkValidator`
|
||||
|
||||
```ts
|
||||
interface ChunkValidator {
|
||||
validate(chunkId: bigint, data: Uint8Array): Promise<ValidationOutcome>;
|
||||
}
|
||||
```
|
||||
|
||||
Implementations MUST throw on failure. The thrown error is wrapped by
|
||||
the queue into `StreamsChunkQueueError(VALIDATION_FAILED)`.
|
||||
|
||||
### `ValidationOutcome`
|
||||
|
||||
`{ data: Uint8Array }`. The returned bytes supersede the inline
|
||||
ciphertext — useful when validation strips a header or signature.
|
||||
|
||||
### `NoopChunkValidator`
|
||||
|
||||
Concrete class implementing `ChunkValidator` as a pass-through.
|
||||
|
||||
### `noopChunkValidator`
|
||||
|
||||
Singleton instance of `NoopChunkValidator`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The interface is async even for synchronous validators because all
|
||||
real-world validators end up wanting WebCrypto / native crypto, which
|
||||
is async. Forcing the contract to be uniformly async means the queue
|
||||
can `await` without per-implementation branches. The `data` return
|
||||
value is intentionally writable — adapters may strip a signature
|
||||
header before passing bytes downstream, so the queue stores the
|
||||
adapter-returned bytes (not the original wire frame) in its in-memory
|
||||
map. A noop validator is included so the queue can call
|
||||
`validator.validate` unconditionally without checking for a
|
||||
configured strategy.
|
||||
Reference in New Issue
Block a user