OpenSpec Change: enterprise-streams-chunk-queue

Change ID: enterprise-streams-chunk-queue
Package: @suaveplan/streams-chunk-queue
Category / plugin: foundation/flow
Target path: packages/foundation/flow/streams-chunk-queue
Status: PROPOSED
Date: 2026-05-23
Design: (none — proposal carries full design)
Spec delta: ./specs/streams-chunk-queue/spec.md
Reference docs:
- /mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/ (upstream source)

Summary

Migrate @suaveplan/streams-chunk-queue onto genesis. The package provides a universal-tier StreamingChunkQueue that accepts file chunks out of order and reconstitutes them into a sequential WHATWG ReadableStream<Uint8Array>. It composes pluggable validation (Ed25519 chunk signatures), pluggable chunk decryption, an injectable rate limiter, an injectable persistent storage backend, a wildcard event bus, eight lifecycle hooks, a binary chunk-request protocol, missing-chunk timeouts, and progress tracking.

It is the canonical reception primitive consumed by @suaveplan/drive-mesh, the WebRTC drive transport, and any chunked-file receiver that must surface a ReadableStream while chunks arrive on jitter-prone or out-of-order transports.

Current-state audit (genesis)

The package does not exist on genesis. STATUS.md lists it with –. This is a migration from upstream suaveplan/packages/universal/streams-chunk-queue.

Verified by: ls /mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/src/.

Parity Checklist

Capability	Upstream	Migrated	Notes
Out-of-order `addChunk(chunkId, data)` ingestion	✅	✅
ReadableStream sequential output	✅	✅
Pluggable signature validation	✅	⬆️	Promoted to first-class `ChunkValidator` strategy interface; `ChunkVerifier` becomes one adapter
Pluggable chunk decryption	✅	⬆️	Promoted to `ChunkDecryptor` strategy interface
Rate limiting (TokenBucket)	✅	✅	Honours genesis `TokenBucket.consume()` result shape (no throw — `{allowed, retryAfter}`)
Persistent storage adapter	✅	✅	`AbstractStorage` from `@suaveplan/storage`
Auto-start streaming on first chunk	✅	✅
8 lifecycle hooks	✅	✅	`onChunkReceived/Validated/Decrypted/Streamed/StreamProgress/StreamComplete/Error/RateLimit`
Wildcard event listeners	✅	✅	`addEventListener(type \| '*')`, `removeEventListener`
Binary chunk-request protocol	✅	✅	12-byte header + bitmap; `requestChunks`, `requestMissing`
`chunkTimeout` with poll-and-error	✅	✅	Default 30000 ms
Progress events (0–100)	✅	✅
`getStatus` snapshot	✅	✅
`loadChunk` / `loadAllChunks` from storage	✅	✅
`getMissingChunkIds`, `isAllChunksReceived`, `isStreamingComplete`	✅	✅
`clear()` reset	✅	✅
`bigint` chunk identifiers (packer v2 u64)	✅	✅
Structured error codes	🚫	⬆️	Upstream used plain `Error`; genesis introduces `StreamsChunkQueueError` + `STREAMS_CHUNK_QUEUE.*` codes via `defineCodes`
OTel instrumentation	🚫	⬆️	Counters + histogram added (`chunks_received_total`, `chunks_streamed_total`, `validation_failures_total`, `decryption_failures_total`, `rate_limited_total`, `stream_duration_ms`)

No 🚫 entries without compensating ⬆️ enhancement.

Available workspace deps on genesis

Package	Role
`@suaveplan/error` (`workspace:^`)	`SuaveplanError` base; `defineCodes` registry
`@suaveplan/types` (`workspace:^`)	shared types
`@suaveplan/storage` (`workspace:^`)	`AbstractStorage` backing for persistence
`@suaveplan/rate-limiter` (`workspace:^`)	`TokenBucket` rate limiter
`@suaveplan/packer` (`workspace:^`)	Optional `ChunkVerifier` adapter for Ed25519 validation
`tslib` (`catalog:build`)	Runtime helpers
`zod` (`catalog:forms`)	Options schema
`@opentelemetry/api` (`catalog:observability`)	OTel counters/histogram

Dev deps: @suaveplan/testing (workspace:^), vitest, @vitest/coverage-istanbul, typescript, vite, vite-plugin-dts.

Public API surface (summary)

import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";

const queue = new StreamingChunkQueue({
  fileId: 12345n,
  totalChunks: 5n,
  validator,       // optional ChunkValidator
  decryptor,       // optional ChunkDecryptor
  persister,       // optional ChunkPersister
  rateLimiter,     // optional ChunkRateLimiter
  hooks: { /* 8 lifecycle callbacks */ },
});

await queue.addChunk(0n, bytes);
const stream = queue.getStream();   // ReadableStream<Uint8Array>
const req    = queue.requestMissing();

Exported surfaces:

StreamingChunkQueue (core)
ChunkValidator, ChunkDecryptor strategy interfaces
ChunkVerifierValidator (adapter onto @suaveplan/packer ChunkVerifier)
TokenBucketRateLimiterAdapter (adapter onto @suaveplan/rate-limiter TokenBucket)
StorageChunkPersister (adapter onto @suaveplan/storage AbstractStorage)
encodeChunkRequest (binary protocol)
STREAMS_CHUNK_QUEUE_CODES, StreamsChunkQueueError
StreamingChunkQueueOptions, StreamingChunkQueueHooks, ChunkMetadata, StoredChunk, ChunkRequest, StreamEvent, StreamingQueueStatus

File layout

packages/foundation/flow/streams-chunk-queue/
  package.json, tsconfig.lib.json, vite.config.ts, vitest.config.ts, README.md
  src/
    index.ts
    errors/streams-chunk-queue-error.{ts,md,test.ts}, index.ts
    types/types.{ts,md}, index.ts
    queue/streaming-chunk-queue.{ts,md,types.ts,test.ts}, pipeline.{ts,md,test.ts}, storage-ops.{ts,md,test.ts}, index.ts
    validator/validator.{ts,md,types.ts,test.ts}, chunk-verifier-validator.{ts,md,test.ts}, index.ts
    decryptor/decryptor.{ts,md,types.ts,test.ts}, index.ts
    rate-limiter/adapter.{ts,md,types.ts,test.ts}, index.ts
    storage-adapter/persister.{ts,md,types.ts,test.ts}, index.ts
    binary-protocol/encoder.{ts,md,types.ts,test.ts}, index.ts
    hooks/hooks.{ts,md,types.ts,test.ts}, index.ts
    events/events.{ts,md,types.ts,test.ts}, index.ts
    telemetry/telemetry.{ts,md,test.ts}, index.ts
  tests/setup/polyfills.ts

Impact

Affected dependents

@suaveplan/drive-mesh — reception primitive on the receiver side; integration arrives in its own change.
@suaveplan/drive-webrtc / @suaveplan/drive-transport-webrtc — out-of-order chunk arrival is the dominant mode over a WebRTC datachannel.
@suaveplan/drive-transport-https — used when range-requested chunks arrive in parallel.

Breaking changes: none (new package on genesis).

6.7 KiB Raw Blame History Unescape Escape