151 lines
6.7 KiB
Markdown
151 lines
6.7 KiB
Markdown
# OpenSpec Change: enterprise-streams-chunk-queue
|
||
|
||
- **Change ID**: enterprise-streams-chunk-queue
|
||
- **Package**: `@suaveplan/streams-chunk-queue`
|
||
- **Category / plugin**: `foundation/flow`
|
||
- **Target path**: `packages/foundation/flow/streams-chunk-queue`
|
||
- **Status**: PROPOSED
|
||
- **Date**: 2026-05-23
|
||
- **Design**: (none — proposal carries full design)
|
||
- **Spec delta**: ./specs/streams-chunk-queue/spec.md
|
||
- **Reference docs**:
|
||
- `/mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/` (upstream source)
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
Migrate `@suaveplan/streams-chunk-queue` onto genesis. The package provides a
|
||
universal-tier `StreamingChunkQueue` that accepts file chunks **out of order**
|
||
and reconstitutes them into a sequential WHATWG `ReadableStream<Uint8Array>`.
|
||
It composes pluggable validation (Ed25519 chunk signatures), pluggable chunk
|
||
decryption, an injectable rate limiter, an injectable persistent storage
|
||
backend, a wildcard event bus, eight lifecycle hooks, a binary chunk-request
|
||
protocol, missing-chunk timeouts, and progress tracking.
|
||
|
||
It is the canonical reception primitive consumed by `@suaveplan/drive-mesh`,
|
||
the WebRTC drive transport, and any chunked-file receiver that must surface a
|
||
ReadableStream while chunks arrive on jitter-prone or out-of-order transports.
|
||
|
||
---
|
||
|
||
## Current-state audit (genesis)
|
||
|
||
The package does not exist on genesis. `STATUS.md` lists it with `–`. This is
|
||
a **migration** from upstream `suaveplan/packages/universal/streams-chunk-queue`.
|
||
|
||
**Verified by:** `ls /mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/src/`.
|
||
|
||
---
|
||
|
||
## Parity Checklist
|
||
|
||
| Capability | Upstream | Migrated | Notes |
|
||
|---|---|---|---|
|
||
| Out-of-order `addChunk(chunkId, data)` ingestion | ✅ | ✅ | |
|
||
| ReadableStream<Uint8Array> sequential output | ✅ | ✅ | |
|
||
| Pluggable signature validation | ✅ | ⬆️ | Promoted to first-class `ChunkValidator` strategy interface; `ChunkVerifier` becomes one adapter |
|
||
| Pluggable chunk decryption | ✅ | ⬆️ | Promoted to `ChunkDecryptor` strategy interface |
|
||
| Rate limiting (TokenBucket) | ✅ | ✅ | Honours genesis `TokenBucket.consume()` result shape (no throw — `{allowed, retryAfter}`) |
|
||
| Persistent storage adapter | ✅ | ✅ | `AbstractStorage` from `@suaveplan/storage` |
|
||
| Auto-start streaming on first chunk | ✅ | ✅ | |
|
||
| 8 lifecycle hooks | ✅ | ✅ | `onChunkReceived/Validated/Decrypted/Streamed/StreamProgress/StreamComplete/Error/RateLimit` |
|
||
| Wildcard event listeners | ✅ | ✅ | `addEventListener(type \| '*')`, `removeEventListener` |
|
||
| Binary chunk-request protocol | ✅ | ✅ | 12-byte header + bitmap; `requestChunks`, `requestMissing` |
|
||
| `chunkTimeout` with poll-and-error | ✅ | ✅ | Default 30000 ms |
|
||
| Progress events (0–100) | ✅ | ✅ | |
|
||
| `getStatus` snapshot | ✅ | ✅ | |
|
||
| `loadChunk` / `loadAllChunks` from storage | ✅ | ✅ | |
|
||
| `getMissingChunkIds`, `isAllChunksReceived`, `isStreamingComplete` | ✅ | ✅ | |
|
||
| `clear()` reset | ✅ | ✅ | |
|
||
| `bigint` chunk identifiers (packer v2 u64) | ✅ | ✅ | |
|
||
| Structured error codes | 🚫 | ⬆️ | Upstream used plain `Error`; genesis introduces `StreamsChunkQueueError` + `STREAMS_CHUNK_QUEUE.*` codes via `defineCodes` |
|
||
| OTel instrumentation | 🚫 | ⬆️ | Counters + histogram added (`chunks_received_total`, `chunks_streamed_total`, `validation_failures_total`, `decryption_failures_total`, `rate_limited_total`, `stream_duration_ms`) |
|
||
|
||
No 🚫 entries without compensating ⬆️ enhancement.
|
||
|
||
---
|
||
|
||
## Available workspace deps on genesis
|
||
|
||
| Package | Role |
|
||
|---|---|
|
||
| `@suaveplan/error` (`workspace:^`) | `SuaveplanError` base; `defineCodes` registry |
|
||
| `@suaveplan/types` (`workspace:^`) | shared types |
|
||
| `@suaveplan/storage` (`workspace:^`) | `AbstractStorage` backing for persistence |
|
||
| `@suaveplan/rate-limiter` (`workspace:^`) | `TokenBucket` rate limiter |
|
||
| `@suaveplan/packer` (`workspace:^`) | Optional `ChunkVerifier` adapter for Ed25519 validation |
|
||
| `tslib` (`catalog:build`) | Runtime helpers |
|
||
| `zod` (`catalog:forms`) | Options schema |
|
||
| `@opentelemetry/api` (`catalog:observability`) | OTel counters/histogram |
|
||
|
||
Dev deps: `@suaveplan/testing` (`workspace:^`), `vitest`, `@vitest/coverage-istanbul`, `typescript`, `vite`, `vite-plugin-dts`.
|
||
|
||
---
|
||
|
||
## Public API surface (summary)
|
||
|
||
```ts
|
||
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
|
||
|
||
const queue = new StreamingChunkQueue({
|
||
fileId: 12345n,
|
||
totalChunks: 5n,
|
||
validator, // optional ChunkValidator
|
||
decryptor, // optional ChunkDecryptor
|
||
persister, // optional ChunkPersister
|
||
rateLimiter, // optional ChunkRateLimiter
|
||
hooks: { /* 8 lifecycle callbacks */ },
|
||
});
|
||
|
||
await queue.addChunk(0n, bytes);
|
||
const stream = queue.getStream(); // ReadableStream<Uint8Array>
|
||
const req = queue.requestMissing();
|
||
```
|
||
|
||
Exported surfaces:
|
||
|
||
- `StreamingChunkQueue` (core)
|
||
- `ChunkValidator`, `ChunkDecryptor` strategy interfaces
|
||
- `ChunkVerifierValidator` (adapter onto `@suaveplan/packer` `ChunkVerifier`)
|
||
- `TokenBucketRateLimiterAdapter` (adapter onto `@suaveplan/rate-limiter` `TokenBucket`)
|
||
- `StorageChunkPersister` (adapter onto `@suaveplan/storage` `AbstractStorage`)
|
||
- `encodeChunkRequest` (binary protocol)
|
||
- `STREAMS_CHUNK_QUEUE_CODES`, `StreamsChunkQueueError`
|
||
- `StreamingChunkQueueOptions`, `StreamingChunkQueueHooks`, `ChunkMetadata`, `StoredChunk`, `ChunkRequest`, `StreamEvent`, `StreamingQueueStatus`
|
||
|
||
---
|
||
|
||
## File layout
|
||
|
||
```text
|
||
packages/foundation/flow/streams-chunk-queue/
|
||
package.json, tsconfig.lib.json, vite.config.ts, vitest.config.ts, README.md
|
||
src/
|
||
index.ts
|
||
errors/streams-chunk-queue-error.{ts,md,test.ts}, index.ts
|
||
types/types.{ts,md}, index.ts
|
||
queue/streaming-chunk-queue.{ts,md,types.ts,test.ts}, pipeline.{ts,md,test.ts}, storage-ops.{ts,md,test.ts}, index.ts
|
||
validator/validator.{ts,md,types.ts,test.ts}, chunk-verifier-validator.{ts,md,test.ts}, index.ts
|
||
decryptor/decryptor.{ts,md,types.ts,test.ts}, index.ts
|
||
rate-limiter/adapter.{ts,md,types.ts,test.ts}, index.ts
|
||
storage-adapter/persister.{ts,md,types.ts,test.ts}, index.ts
|
||
binary-protocol/encoder.{ts,md,types.ts,test.ts}, index.ts
|
||
hooks/hooks.{ts,md,types.ts,test.ts}, index.ts
|
||
events/events.{ts,md,types.ts,test.ts}, index.ts
|
||
telemetry/telemetry.{ts,md,test.ts}, index.ts
|
||
tests/setup/polyfills.ts
|
||
```
|
||
|
||
---
|
||
|
||
## Impact
|
||
|
||
### Affected dependents
|
||
|
||
- `@suaveplan/drive-mesh` — reception primitive on the receiver side; integration arrives in its own change.
|
||
- `@suaveplan/drive-webrtc` / `@suaveplan/drive-transport-webrtc` — out-of-order chunk arrival is the dominant mode over a WebRTC datachannel.
|
||
- `@suaveplan/drive-transport-https` — used when range-requested chunks arrive in parallel.
|
||
|
||
**Breaking changes**: none (new package on genesis).
|