Initial Commit
This commit is contained in:
@@ -0,0 +1,150 @@
|
||||
# OpenSpec Change: enterprise-streams-chunk-queue
|
||||
|
||||
- **Change ID**: enterprise-streams-chunk-queue
|
||||
- **Package**: `@suaveplan/streams-chunk-queue`
|
||||
- **Category / plugin**: `foundation/flow`
|
||||
- **Target path**: `packages/foundation/flow/streams-chunk-queue`
|
||||
- **Status**: PROPOSED
|
||||
- **Date**: 2026-05-23
|
||||
- **Design**: (none — proposal carries full design)
|
||||
- **Spec delta**: ./specs/streams-chunk-queue/spec.md
|
||||
- **Reference docs**:
|
||||
- `/mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/` (upstream source)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Migrate `@suaveplan/streams-chunk-queue` onto genesis. The package provides a
|
||||
universal-tier `StreamingChunkQueue` that accepts file chunks **out of order**
|
||||
and reconstitutes them into a sequential WHATWG `ReadableStream<Uint8Array>`.
|
||||
It composes pluggable validation (Ed25519 chunk signatures), pluggable chunk
|
||||
decryption, an injectable rate limiter, an injectable persistent storage
|
||||
backend, a wildcard event bus, eight lifecycle hooks, a binary chunk-request
|
||||
protocol, missing-chunk timeouts, and progress tracking.
|
||||
|
||||
It is the canonical reception primitive consumed by `@suaveplan/drive-mesh`,
|
||||
the WebRTC drive transport, and any chunked-file receiver that must surface a
|
||||
ReadableStream while chunks arrive on jitter-prone or out-of-order transports.
|
||||
|
||||
---
|
||||
|
||||
## Current-state audit (genesis)
|
||||
|
||||
The package does not exist on genesis. `STATUS.md` lists it with `–`. This is
|
||||
a **migration** from upstream `suaveplan/packages/universal/streams-chunk-queue`.
|
||||
|
||||
**Verified by:** `ls /mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/src/`.
|
||||
|
||||
---
|
||||
|
||||
## Parity Checklist
|
||||
|
||||
| Capability | Upstream | Migrated | Notes |
|
||||
|---|---|---|---|
|
||||
| Out-of-order `addChunk(chunkId, data)` ingestion | ✅ | ✅ | |
|
||||
| ReadableStream<Uint8Array> sequential output | ✅ | ✅ | |
|
||||
| Pluggable signature validation | ✅ | ⬆️ | Promoted to first-class `ChunkValidator` strategy interface; `ChunkVerifier` becomes one adapter |
|
||||
| Pluggable chunk decryption | ✅ | ⬆️ | Promoted to `ChunkDecryptor` strategy interface |
|
||||
| Rate limiting (TokenBucket) | ✅ | ✅ | Honours genesis `TokenBucket.consume()` result shape (no throw — `{allowed, retryAfter}`) |
|
||||
| Persistent storage adapter | ✅ | ✅ | `AbstractStorage` from `@suaveplan/storage` |
|
||||
| Auto-start streaming on first chunk | ✅ | ✅ | |
|
||||
| 8 lifecycle hooks | ✅ | ✅ | `onChunkReceived/Validated/Decrypted/Streamed/StreamProgress/StreamComplete/Error/RateLimit` |
|
||||
| Wildcard event listeners | ✅ | ✅ | `addEventListener(type \| '*')`, `removeEventListener` |
|
||||
| Binary chunk-request protocol | ✅ | ✅ | 12-byte header + bitmap; `requestChunks`, `requestMissing` |
|
||||
| `chunkTimeout` with poll-and-error | ✅ | ✅ | Default 30000 ms |
|
||||
| Progress events (0–100) | ✅ | ✅ | |
|
||||
| `getStatus` snapshot | ✅ | ✅ | |
|
||||
| `loadChunk` / `loadAllChunks` from storage | ✅ | ✅ | |
|
||||
| `getMissingChunkIds`, `isAllChunksReceived`, `isStreamingComplete` | ✅ | ✅ | |
|
||||
| `clear()` reset | ✅ | ✅ | |
|
||||
| `bigint` chunk identifiers (packer v2 u64) | ✅ | ✅ | |
|
||||
| Structured error codes | 🚫 | ⬆️ | Upstream used plain `Error`; genesis introduces `StreamsChunkQueueError` + `STREAMS_CHUNK_QUEUE.*` codes via `defineCodes` |
|
||||
| OTel instrumentation | 🚫 | ⬆️ | Counters + histogram added (`chunks_received_total`, `chunks_streamed_total`, `validation_failures_total`, `decryption_failures_total`, `rate_limited_total`, `stream_duration_ms`) |
|
||||
|
||||
No 🚫 entries without compensating ⬆️ enhancement.
|
||||
|
||||
---
|
||||
|
||||
## Available workspace deps on genesis
|
||||
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| `@suaveplan/error` (`workspace:^`) | `SuaveplanError` base; `defineCodes` registry |
|
||||
| `@suaveplan/types` (`workspace:^`) | shared types |
|
||||
| `@suaveplan/storage` (`workspace:^`) | `AbstractStorage` backing for persistence |
|
||||
| `@suaveplan/rate-limiter` (`workspace:^`) | `TokenBucket` rate limiter |
|
||||
| `@suaveplan/packer` (`workspace:^`) | Optional `ChunkVerifier` adapter for Ed25519 validation |
|
||||
| `tslib` (`catalog:build`) | Runtime helpers |
|
||||
| `zod` (`catalog:forms`) | Options schema |
|
||||
| `@opentelemetry/api` (`catalog:observability`) | OTel counters/histogram |
|
||||
|
||||
Dev deps: `@suaveplan/testing` (`workspace:^`), `vitest`, `@vitest/coverage-istanbul`, `typescript`, `vite`, `vite-plugin-dts`.
|
||||
|
||||
---
|
||||
|
||||
## Public API surface (summary)
|
||||
|
||||
```ts
|
||||
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 12345n,
|
||||
totalChunks: 5n,
|
||||
validator, // optional ChunkValidator
|
||||
decryptor, // optional ChunkDecryptor
|
||||
persister, // optional ChunkPersister
|
||||
rateLimiter, // optional ChunkRateLimiter
|
||||
hooks: { /* 8 lifecycle callbacks */ },
|
||||
});
|
||||
|
||||
await queue.addChunk(0n, bytes);
|
||||
const stream = queue.getStream(); // ReadableStream<Uint8Array>
|
||||
const req = queue.requestMissing();
|
||||
```
|
||||
|
||||
Exported surfaces:
|
||||
|
||||
- `StreamingChunkQueue` (core)
|
||||
- `ChunkValidator`, `ChunkDecryptor` strategy interfaces
|
||||
- `ChunkVerifierValidator` (adapter onto `@suaveplan/packer` `ChunkVerifier`)
|
||||
- `TokenBucketRateLimiterAdapter` (adapter onto `@suaveplan/rate-limiter` `TokenBucket`)
|
||||
- `StorageChunkPersister` (adapter onto `@suaveplan/storage` `AbstractStorage`)
|
||||
- `encodeChunkRequest` (binary protocol)
|
||||
- `STREAMS_CHUNK_QUEUE_CODES`, `StreamsChunkQueueError`
|
||||
- `StreamingChunkQueueOptions`, `StreamingChunkQueueHooks`, `ChunkMetadata`, `StoredChunk`, `ChunkRequest`, `StreamEvent`, `StreamingQueueStatus`
|
||||
|
||||
---
|
||||
|
||||
## File layout
|
||||
|
||||
```text
|
||||
packages/foundation/flow/streams-chunk-queue/
|
||||
package.json, tsconfig.lib.json, vite.config.ts, vitest.config.ts, README.md
|
||||
src/
|
||||
index.ts
|
||||
errors/streams-chunk-queue-error.{ts,md,test.ts}, index.ts
|
||||
types/types.{ts,md}, index.ts
|
||||
queue/streaming-chunk-queue.{ts,md,types.ts,test.ts}, pipeline.{ts,md,test.ts}, storage-ops.{ts,md,test.ts}, index.ts
|
||||
validator/validator.{ts,md,types.ts,test.ts}, chunk-verifier-validator.{ts,md,test.ts}, index.ts
|
||||
decryptor/decryptor.{ts,md,types.ts,test.ts}, index.ts
|
||||
rate-limiter/adapter.{ts,md,types.ts,test.ts}, index.ts
|
||||
storage-adapter/persister.{ts,md,types.ts,test.ts}, index.ts
|
||||
binary-protocol/encoder.{ts,md,types.ts,test.ts}, index.ts
|
||||
hooks/hooks.{ts,md,types.ts,test.ts}, index.ts
|
||||
events/events.{ts,md,types.ts,test.ts}, index.ts
|
||||
telemetry/telemetry.{ts,md,test.ts}, index.ts
|
||||
tests/setup/polyfills.ts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
- `@suaveplan/drive-mesh` — reception primitive on the receiver side; integration arrives in its own change.
|
||||
- `@suaveplan/drive-webrtc` / `@suaveplan/drive-transport-webrtc` — out-of-order chunk arrival is the dominant mode over a WebRTC datachannel.
|
||||
- `@suaveplan/drive-transport-https` — used when range-requested chunks arrive in parallel.
|
||||
|
||||
**Breaking changes**: none (new package on genesis).
|
||||
Reference in New Issue
Block a user