# OpenSpec Change: enterprise-streams-chunk-queue

- **Change ID**: enterprise-streams-chunk-queue
- **Package**: `@suaveplan/streams-chunk-queue`
- **Category / plugin**: `foundation/flow`
- **Target path**: `packages/foundation/flow/streams-chunk-queue`
- **Status**: PROPOSED
- **Date**: 2026-05-23
- **Design**: (none — proposal carries full design)
- **Spec delta**: ./specs/streams-chunk-queue/spec.md
- **Reference docs**:
  - `/mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/` (upstream source)

---

## Summary

Migrate `@suaveplan/streams-chunk-queue` onto genesis. The package provides a
universal-tier `StreamingChunkQueue` that accepts file chunks **out of order**
and reconstitutes them into a sequential WHATWG `ReadableStream<Uint8Array>`.
It composes pluggable validation (Ed25519 chunk signatures), pluggable chunk
decryption, an injectable rate limiter, an injectable persistent storage
backend, a wildcard event bus, eight lifecycle hooks, a binary chunk-request
protocol, missing-chunk timeouts, and progress tracking.

It is the canonical reception primitive consumed by `@suaveplan/drive-mesh`,
the WebRTC drive transport, and any chunked-file receiver that must surface a
ReadableStream while chunks arrive on jitter-prone or out-of-order transports.

---

## Current-state audit (genesis)

The package does not exist on genesis. `STATUS.md` lists it with `–`. This is
a **migration** from upstream `suaveplan/packages/universal/streams-chunk-queue`.

**Verified by:** `ls /mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/src/`.

---

## Parity Checklist

| Capability | Upstream | Migrated | Notes |
|---|---|---|---|
| Out-of-order `addChunk(chunkId, data)` ingestion | ✅ | ✅ | |
| ReadableStream<Uint8Array> sequential output | ✅ | ✅ | |
| Pluggable signature validation | ✅ | ⬆️ | Promoted to first-class `ChunkValidator` strategy interface; `ChunkVerifier` becomes one adapter |
| Pluggable chunk decryption | ✅ | ⬆️ | Promoted to `ChunkDecryptor` strategy interface |
| Rate limiting (TokenBucket) | ✅ | ✅ | Honours genesis `TokenBucket.consume()` result shape (no throw — `{allowed, retryAfter}`) |
| Persistent storage adapter | ✅ | ✅ | `AbstractStorage` from `@suaveplan/storage` |
| Auto-start streaming on first chunk | ✅ | ✅ | |
| 8 lifecycle hooks | ✅ | ✅ | `onChunkReceived/Validated/Decrypted/Streamed/StreamProgress/StreamComplete/Error/RateLimit` |
| Wildcard event listeners | ✅ | ✅ | `addEventListener(type \| '*')`, `removeEventListener` |
| Binary chunk-request protocol | ✅ | ✅ | 12-byte header + bitmap; `requestChunks`, `requestMissing` |
| `chunkTimeout` with poll-and-error | ✅ | ✅ | Default 30000 ms |
| Progress events (0–100) | ✅ | ✅ | |
| `getStatus` snapshot | ✅ | ✅ | |
| `loadChunk` / `loadAllChunks` from storage | ✅ | ✅ | |
| `getMissingChunkIds`, `isAllChunksReceived`, `isStreamingComplete` | ✅ | ✅ | |
| `clear()` reset | ✅ | ✅ | |
| `bigint` chunk identifiers (packer v2 u64) | ✅ | ✅ | |
| Structured error codes | 🚫 | ⬆️ | Upstream used plain `Error`; genesis introduces `StreamsChunkQueueError` + `STREAMS_CHUNK_QUEUE.*` codes via `defineCodes` |
| OTel instrumentation | 🚫 | ⬆️ | Counters + histogram added (`chunks_received_total`, `chunks_streamed_total`, `validation_failures_total`, `decryption_failures_total`, `rate_limited_total`, `stream_duration_ms`) |

No 🚫 entries without compensating ⬆️ enhancement.

---

## Available workspace deps on genesis

| Package | Role |
|---|---|
| `@suaveplan/error` (`workspace:^`) | `SuaveplanError` base; `defineCodes` registry |
| `@suaveplan/types` (`workspace:^`) | shared types |
| `@suaveplan/storage` (`workspace:^`) | `AbstractStorage` backing for persistence |
| `@suaveplan/rate-limiter` (`workspace:^`) | `TokenBucket` rate limiter |
| `@suaveplan/packer` (`workspace:^`) | Optional `ChunkVerifier` adapter for Ed25519 validation |
| `tslib` (`catalog:build`) | Runtime helpers |
| `zod` (`catalog:forms`) | Options schema |
| `@opentelemetry/api` (`catalog:observability`) | OTel counters/histogram |

Dev deps: `@suaveplan/testing` (`workspace:^`), `vitest`, `@vitest/coverage-istanbul`, `typescript`, `vite`, `vite-plugin-dts`.

---

## Public API surface (summary)

```ts
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";

const queue = new StreamingChunkQueue({
  fileId: 12345n,
  totalChunks: 5n,
  validator,       // optional ChunkValidator
  decryptor,       // optional ChunkDecryptor
  persister,       // optional ChunkPersister
  rateLimiter,     // optional ChunkRateLimiter
  hooks: { /* 8 lifecycle callbacks */ },
});

await queue.addChunk(0n, bytes);
const stream = queue.getStream();   // ReadableStream<Uint8Array>
const req    = queue.requestMissing();
```

Exported surfaces:

- `StreamingChunkQueue` (core)
- `ChunkValidator`, `ChunkDecryptor` strategy interfaces
- `ChunkVerifierValidator` (adapter onto `@suaveplan/packer` `ChunkVerifier`)
- `TokenBucketRateLimiterAdapter` (adapter onto `@suaveplan/rate-limiter` `TokenBucket`)
- `StorageChunkPersister` (adapter onto `@suaveplan/storage` `AbstractStorage`)
- `encodeChunkRequest` (binary protocol)
- `STREAMS_CHUNK_QUEUE_CODES`, `StreamsChunkQueueError`
- `StreamingChunkQueueOptions`, `StreamingChunkQueueHooks`, `ChunkMetadata`, `StoredChunk`, `ChunkRequest`, `StreamEvent`, `StreamingQueueStatus`

---

## File layout

```text
packages/foundation/flow/streams-chunk-queue/
  package.json, tsconfig.lib.json, vite.config.ts, vitest.config.ts, README.md
  src/
    index.ts
    errors/streams-chunk-queue-error.{ts,md,test.ts}, index.ts
    types/types.{ts,md}, index.ts
    queue/streaming-chunk-queue.{ts,md,types.ts,test.ts}, pipeline.{ts,md,test.ts}, storage-ops.{ts,md,test.ts}, index.ts
    validator/validator.{ts,md,types.ts,test.ts}, chunk-verifier-validator.{ts,md,test.ts}, index.ts
    decryptor/decryptor.{ts,md,types.ts,test.ts}, index.ts
    rate-limiter/adapter.{ts,md,types.ts,test.ts}, index.ts
    storage-adapter/persister.{ts,md,types.ts,test.ts}, index.ts
    binary-protocol/encoder.{ts,md,types.ts,test.ts}, index.ts
    hooks/hooks.{ts,md,types.ts,test.ts}, index.ts
    events/events.{ts,md,types.ts,test.ts}, index.ts
    telemetry/telemetry.{ts,md,test.ts}, index.ts
  tests/setup/polyfills.ts
```

---

## Impact

### Affected dependents

- `@suaveplan/drive-mesh` — reception primitive on the receiver side; integration arrives in its own change.
- `@suaveplan/drive-webrtc` / `@suaveplan/drive-transport-webrtc` — out-of-order chunk arrival is the dominant mode over a WebRTC datachannel.
- `@suaveplan/drive-transport-https` — used when range-requested chunks arrive in parallel.

**Breaking changes**: none (new package on genesis).