Initial Commit
This commit is contained in:
@@ -0,0 +1,150 @@
|
||||
# OpenSpec Change: enterprise-streams-chunk-queue
|
||||
|
||||
- **Change ID**: enterprise-streams-chunk-queue
|
||||
- **Package**: `@suaveplan/streams-chunk-queue`
|
||||
- **Category / plugin**: `foundation/flow`
|
||||
- **Target path**: `packages/foundation/flow/streams-chunk-queue`
|
||||
- **Status**: PROPOSED
|
||||
- **Date**: 2026-05-23
|
||||
- **Design**: (none — proposal carries full design)
|
||||
- **Spec delta**: ./specs/streams-chunk-queue/spec.md
|
||||
- **Reference docs**:
|
||||
- `/mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/` (upstream source)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Migrate `@suaveplan/streams-chunk-queue` onto genesis. The package provides a
|
||||
universal-tier `StreamingChunkQueue` that accepts file chunks **out of order**
|
||||
and reconstitutes them into a sequential WHATWG `ReadableStream<Uint8Array>`.
|
||||
It composes pluggable validation (Ed25519 chunk signatures), pluggable chunk
|
||||
decryption, an injectable rate limiter, an injectable persistent storage
|
||||
backend, a wildcard event bus, eight lifecycle hooks, a binary chunk-request
|
||||
protocol, missing-chunk timeouts, and progress tracking.
|
||||
|
||||
It is the canonical reception primitive consumed by `@suaveplan/drive-mesh`,
|
||||
the WebRTC drive transport, and any chunked-file receiver that must surface a
|
||||
ReadableStream while chunks arrive on jitter-prone or out-of-order transports.
|
||||
|
||||
---
|
||||
|
||||
## Current-state audit (genesis)
|
||||
|
||||
The package does not exist on genesis. `STATUS.md` lists it with `–`. This is
|
||||
a **migration** from upstream `suaveplan/packages/universal/streams-chunk-queue`.
|
||||
|
||||
**Verified by:** `ls /mnt/s/web/suaveplan/packages/universal/streams-chunk-queue/src/`.
|
||||
|
||||
---
|
||||
|
||||
## Parity Checklist
|
||||
|
||||
| Capability | Upstream | Migrated | Notes |
|
||||
|---|---|---|---|
|
||||
| Out-of-order `addChunk(chunkId, data)` ingestion | ✅ | ✅ | |
|
||||
| ReadableStream<Uint8Array> sequential output | ✅ | ✅ | |
|
||||
| Pluggable signature validation | ✅ | ⬆️ | Promoted to first-class `ChunkValidator` strategy interface; `ChunkVerifier` becomes one adapter |
|
||||
| Pluggable chunk decryption | ✅ | ⬆️ | Promoted to `ChunkDecryptor` strategy interface |
|
||||
| Rate limiting (TokenBucket) | ✅ | ✅ | Honours genesis `TokenBucket.consume()` result shape (no throw — `{allowed, retryAfter}`) |
|
||||
| Persistent storage adapter | ✅ | ✅ | `AbstractStorage` from `@suaveplan/storage` |
|
||||
| Auto-start streaming on first chunk | ✅ | ✅ | |
|
||||
| 8 lifecycle hooks | ✅ | ✅ | `onChunkReceived/Validated/Decrypted/Streamed/StreamProgress/StreamComplete/Error/RateLimit` |
|
||||
| Wildcard event listeners | ✅ | ✅ | `addEventListener(type \| '*')`, `removeEventListener` |
|
||||
| Binary chunk-request protocol | ✅ | ✅ | 12-byte header + bitmap; `requestChunks`, `requestMissing` |
|
||||
| `chunkTimeout` with poll-and-error | ✅ | ✅ | Default 30000 ms |
|
||||
| Progress events (0–100) | ✅ | ✅ | |
|
||||
| `getStatus` snapshot | ✅ | ✅ | |
|
||||
| `loadChunk` / `loadAllChunks` from storage | ✅ | ✅ | |
|
||||
| `getMissingChunkIds`, `isAllChunksReceived`, `isStreamingComplete` | ✅ | ✅ | |
|
||||
| `clear()` reset | ✅ | ✅ | |
|
||||
| `bigint` chunk identifiers (packer v2 u64) | ✅ | ✅ | |
|
||||
| Structured error codes | 🚫 | ⬆️ | Upstream used plain `Error`; genesis introduces `StreamsChunkQueueError` + `STREAMS_CHUNK_QUEUE.*` codes via `defineCodes` |
|
||||
| OTel instrumentation | 🚫 | ⬆️ | Counters + histogram added (`chunks_received_total`, `chunks_streamed_total`, `validation_failures_total`, `decryption_failures_total`, `rate_limited_total`, `stream_duration_ms`) |
|
||||
|
||||
No 🚫 entries without compensating ⬆️ enhancement.
|
||||
|
||||
---
|
||||
|
||||
## Available workspace deps on genesis
|
||||
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| `@suaveplan/error` (`workspace:^`) | `SuaveplanError` base; `defineCodes` registry |
|
||||
| `@suaveplan/types` (`workspace:^`) | shared types |
|
||||
| `@suaveplan/storage` (`workspace:^`) | `AbstractStorage` backing for persistence |
|
||||
| `@suaveplan/rate-limiter` (`workspace:^`) | `TokenBucket` rate limiter |
|
||||
| `@suaveplan/packer` (`workspace:^`) | Optional `ChunkVerifier` adapter for Ed25519 validation |
|
||||
| `tslib` (`catalog:build`) | Runtime helpers |
|
||||
| `zod` (`catalog:forms`) | Options schema |
|
||||
| `@opentelemetry/api` (`catalog:observability`) | OTel counters/histogram |
|
||||
|
||||
Dev deps: `@suaveplan/testing` (`workspace:^`), `vitest`, `@vitest/coverage-istanbul`, `typescript`, `vite`, `vite-plugin-dts`.
|
||||
|
||||
---
|
||||
|
||||
## Public API surface (summary)
|
||||
|
||||
```ts
|
||||
import { StreamingChunkQueue } from "@suaveplan/streams-chunk-queue";
|
||||
|
||||
const queue = new StreamingChunkQueue({
|
||||
fileId: 12345n,
|
||||
totalChunks: 5n,
|
||||
validator, // optional ChunkValidator
|
||||
decryptor, // optional ChunkDecryptor
|
||||
persister, // optional ChunkPersister
|
||||
rateLimiter, // optional ChunkRateLimiter
|
||||
hooks: { /* 8 lifecycle callbacks */ },
|
||||
});
|
||||
|
||||
await queue.addChunk(0n, bytes);
|
||||
const stream = queue.getStream(); // ReadableStream<Uint8Array>
|
||||
const req = queue.requestMissing();
|
||||
```
|
||||
|
||||
Exported surfaces:
|
||||
|
||||
- `StreamingChunkQueue` (core)
|
||||
- `ChunkValidator`, `ChunkDecryptor` strategy interfaces
|
||||
- `ChunkVerifierValidator` (adapter onto `@suaveplan/packer` `ChunkVerifier`)
|
||||
- `TokenBucketRateLimiterAdapter` (adapter onto `@suaveplan/rate-limiter` `TokenBucket`)
|
||||
- `StorageChunkPersister` (adapter onto `@suaveplan/storage` `AbstractStorage`)
|
||||
- `encodeChunkRequest` (binary protocol)
|
||||
- `STREAMS_CHUNK_QUEUE_CODES`, `StreamsChunkQueueError`
|
||||
- `StreamingChunkQueueOptions`, `StreamingChunkQueueHooks`, `ChunkMetadata`, `StoredChunk`, `ChunkRequest`, `StreamEvent`, `StreamingQueueStatus`
|
||||
|
||||
---
|
||||
|
||||
## File layout
|
||||
|
||||
```text
|
||||
packages/foundation/flow/streams-chunk-queue/
|
||||
package.json, tsconfig.lib.json, vite.config.ts, vitest.config.ts, README.md
|
||||
src/
|
||||
index.ts
|
||||
errors/streams-chunk-queue-error.{ts,md,test.ts}, index.ts
|
||||
types/types.{ts,md}, index.ts
|
||||
queue/streaming-chunk-queue.{ts,md,types.ts,test.ts}, pipeline.{ts,md,test.ts}, storage-ops.{ts,md,test.ts}, index.ts
|
||||
validator/validator.{ts,md,types.ts,test.ts}, chunk-verifier-validator.{ts,md,test.ts}, index.ts
|
||||
decryptor/decryptor.{ts,md,types.ts,test.ts}, index.ts
|
||||
rate-limiter/adapter.{ts,md,types.ts,test.ts}, index.ts
|
||||
storage-adapter/persister.{ts,md,types.ts,test.ts}, index.ts
|
||||
binary-protocol/encoder.{ts,md,types.ts,test.ts}, index.ts
|
||||
hooks/hooks.{ts,md,types.ts,test.ts}, index.ts
|
||||
events/events.{ts,md,types.ts,test.ts}, index.ts
|
||||
telemetry/telemetry.{ts,md,test.ts}, index.ts
|
||||
tests/setup/polyfills.ts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
- `@suaveplan/drive-mesh` — reception primitive on the receiver side; integration arrives in its own change.
|
||||
- `@suaveplan/drive-webrtc` / `@suaveplan/drive-transport-webrtc` — out-of-order chunk arrival is the dominant mode over a WebRTC datachannel.
|
||||
- `@suaveplan/drive-transport-https` — used when range-requested chunks arrive in parallel.
|
||||
|
||||
**Breaking changes**: none (new package on genesis).
|
||||
@@ -0,0 +1,320 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: streams-chunk-queue (streams-chunk-queue)
|
||||
|
||||
**Package**: `@suaveplan/streams-chunk-queue`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. No exceptions.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Universal-tier purity
|
||||
|
||||
The `@suaveplan/streams-chunk-queue` package SHALL be universal-tier: zero
|
||||
imports of browser-only globals (`window`, `document`, `navigator`,
|
||||
`localStorage`, `IndexedDB`) and zero imports of Node-only modules
|
||||
(`node:fs`, `node:path`, `node:crypto`, `process`). Runtime dependencies
|
||||
SHALL be limited to `@suaveplan/error`, `@suaveplan/types`,
|
||||
`@suaveplan/storage`, `@suaveplan/rate-limiter`, `@suaveplan/packer`,
|
||||
`tslib`, `zod`, and `@opentelemetry/api`.
|
||||
|
||||
#### Scenario: Source tree references no platform globals
|
||||
|
||||
- **WHEN** `grep -rn "window\|document\|node:fs\|node:path\|node:crypto" packages/foundation/flow/streams-chunk-queue/src/` is run
|
||||
- **THEN** it produces zero matches
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"is universal-tier and avoids browser+node globals"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Bigint chunk identifiers (packer v2 u64)
|
||||
|
||||
`fileId`, `totalChunks`, `chunkId`, `nextChunkId`, and every chunk-id field
|
||||
on emitted events SHALL be `bigint`.
|
||||
|
||||
#### Scenario: Public API accepts and returns bigint
|
||||
|
||||
- **WHEN** a caller constructs `new StreamingChunkQueue({ fileId: 12345n, totalChunks: 5n })` and calls `addChunk(0n, bytes)`
|
||||
- **THEN** `getStatus().fileId === 12345n`, `getStatus().nextChunkId` is `bigint`, and `getMissingChunkIds()` returns `bigint[]`
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"exposes bigint identifiers on status and missing-chunk APIs"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Structured error registry
|
||||
|
||||
The package SHALL export `STREAMS_CHUNK_QUEUE_CODES` registered via
|
||||
`defineCodes` from `@suaveplan/error`, and a `StreamsChunkQueueError`
|
||||
subclass of `SuaveplanError`. The registry SHALL include the codes
|
||||
`INVALID_TOTAL_CHUNKS`, `INVALID_CHUNK_ID`, `STORAGE_NOT_CONFIGURED`,
|
||||
`STORAGE_CHUNK_MISSING`, `VALIDATION_FAILED`, `DECRYPTION_FAILED`,
|
||||
`STREAM_TIMEOUT`, and `RATE_LIMIT_EXHAUSTED`.
|
||||
|
||||
#### Scenario: Constructor rejects non-positive totalChunks
|
||||
|
||||
- **WHEN** a caller constructs with `totalChunks <= 0n`
|
||||
- **THEN** the constructor throws `StreamsChunkQueueError` whose `code === STREAMS_CHUNK_QUEUE_CODES.INVALID_TOTAL_CHUNKS`
|
||||
|
||||
#### Scenario: addChunk rejects out-of-range chunkId
|
||||
|
||||
- **WHEN** a caller calls `addChunk(totalChunks, data)`
|
||||
- **THEN** the call rejects with `StreamsChunkQueueError` whose `code === STREAMS_CHUNK_QUEUE_CODES.INVALID_CHUNK_ID`
|
||||
|
||||
**Verified by:** `src/errors/streams-chunk-queue-error.test.ts::"registers every documented STREAMS_CHUNK_QUEUE code"` AND `src/queue/streaming-chunk-queue.test.ts::"rejects out-of-range chunk ids"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Out-of-order chunk ingestion
|
||||
|
||||
`addChunk(chunkId, data)` SHALL accept chunks in any order, deduplicate
|
||||
identical chunk ids, build `ChunkMetadata` per arrival, emit
|
||||
`chunk:received`, invoke `onChunkReceived`, and update progress.
|
||||
|
||||
#### Scenario: Out-of-order assembly
|
||||
|
||||
- **WHEN** chunks arrive `[2, 0, 1, 3, 4]`
|
||||
- **THEN** the reader of `getStream()` receives them in order `[0, 1, 2, 3, 4]`
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"deduplicates and assembles out-of-order chunks"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: ReadableStream consumer surface
|
||||
|
||||
`getStream()` SHALL return a WHATWG `ReadableStream<Uint8Array>` that
|
||||
outputs chunks in monotonically increasing `chunkId` order, closes when
|
||||
the last chunk has been streamed, and emits `stream:complete` plus
|
||||
invokes `onStreamComplete`. Calling `getStream()` twice SHALL return the
|
||||
same stream instance.
|
||||
|
||||
#### Scenario: Stream closes on completion
|
||||
|
||||
- **WHEN** every chunk has been added and read
|
||||
- **THEN** the stream reader observes `done: true` and `isStreamingComplete()` returns `true`
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"closes the readable stream after the final chunk"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Auto-start
|
||||
|
||||
When `autoStart` is unset or `true` (default), the queue SHALL begin
|
||||
streaming as soon as the first chunk arrives. When `autoStart === false`,
|
||||
the stream materialises only on `getStream()`.
|
||||
|
||||
#### Scenario: Auto-start materialises the stream on first addChunk
|
||||
|
||||
- **WHEN** `autoStart` is left default and `addChunk(0n, bytes)` is called
|
||||
- **THEN** `getStatus().isStreaming` flips to `true` (or `isComplete` is `true` if the file has a single chunk)
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"auto-starts streaming on the first chunk when autoStart defaults to true"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Pluggable validator
|
||||
|
||||
The package SHALL define `ChunkValidator` and SHALL ship
|
||||
`ChunkVerifierValidator` wrapping `@suaveplan/packer` `ChunkVerifier`. On
|
||||
validator failure the queue SHALL emit `validation:failed`, fire
|
||||
`onError`, and reject `addChunk` with `StreamsChunkQueueError(VALIDATION_FAILED)`.
|
||||
|
||||
#### Scenario: ChunkVerifier failure propagates a typed error
|
||||
|
||||
- **WHEN** an injected verifier returns `{ ok: false }`
|
||||
- **THEN** the adapter rejects with `StreamsChunkQueueError(VALIDATION_FAILED)` with `context.chunkId` set
|
||||
|
||||
**Verified by:** `src/validator/chunk-verifier-validator.test.ts::"surfaces verifier failure as StreamsChunkQueueError"` AND `src/queue/streaming-chunk-queue.test.ts::"emits validation:failed and rejects when validator throws"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Pluggable decryptor
|
||||
|
||||
The package SHALL define `ChunkDecryptor`. On failure the queue SHALL
|
||||
emit `decryption:failed` and reject with
|
||||
`StreamsChunkQueueError(DECRYPTION_FAILED)`.
|
||||
|
||||
#### Scenario: Decryptor failure propagates a typed error
|
||||
|
||||
- **WHEN** a decryptor throws
|
||||
- **THEN** `addChunk` rejects with `StreamsChunkQueueError(DECRYPTION_FAILED)` and `decryption:failed` is emitted
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"emits decryption:failed and rejects when decryptor throws"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Rate-limit-aware streaming
|
||||
|
||||
When a `ChunkRateLimiter` is configured, `getStream()` SHALL call
|
||||
`consume(bytes)` before enqueueing each chunk. On denial the queue
|
||||
SHALL emit `rate:limited`, fire `onRateLimit`, sleep for the returned
|
||||
delay, then retry. Rate-limit admission SHALL NOT drop chunks.
|
||||
|
||||
#### Scenario: Rate-limit denial defers but does not drop
|
||||
|
||||
- **WHEN** the limiter denies the first request and admits the second
|
||||
- **THEN** the consumer still receives every chunk and at least one `rate:limited` event is observed
|
||||
|
||||
**Verified by:** `src/rate-limiter/adapter.test.ts::"defers and retries when the bucket denies a request"` AND `src/queue/streaming-chunk-queue.test.ts::"defers streaming under rate-limit pressure without dropping chunks"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Persistent storage adapter
|
||||
|
||||
The queue SHALL accept an optional `ChunkPersister` and SHALL ship
|
||||
`StorageChunkPersister` over `AbstractStorage`. Every `addChunk` SHALL
|
||||
persist the chunk under `file:<fileId>:chunk:<chunkId>`. `loadChunk`
|
||||
and `loadAllChunks` SHALL hydrate the queue, throwing
|
||||
`STORAGE_NOT_CONFIGURED` or `STORAGE_CHUNK_MISSING` on failure.
|
||||
|
||||
#### Scenario: Round-trip through storage hydrates a new queue
|
||||
|
||||
- **WHEN** a queue persists three chunks and a fresh queue calls `loadAllChunks()`
|
||||
- **THEN** `getStatus().receivedChunks === 3` on the new queue
|
||||
|
||||
**Verified by:** `src/storage-adapter/persister.test.ts::"round-trips chunks through AbstractStorage"` AND `src/queue/streaming-chunk-queue.test.ts::"hydrates from storage via loadAllChunks"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Missing-chunk timeout
|
||||
|
||||
If a sequential chunk is still missing after `chunkTimeout` ms (default
|
||||
30 000) the readable stream SHALL error with
|
||||
`StreamsChunkQueueError(STREAM_TIMEOUT)`, emit `stream:error`, and
|
||||
invoke `onError`.
|
||||
|
||||
#### Scenario: Stream errors when next chunk never arrives
|
||||
|
||||
- **WHEN** `chunkTimeout` is small and chunk `1n` never arrives after chunk `0n`
|
||||
- **THEN** the reader rejects with a `StreamsChunkQueueError(STREAM_TIMEOUT)`
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"errors the readable stream after chunkTimeout"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Hook system (8 lifecycle hooks)
|
||||
|
||||
The queue SHALL accept eight hooks: `onChunkReceived`,
|
||||
`onChunkValidated`, `onChunkDecrypted`, `onChunkStreamed`,
|
||||
`onStreamProgress`, `onStreamComplete`, `onError`, `onRateLimit`. Hooks
|
||||
returning a `Promise` SHALL be awaited; a thrown hook SHALL NOT crash
|
||||
the queue — the error is re-routed via `onError`.
|
||||
|
||||
#### Scenario: Every hook fires on the matching lifecycle event
|
||||
|
||||
- **WHEN** a queue with all eight hooks ingests one chunk
|
||||
- **THEN** at minimum `onChunkReceived`, `onChunkValidated`, `onChunkDecrypted`, `onChunkStreamed`, `onStreamProgress`, `onStreamComplete` all fire
|
||||
|
||||
**Verified by:** `src/hooks/hooks.test.ts::"safely invokes async and sync hooks and re-routes errors"` AND `src/queue/streaming-chunk-queue.test.ts::"fires every lifecycle hook"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Event system with wildcard
|
||||
|
||||
`addEventListener(type, listener)` and `removeEventListener(type, listener)`
|
||||
SHALL accept any concrete `StreamEvent['type']` or `'*'`. Wildcard
|
||||
listeners SHALL receive every emitted event.
|
||||
|
||||
#### Scenario: Wildcard listener observes every concrete event
|
||||
|
||||
- **WHEN** a wildcard listener and a `chunk:received` listener are attached and a chunk is added
|
||||
- **THEN** both listeners observe the `chunk:received` event
|
||||
|
||||
**Verified by:** `src/events/events.test.ts::"delivers events to wildcard and concrete listeners"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Binary chunk-request protocol
|
||||
|
||||
`requestChunks(chunkIds?)` and `requestMissing()` SHALL return a
|
||||
`Uint8Array` whose layout is 8 bytes BE `fileId` u64, 4 bytes BE
|
||||
`totalChunks` u32, then `⌈totalChunks/8⌉` bytes bitmap where bit `i` of
|
||||
byte `floor(i/8)` is `1` iff chunk `i` is requested.
|
||||
|
||||
#### Scenario: Bitmap encodes the requested chunk ids correctly
|
||||
|
||||
- **WHEN** `encodeChunkRequest({fileId: 12345n, totalChunks: 10n, chunkIds: [0n, 2n, 5n]})` is called
|
||||
- **THEN** byte 12 equals `0b00100101` and byte 13 equals `0x00`
|
||||
|
||||
**Verified by:** `src/binary-protocol/encoder.test.ts::"encodes file id, total chunks, and bitmap exactly"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Missing chunks + status snapshot
|
||||
|
||||
`getMissingChunkIds()` SHALL return ascending `bigint[]` of un-received
|
||||
chunk ids. `getStatus()` SHALL return a frozen snapshot containing
|
||||
`fileId`, `totalChunks`, `receivedChunks`, `validatedChunks`,
|
||||
`decryptedChunks`, `streamedChunks`, `progress`, `isStreaming`,
|
||||
`isComplete`, `missingChunks`, and `nextChunkId`.
|
||||
`isAllChunksReceived()` and `isStreamingComplete()` SHALL be derivable
|
||||
from the same state.
|
||||
|
||||
#### Scenario: Status reflects partial progress
|
||||
|
||||
- **WHEN** 3 of 5 chunks have been ingested
|
||||
- **THEN** `getStatus()` reports `receivedChunks === 3`, `progress === 60`, and `missingChunks.length === 2`
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"reports a precise status snapshot during partial ingestion"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: clear() resets queue state
|
||||
|
||||
`clear()` SHALL drop every stored chunk and reset internal state so the
|
||||
queue can be reused.
|
||||
|
||||
#### Scenario: Cleared queue accepts a fresh ingest
|
||||
|
||||
- **WHEN** a queue completes a transfer and then `clear()` is called
|
||||
- **THEN** `getStatus().receivedChunks === 0` and `addChunk(0n, …)` proceeds without error
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"clears all state and is reusable"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: OTel instrumentation
|
||||
|
||||
The package SHALL acquire the meter `@suaveplan/streams-chunk-queue` and
|
||||
SHALL record five counters
|
||||
(`streams_chunk_queue.chunks_received_total`,
|
||||
`chunks_streamed_total`, `validation_failures_total`,
|
||||
`decryption_failures_total`, `rate_limited_total`) and one histogram
|
||||
(`stream_duration_ms`). Instruments SHALL be created lazily and the
|
||||
meter SHALL be reused.
|
||||
|
||||
#### Scenario: Telemetry module re-uses one meter
|
||||
|
||||
- **WHEN** `getTelemetry()` is invoked twice
|
||||
- **THEN** both calls return the same object reference
|
||||
|
||||
**Verified by:** `src/telemetry/telemetry.test.ts::"creates instruments lazily and reuses the meter"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Test-runner imports go through `@suaveplan/testing/runner`
|
||||
|
||||
Every test file under `packages/foundation/flow/streams-chunk-queue/`
|
||||
SHALL import test runner primitives from `@suaveplan/testing/runner`,
|
||||
never directly from `vitest` or `bun:test`.
|
||||
|
||||
#### Scenario: No direct runner imports
|
||||
|
||||
- **WHEN** `grep -rnE "from ['\"](vitest|bun:test)['\"]" packages/foundation/flow/streams-chunk-queue/src/` is run
|
||||
- **THEN** it produces zero output
|
||||
|
||||
**Verified by:** `src/queue/streaming-chunk-queue.test.ts::"executes under @suaveplan/testing/runner"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 100 / 100 / 100 / 100 coverage
|
||||
|
||||
The package SHALL ship with `vitest.config.ts` whose `coverage.thresholds`
|
||||
are `statements: 100, branches: 100, functions: 100, lines: 100`.
|
||||
|
||||
#### Scenario: Coverage thresholds enforce 100 across all axes
|
||||
|
||||
- **WHEN** the coverage gate runs
|
||||
- **THEN** every axis is 100 and the gate exits zero
|
||||
|
||||
**Verified by:** `vitest.config.ts::"coverage.thresholds equal 100/100/100/100"`
|
||||
@@ -0,0 +1,78 @@
|
||||
# Tasks: enterprise-streams-chunk-queue
|
||||
|
||||
- **Change ID**: enterprise-streams-chunk-queue
|
||||
- **Package**: `@suaveplan/streams-chunk-queue`
|
||||
- **Date**: 2026-05-23
|
||||
|
||||
## 0. Pre-flight (non-negotiable)
|
||||
|
||||
- [x] 0.1 `git status` clean in worktree; branch renamed to `feat/streams-chunk-queue`.
|
||||
- [x] 0.2 Worktree under `.claude/worktrees/agent-a42db4238ae08d309` accounted for (this change).
|
||||
- [x] 0.3 `STATUS.md` lists `@suaveplan/streams-chunk-queue` as `–` (pending) → confirms genesis-side absence.
|
||||
- [x] 0.4 Upstream source read end-to-end: `src/core/streaming-chunk-queue.ts`, `src/types/types.ts`, `README.md`.
|
||||
- [x] 0.5 Genesis dependency APIs audited: `AbstractStorage`, `TokenBucket.consume()`, `ChunkVerifier.verify()` shapes confirmed before authoring adapters.
|
||||
|
||||
## 1. Errors
|
||||
|
||||
- [x] 1.1 `errors/streams-chunk-queue-error.ts` — `STREAMS_CHUNK_QUEUE_CODES` via `defineCodes`, `StreamsChunkQueueError`.
|
||||
- [x] 1.2 Tests cover construction, code preservation, cause chaining.
|
||||
|
||||
## 2. Shared types
|
||||
|
||||
- [x] 2.1 `types/types.ts` — `ChunkMetadata`, `StoredChunk`, `ChunkRequest`, `StreamEvent`, `StreamingQueueStatus`, `StreamingChunkQueueHooks`. All chunk IDs `bigint`.
|
||||
|
||||
## 3. Validator subsystem
|
||||
|
||||
- [x] 3.1 `validator/validator.ts` — `ChunkValidator` strategy + `NoopChunkValidator`.
|
||||
- [x] 3.2 `validator/chunk-verifier-validator.ts` — adapter wrapping `@suaveplan/packer` `ChunkVerifier`.
|
||||
|
||||
## 4. Decryptor subsystem
|
||||
|
||||
- [x] 4.1 `decryptor/decryptor.ts` — `ChunkDecryptor` strategy + `NoopChunkDecryptor`.
|
||||
|
||||
## 5. Rate-limiter adapter
|
||||
|
||||
- [x] 5.1 `rate-limiter/adapter.ts` — `TokenBucketRateLimiterAdapter` adapting genesis `TokenBucket.consume()`.
|
||||
|
||||
## 6. Storage adapter
|
||||
|
||||
- [x] 6.1 `storage-adapter/persister.ts` — `StorageChunkPersister` over `AbstractStorage`.
|
||||
|
||||
## 7. Binary protocol
|
||||
|
||||
- [x] 7.1 `binary-protocol/encoder.ts` — `encodeChunkRequest` u64+u32+bitmap layout.
|
||||
|
||||
## 8. Hooks + events
|
||||
|
||||
- [x] 8.1 `hooks/hooks.ts` — `invokeHook` safe-invocation helper.
|
||||
- [x] 8.2 `events/events.ts` — `StreamEventEmitter` with wildcard support.
|
||||
|
||||
## 9. Telemetry
|
||||
|
||||
- [x] 9.1 `telemetry/telemetry.ts` — OTel meter with counters and histogram.
|
||||
|
||||
## 10. Core `StreamingChunkQueue`
|
||||
|
||||
- [x] 10.1 `queue/streaming-chunk-queue.ts` — composes validator, decryptor, rate limiter, persister, telemetry, hooks, events.
|
||||
- [x] 10.2 Tests cover construction validation, out-of-range chunkId, duplicate ignore, out-of-order assembly, autostart, rate-limit retry, validation/decryption failure, persist/load round-trip, timeout, status, wildcard listener, hook invocation, cancel + clear.
|
||||
|
||||
## 11. Docs
|
||||
|
||||
- [x] 11.1 Every implementation `.ts` has a sibling `.md` (≥200 words).
|
||||
- [x] 11.2 Root `README.md` with Modules section.
|
||||
|
||||
## 12. Pre-publish gate
|
||||
|
||||
- [x] 12.1 `bunx turbo lint typecheck test --filter=@suaveplan/streams-chunk-queue` green.
|
||||
- [x] 12.2 `bunx turbo test:coverage --filter=@suaveplan/streams-chunk-queue` at 100/100/100/100.
|
||||
- [x] 12.3 `docs:check` 100%.
|
||||
|
||||
## 13. Spec fold + archive
|
||||
|
||||
- [x] 13.1 Fold spec delta into `openspec/specs/streams-chunk-queue/spec.md`.
|
||||
- [x] 13.2 Move folder to `openspec/archive/2026-05-23-enterprise-streams-chunk-queue/`.
|
||||
- [x] 13.3 Run `bun scripts/folder-reorg/generate-manifest.ts`.
|
||||
|
||||
## 14. Commit
|
||||
|
||||
- [x] 14.1 Commit on `feat/streams-chunk-queue`. No push, no main, no `--no-verify`.
|
||||
Reference in New Issue
Block a user