Initial Commit
This commit is contained in:
228
openspec/changes/enterprise-streams-redesign/proposal.md
Normal file
228
openspec/changes/enterprise-streams-redesign/proposal.md
Normal file
@@ -0,0 +1,228 @@
|
||||
# enterprise-streams-redesign
|
||||
|
||||
**Package**: `@suaveplan/streams`
|
||||
**Status**: PROPOSED
|
||||
**Type**: refactor
|
||||
|
||||
## Why
|
||||
|
||||
The genesis-original `enterprise-streams` change (archived 2026-04-24) shipped a functional, WHATWG-`TransformStream`-only API. After auditing `../suaveplan/packages/universal/streams/src/` — the canonical implementation on main — three structural gaps surfaced that block downstream packages (`streams-metadata`, `streams-chunk-queue`, future `audio-engine`, future `streams-worker`) from being built consistently:
|
||||
|
||||
1. **No plugin contract.** Hooks (`crypto:before-chunk`, `crypto:after-chunk`), events (`start` / `chunk` / `complete` / `error`), parameter introspection, and lifecycle (`init` / `flush` / `dispose`) cannot be expressed in pure `TransformStream` factories. Main's `BaseTransform` abstract class — with `process<T>(input)`, `setOperation`, `setParams`, `hook(name, fn)`, `on(name, fn)`, and `toTransformStream()` bridge — is the canonical plugin shape and is required for the DAW-style audio plugin ambition.
|
||||
2. **No streaming AEAD transform.** Main ships `CryptoTransform` with pluggable `xchacha20StreamingEngine` / `aesGcmStreamingEngine` that bind a counter and final-flag into the AEAD AAD, detecting reorder/truncate/tamper. Genesis has no equivalent. Encryption-on-the-fly is one of the explicit driving use cases.
|
||||
3. **No bidirectional / network primitives.** Main ships `BaseDuplexStream` and `NetworkStreamImpl` (id + protocol + metadata Map) for WebRTC / WebSocket / TCP integration. Genesis has none. These are foundational for the planned drive / network-discovery / audio-network packages.
|
||||
|
||||
Additionally, the existing genesis layout uses redundant `-stream` suffixes on every transform module (`transforms/throttle-stream/`, `transforms/map-stream/`, etc.) — confusing inside a package literally named `streams`. This change normalises naming as part of the redesign.
|
||||
|
||||
The 11 existing functional transforms, 3 sources, 3 sinks, fan-out, merge, pipeline-builder, instrument-transform, and node-adapters are all preserved as the implementation backbone — every `BaseTransform` subclass either subclasses or composes the equivalent functional logic via `toTransformStream()`, so no behavioural regression is introduced.
|
||||
|
||||
## What changes
|
||||
|
||||
### ADDED — plugin substrate (ported from main)
|
||||
|
||||
- ADDED: `BaseTransform` — abstract class with `name`, `setOperation`, `setParams`, `getOperation`, `getParams`, `hook(name, fn)`, `removeHook(name, fn)`, `on(name, fn)`, `off(name, fn)`, `process<T>(input, options?)` abstract, `toTransformStream<TIn, TOut>(): TransformStream<TIn, TOut>` bridge to WHATWG, `dispose()` for cleanup
|
||||
- ADDED: `TransformOptions`, `IBaseTransform`, `TransformHookPayload`, `TransformEventPayload` types
|
||||
- ADDED: `StreamManager` — `pipe(...transforms)`, `execute<T>(input, options?)`, `clear()`, `getTransforms()`
|
||||
- ADDED: `TransformFactory` — registry + `create(type, ...args)` with `TransformType` discriminated union; built-in registrations for every bundled transform
|
||||
- ADDED: `BaseDuplexStream<R, W>` primitive with `StreamSource<R>` and `StreamSink<W>` configuration
|
||||
- ADDED: `NetworkStreamImpl` with `id`, `protocol`, `metadata: Map<string, unknown>`, `closed` flag, `close()` and `reset()`
|
||||
- ADDED: factory functions `createDuplexStream`, `createNetworkStream`, `createTransformStream`
|
||||
|
||||
### ADDED — bundled BaseTransform subclasses (port + new)
|
||||
|
||||
Ported from main:
|
||||
|
||||
- ADDED: `CryptoTransform` — pluggable streaming AEAD with `engine`, `key`, `operation` ("encrypt" | "decrypt"), hooks (`crypto:before-chunk`, `crypto:after-chunk`), events (`crypto:start`, `crypto:chunk`, `crypto:complete`, `crypto:error`), `process(asyncIterable)` for stream-of-frames, `toTransformStream()` for WHATWG bridge. Depends on `@suaveplan/crypto` streaming AEAD engines.
|
||||
- ADDED: `SplitJoinTransform` — object-mode split with `chunkSize` + `overlap`; `join` reassembles with optional `separator` and `removeOverlap`; supports string / `ArrayBuffer` / array; type guards `isSplittable`, `isJoinable`
|
||||
|
||||
Existing 11 functional transforms migrated to `BaseTransform` subclasses (see § Renames):
|
||||
|
||||
- ADDED: `MapTransform`, `FilterTransform`, `FlatMapTransform`, `BatchTransform`, `WindowTransform`, `ThrottleTransform`, `LimitTransform`, `DedupTransform`, `TapTransform`, `TextCodecTransform` (encode/decode operations), `CompressTransform` (compress/decompress operations)
|
||||
|
||||
Genesis-original gap-fillers (new):
|
||||
|
||||
- ADDED: `ChunkBytesTransform` — byte-level fixed-frame re-framing with tail policy (`emit` | `pad` | `drop`)
|
||||
- ADDED: `CoalesceBytesTransform` — merges small `Uint8Array` chunks until a byte threshold or max-latency timeout
|
||||
- ADDED: `SlidingBytesTransform` — overlapping byte windows with size + stride
|
||||
- ADDED: `LengthPrefixEncodeTransform`, `LengthPrefixDecodeTransform` — wire-protocol framing with 1/2/4/8-byte prefix and configurable byte-order
|
||||
|
||||
### ADDED — pipeline ops (genesis-original)
|
||||
|
||||
- ADDED: `partition<T>(source, predicate)` → `{ truthy, falsy }` single-pass split (no tee duplication)
|
||||
- ADDED: `route<T, K extends string>(source, keyFn, keys, options?)` → `Record<K, ReadableStream<T>>`; `onUnknownKey: 'drop' | 'error'`
|
||||
- ADDED: `routeBy<T, K>(source, keyFn, keys, options?)` → `Map<K, ReadableStream<T>>` (non-string-key variant)
|
||||
- ADDED: `zip<T extends readonly unknown[]>(streams)` → tuple stream with `onShortInput: 'close' | 'error' | 'pad'` option
|
||||
- ADDED: `concat<T>(streams)` → sequential drain
|
||||
- ADDED: `mergeOrdered<T>(streams, compare)` → k-way merge of pre-sorted streams
|
||||
- ADDED: `safeBranch<TIn, TOut>(factory, options?)` → wraps a sub-pipeline so its errors emit on `EventBusLike` instead of propagating
|
||||
|
||||
### ADDED — sources, sinks, utilities
|
||||
|
||||
- ADDED: `toAsyncIterable<T>(stream, signal?)` sink
|
||||
- ADDED: `getDataFromFile(filePath)` — Node-only file reader with runtime environment detection (throws in browser)
|
||||
- ADDED: `bufferToUint8Array(buffer)` — handles both Node `Buffer` and `ArrayBuffer`
|
||||
- ADDED: `createUint8ArraySlice(array, start, end)` — explicit-typed copy helper
|
||||
- ADDED: `createDataStream<T>(data, transformer, options?)` — generic source factory used by `StreamManager` for non-stream inputs
|
||||
- ADDED: `processDataStream<T>(stream, options?)` — generic sink factory
|
||||
- ADDED: `streamToUint8Array(stream)`, `uint8ArrayToStream(data, options?)`, `bufferToStream`, `streamToBuffer` cross-runtime helpers
|
||||
- ADDED: `validateStreamIntegrity(stream)` — chunk-validity sweep helper
|
||||
|
||||
### MODIFIED — naming convention
|
||||
|
||||
Drop redundant `-stream` suffix on every module name inside the streams package. Implementation file basename matches the parent folder; class and function names follow PascalCase / camelCase conventions without the redundant suffix.
|
||||
|
||||
| Old path | New path | Rationale |
|
||||
|---|---|---|
|
||||
| `transforms/map-stream/map-stream.ts` | `transforms/map/map.ts` | inside a streams package, "stream" suffix is redundant |
|
||||
| `transforms/filter-stream/filter-stream.ts` | `transforms/filter/filter.ts` | same |
|
||||
| `transforms/flat-map-stream/` | `transforms/flat-map/` | same |
|
||||
| `transforms/batch-stream/` | `transforms/batch/` | same |
|
||||
| `transforms/window-stream/` | `transforms/window/` | same |
|
||||
| `transforms/throttle-stream/` | `transforms/throttle/` | same |
|
||||
| `transforms/limit-stream/` | `transforms/limit/` | same |
|
||||
| `transforms/dedup-stream/` | `transforms/dedup/` | same |
|
||||
| `transforms/tap-stream/` | `transforms/tap/` | same |
|
||||
| `transforms/text-codec/` | unchanged | already conforms |
|
||||
| `transforms/compress/` | unchanged | already conforms |
|
||||
| `pipeline/pipeline-builder/` | `pipeline/builder/` | redundant package-name prefix |
|
||||
| `instrument/instrument-transform/` | `instrument/transform/` | redundant prefix |
|
||||
| `node/node-adapters.ts` | `node/adapters/adapters.ts` | promoted to feature dir + redundant prefix dropped |
|
||||
| `errors/stream-error.ts` | `errors/error/error.ts` | redundant prefix dropped, promoted to feature dir |
|
||||
|
||||
The corresponding exported symbols are renamed in lockstep:
|
||||
|
||||
| Old export | New export |
|
||||
|---|---|
|
||||
| `mapStream` | `mapStream` (kept — describes the *function* shape, not file naming) plus new `MapTransform` class |
|
||||
| `filterStream` | `filterStream` plus `FilterTransform` |
|
||||
| `flatMapStream` | `flatMapStream` plus `FlatMapTransform` |
|
||||
| `batchStream` | `batchStream` plus `BatchTransform` |
|
||||
| `windowStream` | `windowStream` plus `WindowTransform` |
|
||||
| `throttleStream` | `throttleStream` plus `ThrottleTransform` |
|
||||
| `limitStream` | `limitStream` plus `LimitTransform` |
|
||||
| `dedupStream` | `dedupStream` plus `DedupTransform` |
|
||||
| `tapStream` | `tapStream` plus `TapTransform` |
|
||||
| `compressStream`, `decompressStream` | unchanged plus `CompressTransform` (operation = "compress" \| "decompress") |
|
||||
| `encodeTextStream`, `decodeTextStream` | unchanged plus `TextCodecTransform` |
|
||||
| `pipeline()` | unchanged |
|
||||
| `fanOut`, `merge` | unchanged |
|
||||
| `instrumentTransform` | unchanged |
|
||||
| `fromNodeReadable`, `toNodeReadable`, `fromNodeWritable` | unchanged |
|
||||
| `StreamPipeFailedError` | renamed to `StreamPipeFailedError` (lives in `errors/error/error.ts` now) |
|
||||
|
||||
The function-form factories (`mapStream`, `filterStream`, …) remain exported for tree-shaking and for direct WHATWG composition; the OO subclasses delegate to them via `toTransformStream()`. **Both call paths are first-class.** This is the only "two APIs" pattern this package permits, and it is justified because the OO subclass *uses* the functional factory under the hood — no parallel implementation, no drift surface.
|
||||
|
||||
### ADDED — error codes
|
||||
|
||||
- ADDED: `STREAM.FRAME_TOO_LARGE`, `STREAM.TRUNCATED_FRAME`, `STREAM.ROUTE_UNKNOWN_KEY`, `STREAM.ZIP_LENGTH_MISMATCH`, `STREAM.BRANCH_FAILED`, `STREAM.CRYPTO_DECRYPT_FAILED`, `STREAM.CRYPTO_INVALID_FRAME`, `STREAM.DUPLEX_CLOSED`, `STREAM.NETWORK_RESET`, `STREAM.FILE_READ_FAILED`, `STREAM.FILE_BROWSER_NOT_SUPPORTED`
|
||||
|
||||
### REMOVED
|
||||
|
||||
- REMOVED: nothing — the previous spec is preserved verbatim under `MODIFIED Requirements`. Symbols are renamed (with old paths gone) but exported names are kept on the function-form API.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Audio-specific anything** (sample frames, channel-aware ops, mixing, pitch-shift, dynamics-processor) — belongs in the upcoming `audio-engine` package as plugins, not in streams. Karaoke/radio transforms from main do **not** port over.
|
||||
- **Image / video / metadata transforms** — already split into siblings (`streams-metadata` proposed; image/video to follow as separate changes).
|
||||
- **Worker offload** (`pipeInWorker`) — deferred to a future `streams-worker` sibling package once audio-engine demands prove the transport shape.
|
||||
- **`schema-registry` integration** of stream chunk contracts — deferred until `@suaveplan/schema-registry` ships the consumer hooks the streams chunk format would need.
|
||||
- **Mutation testing & contract testing layers** — covered by separate `enterprise-testing-mutation` and `enterprise-testing-contract` changes; this change ships 100% line/branch/function coverage as the baseline.
|
||||
|
||||
## Source mapping
|
||||
|
||||
This change derives from `../suaveplan/enterprise/streams.md` (the generic enterprise-quality template applied to streams) and the canonical implementation at `../suaveplan/packages/universal/streams/src/` (the ground-truth API surface).
|
||||
|
||||
| Requirement | Source | Notes |
|
||||
|---|---|---|
|
||||
| Pure TypeScript, runs in Bun + Node + browser | `enterprise/streams.md` L38 | verbatim |
|
||||
| WHATWG Streams baseline | `enterprise/streams.md` L40 | verbatim |
|
||||
| Zero-copy `Uint8Array` operations on hot paths | `enterprise/streams.md` L41 | verbatim |
|
||||
| Backpressure-first; bounded queues with declarative high/low watermarks | `enterprise/streams.md` L42 | verbatim |
|
||||
| `AbortSignal` propagates end-to-end | `enterprise/streams.md` L43 | verbatim |
|
||||
| Zod schemas for stream configuration and chunk contracts | `enterprise/streams.md` L44 | partial — schema-registry integration deferred (see Out of scope) |
|
||||
| Property-based tests via fast-check | `enterprise/streams.md` L45 | verbatim |
|
||||
| Deterministic — no `Math.random()` / `Date.now()` at module scope, injectable clock + RNG | `enterprise/streams.md` L46 | verbatim |
|
||||
| OpenTelemetry instrumentation | `enterprise/streams.md` L47 | verbatim |
|
||||
| Structured errors extending `@suaveplan/error` with stable codes | `enterprise/streams.md` L48 | verbatim |
|
||||
| Idempotency + retry primitives for network-backed adapters | `enterprise/streams.md` L49 | applied to `NetworkStreamImpl`'s reset semantics |
|
||||
| Feature flags for risky transform rollouts | `enterprise/streams.md` L50 | applied via `FlagRegistry` injection on factory; opt-in |
|
||||
| Plugin / extension interface | `enterprise/streams.md` L74 | satisfied by `BaseTransform` + `TransformFactory` |
|
||||
| Observability surface — counters, histograms, spans on every externally-visible op | `enterprise/streams.md` L75 | satisfied by `instrumentTransform` + per-class hook surface |
|
||||
| Configurability — Zod-validated options with sensible defaults | `enterprise/streams.md` L76 | verbatim |
|
||||
| Performance — benchmark suite + CI regression gate | `enterprise/streams.md` L77 | verbatim |
|
||||
| Resilience — retries / timeouts / cancellation / circuit breakers | `enterprise/streams.md` L78 | applied via `@suaveplan/retry` composition (caller-side) |
|
||||
| Security — input validation, output encoding, zeroing of sensitive buffers | `enterprise/streams.md` L79 | applied to `CryptoTransform` key zeroize on dispose |
|
||||
| Migration story for public API changes | `enterprise/streams.md` L80 | satisfied by the rename matrix above |
|
||||
| Example app / story demonstrating full feature set | `enterprise/streams.md` L82 | satisfied by `examples/` runnable scripts |
|
||||
|
||||
### Requirements with no `enterprise/streams.md` line
|
||||
|
||||
These derive from `../suaveplan/packages/universal/streams/src/` (canonical implementation on main) and from the `enterprise-streams-redesign` audit conducted in the brainstorming session.
|
||||
|
||||
| Requirement | Source | Justification |
|
||||
|---|---|---|
|
||||
| `BaseTransform` abstract class with hook + event surface | `../suaveplan/packages/universal/streams/src/transforms/base/base.ts` + `base.md` | canonical plugin shape — required for hooks/events that no functional API can express |
|
||||
| `StreamManager` chainable pipeline | `../suaveplan/packages/universal/streams/src/manager/manager.ts` + `manager.md` | canonical |
|
||||
| `TransformFactory` registry | `../suaveplan/packages/universal/streams/src/transforms/factory/factory.ts` + `factory.md` | canonical |
|
||||
| `BaseDuplexStream` + `NetworkStreamImpl` | `../suaveplan/packages/universal/streams/src/primitives/duplex.ts` + `duplex.md` | canonical |
|
||||
| `CryptoTransform` with pluggable streaming AEAD | `../suaveplan/packages/universal/streams/src/transforms/crypto/crypto.ts` + `crypto.md` | canonical; depends on `@suaveplan/crypto` streaming AEAD engines |
|
||||
| `SplitJoinTransform` | `../suaveplan/packages/universal/streams/src/transforms/split-join/split-join.ts` + `split-join.md` | canonical |
|
||||
| Buffer utilities (`getDataFromFile`, `bufferToUint8Array`, `createUint8ArraySlice`) | `../suaveplan/packages/universal/streams/src/utils/buffer/buffer.md` | canonical |
|
||||
| Stream utilities (`createDataStream`, `processDataStream`, `streamToUint8Array`, `uint8ArrayToStream`, `validateStreamIntegrity`) | `../suaveplan/packages/universal/streams/src/utils/stream/stream.md` | canonical |
|
||||
| `chunkBytes`, `coalesceBytes`, `slidingBytes`, `lengthPrefixEncode/Decode` | genesis audit (this change) | byte-level framing primitives; AEAD requires fixed-byte frames; main's `SplitJoinTransform` is object-mode only |
|
||||
| `partition`, `route`, `routeBy` | genesis audit (this change) | single-pass split (`fanOut` duplicates via `tee`); essential for "split into different plugin chains by criterion" |
|
||||
| `zip`, `concat`, `mergeOrdered` | genesis audit (this change) | merge variants beyond arrival-order race; `zip` enables sample-aligned recombination, `mergeOrdered` enables k-way time-sorted merge |
|
||||
| `safeBranch` | genesis audit (this change) | branch error isolation — without it, one failing FX rack tears down the whole graph |
|
||||
| `toAsyncIterable` sink | genesis audit (this change) | async iterator interop, missing from main |
|
||||
| Naming convention: drop redundant `-stream` suffix | user direction (brainstorming session) | clarity inside a streams package |
|
||||
| 100% test coverage threshold | genesis-wide policy (`openspec/AGENTS.md` § 7) | non-package-specific |
|
||||
| Source-mapped `**Verified by:**` lines | genesis-wide policy (`openspec/AGENTS.md` § 13) | non-package-specific |
|
||||
|
||||
## Verification approach
|
||||
|
||||
Every requirement in the spec delta carries a `**Verified by:**` line pointing at a concrete `it(...)` title in one of the test files below. Tests are co-located with implementation per the genesis 5-file convention.
|
||||
|
||||
- `src/base/transform/transform.test.ts` — `BaseTransform` lifecycle, hook + event surfaces, `toTransformStream()` bridge, `dispose()` semantics
|
||||
- `src/base/transform/transform.spec-conformance.test.ts` — type-level: generic narrowing, hook-payload type inference, factory return-type preservation
|
||||
- `src/manager/manager/manager.test.ts` — pipeline chaining, sequential execution, `clear()` reset, error propagation
|
||||
- `src/factory/factory/factory.test.ts` — type-discriminated `create()`, registry lookup, unknown-type error
|
||||
- `src/primitives/duplex/duplex.test.ts` — bidirectional flow, byte vs object mode, `cancel()` semantics
|
||||
- `src/primitives/network/network.test.ts` — id, protocol, metadata, `close()`, `reset()` with error
|
||||
- `src/transforms/<name>/<name>.test.ts` — per-transform behaviour (one file per BaseTransform subclass)
|
||||
- `src/transforms/crypto/crypto.test.ts` + `crypto.spec-conformance.test.ts` — round-trip encrypt/decrypt, hook payloads, event firing order, reorder/truncate detection
|
||||
- `src/pipeline/<op>/<op>.test.ts` — per-op behaviour
|
||||
- `src/sinks/to-async-iterable/to-async-iterable.test.ts` — async-iterator yields all chunks; `signal` cancels
|
||||
- `src/utils/buffer/<helper>/<helper>.test.ts` — Node + browser branches; env-detection error
|
||||
- `src/utils/stream/<helper>/<helper>.test.ts` — round-trip identity, integrity validation
|
||||
- `src/instrument/transform/transform.test.ts` — span + counter + histogram emitted via in-memory OTel `MeterProvider`
|
||||
- `src/node/adapters/adapters.test.ts` — Node ↔ WHATWG bridge round-trip
|
||||
- `tests/integration.test.ts` — full pipelines: encrypt → length-prefix → decrypt round-trip; partition + safe-branch + merge composition
|
||||
- `tests/property.test.ts` — `chunkBytes` byte-conservation, `partition` order preservation, `zip`/`unzip` round-trip, `mergeOrdered` sortedness invariant
|
||||
- `tests/dep-graph.test.ts` — only declared workspace deps reachable; no `node:fs` import in browser entry
|
||||
- `tests/tree-shake.test.ts` — `sideEffects: false` round-trip via `Bun.build`
|
||||
|
||||
## Migration impact
|
||||
|
||||
**This package has consumers on genesis already** — every package that imports from `@suaveplan/streams`. Audit:
|
||||
|
||||
```bash
|
||||
grep -rln "@suaveplan/streams" packages/ --include="package.json"
|
||||
```
|
||||
|
||||
The function-form API (`mapStream`, `filterStream`, `pipeline`, `fanOut`, `merge`, `instrumentTransform`, `fromNodeReadable`, `toNodeReadable`, `fromNodeWritable`, `compressStream`, `decompressStream`, `encodeTextStream`, `decodeTextStream`, `toArray`, `toBytes`, `forEach`, `fromArray`, `fromBytes`, `fromAsyncGenerator`) is preserved. Internal subpath imports change:
|
||||
|
||||
| Old subpath import | New subpath import |
|
||||
|---|---|
|
||||
| `@suaveplan/streams/transforms/map-stream/map-stream` | `@suaveplan/streams/transforms/map/map` |
|
||||
| `@suaveplan/streams/transforms/filter-stream/filter-stream` | `@suaveplan/streams/transforms/filter/filter` |
|
||||
| (… same pattern for the renamed transforms above) | |
|
||||
| `@suaveplan/streams/pipeline/pipeline-builder/pipeline-builder` | `@suaveplan/streams/pipeline/builder/builder` |
|
||||
| `@suaveplan/streams/instrument/instrument-transform/instrument-transform` | `@suaveplan/streams/instrument/transform/transform` |
|
||||
| `@suaveplan/streams/node/node-adapters` | `@suaveplan/streams/node/adapters/adapters` |
|
||||
| `@suaveplan/streams/errors/stream-error` | `@suaveplan/streams/errors/error/error` |
|
||||
|
||||
**Consumer migration is part of this change.** Every internal consumer is updated in the same change folder; no shim or deprecation alias remains. The barrel `@suaveplan/streams` (root export) is unaffected — top-level imports keep working.
|
||||
|
||||
**Breaking changes:** subpath imports only (top-level barrel preserved). New OO subclasses are pure additions.
|
||||
|
||||
**Prerequisites:** `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/events`, `@suaveplan/crypto` (for `CryptoTransform`) must be at their current genesis versions on Verdaccio.
|
||||
@@ -0,0 +1,624 @@
|
||||
# Capability Spec Delta: streams (enterprise-streams-redesign)
|
||||
|
||||
**Package**: `@suaveplan/streams`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement carries a `**Verified by:**` line at authoring time. (See `openspec/AGENTS.md` Section 13.)
|
||||
|
||||
This delta is authored against an empty folded spec (`openspec/specs/streams/spec.md` does not exist on genesis — the prior `enterprise-streams` change archived without folding). All requirements below land as **ADDED** on archive.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: BaseTransform abstract class
|
||||
|
||||
The package SHALL export an abstract `BaseTransform` class providing the canonical plugin contract for stream transforms. Subclasses implement `process<T>(input: T, options?: TransformOptions): Promise<T>`. Every subclass inherits a uniform hook surface, event surface, parameter introspection, and a `toTransformStream<TIn, TOut>(): TransformStream<TIn, TOut>` bridge to WHATWG Streams.
|
||||
|
||||
#### Scenario: Subclass implements process and is invokable
|
||||
|
||||
- **WHEN** a consumer defines `class JsonTransform extends BaseTransform` with `name = "json"` and a concrete `process(input)` that JSON-stringifies the input
|
||||
- **THEN** `new JsonTransform().process({ a: 1 })` returns `'{"a":1}'`
|
||||
|
||||
#### Scenario: Hooks fire in registration order around process
|
||||
|
||||
- **WHEN** `t.hook("before", payloadA => payloadA)` and `t.hook("before", payloadB => payloadB)` are registered, then `t.process(...)` runs
|
||||
- **THEN** the first hook receives the original payload and its return value is passed to the second hook before the wrapped operation runs
|
||||
|
||||
#### Scenario: Events fire without affecting return value
|
||||
|
||||
- **WHEN** a consumer registers `t.on("complete", payload => sideEffect(payload))` and `process` runs to completion
|
||||
- **THEN** the side effect is observed and `process` still resolves with the operation's actual return value
|
||||
|
||||
#### Scenario: toTransformStream returns a usable WHATWG TransformStream
|
||||
|
||||
- **WHEN** a consumer calls `t.toTransformStream()` and pipes a `ReadableStream` through it
|
||||
- **THEN** every chunk is processed by `t.process(chunk)` and the result is enqueued to the readable side
|
||||
|
||||
#### Scenario: dispose releases resources idempotently
|
||||
|
||||
- **WHEN** `t.dispose()` is called twice in succession
|
||||
- **THEN** internal hooks and event listeners are cleared on the first call; the second call is a no-op and does not throw
|
||||
|
||||
**Verified by:** `src/base/transform/transform.test.ts::"subclass process returns transformed value"` AND `src/base/transform/transform.test.ts::"hooks fire in registration order with payload chaining"` AND `src/base/transform/transform.test.ts::"events fire without affecting return value"` AND `src/base/transform/transform.test.ts::"toTransformStream pipes through process"` AND `src/base/transform/transform.test.ts::"dispose is idempotent and clears hooks and listeners"` AND `src/base/transform/transform.spec-conformance.test.ts::"hook payload type is inferred from event name literal"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: TransformOptions parameter and Zod schema
|
||||
|
||||
The package SHALL accept `TransformOptions` on every `process()` call: `{ signal?: AbortSignal; clock?: Clock; logger?: LoggerLike }`. The schema is exported as `TransformOptionsSchema` from a sibling `*.schemas.ts`.
|
||||
|
||||
#### Scenario: Aborted signal rejects mid-process
|
||||
|
||||
- **WHEN** `t.process(input, { signal })` runs and `signal.abort(reason)` fires before completion
|
||||
- **THEN** the returned promise rejects with `StreamError("STREAM.ABORTED")` whose `cause` is the abort reason
|
||||
|
||||
#### Scenario: Schema rejects unknown options at the boundary
|
||||
|
||||
- **WHEN** `TransformOptionsSchema.parse({ unknownKey: 1 })` is invoked
|
||||
- **THEN** Zod throws with a `ZodError` listing `unknownKey` (strict object schema)
|
||||
|
||||
**Verified by:** `src/base/transform/transform.test.ts::"aborted signal rejects process with STREAM.ABORTED and propagates cause"` AND `src/base/transform/transform.schemas.test.ts::"TransformOptionsSchema rejects unknown keys"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: StreamManager chainable pipeline
|
||||
|
||||
The package SHALL export `StreamManager` with `pipe(...transforms: BaseTransform[]): StreamManager`, `execute<T>(input: T, options?: ExecuteOptions): Promise<T>`, `clear(): StreamManager`, and `getTransforms(): readonly BaseTransform[]`.
|
||||
|
||||
#### Scenario: Sequential execution feeds output into next transform
|
||||
|
||||
- **WHEN** `manager.pipe(a, b).execute(input)` runs
|
||||
- **THEN** `a.process(input)` resolves to `x`, then `b.process(x)` resolves to the final value
|
||||
|
||||
#### Scenario: Empty pipeline returns input unchanged
|
||||
|
||||
- **WHEN** `new StreamManager().execute(input)` runs without any pipe
|
||||
- **THEN** the result is exactly `input` (referential equality)
|
||||
|
||||
#### Scenario: Error in middle transform aborts downstream
|
||||
|
||||
- **WHEN** transform `b` throws and pipeline is `pipe(a, b, c)`
|
||||
- **THEN** `c.process` is never invoked; the error from `b` propagates to the caller wrapped in `StreamError("STREAM.TRANSFORM_FAILED")`
|
||||
|
||||
#### Scenario: clear() empties the pipeline
|
||||
|
||||
- **WHEN** `manager.pipe(a).clear().getTransforms()` is called
|
||||
- **THEN** the returned array has length 0
|
||||
|
||||
**Verified by:** `src/manager/manager/manager.test.ts::"pipe chains transforms in order"` AND `src/manager/manager/manager.test.ts::"empty pipeline returns input unchanged"` AND `src/manager/manager/manager.test.ts::"error in middle transform aborts downstream and wraps in STREAM.TRANSFORM_FAILED"` AND `src/manager/manager/manager.test.ts::"clear() empties pipeline"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: TransformFactory registry
|
||||
|
||||
The package SHALL export `TransformFactory` with static `create(type, ...args)` and instance `create(type, ...args)`, where `type` is a literal string keyed against a registry of bundled transforms. Unknown types raise `StreamError("STREAM.UNKNOWN_TRANSFORM_TYPE")`. The registry is open: `TransformFactory.register(type, factory)` adds new types at runtime.
|
||||
|
||||
#### Scenario: Built-in type resolves to correct subclass instance
|
||||
|
||||
- **WHEN** `TransformFactory.create("crypto", { engine, key, operation: "encrypt" })` is called
|
||||
- **THEN** the returned instance is `instanceof CryptoTransform` with the supplied options
|
||||
|
||||
#### Scenario: Unknown type throws structured error
|
||||
|
||||
- **WHEN** `TransformFactory.create("nonexistent")` is called
|
||||
- **THEN** the call throws `StreamError` with code `"STREAM.UNKNOWN_TRANSFORM_TYPE"` and a message naming `"nonexistent"`
|
||||
|
||||
#### Scenario: Custom registration is honoured
|
||||
|
||||
- **WHEN** a consumer calls `TransformFactory.register("upper", () => new UpperCaseTransform())` then `create("upper")`
|
||||
- **THEN** the returned instance is the registered subclass
|
||||
|
||||
**Verified by:** `src/factory/factory/factory.test.ts::"built-in type resolves to subclass instance"` AND `src/factory/factory/factory.test.ts::"unknown type throws STREAM.UNKNOWN_TRANSFORM_TYPE"` AND `src/factory/factory/factory.test.ts::"custom registration is honoured"` AND `src/factory/factory/factory.spec-conformance.test.ts::"create() narrows return type by literal type argument"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: BaseDuplexStream primitive
|
||||
|
||||
The package SHALL export `BaseDuplexStream<R, W>` with constructor `(source: StreamSource<R>, sink: StreamSink<W>, options?: DuplexStreamOptions)` exposing `readable: ReadableStream<R>` and `writable: WritableStream<W>`. Supports byte-mode and object-mode via `options.objectMode`. Honours `highWaterMark`.
|
||||
|
||||
#### Scenario: Bidirectional flow
|
||||
|
||||
- **WHEN** a consumer writes a chunk to `duplex.writable` and reads from `duplex.readable` whose `pull` is wired to enqueue an echo
|
||||
- **THEN** the echoed chunk is read back
|
||||
|
||||
#### Scenario: cancel propagates to source
|
||||
|
||||
- **WHEN** `duplex.readable.cancel(reason)` is called
|
||||
- **THEN** the source's `cancel(reason)` is invoked exactly once with the same reason
|
||||
|
||||
#### Scenario: Object-mode bypasses byteLength accounting
|
||||
|
||||
- **WHEN** `objectMode: true` and 1000 non-byte objects are written
|
||||
- **THEN** the writable accepts them subject to `highWaterMark` count, not byte size
|
||||
|
||||
**Verified by:** `src/primitives/duplex/duplex.test.ts::"bidirectional flow round-trips a chunk"` AND `src/primitives/duplex/duplex.test.ts::"cancel propagates reason to source.cancel"` AND `src/primitives/duplex/duplex.test.ts::"object-mode honours count-based highWaterMark"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: NetworkStreamImpl with metadata
|
||||
|
||||
The package SHALL export `NetworkStreamImpl` extending the duplex pattern with `id: string`, optional `protocol?: string`, `metadata: Map<string, unknown>`, `closed: boolean`, `close(): Promise<void>`, and `reset(): Promise<void>`. `close()` cleanly closes both sides; `reset()` aborts both sides with `StreamError("STREAM.NETWORK_RESET")`.
|
||||
|
||||
#### Scenario: Metadata is accessible after construction
|
||||
|
||||
- **WHEN** `createNetworkStream("peer-1", "webrtc", new Map([["userId", "u-42"]]))` is called
|
||||
- **THEN** `stream.id === "peer-1"`, `stream.protocol === "webrtc"`, and `stream.metadata.get("userId") === "u-42"`
|
||||
|
||||
#### Scenario: close() flips closed flag and prevents further writes
|
||||
|
||||
- **WHEN** `await stream.close()` runs and a subsequent writer attempts to write
|
||||
- **THEN** `stream.closed === true` and the write rejects
|
||||
|
||||
#### Scenario: reset() aborts with STREAM.NETWORK_RESET
|
||||
|
||||
- **WHEN** `await stream.reset()` runs while a reader is pending
|
||||
- **THEN** the reader rejects with `StreamError("STREAM.NETWORK_RESET")`
|
||||
|
||||
**Verified by:** `src/primitives/network/network.test.ts::"metadata id and protocol accessible after construction"` AND `src/primitives/network/network.test.ts::"close() flips closed flag and rejects subsequent writes"` AND `src/primitives/network/network.test.ts::"reset() aborts pending reader with STREAM.NETWORK_RESET"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: CryptoTransform with pluggable streaming AEAD
|
||||
|
||||
The package SHALL export `CryptoTransform` taking `{ engine: StreamingAeadEngine, key: Uint8Array, operation: "encrypt" | "decrypt" }` and exposing the `BaseTransform` plugin surface plus `crypto:before-chunk` and `crypto:after-chunk` hooks and `crypto:start` / `crypto:chunk` / `crypto:complete` / `crypto:error` events. The transform binds an 8-byte counter and 1-byte final-flag into the AEAD AAD, so reorder, truncation, duplication, and tamper all surface as decryption failure.
|
||||
|
||||
#### Scenario: Encrypt-decrypt round-trip yields identity
|
||||
|
||||
- **WHEN** the same key is used to construct an `encrypt` and a `decrypt` transform, and a sequence of plaintext frames pipes through both via `toTransformStream()`
|
||||
- **THEN** the output equals the input frame-by-frame
|
||||
|
||||
#### Scenario: Truncation is detected
|
||||
|
||||
- **WHEN** an encrypted stream's last frame is dropped before decryption
|
||||
- **THEN** the `decrypt` transform rejects with `StreamError("STREAM.CRYPTO_DECRYPT_FAILED")` referencing AAD verification failure
|
||||
|
||||
#### Scenario: Reorder is detected
|
||||
|
||||
- **WHEN** two encrypted middle frames are swapped before decryption
|
||||
- **THEN** the `decrypt` transform rejects with `StreamError("STREAM.CRYPTO_DECRYPT_FAILED")`
|
||||
|
||||
#### Scenario: Hooks observe and may modify chunks
|
||||
|
||||
- **WHEN** `t.hook("crypto:before-chunk", p => ({ ...p, chunk: redact(p.chunk) }))` is registered and a frame passes through
|
||||
- **THEN** the redacted chunk is what gets sealed
|
||||
|
||||
#### Scenario: Events fire in canonical order
|
||||
|
||||
- **WHEN** an `encrypt` transform processes 3 frames to completion
|
||||
- **THEN** events fire as `crypto:start`, `crypto:chunk` ×3, `crypto:complete` in that order
|
||||
|
||||
#### Scenario: dispose zeroizes key buffer
|
||||
|
||||
- **WHEN** `t.dispose()` is called on an encrypt transform
|
||||
- **THEN** the internal copy of the key buffer is overwritten with zeros (via `@suaveplan/crypto-utils.zeroize`)
|
||||
|
||||
**Verified by:** `src/transforms/crypto/crypto.test.ts::"encrypt-decrypt round-trip preserves frame sequence"` AND `src/transforms/crypto/crypto.test.ts::"truncation produces STREAM.CRYPTO_DECRYPT_FAILED"` AND `src/transforms/crypto/crypto.test.ts::"reorder produces STREAM.CRYPTO_DECRYPT_FAILED"` AND `src/transforms/crypto/crypto.test.ts::"crypto:before-chunk hook can replace plaintext before sealing"` AND `src/transforms/crypto/crypto.test.ts::"events fire in canonical order start/chunk*N/complete"` AND `src/transforms/crypto/crypto.spec-conformance.test.ts::"dispose() zeroizes key buffer"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: SplitJoinTransform object-mode chunking
|
||||
|
||||
The package SHALL export `SplitJoinTransform` with operations `"split"` and `"join"`. Split takes `{ chunkSize, overlap }` and accepts `string`, `ArrayBuffer`, or `T[]`. Join accepts `{ chunks, metadata? }` shape and reconstructs the original. Type guards `isSplittable(value)` and `isJoinable(value)` are exported.
|
||||
|
||||
#### Scenario: String split with overlap reproduces every character
|
||||
|
||||
- **WHEN** a 100-char string is split with `chunkSize: 30, overlap: 5` and joined back with `removeOverlap: true`
|
||||
- **THEN** the joined string equals the original
|
||||
|
||||
#### Scenario: ArrayBuffer split preserves byte-for-byte content
|
||||
|
||||
- **WHEN** a 1024-byte `ArrayBuffer` is split with `chunkSize: 256, overlap: 0` and joined back
|
||||
- **THEN** the joined buffer equals the original byte-for-byte
|
||||
|
||||
#### Scenario: Type guard rejects non-splittable input
|
||||
|
||||
- **WHEN** `isSplittable({ foo: 1 })` is called
|
||||
- **THEN** the result is `false`
|
||||
|
||||
**Verified by:** `src/transforms/split-join/split-join.test.ts::"string split with overlap round-trips via removeOverlap"` AND `src/transforms/split-join/split-join.test.ts::"ArrayBuffer split round-trips byte-for-byte"` AND `src/transforms/split-join/split-join.test.ts::"isSplittable rejects plain objects"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Functional transform factories preserved
|
||||
|
||||
The package SHALL preserve the function-form factories from the prior `enterprise-streams` archive: `mapStream`, `filterStream`, `flatMapStream`, `batchStream`, `windowStream`, `throttleStream`, `limitStream`, `dedupStream`, `tapStream`, `encodeTextStream`, `decodeTextStream`, `compressStream`, `decompressStream`, `fromArray`, `fromBytes`, `fromAsyncGenerator`, `toArray`, `toBytes`, `forEach`, `pipeline`, `fanOut`, `merge`, `instrumentTransform`, `fromNodeReadable`, `toNodeReadable`, `fromNodeWritable`. The corresponding `BaseTransform` subclasses delegate to these factories via `toTransformStream()` — no parallel implementation.
|
||||
|
||||
#### Scenario: MapTransform.toTransformStream() and mapStream(fn) are observably equivalent
|
||||
|
||||
- **WHEN** the same chunk is piped through `mapStream(fn)` directly and through `new MapTransform({ fn }).toTransformStream()`
|
||||
- **THEN** the output chunks are equal
|
||||
|
||||
#### Scenario: Functional factory is tree-shakeable
|
||||
|
||||
- **WHEN** a consumer imports only `mapStream` from `@suaveplan/streams/transforms/map/map`
|
||||
- **THEN** the resulting bundle does not include `BaseTransform` or any unrelated transform
|
||||
|
||||
**Verified by:** `src/transforms/map/map.test.ts::"MapTransform.toTransformStream and mapStream(fn) produce equivalent output"` AND `tests/tree-shake.test.ts::"importing only mapStream excludes BaseTransform"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: ChunkBytes byte-level re-framing
|
||||
|
||||
The package SHALL export `chunkBytes(frameSize: number, options?: { tail?: 'emit' | 'pad' | 'drop' }): TransformStream<Uint8Array, Uint8Array>` and `ChunkBytesTransform` class. Re-emits any input byte stream as a stream of frames of exactly `frameSize` bytes; the final partial frame is handled per the `tail` policy (default: `'emit'`).
|
||||
|
||||
#### Scenario: All-equal-size frames concatenate to original input
|
||||
|
||||
- **WHEN** an input stream of total length `n × frameSize` bytes (chunked arbitrarily) is piped through `chunkBytes(frameSize)`
|
||||
- **THEN** every output frame has length `frameSize` and concatenated output equals input byte-for-byte
|
||||
|
||||
#### Scenario: Tail policy 'emit' yields a final partial frame
|
||||
|
||||
- **WHEN** an input of `frameSize × 2 + 7` bytes is piped through `chunkBytes(frameSize, { tail: 'emit' })`
|
||||
- **THEN** the third emitted frame has length 7 and concatenated output equals input
|
||||
|
||||
#### Scenario: Tail policy 'pad' zero-pads the final frame
|
||||
|
||||
- **WHEN** the same input is piped through `chunkBytes(frameSize, { tail: 'pad' })`
|
||||
- **THEN** the third frame has length `frameSize` and bytes `frameSize - 7..frameSize - 1` are 0
|
||||
|
||||
#### Scenario: Tail policy 'drop' discards incomplete final frame
|
||||
|
||||
- **WHEN** the same input is piped through `chunkBytes(frameSize, { tail: 'drop' })`
|
||||
- **THEN** only two frames are emitted
|
||||
|
||||
**Verified by:** `src/transforms/chunk-bytes/chunk-bytes.test.ts::"all frames are exactly frameSize bytes"` AND `src/transforms/chunk-bytes/chunk-bytes.test.ts::"tail emit yields final partial frame and round-trips"` AND `src/transforms/chunk-bytes/chunk-bytes.test.ts::"tail pad zero-pads final frame to frameSize"` AND `src/transforms/chunk-bytes/chunk-bytes.test.ts::"tail drop discards incomplete final frame"` AND `tests/property.test.ts::"chunkBytes byte-conservation under arbitrary input chunking"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: CoalesceBytes batching with latency cap
|
||||
|
||||
The package SHALL export `coalesceBytes(minBytes: number, options?: { maxLatencyMs?: number; clock?: Clock }): TransformStream<Uint8Array, Uint8Array>`. Accumulates incoming chunks until the accumulator reaches `minBytes` or `maxLatencyMs` has elapsed since the first buffered chunk, then emits a single coalesced frame.
|
||||
|
||||
#### Scenario: Below-threshold inputs are buffered and emitted on close
|
||||
|
||||
- **WHEN** three 100-byte chunks pipe in (total 300 bytes) with `coalesceBytes(1024)` and the input then closes
|
||||
- **THEN** a single 300-byte frame is emitted before the output closes
|
||||
|
||||
#### Scenario: Threshold reached emits immediately
|
||||
|
||||
- **WHEN** chunks accumulate to ≥ `minBytes` mid-stream
|
||||
- **THEN** a single coalesced frame is emitted at that point and the accumulator resets
|
||||
|
||||
#### Scenario: maxLatencyMs forces an early flush
|
||||
|
||||
- **WHEN** a single 50-byte chunk is buffered with `coalesceBytes(1024, { maxLatencyMs: 100 })` and 100 ms elapses (FakeClock advance)
|
||||
- **THEN** a 50-byte frame is emitted
|
||||
|
||||
**Verified by:** `src/transforms/coalesce-bytes/coalesce-bytes.test.ts::"below-threshold buffers and flushes on close"` AND `src/transforms/coalesce-bytes/coalesce-bytes.test.ts::"threshold reached emits coalesced frame and resets"` AND `src/transforms/coalesce-bytes/coalesce-bytes.test.ts::"maxLatencyMs forces early flush via FakeClock"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: SlidingBytes overlapping window
|
||||
|
||||
The package SHALL export `slidingBytes(size: number, stride: number): TransformStream<Uint8Array, Uint8Array>`. Emits overlapping byte windows of length `size` advancing by `stride` bytes per emission.
|
||||
|
||||
#### Scenario: Adjacent windows overlap by size − stride bytes
|
||||
|
||||
- **WHEN** a 16-byte input is piped through `slidingBytes(8, 4)`
|
||||
- **THEN** windows are emitted at offsets 0, 4, 8, each of length 8; consecutive windows share their last `size − stride` bytes
|
||||
|
||||
#### Scenario: Stride larger than size skips bytes
|
||||
|
||||
- **WHEN** `slidingBytes(4, 8)` runs over a 24-byte input
|
||||
- **THEN** non-overlapping windows are emitted at offsets 0, 8, 16; the 4 bytes between each window are discarded
|
||||
|
||||
**Verified by:** `src/transforms/sliding-bytes/sliding-bytes.test.ts::"adjacent windows overlap by size minus stride"` AND `src/transforms/sliding-bytes/sliding-bytes.test.ts::"stride larger than size skips bytes between windows"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: LengthPrefix encode and decode
|
||||
|
||||
The package SHALL export `lengthPrefixEncode(prefixBytes: 1 | 2 | 4 | 8, options?: { byteOrder?: 'le' | 'be' })` and a matching `lengthPrefixDecode(prefixBytes, options?: { byteOrder?: 'le' | 'be'; maxFrameBytes?: number })`. Encode prepends a length prefix to each chunk. Decode reads a length prefix, accumulates that many bytes, and emits the framed payload.
|
||||
|
||||
#### Scenario: Encode-decode round-trip preserves frame boundaries
|
||||
|
||||
- **WHEN** three frames of varying sizes are piped through `lengthPrefixEncode(4) → lengthPrefixDecode(4)`
|
||||
- **THEN** the three output frames equal the three input frames byte-for-byte
|
||||
|
||||
#### Scenario: maxFrameBytes overflow throws STREAM.FRAME_TOO_LARGE
|
||||
|
||||
- **WHEN** decode encounters a length prefix exceeding `maxFrameBytes`
|
||||
- **THEN** the transform errors the readable side with `StreamError("STREAM.FRAME_TOO_LARGE")` containing the prefix value
|
||||
|
||||
#### Scenario: Truncated frame at close throws STREAM.TRUNCATED_FRAME
|
||||
|
||||
- **WHEN** the input closes mid-frame with fewer bytes than the declared length prefix
|
||||
- **THEN** the transform errors with `StreamError("STREAM.TRUNCATED_FRAME")`
|
||||
|
||||
#### Scenario: Big-endian and little-endian round-trip
|
||||
|
||||
- **WHEN** encode/decode is run with `byteOrder: 'be'` and a frame whose length is `0x01_00`
|
||||
- **THEN** the prefix bytes on the wire are `[0x00, 0x00, 0x01, 0x00]` (4-byte BE) and decode recovers the frame
|
||||
|
||||
**Verified by:** `src/transforms/length-prefix-encode/length-prefix-encode.test.ts::"encode-decode round-trip preserves three frames"` AND `src/transforms/length-prefix-decode/length-prefix-decode.test.ts::"maxFrameBytes overflow throws STREAM.FRAME_TOO_LARGE"` AND `src/transforms/length-prefix-decode/length-prefix-decode.test.ts::"truncated frame at close throws STREAM.TRUNCATED_FRAME"` AND `src/transforms/length-prefix-encode/length-prefix-encode.test.ts::"big-endian prefix encoding produces expected bytes"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Partition single-pass split
|
||||
|
||||
The package SHALL export `partition<T>(source: ReadableStream<T>, predicate: (chunk: T) => boolean): { truthy: ReadableStream<T>; falsy: ReadableStream<T> }`. Each input chunk is routed to exactly one of the two output streams (no duplication, unlike `tee`/`fanOut`). Backpressure on either output applies to the source.
|
||||
|
||||
#### Scenario: Union of outputs equals input, in order per side
|
||||
|
||||
- **WHEN** `partition(source, n => n % 2 === 0)` is applied to `[1,2,3,4,5,6]`
|
||||
- **THEN** the truthy stream yields `[2,4,6]` and the falsy stream yields `[1,3,5]`
|
||||
|
||||
#### Scenario: Slow consumer on one side applies backpressure to source
|
||||
|
||||
- **WHEN** the truthy consumer is held while the source has 1000 items
|
||||
- **THEN** `source` is read at most up to the implementation's high-water-mark before pulls block
|
||||
|
||||
**Verified by:** `src/pipeline/partition/partition.test.ts::"union of outputs equals input with per-side ordering preserved"` AND `src/pipeline/partition/partition.test.ts::"slow consumer applies backpressure to source"` AND `tests/property.test.ts::"partition order preservation per side under arbitrary inputs"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Route N-way single-pass split
|
||||
|
||||
The package SHALL export `route<T, K extends string>(source, keyFn, keys, options?: { onUnknownKey?: 'drop' | 'error' }): Record<K, ReadableStream<T>>` and a non-string-key variant `routeBy<T, K>(source, keyFn, keys, options?): Map<K, ReadableStream<T>>`.
|
||||
|
||||
#### Scenario: Each chunk goes to exactly one keyed branch
|
||||
|
||||
- **WHEN** `route(source, c => c.kind, ["a", "b"])` is applied
|
||||
- **THEN** every input chunk arrives in exactly one branch matching its `kind`
|
||||
|
||||
#### Scenario: Unknown key with onUnknownKey 'drop' silently discards
|
||||
|
||||
- **WHEN** a chunk maps to a key not in `keys` and `onUnknownKey: 'drop'`
|
||||
- **THEN** the chunk is consumed from source but emitted nowhere
|
||||
|
||||
#### Scenario: Unknown key with onUnknownKey 'error' fails the source
|
||||
|
||||
- **WHEN** a chunk maps to a key not in `keys` and `onUnknownKey: 'error'`
|
||||
- **THEN** every output stream errors with `StreamError("STREAM.ROUTE_UNKNOWN_KEY")` containing the offending key
|
||||
|
||||
**Verified by:** `src/pipeline/route/route.test.ts::"each chunk routed to exactly one keyed branch"` AND `src/pipeline/route/route.test.ts::"unknown key with drop policy silently discards"` AND `src/pipeline/route/route.test.ts::"unknown key with error policy errors all branches with STREAM.ROUTE_UNKNOWN_KEY"` AND `src/pipeline/route/route.spec-conformance.test.ts::"route() return type is Record keyed by literal keys argument"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Zip sample-aligned merge
|
||||
|
||||
The package SHALL export `zip<T extends readonly unknown[]>(streams: { [K in keyof T]: ReadableStream<T[K]> }, options?: { onShortInput?: 'close' | 'error' | 'pad'; padValue?: unknown }): ReadableStream<T>`. Emits one tuple per round once every input has produced a chunk.
|
||||
|
||||
#### Scenario: Equal-length streams zip to tuples
|
||||
|
||||
- **WHEN** `zip([fromArray([1,2,3]), fromArray(["a","b","c"])])` is collected
|
||||
- **THEN** the result is `[[1,"a"], [2,"b"], [3,"c"]]`
|
||||
|
||||
#### Scenario: Short input with onShortInput 'close' ends output at first end
|
||||
|
||||
- **WHEN** one input stream ends after 2 chunks, the other after 5, with default `onShortInput: 'close'`
|
||||
- **THEN** the zipped output emits 2 tuples and closes
|
||||
|
||||
#### Scenario: Short input with onShortInput 'error' rejects
|
||||
|
||||
- **WHEN** any input ends before others with `onShortInput: 'error'`
|
||||
- **THEN** the zipped output errors with `StreamError("STREAM.ZIP_LENGTH_MISMATCH")`
|
||||
|
||||
**Verified by:** `src/pipeline/zip/zip.test.ts::"equal-length streams zip to tuples"` AND `src/pipeline/zip/zip.test.ts::"short input with close policy ends at first end"` AND `src/pipeline/zip/zip.test.ts::"short input with error policy throws STREAM.ZIP_LENGTH_MISMATCH"` AND `tests/property.test.ts::"zip then unzip round-trips for equal-length inputs"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Concat sequential drain
|
||||
|
||||
The package SHALL export `concat<T>(streams: ReadableStream<T>[]): ReadableStream<T>`. Drains each stream to completion in order before pulling from the next.
|
||||
|
||||
#### Scenario: Order is preserved across boundaries
|
||||
|
||||
- **WHEN** `concat([fromArray([1,2,3]), fromArray([4,5,6])])` is collected
|
||||
- **THEN** the result is `[1,2,3,4,5,6]`
|
||||
|
||||
#### Scenario: Error in one stream propagates and halts
|
||||
|
||||
- **WHEN** the second of three streams errors mid-drain
|
||||
- **THEN** the third stream is never read; the error propagates wrapped in `StreamError("STREAM.PIPE_FAILED")`
|
||||
|
||||
**Verified by:** `src/pipeline/concat/concat.test.ts::"order preserved across stream boundaries"` AND `src/pipeline/concat/concat.test.ts::"error in middle stream halts and propagates STREAM.PIPE_FAILED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: MergeOrdered k-way merge of sorted streams
|
||||
|
||||
The package SHALL export `mergeOrdered<T>(streams: ReadableStream<T>[], compare: (a: T, b: T) => number): ReadableStream<T>`. Performs a k-way merge of pre-sorted input streams.
|
||||
|
||||
#### Scenario: Three sorted streams merge to a globally sorted output
|
||||
|
||||
- **WHEN** three streams `[1,4,7]`, `[2,5,8]`, `[3,6,9]` are merged with ascending compare
|
||||
- **THEN** the output is `[1,2,3,4,5,6,7,8,9]`
|
||||
|
||||
#### Scenario: Output is sorted iff inputs are sorted
|
||||
|
||||
- **WHEN** any input is unsorted
|
||||
- **THEN** the output may not be sorted (documented limitation; not validated at runtime)
|
||||
|
||||
**Verified by:** `src/pipeline/merge-ordered/merge-ordered.test.ts::"three sorted streams merge to globally sorted output"` AND `tests/property.test.ts::"mergeOrdered output is sorted when all inputs are sorted"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: SafeBranch error isolation
|
||||
|
||||
The package SHALL export `safeBranch<TIn, TOut>(factory: (input: ReadableStream<TIn>) => ReadableStream<TOut>, options?: { bus?: EventBusLike; onError?: (err: unknown) => void }): (input: ReadableStream<TIn>) => ReadableStream<TOut>`. Wraps a sub-pipeline so that errors inside it do not propagate up; instead they emit `streams:branch-error` on the bus and/or invoke `onError`.
|
||||
|
||||
#### Scenario: Branch error becomes a bus event without propagation
|
||||
|
||||
- **WHEN** a branch factory throws on the third chunk and `bus` is provided
|
||||
- **THEN** the bus receives `streams:branch-error` with the error; the outer pipeline does not error
|
||||
|
||||
#### Scenario: Output stream closes cleanly after branch error
|
||||
|
||||
- **WHEN** the branch errors and the wrapped factory's output stream is being read
|
||||
- **THEN** the read completes (close) without throwing and no further chunks are emitted
|
||||
|
||||
**Verified by:** `src/pipeline/safe-branch/safe-branch.test.ts::"branch error emits bus event without propagating to outer pipeline"` AND `src/pipeline/safe-branch/safe-branch.test.ts::"output closes cleanly after branch error"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: ToAsyncIterable sink
|
||||
|
||||
The package SHALL export `toAsyncIterable<T>(stream: ReadableStream<T>, signal?: AbortSignal): AsyncIterable<T>`. Yields chunks until the stream closes or `signal` aborts.
|
||||
|
||||
#### Scenario: All chunks are yielded in order
|
||||
|
||||
- **WHEN** `toAsyncIterable(fromArray([1,2,3]))` is iterated with `for await`
|
||||
- **THEN** the loop sees `1, 2, 3` in order
|
||||
|
||||
#### Scenario: AbortSignal stops iteration
|
||||
|
||||
- **WHEN** `signal.abort()` fires mid-iteration
|
||||
- **THEN** the iterator's next `next()` rejects with `StreamError("STREAM.ABORTED")`
|
||||
|
||||
**Verified by:** `src/sinks/to-async-iterable/to-async-iterable.test.ts::"yields chunks in order"` AND `src/sinks/to-async-iterable/to-async-iterable.test.ts::"abort signal stops iteration with STREAM.ABORTED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Buffer utilities
|
||||
|
||||
The package SHALL export `getDataFromFile(filePath: string): Promise<Uint8Array>` (Node-only with runtime env detection), `bufferToUint8Array(buffer: Buffer | ArrayBuffer): Uint8Array`, and `createUint8ArraySlice(array: Uint8Array, start: number, end: number): Uint8Array`.
|
||||
|
||||
#### Scenario: getDataFromFile reads a Node file as Uint8Array
|
||||
|
||||
- **WHEN** in Node environment, `await getDataFromFile(path)` is called on an existing 4-byte file with bytes `[1,2,3,4]`
|
||||
- **THEN** the result is a `Uint8Array` of length 4 with bytes `[1,2,3,4]`
|
||||
|
||||
#### Scenario: getDataFromFile throws in browser environment
|
||||
|
||||
- **WHEN** `globalThis.process` is undefined and `getDataFromFile(path)` is called
|
||||
- **THEN** the call rejects with `StreamError("STREAM.FILE_BROWSER_NOT_SUPPORTED")`
|
||||
|
||||
#### Scenario: getDataFromFile throws on missing file
|
||||
|
||||
- **WHEN** `await getDataFromFile("/no/such/file")` is called in Node
|
||||
- **THEN** the call rejects with `StreamError("STREAM.FILE_READ_FAILED")` whose `cause` is the underlying ENOENT error
|
||||
|
||||
#### Scenario: bufferToUint8Array preserves bytes from a Node Buffer
|
||||
|
||||
- **WHEN** `bufferToUint8Array(Buffer.from([0x10, 0x20]))` is called in Node
|
||||
- **THEN** the returned `Uint8Array` has bytes `[0x10, 0x20]`
|
||||
|
||||
#### Scenario: createUint8ArraySlice produces an independent copy
|
||||
|
||||
- **WHEN** a slice is created and mutated
|
||||
- **THEN** the original array is unchanged
|
||||
|
||||
**Verified by:** `src/utils/buffer/get-data-from-file/get-data-from-file.test.ts::"reads a Node file as Uint8Array"` AND `src/utils/buffer/get-data-from-file/get-data-from-file.test.ts::"throws STREAM.FILE_BROWSER_NOT_SUPPORTED when process is undefined"` AND `src/utils/buffer/get-data-from-file/get-data-from-file.test.ts::"throws STREAM.FILE_READ_FAILED with ENOENT cause for missing path"` AND `src/utils/buffer/buffer-to-uint8/buffer-to-uint8.test.ts::"preserves bytes from a Node Buffer"` AND `src/utils/buffer/uint8-slice/uint8-slice.test.ts::"slice mutation does not affect original"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Stream utilities
|
||||
|
||||
The package SHALL export `streamToUint8Array(stream: ReadableStream<Uint8Array>, signal?: AbortSignal): Promise<Uint8Array>`, `uint8ArrayToStream(data: Uint8Array, options?: { chunkSize?: number }): ReadableStream<Uint8Array>`, `createDataStream<T>(data: T, transformer: (data: T) => Uint8Array | Promise<Uint8Array>, options?: { chunkSize?: number }): Promise<ReadableStream<Uint8Array>>`, `processDataStream<T>(stream: ReadableStream<Uint8Array>, processor: (data: Uint8Array) => T | Promise<T>): Promise<T>`, and `validateStreamIntegrity(stream: ReadableStream<Uint8Array>, validate: (chunk: Uint8Array) => boolean): Promise<boolean>`.
|
||||
|
||||
#### Scenario: Round-trip identity
|
||||
|
||||
- **WHEN** `streamToUint8Array(uint8ArrayToStream(input))` is awaited
|
||||
- **THEN** the result equals `input` byte-for-byte
|
||||
|
||||
#### Scenario: validateStreamIntegrity returns false on first invalid chunk
|
||||
|
||||
- **WHEN** any chunk fails the predicate
|
||||
- **THEN** the call resolves to `false` without consuming further chunks
|
||||
|
||||
**Verified by:** `src/utils/stream/round-trip/round-trip.test.ts::"streamToUint8Array of uint8ArrayToStream is identity"` AND `src/utils/stream/validate-integrity/validate-integrity.test.ts::"returns false on first invalid chunk and short-circuits"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Instrumented transform with OTel
|
||||
|
||||
The package SHALL export `instrumentTransform<TIn, TOut>(transform: TransformStream<TIn, TOut>, name: string, options?: InstrumentOptions): TransformStream<TIn, TOut>`. Wraps the input transform so each chunk produces a span `stream.transform.<name>`, increments counter `stream.chunks_total`, and records histogram `stream.transform_duration_ms`. Attributes include `stream.stage = name` plus user-supplied `options.attributes`.
|
||||
|
||||
#### Scenario: Each chunk emits a span and counter increment
|
||||
|
||||
- **WHEN** an in-memory `MeterProvider` is registered and 5 chunks pass through an instrumented map transform named `"upper"`
|
||||
- **THEN** `stream.chunks_total` is incremented 5 times with attribute `stream.stage = "upper"`; 5 spans named `stream.transform.upper` are emitted
|
||||
|
||||
#### Scenario: Duration histogram records non-negative values
|
||||
|
||||
- **WHEN** chunks pass through an instrumented transform that sleeps 10 ms via `FakeClock`
|
||||
- **THEN** every recorded `stream.transform_duration_ms` value is ≥ 0 (FakeClock returns 0; real clock returns positive)
|
||||
|
||||
**Verified by:** `src/instrument/transform/transform.test.ts::"each chunk emits a span and increments stream.chunks_total"` AND `src/instrument/transform/transform.spec-conformance.test.ts::"in-memory MeterProvider observes counter and histogram emissions"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Node interop adapters
|
||||
|
||||
The package SHALL export `fromNodeReadable(readable: NodeReadable): ReadableStream<Uint8Array>`, `toNodeReadable(stream: ReadableStream<Uint8Array>): NodeReadable`, and `fromNodeWritable(writable: NodeWritable): WritableStream<Uint8Array>` from the `@suaveplan/streams/node/adapters/adapters` subpath.
|
||||
|
||||
#### Scenario: Node Readable round-trips through WHATWG
|
||||
|
||||
- **WHEN** a Node `Readable` emitting `[Buffer.from("a"), Buffer.from("b")]` is converted via `fromNodeReadable`, drained, and re-emitted via `toNodeReadable`
|
||||
- **THEN** the final Node Readable yields the bytes of `"ab"`
|
||||
|
||||
#### Scenario: Node Writable receives all chunks
|
||||
|
||||
- **WHEN** a WHATWG `WritableStream` returned by `fromNodeWritable` is fed three chunks
|
||||
- **THEN** the underlying Node Writable observes three `write` calls in order
|
||||
|
||||
**Verified by:** `src/node/adapters/adapters.test.ts::"Node Readable round-trips through fromNodeReadable and toNodeReadable"` AND `src/node/adapters/adapters.test.ts::"fromNodeWritable forwards chunks to underlying Writable"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Structured error class with stable code registry
|
||||
|
||||
The package SHALL export `StreamError extends SuaveplanError<StreamErrorCode>` and a stable code registry registered via `defineCodes`. Codes:
|
||||
|
||||
- `STREAM.ABORTED`, `STREAM.BACKPRESSURE_EXCEEDED`, `STREAM.CHUNK_INVALID`, `STREAM.PIPE_FAILED`, `STREAM.TRANSFORM_FAILED`, `STREAM.SOURCE_EXHAUSTED`, `STREAM.READ_FAILED`, `STREAM.WRITE_FAILED`, `STREAM.CLOSED`
|
||||
- `STREAM.FRAME_TOO_LARGE`, `STREAM.TRUNCATED_FRAME`, `STREAM.ROUTE_UNKNOWN_KEY`, `STREAM.ZIP_LENGTH_MISMATCH`, `STREAM.BRANCH_FAILED`
|
||||
- `STREAM.CRYPTO_DECRYPT_FAILED`, `STREAM.CRYPTO_INVALID_FRAME`
|
||||
- `STREAM.DUPLEX_CLOSED`, `STREAM.NETWORK_RESET`
|
||||
- `STREAM.FILE_READ_FAILED`, `STREAM.FILE_BROWSER_NOT_SUPPORTED`
|
||||
- `STREAM.UNKNOWN_TRANSFORM_TYPE`
|
||||
|
||||
#### Scenario: Every error code is thrown by at least one production path
|
||||
|
||||
- **WHEN** the code registry is enumerated and cross-checked against grep of `throw new StreamError`
|
||||
- **THEN** every code appears in at least one production-path throw site (verified by `tests/error-coverage.test.ts`)
|
||||
|
||||
#### Scenario: Subclass narrows code union
|
||||
|
||||
- **WHEN** a consumer narrows `err.code === "STREAM.ABORTED"` with TypeScript
|
||||
- **THEN** the narrowed branch type is the literal `"STREAM.ABORTED"`
|
||||
|
||||
**Verified by:** `tests/error-coverage.test.ts::"every StreamErrorCode appears in at least one production throw site"` AND `src/errors/error/error.spec-conformance.test.ts::"narrowing err.code preserves literal type"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Subpath exports per feature module
|
||||
|
||||
The `package.json` `exports` map SHALL include one entry per feature module so consumers can tree-shake. The barrel `"."` is preserved for ergonomic top-level imports.
|
||||
|
||||
#### Scenario: Every feature directory under src/ has a corresponding subpath export
|
||||
|
||||
- **WHEN** `tests/dep-graph.test.ts` enumerates `src/**/index.ts` and matches against `package.json` `exports`
|
||||
- **THEN** every feature index has a matching subpath export
|
||||
|
||||
#### Scenario: Side-effect-free imports
|
||||
|
||||
- **WHEN** the entry point `src/index.ts` is loaded
|
||||
- **THEN** no side effects (OTel registration, global mutation) occur (verified via `tests/tree-shake.test.ts`)
|
||||
|
||||
**Verified by:** `tests/dep-graph.test.ts::"every feature index has a matching subpath export in package.json"` AND `tests/tree-shake.test.ts::"entry-point load produces no observable side effects"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 100% coverage threshold
|
||||
|
||||
The `vitest.config.ts` SHALL set `thresholds: { statements: 100, branches: 100, functions: 100, lines: 100 }` with `provider: 'istanbul'`. The pre-publish gate `bunx turbo test:coverage --filter=@suaveplan/streams` SHALL hard-fail below 100% on any of the four metrics.
|
||||
|
||||
#### Scenario: Coverage configuration is set
|
||||
|
||||
- **WHEN** the vitest config is parsed
|
||||
- **THEN** all four thresholds equal 100 and provider is `'istanbul'`
|
||||
|
||||
**Verified by:** `tests/dep-graph.test.ts::"vitest.config.ts coverage thresholds are 100/100/100/100 with istanbul provider"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: AbortSignal propagation end-to-end
|
||||
|
||||
The package SHALL accept an `AbortSignal` on every async public entry point: `process`, `execute`, `toArray`, `toBytes`, `forEach`, `pipeline.collect`, `pipeline.drain`, `pipeline.forEach`, `streamToUint8Array`, `validateStreamIntegrity`, `toAsyncIterable`. On abort, in-flight operations cancel underlying readers/writers and reject pending promises with `StreamError("STREAM.ABORTED")`.
|
||||
|
||||
#### Scenario: Aborting collect cancels source
|
||||
|
||||
- **WHEN** `pipeline(fromArray([1,2,3,4,5]).pipeThrough(throttle({ everyMs: 100 }))).collect(signal)` runs and `signal.abort()` fires after 50 ms (FakeClock)
|
||||
- **THEN** the promise rejects with `STREAM.ABORTED` and the source has its `cancel()` invoked
|
||||
|
||||
**Verified by:** `tests/integration.test.ts::"abort during pipeline.collect cancels source and rejects with STREAM.ABORTED"` AND `tests/integration.test.ts::"abort during streamToUint8Array cancels reader"`
|
||||
134
openspec/changes/enterprise-streams-redesign/tasks.md
Normal file
134
openspec/changes/enterprise-streams-redesign/tasks.md
Normal file
@@ -0,0 +1,134 @@
|
||||
# Tasks — enterprise-streams-redesign
|
||||
|
||||
> Every checkbox MUST reflect work actually done in this change. Retroactive ticking is forbidden.
|
||||
|
||||
## 0. Pre-flight (non-negotiable)
|
||||
|
||||
- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
|
||||
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
|
||||
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
|
||||
|
||||
## 1. Spec authoring
|
||||
|
||||
- [x] 1.1 Read `../suaveplan/enterprise/streams.md` end-to-end
|
||||
- [x] 1.2 Read `openspec/specs/streams/spec.md` (does not exist on genesis — clean slate)
|
||||
- [x] 1.3 Read `openspec/AGENTS.md` Sections 13 and 14
|
||||
- [x] 1.4 Draft `proposal.md` from `openspec/templates/proposal.md`
|
||||
- [x] 1.5 Fill in `## Source mapping` table — every requirement cites a line range OR appears in the "no source line" table
|
||||
- [x] 1.6 Draft `specs/streams/spec.md` from `openspec/templates/spec-delta.md`
|
||||
- [x] 1.7 Every requirement in the spec delta carries a `**Verified by:**` line naming concrete `it(...)` titles
|
||||
|
||||
## 2. Implementation — substrate
|
||||
|
||||
- [ ] 2.1 Reset `packages/flow/streams/src/` layout — create new directory tree per the rename matrix in `proposal.md`; delete old `-stream`-suffixed directories
|
||||
- [ ] 2.2 `src/errors/error/error.ts` + `error.md` — `StreamError` base class + subclasses for every code
|
||||
- [ ] 2.3 `src/error-codes.ts` — `defineCodes` registry; named exports for each code constant
|
||||
- [ ] 2.4 `src/types/common.ts` + `common.md` — `TransformOptions`, `StreamSource`, `StreamSink`, `DuplexStreamOptions`, `ReadableOptions`, `InstrumentOptions` shared interfaces
|
||||
- [ ] 2.5 `src/types/common.schemas.ts` — Zod schemas for every public option type
|
||||
- [ ] 2.6 `src/base/transform/transform.ts` + `.md` + `.types.ts` + `.test.ts` + `.spec-conformance.test.ts` — `BaseTransform` abstract class
|
||||
- [ ] 2.7 `src/manager/manager/manager.ts` + `.md` + `.types.ts` + `.test.ts` — `StreamManager`
|
||||
- [ ] 2.8 `src/factory/factory/factory.ts` + `.md` + `.types.ts` + `.test.ts` + `.spec-conformance.test.ts` — `TransformFactory` registry
|
||||
- [ ] 2.9 `src/primitives/duplex/duplex.ts` + `.md` + `.types.ts` + `.test.ts` — `BaseDuplexStream` + `createDuplexStream`
|
||||
- [ ] 2.10 `src/primitives/network/network.ts` + `.md` + `.types.ts` + `.test.ts` — `NetworkStreamImpl` + `createNetworkStream`
|
||||
|
||||
## 3. Implementation — bundled BaseTransform subclasses (port + new)
|
||||
|
||||
- [ ] 3.1 `src/transforms/map/` — `MapTransform` + `mapStream` factory + .md + .test.ts (rename from `map-stream`)
|
||||
- [ ] 3.2 `src/transforms/filter/` — `FilterTransform` + `filterStream` (rename)
|
||||
- [ ] 3.3 `src/transforms/flat-map/` — `FlatMapTransform` + `flatMapStream` (rename)
|
||||
- [ ] 3.4 `src/transforms/batch/` — `BatchTransform` + `batchStream` (rename)
|
||||
- [ ] 3.5 `src/transforms/window/` — `WindowTransform` + `windowStream` (rename)
|
||||
- [ ] 3.6 `src/transforms/throttle/` — `ThrottleTransform` + `throttleStream` (rename)
|
||||
- [ ] 3.7 `src/transforms/limit/` — `LimitTransform` + `limitStream` (rename)
|
||||
- [ ] 3.8 `src/transforms/dedup/` — `DedupTransform` + `dedupStream` (rename)
|
||||
- [ ] 3.9 `src/transforms/tap/` — `TapTransform` + `tapStream` (rename)
|
||||
- [ ] 3.10 `src/transforms/text-codec/` — `TextCodecTransform` + `encodeTextStream`/`decodeTextStream` (no rename)
|
||||
- [ ] 3.11 `src/transforms/compress/` — `CompressTransform` + `compressStream`/`decompressStream` (no rename)
|
||||
- [ ] 3.12 `src/transforms/crypto/` — `CryptoTransform` (port from main, depends on `@suaveplan/crypto` streaming AEAD)
|
||||
- [ ] 3.13 `src/transforms/split-join/` — `SplitJoinTransform` (port from main)
|
||||
|
||||
## 4. Implementation — byte-framing (genesis-original)
|
||||
|
||||
- [ ] 4.1 `src/transforms/chunk-bytes/` — `ChunkBytesTransform` + `chunkBytes` factory
|
||||
- [ ] 4.2 `src/transforms/coalesce-bytes/` — `CoalesceBytesTransform` + `coalesceBytes` factory
|
||||
- [ ] 4.3 `src/transforms/sliding-bytes/` — `SlidingBytesTransform` + `slidingBytes` factory
|
||||
- [ ] 4.4 `src/transforms/length-prefix-encode/` — `LengthPrefixEncodeTransform` + `lengthPrefixEncode` factory
|
||||
- [ ] 4.5 `src/transforms/length-prefix-decode/` — `LengthPrefixDecodeTransform` + `lengthPrefixDecode` factory
|
||||
|
||||
## 5. Implementation — pipeline ops
|
||||
|
||||
- [ ] 5.1 `src/pipeline/builder/builder.ts` — `pipeline()` + `PipelineBuilder` (rename from `pipeline-builder`)
|
||||
- [ ] 5.2 `src/pipeline/fan-out/` — preserved as-is (no rename)
|
||||
- [ ] 5.3 `src/pipeline/merge/` — preserved as-is (no rename)
|
||||
- [ ] 5.4 `src/pipeline/partition/` — `partition()` single-pass split
|
||||
- [ ] 5.5 `src/pipeline/route/` — `route()` + `routeBy()` N-way split
|
||||
- [ ] 5.6 `src/pipeline/zip/` — `zip()` sample-aligned merge
|
||||
- [ ] 5.7 `src/pipeline/concat/` — `concat()` sequential drain
|
||||
- [ ] 5.8 `src/pipeline/merge-ordered/` — `mergeOrdered()` k-way merge
|
||||
- [ ] 5.9 `src/pipeline/safe-branch/` — `safeBranch()` error isolation
|
||||
|
||||
## 6. Implementation — sources, sinks, utils, instrumentation
|
||||
|
||||
- [ ] 6.1 `src/readable/from-array/`, `src/readable/from-bytes/`, `src/readable/from-generator/` — preserved
|
||||
- [ ] 6.2 `src/sinks/to-array/`, `src/sinks/to-bytes/`, `src/sinks/for-each/` — preserved
|
||||
- [ ] 6.3 `src/sinks/to-async-iterable/` — new
|
||||
- [ ] 6.4 `src/instrument/transform/` — preserved (rename from `instrument-transform`)
|
||||
- [ ] 6.5 `src/node/adapters/` — preserved (rename from `node-adapters.ts` to feature dir)
|
||||
- [ ] 6.6 `src/utils/buffer/get-data-from-file/`, `src/utils/buffer/buffer-to-uint8/`, `src/utils/buffer/uint8-slice/` — port from main
|
||||
- [ ] 6.7 `src/utils/stream/round-trip/` (`streamToUint8Array` + `uint8ArrayToStream`), `src/utils/stream/create-data-stream/`, `src/utils/stream/process-data-stream/`, `src/utils/stream/validate-integrity/` — port from main
|
||||
|
||||
## 7. Tests
|
||||
|
||||
- [ ] 7.1 Unit tests at every `src/<feature>/<feature>.test.ts` covering every scenario in the spec delta
|
||||
- [ ] 7.2 Spec-conformance tests at `*.spec-conformance.test.ts` for type-level / observability / lifecycle scenarios
|
||||
- [ ] 7.3 `tests/integration.test.ts` — end-to-end pipelines (encrypt + length-prefix + decrypt round-trip; partition + safe-branch + merge composition)
|
||||
- [ ] 7.4 `tests/property.test.ts` — fast-check invariants (`chunkBytes` byte-conservation, `partition` order preservation, `zip`/unzip round-trip, `mergeOrdered` sortedness)
|
||||
- [ ] 7.5 `tests/dep-graph.test.ts` — workspace-dep audit, side-effect audit, no `node:fs` in browser entry
|
||||
- [ ] 7.6 `tests/tree-shake.test.ts` — `sideEffects: false` round-trip
|
||||
- [ ] 7.7 `tests/error-coverage.test.ts` — every `StreamErrorCode` thrown by at least one production path
|
||||
- [ ] 7.8 Every `it(...)` carries `{ timeout: 5000 }` (or 30000 for genuinely async)
|
||||
|
||||
## 8. Documentation
|
||||
|
||||
- [ ] 8.1 Sibling `.md` for every implementation `.ts` (≥200 words, basic + advanced examples, full API reference)
|
||||
- [ ] 8.2 `README.md` at package root rewritten with all new modules linked under `## Modules`
|
||||
- [ ] 8.3 Update `CLAUDE.md` `streams` package listing to reflect new surface (BaseTransform/StreamManager/TransformFactory/CryptoTransform/etc.)
|
||||
|
||||
## 9. Package configuration
|
||||
|
||||
- [ ] 9.1 Update `package.json` — add `@suaveplan/crypto` (`workspace:^`), `@suaveplan/events` (`workspace:^`); keep `@suaveplan/error`, `@suaveplan/types`; ensure `tslib`, `zod`, `@opentelemetry/api` via catalog
|
||||
- [ ] 9.2 Update `package.json` `exports` — one subpath per feature module per `openspec/AGENTS.md` Section 6
|
||||
- [ ] 9.3 Confirm `sideEffects: false` and tree-shake guard test passes
|
||||
- [ ] 9.4 `vitest.config.ts` thresholds remain `100/100/100/100` with `provider: 'istanbul'`
|
||||
|
||||
## 10. Consumer migration
|
||||
|
||||
- [ ] 10.1 `grep -rln "@suaveplan/streams/transforms/.*-stream" packages/` — update any internal subpath consumer to new paths
|
||||
- [ ] 10.2 `grep -rln "@suaveplan/streams/pipeline/pipeline-builder" packages/` — update to `pipeline/builder`
|
||||
- [ ] 10.3 `grep -rln "@suaveplan/streams/instrument/instrument-transform" packages/` — update to `instrument/transform`
|
||||
- [ ] 10.4 `grep -rln "@suaveplan/streams/node/node-adapters" packages/` — update to `node/adapters/adapters`
|
||||
- [ ] 10.5 `grep -rln "@suaveplan/streams/errors/stream-error" packages/` — update to `errors/error/error`
|
||||
|
||||
## 11. Pre-archive verification (the 5-gate triple)
|
||||
|
||||
- [ ] 11.1 `bun run check` — Biome 0 errors / 0 warnings / 0 info across the whole monorepo
|
||||
- [ ] 11.2 `bunx turbo lint typecheck test --filter=@suaveplan/streams` — green
|
||||
- [ ] 11.3 `bunx turbo test:coverage --filter=@suaveplan/streams` — 100/100/100/100
|
||||
- [ ] 11.4 `bun scripts/openspec-conformance.ts @suaveplan/streams` — exit 0, no `error` findings
|
||||
- [ ] 11.5 `bunx turbo build --filter=@suaveplan/streams` — green
|
||||
|
||||
## 12. Pre-archive 7-row checklist (AGENTS.md Section 13)
|
||||
|
||||
- [ ] 12.1 Generics threaded; `expectTypeOf` assertions on `BaseTransform.process<T>`, `TransformFactory.create<T>` literal-type narrowing, `route()` `Record<K, …>` keying
|
||||
- [ ] 12.2 Zod schemas wired at every declared boundary (`TransformOptions`, `ChunkQueueOptions`-style option bundles, public factories)
|
||||
- [ ] 12.3 Every error code in `error-codes.ts` thrown by at least one production path (verified by `tests/error-coverage.test.ts`)
|
||||
- [ ] 12.4 Strict-mode behaviors tested: `onUnknownKey: 'drop' | 'error'`, `tail: 'emit' | 'pad' | 'drop'`, `onShortInput: 'close' | 'error' | 'pad'`
|
||||
- [ ] 12.5 OTel instruments emitted from production paths (`stream.chunks_total`, `stream.transform_duration_ms`) and asserted with a real in-memory `MeterProvider`
|
||||
- [ ] 12.6 Lifecycle dedup tested: `dispose()` idempotent on `BaseTransform` and `CryptoTransform`; `clear()` idempotent on `StreamManager`
|
||||
- [ ] 12.7 No retroactive tick — every box above reflects work done in this change
|
||||
|
||||
## 13. Archive
|
||||
|
||||
- [ ] 13.1 Fold spec delta into `openspec/specs/streams/spec.md` (creates the folded spec for the first time)
|
||||
- [ ] 13.2 Move `openspec/changes/enterprise-streams-redesign/` → `openspec/archive/<yyyy-mm-dd>-enterprise-streams-redesign/`
|
||||
- [ ] 13.3 Refresh `MANIFEST.md` via `bun scripts/folder-reorg/generate-manifest.ts` (deps changed: `@suaveplan/crypto`, `@suaveplan/events` added)
|
||||
Reference in New Issue
Block a user