Initial Commit
This commit is contained in:
228
openspec/changes/enterprise-streams-redesign/proposal.md
Normal file
228
openspec/changes/enterprise-streams-redesign/proposal.md
Normal file
@@ -0,0 +1,228 @@
|
||||
# enterprise-streams-redesign
|
||||
|
||||
**Package**: `@suaveplan/streams`
|
||||
**Status**: PROPOSED
|
||||
**Type**: refactor
|
||||
|
||||
## Why
|
||||
|
||||
The genesis-original `enterprise-streams` change (archived 2026-04-24) shipped a functional, WHATWG-`TransformStream`-only API. After auditing `../suaveplan/packages/universal/streams/src/` — the canonical implementation on main — three structural gaps surfaced that block downstream packages (`streams-metadata`, `streams-chunk-queue`, future `audio-engine`, future `streams-worker`) from being built consistently:
|
||||
|
||||
1. **No plugin contract.** Hooks (`crypto:before-chunk`, `crypto:after-chunk`), events (`start` / `chunk` / `complete` / `error`), parameter introspection, and lifecycle (`init` / `flush` / `dispose`) cannot be expressed in pure `TransformStream` factories. Main's `BaseTransform` abstract class — with `process<T>(input)`, `setOperation`, `setParams`, `hook(name, fn)`, `on(name, fn)`, and `toTransformStream()` bridge — is the canonical plugin shape and is required for the DAW-style audio plugin ambition.
|
||||
2. **No streaming AEAD transform.** Main ships `CryptoTransform` with pluggable `xchacha20StreamingEngine` / `aesGcmStreamingEngine` that bind a counter and final-flag into the AEAD AAD, detecting reorder/truncate/tamper. Genesis has no equivalent. Encryption-on-the-fly is one of the explicit driving use cases.
|
||||
3. **No bidirectional / network primitives.** Main ships `BaseDuplexStream` and `NetworkStreamImpl` (id + protocol + metadata Map) for WebRTC / WebSocket / TCP integration. Genesis has none. These are foundational for the planned drive / network-discovery / audio-network packages.
|
||||
|
||||
Additionally, the existing genesis layout uses redundant `-stream` suffixes on every transform module (`transforms/throttle-stream/`, `transforms/map-stream/`, etc.) — confusing inside a package literally named `streams`. This change normalises naming as part of the redesign.
|
||||
|
||||
The 11 existing functional transforms, 3 sources, 3 sinks, fan-out, merge, pipeline-builder, instrument-transform, and node-adapters are all preserved as the implementation backbone — every `BaseTransform` subclass either subclasses or composes the equivalent functional logic via `toTransformStream()`, so no behavioural regression is introduced.
|
||||
|
||||
## What changes
|
||||
|
||||
### ADDED — plugin substrate (ported from main)
|
||||
|
||||
- ADDED: `BaseTransform` — abstract class with `name`, `setOperation`, `setParams`, `getOperation`, `getParams`, `hook(name, fn)`, `removeHook(name, fn)`, `on(name, fn)`, `off(name, fn)`, `process<T>(input, options?)` abstract, `toTransformStream<TIn, TOut>(): TransformStream<TIn, TOut>` bridge to WHATWG, `dispose()` for cleanup
|
||||
- ADDED: `TransformOptions`, `IBaseTransform`, `TransformHookPayload`, `TransformEventPayload` types
|
||||
- ADDED: `StreamManager` — `pipe(...transforms)`, `execute<T>(input, options?)`, `clear()`, `getTransforms()`
|
||||
- ADDED: `TransformFactory` — registry + `create(type, ...args)` with `TransformType` discriminated union; built-in registrations for every bundled transform
|
||||
- ADDED: `BaseDuplexStream<R, W>` primitive with `StreamSource<R>` and `StreamSink<W>` configuration
|
||||
- ADDED: `NetworkStreamImpl` with `id`, `protocol`, `metadata: Map<string, unknown>`, `closed` flag, `close()` and `reset()`
|
||||
- ADDED: factory functions `createDuplexStream`, `createNetworkStream`, `createTransformStream`
|
||||
|
||||
### ADDED — bundled BaseTransform subclasses (port + new)
|
||||
|
||||
Ported from main:
|
||||
|
||||
- ADDED: `CryptoTransform` — pluggable streaming AEAD with `engine`, `key`, `operation` ("encrypt" | "decrypt"), hooks (`crypto:before-chunk`, `crypto:after-chunk`), events (`crypto:start`, `crypto:chunk`, `crypto:complete`, `crypto:error`), `process(asyncIterable)` for stream-of-frames, `toTransformStream()` for WHATWG bridge. Depends on `@suaveplan/crypto` streaming AEAD engines.
|
||||
- ADDED: `SplitJoinTransform` — object-mode split with `chunkSize` + `overlap`; `join` reassembles with optional `separator` and `removeOverlap`; supports string / `ArrayBuffer` / array; type guards `isSplittable`, `isJoinable`
|
||||
|
||||
Existing 11 functional transforms migrated to `BaseTransform` subclasses (see § Renames):
|
||||
|
||||
- ADDED: `MapTransform`, `FilterTransform`, `FlatMapTransform`, `BatchTransform`, `WindowTransform`, `ThrottleTransform`, `LimitTransform`, `DedupTransform`, `TapTransform`, `TextCodecTransform` (encode/decode operations), `CompressTransform` (compress/decompress operations)
|
||||
|
||||
Genesis-original gap-fillers (new):
|
||||
|
||||
- ADDED: `ChunkBytesTransform` — byte-level fixed-frame re-framing with tail policy (`emit` | `pad` | `drop`)
|
||||
- ADDED: `CoalesceBytesTransform` — merges small `Uint8Array` chunks until a byte threshold or max-latency timeout
|
||||
- ADDED: `SlidingBytesTransform` — overlapping byte windows with size + stride
|
||||
- ADDED: `LengthPrefixEncodeTransform`, `LengthPrefixDecodeTransform` — wire-protocol framing with 1/2/4/8-byte prefix and configurable byte-order
|
||||
|
||||
### ADDED — pipeline ops (genesis-original)
|
||||
|
||||
- ADDED: `partition<T>(source, predicate)` → `{ truthy, falsy }` single-pass split (no tee duplication)
|
||||
- ADDED: `route<T, K extends string>(source, keyFn, keys, options?)` → `Record<K, ReadableStream<T>>`; `onUnknownKey: 'drop' | 'error'`
|
||||
- ADDED: `routeBy<T, K>(source, keyFn, keys, options?)` → `Map<K, ReadableStream<T>>` (non-string-key variant)
|
||||
- ADDED: `zip<T extends readonly unknown[]>(streams)` → tuple stream with `onShortInput: 'close' | 'error' | 'pad'` option
|
||||
- ADDED: `concat<T>(streams)` → sequential drain
|
||||
- ADDED: `mergeOrdered<T>(streams, compare)` → k-way merge of pre-sorted streams
|
||||
- ADDED: `safeBranch<TIn, TOut>(factory, options?)` → wraps a sub-pipeline so its errors emit on `EventBusLike` instead of propagating
|
||||
|
||||
### ADDED — sources, sinks, utilities
|
||||
|
||||
- ADDED: `toAsyncIterable<T>(stream, signal?)` sink
|
||||
- ADDED: `getDataFromFile(filePath)` — Node-only file reader with runtime environment detection (throws in browser)
|
||||
- ADDED: `bufferToUint8Array(buffer)` — handles both Node `Buffer` and `ArrayBuffer`
|
||||
- ADDED: `createUint8ArraySlice(array, start, end)` — explicit-typed copy helper
|
||||
- ADDED: `createDataStream<T>(data, transformer, options?)` — generic source factory used by `StreamManager` for non-stream inputs
|
||||
- ADDED: `processDataStream<T>(stream, options?)` — generic sink factory
|
||||
- ADDED: `streamToUint8Array(stream)`, `uint8ArrayToStream(data, options?)`, `bufferToStream`, `streamToBuffer` cross-runtime helpers
|
||||
- ADDED: `validateStreamIntegrity(stream)` — chunk-validity sweep helper
|
||||
|
||||
### MODIFIED — naming convention
|
||||
|
||||
Drop redundant `-stream` suffix on every module name inside the streams package. Implementation file basename matches the parent folder; class and function names follow PascalCase / camelCase conventions without the redundant suffix.
|
||||
|
||||
| Old path | New path | Rationale |
|
||||
|---|---|---|
|
||||
| `transforms/map-stream/map-stream.ts` | `transforms/map/map.ts` | inside a streams package, "stream" suffix is redundant |
|
||||
| `transforms/filter-stream/filter-stream.ts` | `transforms/filter/filter.ts` | same |
|
||||
| `transforms/flat-map-stream/` | `transforms/flat-map/` | same |
|
||||
| `transforms/batch-stream/` | `transforms/batch/` | same |
|
||||
| `transforms/window-stream/` | `transforms/window/` | same |
|
||||
| `transforms/throttle-stream/` | `transforms/throttle/` | same |
|
||||
| `transforms/limit-stream/` | `transforms/limit/` | same |
|
||||
| `transforms/dedup-stream/` | `transforms/dedup/` | same |
|
||||
| `transforms/tap-stream/` | `transforms/tap/` | same |
|
||||
| `transforms/text-codec/` | unchanged | already conforms |
|
||||
| `transforms/compress/` | unchanged | already conforms |
|
||||
| `pipeline/pipeline-builder/` | `pipeline/builder/` | redundant package-name prefix |
|
||||
| `instrument/instrument-transform/` | `instrument/transform/` | redundant prefix |
|
||||
| `node/node-adapters.ts` | `node/adapters/adapters.ts` | promoted to feature dir + redundant prefix dropped |
|
||||
| `errors/stream-error.ts` | `errors/error/error.ts` | redundant prefix dropped, promoted to feature dir |
|
||||
|
||||
The corresponding exported symbols are renamed in lockstep:
|
||||
|
||||
| Old export | New export |
|
||||
|---|---|
|
||||
| `mapStream` | `mapStream` (kept — describes the *function* shape, not file naming) plus new `MapTransform` class |
|
||||
| `filterStream` | `filterStream` plus `FilterTransform` |
|
||||
| `flatMapStream` | `flatMapStream` plus `FlatMapTransform` |
|
||||
| `batchStream` | `batchStream` plus `BatchTransform` |
|
||||
| `windowStream` | `windowStream` plus `WindowTransform` |
|
||||
| `throttleStream` | `throttleStream` plus `ThrottleTransform` |
|
||||
| `limitStream` | `limitStream` plus `LimitTransform` |
|
||||
| `dedupStream` | `dedupStream` plus `DedupTransform` |
|
||||
| `tapStream` | `tapStream` plus `TapTransform` |
|
||||
| `compressStream`, `decompressStream` | unchanged plus `CompressTransform` (operation = "compress" \| "decompress") |
|
||||
| `encodeTextStream`, `decodeTextStream` | unchanged plus `TextCodecTransform` |
|
||||
| `pipeline()` | unchanged |
|
||||
| `fanOut`, `merge` | unchanged |
|
||||
| `instrumentTransform` | unchanged |
|
||||
| `fromNodeReadable`, `toNodeReadable`, `fromNodeWritable` | unchanged |
|
||||
| `StreamPipeFailedError` | renamed to `StreamPipeFailedError` (lives in `errors/error/error.ts` now) |
|
||||
|
||||
The function-form factories (`mapStream`, `filterStream`, …) remain exported for tree-shaking and for direct WHATWG composition; the OO subclasses delegate to them via `toTransformStream()`. **Both call paths are first-class.** This is the only "two APIs" pattern this package permits, and it is justified because the OO subclass *uses* the functional factory under the hood — no parallel implementation, no drift surface.
|
||||
|
||||
### ADDED — error codes
|
||||
|
||||
- ADDED: `STREAM.FRAME_TOO_LARGE`, `STREAM.TRUNCATED_FRAME`, `STREAM.ROUTE_UNKNOWN_KEY`, `STREAM.ZIP_LENGTH_MISMATCH`, `STREAM.BRANCH_FAILED`, `STREAM.CRYPTO_DECRYPT_FAILED`, `STREAM.CRYPTO_INVALID_FRAME`, `STREAM.DUPLEX_CLOSED`, `STREAM.NETWORK_RESET`, `STREAM.FILE_READ_FAILED`, `STREAM.FILE_BROWSER_NOT_SUPPORTED`
|
||||
|
||||
### REMOVED
|
||||
|
||||
- REMOVED: nothing — the previous spec is preserved verbatim under `MODIFIED Requirements`. Symbols are renamed (with old paths gone) but exported names are kept on the function-form API.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Audio-specific anything** (sample frames, channel-aware ops, mixing, pitch-shift, dynamics-processor) — belongs in the upcoming `audio-engine` package as plugins, not in streams. Karaoke/radio transforms from main do **not** port over.
|
||||
- **Image / video / metadata transforms** — already split into siblings (`streams-metadata` proposed; image/video to follow as separate changes).
|
||||
- **Worker offload** (`pipeInWorker`) — deferred to a future `streams-worker` sibling package once audio-engine demands prove the transport shape.
|
||||
- **`schema-registry` integration** of stream chunk contracts — deferred until `@suaveplan/schema-registry` ships the consumer hooks the streams chunk format would need.
|
||||
- **Mutation testing & contract testing layers** — covered by separate `enterprise-testing-mutation` and `enterprise-testing-contract` changes; this change ships 100% line/branch/function coverage as the baseline.
|
||||
|
||||
## Source mapping
|
||||
|
||||
This change derives from `../suaveplan/enterprise/streams.md` (the generic enterprise-quality template applied to streams) and the canonical implementation at `../suaveplan/packages/universal/streams/src/` (the ground-truth API surface).
|
||||
|
||||
| Requirement | Source | Notes |
|
||||
|---|---|---|
|
||||
| Pure TypeScript, runs in Bun + Node + browser | `enterprise/streams.md` L38 | verbatim |
|
||||
| WHATWG Streams baseline | `enterprise/streams.md` L40 | verbatim |
|
||||
| Zero-copy `Uint8Array` operations on hot paths | `enterprise/streams.md` L41 | verbatim |
|
||||
| Backpressure-first; bounded queues with declarative high/low watermarks | `enterprise/streams.md` L42 | verbatim |
|
||||
| `AbortSignal` propagates end-to-end | `enterprise/streams.md` L43 | verbatim |
|
||||
| Zod schemas for stream configuration and chunk contracts | `enterprise/streams.md` L44 | partial — schema-registry integration deferred (see Out of scope) |
|
||||
| Property-based tests via fast-check | `enterprise/streams.md` L45 | verbatim |
|
||||
| Deterministic — no `Math.random()` / `Date.now()` at module scope, injectable clock + RNG | `enterprise/streams.md` L46 | verbatim |
|
||||
| OpenTelemetry instrumentation | `enterprise/streams.md` L47 | verbatim |
|
||||
| Structured errors extending `@suaveplan/error` with stable codes | `enterprise/streams.md` L48 | verbatim |
|
||||
| Idempotency + retry primitives for network-backed adapters | `enterprise/streams.md` L49 | applied to `NetworkStreamImpl`'s reset semantics |
|
||||
| Feature flags for risky transform rollouts | `enterprise/streams.md` L50 | applied via `FlagRegistry` injection on factory; opt-in |
|
||||
| Plugin / extension interface | `enterprise/streams.md` L74 | satisfied by `BaseTransform` + `TransformFactory` |
|
||||
| Observability surface — counters, histograms, spans on every externally-visible op | `enterprise/streams.md` L75 | satisfied by `instrumentTransform` + per-class hook surface |
|
||||
| Configurability — Zod-validated options with sensible defaults | `enterprise/streams.md` L76 | verbatim |
|
||||
| Performance — benchmark suite + CI regression gate | `enterprise/streams.md` L77 | verbatim |
|
||||
| Resilience — retries / timeouts / cancellation / circuit breakers | `enterprise/streams.md` L78 | applied via `@suaveplan/retry` composition (caller-side) |
|
||||
| Security — input validation, output encoding, zeroing of sensitive buffers | `enterprise/streams.md` L79 | applied to `CryptoTransform` key zeroize on dispose |
|
||||
| Migration story for public API changes | `enterprise/streams.md` L80 | satisfied by the rename matrix above |
|
||||
| Example app / story demonstrating full feature set | `enterprise/streams.md` L82 | satisfied by `examples/` runnable scripts |
|
||||
|
||||
### Requirements with no `enterprise/streams.md` line
|
||||
|
||||
These derive from `../suaveplan/packages/universal/streams/src/` (canonical implementation on main) and from the `enterprise-streams-redesign` audit conducted in the brainstorming session.
|
||||
|
||||
| Requirement | Source | Justification |
|
||||
|---|---|---|
|
||||
| `BaseTransform` abstract class with hook + event surface | `../suaveplan/packages/universal/streams/src/transforms/base/base.ts` + `base.md` | canonical plugin shape — required for hooks/events that no functional API can express |
|
||||
| `StreamManager` chainable pipeline | `../suaveplan/packages/universal/streams/src/manager/manager.ts` + `manager.md` | canonical |
|
||||
| `TransformFactory` registry | `../suaveplan/packages/universal/streams/src/transforms/factory/factory.ts` + `factory.md` | canonical |
|
||||
| `BaseDuplexStream` + `NetworkStreamImpl` | `../suaveplan/packages/universal/streams/src/primitives/duplex.ts` + `duplex.md` | canonical |
|
||||
| `CryptoTransform` with pluggable streaming AEAD | `../suaveplan/packages/universal/streams/src/transforms/crypto/crypto.ts` + `crypto.md` | canonical; depends on `@suaveplan/crypto` streaming AEAD engines |
|
||||
| `SplitJoinTransform` | `../suaveplan/packages/universal/streams/src/transforms/split-join/split-join.ts` + `split-join.md` | canonical |
|
||||
| Buffer utilities (`getDataFromFile`, `bufferToUint8Array`, `createUint8ArraySlice`) | `../suaveplan/packages/universal/streams/src/utils/buffer/buffer.md` | canonical |
|
||||
| Stream utilities (`createDataStream`, `processDataStream`, `streamToUint8Array`, `uint8ArrayToStream`, `validateStreamIntegrity`) | `../suaveplan/packages/universal/streams/src/utils/stream/stream.md` | canonical |
|
||||
| `chunkBytes`, `coalesceBytes`, `slidingBytes`, `lengthPrefixEncode/Decode` | genesis audit (this change) | byte-level framing primitives; AEAD requires fixed-byte frames; main's `SplitJoinTransform` is object-mode only |
|
||||
| `partition`, `route`, `routeBy` | genesis audit (this change) | single-pass split (`fanOut` duplicates via `tee`); essential for "split into different plugin chains by criterion" |
|
||||
| `zip`, `concat`, `mergeOrdered` | genesis audit (this change) | merge variants beyond arrival-order race; `zip` enables sample-aligned recombination, `mergeOrdered` enables k-way time-sorted merge |
|
||||
| `safeBranch` | genesis audit (this change) | branch error isolation — without it, one failing FX rack tears down the whole graph |
|
||||
| `toAsyncIterable` sink | genesis audit (this change) | async iterator interop, missing from main |
|
||||
| Naming convention: drop redundant `-stream` suffix | user direction (brainstorming session) | clarity inside a streams package |
|
||||
| 100% test coverage threshold | genesis-wide policy (`openspec/AGENTS.md` § 7) | non-package-specific |
|
||||
| Source-mapped `**Verified by:**` lines | genesis-wide policy (`openspec/AGENTS.md` § 13) | non-package-specific |
|
||||
|
||||
## Verification approach
|
||||
|
||||
Every requirement in the spec delta carries a `**Verified by:**` line pointing at a concrete `it(...)` title in one of the test files below. Tests are co-located with implementation per the genesis 5-file convention.
|
||||
|
||||
- `src/base/transform/transform.test.ts` — `BaseTransform` lifecycle, hook + event surfaces, `toTransformStream()` bridge, `dispose()` semantics
|
||||
- `src/base/transform/transform.spec-conformance.test.ts` — type-level: generic narrowing, hook-payload type inference, factory return-type preservation
|
||||
- `src/manager/manager/manager.test.ts` — pipeline chaining, sequential execution, `clear()` reset, error propagation
|
||||
- `src/factory/factory/factory.test.ts` — type-discriminated `create()`, registry lookup, unknown-type error
|
||||
- `src/primitives/duplex/duplex.test.ts` — bidirectional flow, byte vs object mode, `cancel()` semantics
|
||||
- `src/primitives/network/network.test.ts` — id, protocol, metadata, `close()`, `reset()` with error
|
||||
- `src/transforms/<name>/<name>.test.ts` — per-transform behaviour (one file per BaseTransform subclass)
|
||||
- `src/transforms/crypto/crypto.test.ts` + `crypto.spec-conformance.test.ts` — round-trip encrypt/decrypt, hook payloads, event firing order, reorder/truncate detection
|
||||
- `src/pipeline/<op>/<op>.test.ts` — per-op behaviour
|
||||
- `src/sinks/to-async-iterable/to-async-iterable.test.ts` — async-iterator yields all chunks; `signal` cancels
|
||||
- `src/utils/buffer/<helper>/<helper>.test.ts` — Node + browser branches; env-detection error
|
||||
- `src/utils/stream/<helper>/<helper>.test.ts` — round-trip identity, integrity validation
|
||||
- `src/instrument/transform/transform.test.ts` — span + counter + histogram emitted via in-memory OTel `MeterProvider`
|
||||
- `src/node/adapters/adapters.test.ts` — Node ↔ WHATWG bridge round-trip
|
||||
- `tests/integration.test.ts` — full pipelines: encrypt → length-prefix → decrypt round-trip; partition + safe-branch + merge composition
|
||||
- `tests/property.test.ts` — `chunkBytes` byte-conservation, `partition` order preservation, `zip`/`unzip` round-trip, `mergeOrdered` sortedness invariant
|
||||
- `tests/dep-graph.test.ts` — only declared workspace deps reachable; no `node:fs` import in browser entry
|
||||
- `tests/tree-shake.test.ts` — `sideEffects: false` round-trip via `Bun.build`
|
||||
|
||||
## Migration impact
|
||||
|
||||
**This package has consumers on genesis already** — every package that imports from `@suaveplan/streams`. Audit:
|
||||
|
||||
```bash
|
||||
grep -rln "@suaveplan/streams" packages/ --include="package.json"
|
||||
```
|
||||
|
||||
The function-form API (`mapStream`, `filterStream`, `pipeline`, `fanOut`, `merge`, `instrumentTransform`, `fromNodeReadable`, `toNodeReadable`, `fromNodeWritable`, `compressStream`, `decompressStream`, `encodeTextStream`, `decodeTextStream`, `toArray`, `toBytes`, `forEach`, `fromArray`, `fromBytes`, `fromAsyncGenerator`) is preserved. Internal subpath imports change:
|
||||
|
||||
| Old subpath import | New subpath import |
|
||||
|---|---|
|
||||
| `@suaveplan/streams/transforms/map-stream/map-stream` | `@suaveplan/streams/transforms/map/map` |
|
||||
| `@suaveplan/streams/transforms/filter-stream/filter-stream` | `@suaveplan/streams/transforms/filter/filter` |
|
||||
| (… same pattern for the renamed transforms above) | |
|
||||
| `@suaveplan/streams/pipeline/pipeline-builder/pipeline-builder` | `@suaveplan/streams/pipeline/builder/builder` |
|
||||
| `@suaveplan/streams/instrument/instrument-transform/instrument-transform` | `@suaveplan/streams/instrument/transform/transform` |
|
||||
| `@suaveplan/streams/node/node-adapters` | `@suaveplan/streams/node/adapters/adapters` |
|
||||
| `@suaveplan/streams/errors/stream-error` | `@suaveplan/streams/errors/error/error` |
|
||||
|
||||
**Consumer migration is part of this change.** Every internal consumer is updated in the same change folder; no shim or deprecation alias remains. The barrel `@suaveplan/streams` (root export) is unaffected — top-level imports keep working.
|
||||
|
||||
**Breaking changes:** subpath imports only (top-level barrel preserved). New OO subclasses are pure additions.
|
||||
|
||||
**Prerequisites:** `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/events`, `@suaveplan/crypto` (for `CryptoTransform`) must be at their current genesis versions on Verdaccio.
|
||||
Reference in New Issue
Block a user