suaveplan/openspec/changes/enterprise-streams-redesign/proposal.md

enterprise-streams-redesign

Package: @suaveplan/streams Status: PROPOSED Type: refactor

Why

The genesis-original enterprise-streams change (archived 2026-04-24) shipped a functional, WHATWG-TransformStream-only API. After auditing ../suaveplan/packages/universal/streams/src/ — the canonical implementation on main — three structural gaps surfaced that block downstream packages (streams-metadata, streams-chunk-queue, future audio-engine, future streams-worker) from being built consistently:

1. No plugin contract. Hooks (crypto:before-chunk, crypto:after-chunk), events (start / chunk / complete / error), parameter introspection, and lifecycle (init / flush / dispose) cannot be expressed in pure TransformStream factories. Main's BaseTransform abstract class — with process<T>(input), setOperation, setParams, hook(name, fn), on(name, fn), and toTransformStream() bridge — is the canonical plugin shape and is required for the DAW-style audio plugin ambition.
2. No streaming AEAD transform. Main ships CryptoTransform with pluggable xchacha20StreamingEngine / aesGcmStreamingEngine that bind a counter and final-flag into the AEAD AAD, detecting reorder/truncate/tamper. Genesis has no equivalent. Encryption-on-the-fly is one of the explicit driving use cases.
3. No bidirectional / network primitives. Main ships BaseDuplexStream and NetworkStreamImpl (id + protocol + metadata Map) for WebRTC / WebSocket / TCP integration. Genesis has none. These are foundational for the planned drive / network-discovery / audio-network packages.

Additionally, the existing genesis layout uses redundant -stream suffixes on every transform module (transforms/throttle-stream/, transforms/map-stream/, etc.) — confusing inside a package literally named streams. This change normalises naming as part of the redesign.

The 11 existing functional transforms, 3 sources, 3 sinks, fan-out, merge, pipeline-builder, instrument-transform, and node-adapters are all preserved as the implementation backbone — every BaseTransform subclass either subclasses or composes the equivalent functional logic via toTransformStream(), so no behavioural regression is introduced.

What changes

ADDED — plugin substrate (ported from main)

• ADDED: BaseTransform — abstract class with name, setOperation, setParams, getOperation, getParams, hook(name, fn), removeHook(name, fn), on(name, fn), off(name, fn), process<T>(input, options?) abstract, toTransformStream<TIn, TOut>(): TransformStream<TIn, TOut> bridge to WHATWG, dispose() for cleanup
• ADDED: TransformOptions, IBaseTransform, TransformHookPayload, TransformEventPayload types
• ADDED: StreamManager — pipe(...transforms), execute<T>(input, options?), clear(), getTransforms()
• ADDED: TransformFactory — registry + create(type, ...args) with TransformType discriminated union; built-in registrations for every bundled transform
• ADDED: BaseDuplexStream<R, W> primitive with StreamSource<R> and StreamSink<W> configuration
• ADDED: NetworkStreamImpl with id, protocol, metadata: Map<string, unknown>, closed flag, close() and reset()
• ADDED: factory functions createDuplexStream, createNetworkStream, createTransformStream

ADDED — bundled BaseTransform subclasses (port + new)

Ported from main:

• ADDED: CryptoTransform — pluggable streaming AEAD with engine, key, operation ("encrypt" | "decrypt"), hooks (crypto:before-chunk, crypto:after-chunk), events (crypto:start, crypto:chunk, crypto:complete, crypto:error), process(asyncIterable) for stream-of-frames, toTransformStream() for WHATWG bridge. Depends on @suaveplan/crypto streaming AEAD engines.
• ADDED: SplitJoinTransform — object-mode split with chunkSize + overlap; join reassembles with optional separator and removeOverlap; supports string / ArrayBuffer / array; type guards isSplittable, isJoinable

Existing 11 functional transforms migrated to BaseTransform subclasses (see § Renames):

• ADDED: MapTransform, FilterTransform, FlatMapTransform, BatchTransform, WindowTransform, ThrottleTransform, LimitTransform, DedupTransform, TapTransform, TextCodecTransform (encode/decode operations), CompressTransform (compress/decompress operations)

Genesis-original gap-fillers (new):

• ADDED: ChunkBytesTransform — byte-level fixed-frame re-framing with tail policy (emit | pad | drop)
• ADDED: CoalesceBytesTransform — merges small Uint8Array chunks until a byte threshold or max-latency timeout
• ADDED: SlidingBytesTransform — overlapping byte windows with size + stride
• ADDED: LengthPrefixEncodeTransform, LengthPrefixDecodeTransform — wire-protocol framing with 1/2/4/8-byte prefix and configurable byte-order

ADDED — pipeline ops (genesis-original)

• ADDED: partition<T>(source, predicate) → { truthy, falsy } single-pass split (no tee duplication)
• ADDED: route<T, K extends string>(source, keyFn, keys, options?) → Record<K, ReadableStream<T>>; onUnknownKey: 'drop' | 'error'
• ADDED: routeBy<T, K>(source, keyFn, keys, options?) → Map<K, ReadableStream<T>> (non-string-key variant)
• ADDED: zip<T extends readonly unknown[]>(streams) → tuple stream with onShortInput: 'close' | 'error' | 'pad' option
• ADDED: concat<T>(streams) → sequential drain
• ADDED: mergeOrdered<T>(streams, compare) → k-way merge of pre-sorted streams
• ADDED: safeBranch<TIn, TOut>(factory, options?) → wraps a sub-pipeline so its errors emit on EventBusLike instead of propagating

ADDED — sources, sinks, utilities

• ADDED: toAsyncIterable<T>(stream, signal?) sink
• ADDED: getDataFromFile(filePath) — Node-only file reader with runtime environment detection (throws in browser)
• ADDED: bufferToUint8Array(buffer) — handles both Node Buffer and ArrayBuffer
• ADDED: createUint8ArraySlice(array, start, end) — explicit-typed copy helper
• ADDED: createDataStream<T>(data, transformer, options?) — generic source factory used by StreamManager for non-stream inputs
• ADDED: processDataStream<T>(stream, options?) — generic sink factory
• ADDED: streamToUint8Array(stream), uint8ArrayToStream(data, options?), bufferToStream, streamToBuffer cross-runtime helpers
• ADDED: validateStreamIntegrity(stream) — chunk-validity sweep helper

MODIFIED — naming convention

Drop redundant -stream suffix on every module name inside the streams package. Implementation file basename matches the parent folder; class and function names follow PascalCase / camelCase conventions without the redundant suffix.

Old path	New path	Rationale
transforms/map-stream/map-stream.ts	transforms/map/map.ts	inside a streams package, "stream" suffix is redundant
transforms/filter-stream/filter-stream.ts	transforms/filter/filter.ts	same
transforms/flat-map-stream/	transforms/flat-map/	same
transforms/batch-stream/	transforms/batch/	same
transforms/window-stream/	transforms/window/	same
transforms/throttle-stream/	transforms/throttle/	same
transforms/limit-stream/	transforms/limit/	same
transforms/dedup-stream/	transforms/dedup/	same
transforms/tap-stream/	transforms/tap/	same
transforms/text-codec/	unchanged	already conforms
transforms/compress/	unchanged	already conforms
pipeline/pipeline-builder/	pipeline/builder/	redundant package-name prefix
instrument/instrument-transform/	instrument/transform/	redundant prefix
node/node-adapters.ts	node/adapters/adapters.ts	promoted to feature dir + redundant prefix dropped
errors/stream-error.ts	errors/error/error.ts	redundant prefix dropped, promoted to feature dir

The corresponding exported symbols are renamed in lockstep:

Old export	New export
mapStream	mapStream (kept — describes the function shape, not file naming) plus new MapTransform class
filterStream	filterStream plus FilterTransform
flatMapStream	flatMapStream plus FlatMapTransform
batchStream	batchStream plus BatchTransform
windowStream	windowStream plus WindowTransform
throttleStream	throttleStream plus ThrottleTransform
limitStream	limitStream plus LimitTransform
dedupStream	dedupStream plus DedupTransform
tapStream	tapStream plus TapTransform
compressStream, decompressStream	unchanged plus CompressTransform (operation = "compress" | "decompress")
encodeTextStream, decodeTextStream	unchanged plus TextCodecTransform
pipeline()	unchanged
fanOut, merge	unchanged
instrumentTransform	unchanged
fromNodeReadable, toNodeReadable, fromNodeWritable	unchanged
StreamPipeFailedError	renamed to StreamPipeFailedError (lives in errors/error/error.ts now)

The function-form factories (mapStream, filterStream, …) remain exported for tree-shaking and for direct WHATWG composition; the OO subclasses delegate to them via toTransformStream(). Both call paths are first-class. This is the only "two APIs" pattern this package permits, and it is justified because the OO subclass uses the functional factory under the hood — no parallel implementation, no drift surface.

ADDED — error codes

• ADDED: STREAM.FRAME_TOO_LARGE, STREAM.TRUNCATED_FRAME, STREAM.ROUTE_UNKNOWN_KEY, STREAM.ZIP_LENGTH_MISMATCH, STREAM.BRANCH_FAILED, STREAM.CRYPTO_DECRYPT_FAILED, STREAM.CRYPTO_INVALID_FRAME, STREAM.DUPLEX_CLOSED, STREAM.NETWORK_RESET, STREAM.FILE_READ_FAILED, STREAM.FILE_BROWSER_NOT_SUPPORTED

REMOVED

• REMOVED: nothing — the previous spec is preserved verbatim under MODIFIED Requirements. Symbols are renamed (with old paths gone) but exported names are kept on the function-form API.

Out of scope

• Audio-specific anything (sample frames, channel-aware ops, mixing, pitch-shift, dynamics-processor) — belongs in the upcoming audio-engine package as plugins, not in streams. Karaoke/radio transforms from main do not port over.
• Image / video / metadata transforms — already split into siblings (streams-metadata proposed; image/video to follow as separate changes).
• Worker offload (pipeInWorker) — deferred to a future streams-worker sibling package once audio-engine demands prove the transport shape.
• schema-registry integration of stream chunk contracts — deferred until @suaveplan/schema-registry ships the consumer hooks the streams chunk format would need.
• Mutation testing & contract testing layers — covered by separate enterprise-testing-mutation and enterprise-testing-contract changes; this change ships 100% line/branch/function coverage as the baseline.

Source mapping

This change derives from ../suaveplan/enterprise/streams.md (the generic enterprise-quality template applied to streams) and the canonical implementation at ../suaveplan/packages/universal/streams/src/ (the ground-truth API surface).

Requirement	Source	Notes
Pure TypeScript, runs in Bun + Node + browser	enterprise/streams.md L38	verbatim
WHATWG Streams baseline	enterprise/streams.md L40	verbatim
Zero-copy Uint8Array operations on hot paths	enterprise/streams.md L41	verbatim
Backpressure-first; bounded queues with declarative high/low watermarks	enterprise/streams.md L42	verbatim
AbortSignal propagates end-to-end	enterprise/streams.md L43	verbatim
Zod schemas for stream configuration and chunk contracts	enterprise/streams.md L44	partial — schema-registry integration deferred (see Out of scope)
Property-based tests via fast-check	enterprise/streams.md L45	verbatim
Deterministic — no Math.random() / Date.now() at module scope, injectable clock + RNG	enterprise/streams.md L46	verbatim
OpenTelemetry instrumentation	enterprise/streams.md L47	verbatim
Structured errors extending @suaveplan/error with stable codes	enterprise/streams.md L48	verbatim
Idempotency + retry primitives for network-backed adapters	enterprise/streams.md L49	applied to NetworkStreamImpl's reset semantics
Feature flags for risky transform rollouts	enterprise/streams.md L50	applied via FlagRegistry injection on factory; opt-in
Plugin / extension interface	enterprise/streams.md L74	satisfied by BaseTransform + TransformFactory
Observability surface — counters, histograms, spans on every externally-visible op	enterprise/streams.md L75	satisfied by instrumentTransform + per-class hook surface
Configurability — Zod-validated options with sensible defaults	enterprise/streams.md L76	verbatim
Performance — benchmark suite + CI regression gate	enterprise/streams.md L77	verbatim
Resilience — retries / timeouts / cancellation / circuit breakers	enterprise/streams.md L78	applied via @suaveplan/retry composition (caller-side)
Security — input validation, output encoding, zeroing of sensitive buffers	enterprise/streams.md L79	applied to CryptoTransform key zeroize on dispose
Migration story for public API changes	enterprise/streams.md L80	satisfied by the rename matrix above
Example app / story demonstrating full feature set	enterprise/streams.md L82	satisfied by examples/ runnable scripts

Requirements with no enterprise/streams.md line

These derive from ../suaveplan/packages/universal/streams/src/ (canonical implementation on main) and from the enterprise-streams-redesign audit conducted in the brainstorming session.

Requirement	Source	Justification
BaseTransform abstract class with hook + event surface	../suaveplan/packages/universal/streams/src/transforms/base/base.ts + base.md	canonical plugin shape — required for hooks/events that no functional API can express
StreamManager chainable pipeline	../suaveplan/packages/universal/streams/src/manager/manager.ts + manager.md	canonical
TransformFactory registry	../suaveplan/packages/universal/streams/src/transforms/factory/factory.ts + factory.md	canonical
BaseDuplexStream + NetworkStreamImpl	../suaveplan/packages/universal/streams/src/primitives/duplex.ts + duplex.md	canonical
CryptoTransform with pluggable streaming AEAD	../suaveplan/packages/universal/streams/src/transforms/crypto/crypto.ts + crypto.md	canonical; depends on @suaveplan/crypto streaming AEAD engines
SplitJoinTransform	../suaveplan/packages/universal/streams/src/transforms/split-join/split-join.ts + split-join.md	canonical
Buffer utilities (getDataFromFile, bufferToUint8Array, createUint8ArraySlice)	../suaveplan/packages/universal/streams/src/utils/buffer/buffer.md	canonical
Stream utilities (createDataStream, processDataStream, streamToUint8Array, uint8ArrayToStream, validateStreamIntegrity)	../suaveplan/packages/universal/streams/src/utils/stream/stream.md	canonical
chunkBytes, coalesceBytes, slidingBytes, lengthPrefixEncode/Decode	genesis audit (this change)	byte-level framing primitives; AEAD requires fixed-byte frames; main's SplitJoinTransform is object-mode only
partition, route, routeBy	genesis audit (this change)	single-pass split (fanOut duplicates via tee); essential for "split into different plugin chains by criterion"
zip, concat, mergeOrdered	genesis audit (this change)	merge variants beyond arrival-order race; zip enables sample-aligned recombination, mergeOrdered enables k-way time-sorted merge
safeBranch	genesis audit (this change)	branch error isolation — without it, one failing FX rack tears down the whole graph
toAsyncIterable sink	genesis audit (this change)	async iterator interop, missing from main
Naming convention: drop redundant -stream suffix	user direction (brainstorming session)	clarity inside a streams package
100% test coverage threshold	genesis-wide policy (openspec/AGENTS.md § 7)	non-package-specific
Source-mapped **Verified by:** lines	genesis-wide policy (openspec/AGENTS.md § 13)	non-package-specific

Verification approach

Every requirement in the spec delta carries a **Verified by:** line pointing at a concrete it(...) title in one of the test files below. Tests are co-located with implementation per the genesis 5-file convention.

• src/base/transform/transform.test.ts — BaseTransform lifecycle, hook + event surfaces, toTransformStream() bridge, dispose() semantics
• src/base/transform/transform.spec-conformance.test.ts — type-level: generic narrowing, hook-payload type inference, factory return-type preservation
• src/manager/manager/manager.test.ts — pipeline chaining, sequential execution, clear() reset, error propagation
• src/factory/factory/factory.test.ts — type-discriminated create(), registry lookup, unknown-type error
• src/primitives/duplex/duplex.test.ts — bidirectional flow, byte vs object mode, cancel() semantics
• src/primitives/network/network.test.ts — id, protocol, metadata, close(), reset() with error
• src/transforms/<name>/<name>.test.ts — per-transform behaviour (one file per BaseTransform subclass)
• src/transforms/crypto/crypto.test.ts + crypto.spec-conformance.test.ts — round-trip encrypt/decrypt, hook payloads, event firing order, reorder/truncate detection
• src/pipeline/<op>/<op>.test.ts — per-op behaviour
• src/sinks/to-async-iterable/to-async-iterable.test.ts — async-iterator yields all chunks; signal cancels
• src/utils/buffer/<helper>/<helper>.test.ts — Node + browser branches; env-detection error
• src/utils/stream/<helper>/<helper>.test.ts — round-trip identity, integrity validation
• src/instrument/transform/transform.test.ts — span + counter + histogram emitted via in-memory OTel MeterProvider
• src/node/adapters/adapters.test.ts — Node ↔ WHATWG bridge round-trip
• tests/integration.test.ts — full pipelines: encrypt → length-prefix → decrypt round-trip; partition + safe-branch + merge composition
• tests/property.test.ts — chunkBytes byte-conservation, partition order preservation, zip/unzip round-trip, mergeOrdered sortedness invariant
• tests/dep-graph.test.ts — only declared workspace deps reachable; no node:fs import in browser entry
• tests/tree-shake.test.ts — sideEffects: false round-trip via Bun.build

Migration impact

This package has consumers on genesis already — every package that imports from @suaveplan/streams. Audit:

grep -rln "@suaveplan/streams" packages/ --include="package.json"

The function-form API (mapStream, filterStream, pipeline, fanOut, merge, instrumentTransform, fromNodeReadable, toNodeReadable, fromNodeWritable, compressStream, decompressStream, encodeTextStream, decodeTextStream, toArray, toBytes, forEach, fromArray, fromBytes, fromAsyncGenerator) is preserved. Internal subpath imports change:

Old subpath import	New subpath import
@suaveplan/streams/transforms/map-stream/map-stream	@suaveplan/streams/transforms/map/map
@suaveplan/streams/transforms/filter-stream/filter-stream	@suaveplan/streams/transforms/filter/filter
(… same pattern for the renamed transforms above)
@suaveplan/streams/pipeline/pipeline-builder/pipeline-builder	@suaveplan/streams/pipeline/builder/builder
@suaveplan/streams/instrument/instrument-transform/instrument-transform	@suaveplan/streams/instrument/transform/transform
@suaveplan/streams/node/node-adapters	@suaveplan/streams/node/adapters/adapters
@suaveplan/streams/errors/stream-error	@suaveplan/streams/errors/error/error

Consumer migration is part of this change. Every internal consumer is updated in the same change folder; no shim or deprecation alias remains. The barrel @suaveplan/streams (root export) is unaffected — top-level imports keep working.

Breaking changes: subpath imports only (top-level barrel preserved). New OO subclasses are pure additions.

Prerequisites: @suaveplan/error, @suaveplan/types, @suaveplan/events, @suaveplan/crypto (for CryptoTransform) must be at their current genesis versions on Verdaccio.