Move docs/ package mirror into document-navigator plugin

Relocate the ~6k-file docs tree from the repo root into
plugins/document-navigator/docs/ so the plugin is self-contained and
portable. docs/superpowers/ (specs and plans) stays at the repo root.

Moved via cp+rm rather than git mv because directory rename() fails with
EPERM on this WSL /mnt DrvFs mount; git rename-detection pairs the
identical-content files at commit time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-30 01:10:00 +01:00
parent fe3fedf42e
commit 9bf56a806b
6022 changed files with 12437 additions and 0 deletions

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,176 @@
# BaseTransform
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`BaseTransform` is the abstract plugin contract every stream transform in `@suaveplan/streams` extends. It provides a uniform, type-safe surface for hooks, events, abort handling, WHATWG-Streams interop, and disposal — so subclasses (such as `CryptoTransform`, `MapTransform`, `SplitJoinTransform`) can focus on their actual operation without re-implementing pipeline plumbing.
The class is parameterized by two maps: `THooks extends HookMap` declares hook names and their payload types, and `TEvents extends EventMap` declares event names and their payload types. This means consumers who register hooks or listeners get full TypeScript inference on the payload — no `any`, no string-typed callbacks. Subclasses that don't need declared hooks or events can omit both type parameters and use the lenient defaults.
## Features
- Abstract `process<T>(input, options?)` — every subclass declares its own operation
- Type-safe `hook(name, fn)` / `removeHook(name, fn)` with payload-chaining: each hook receives the previous hook's return value
- Type-safe `on(name, fn)` / `off(name, fn)` — fire-and-forget event listeners, sync, with errors swallowed and forwarded to `options.logger.error`
- Protected `applyHooks(name, payload, options?)` and `emit(name, payload, options?)` helpers for subclasses to fire hooks/events at appropriate points in their `process` implementation
- Protected `assertNotAborted(options)` helper that throws `StreamAbortedError` when `options.signal` has fired
- Default `toTransformStream<TIn, TOut>()` bridge — pipes WHATWG `ReadableStream` chunks through `process(chunk)`
- Idempotent `dispose()` — clears all hooks and listeners; safe to call twice
- Strict `TransformOptionsSchema` rejects unknown options at the boundary
## Usage: Basic Example
```ts
import { BaseTransform } from "@suaveplan/streams/base/transform/transform";
class UpperTransform extends BaseTransform {
constructor() {
super("upper");
}
async process<T = unknown>(input: T): Promise<T> {
if (typeof input !== "string") return input;
return input.toUpperCase() as unknown as T;
}
}
const t = new UpperTransform();
const out = await t.process("hello"); // "HELLO"
```
## Usage: Advanced Example
```ts
import { BaseTransform } from "@suaveplan/streams/base/transform/transform";
import type { TransformOptions } from "@suaveplan/streams/base/transform/transform.types";
interface CryptoHooks {
"before-chunk": { chunk: Uint8Array; counter: number };
"after-chunk": { chunk: Uint8Array; counter: number };
}
interface CryptoEvents {
start: { engineId: string };
chunk: { counter: number; bytes: number };
complete: { totalChunks: number };
}
class XorTransform extends BaseTransform<CryptoHooks, CryptoEvents> {
constructor(private readonly key: Uint8Array) {
super("xor-demo");
}
async process<T = unknown>(
input: T,
options?: TransformOptions
): Promise<T> {
if (!(input instanceof Uint8Array)) {
throw new TypeError("XorTransform requires Uint8Array");
}
this.emit("start", { engineId: "xor" }, options);
let counter = 0;
const before = await this.applyHooks(
"before-chunk",
{ chunk: input, counter },
options
);
this.assertNotAborted(options);
const out = before.chunk.map(
(b, i) => b ^ this.key[i % this.key.length]!
) as Uint8Array;
const after = await this.applyHooks(
"after-chunk",
{ chunk: out, counter },
options
);
counter += 1;
this.emit(
"chunk",
{ counter, bytes: after.chunk.byteLength },
options
);
this.emit("complete", { totalChunks: counter }, options);
return after.chunk as unknown as T;
}
}
const t = new XorTransform(new Uint8Array([0xaa]));
t.hook("before-chunk", (p) => ({ ...p, chunk: p.chunk }));
t.on("chunk", (p) => console.log(`chunk ${p.counter}: ${p.bytes} bytes`));
// Direct call
const sealed = await t.process(new Uint8Array([1, 2, 3]));
// WHATWG bridge
const source = new ReadableStream<Uint8Array>({ /* ... */ });
const transformed = source.pipeThrough(
t.toTransformStream<Uint8Array, Uint8Array>()
);
```
## API Reference
### `BaseTransform<THooks, TEvents>`
Abstract class. Type parameters:
- `THooks extends HookMap` — map of hook names to payload types. Default: `HookMap` (loose).
- `TEvents extends EventMap` — map of event names to payload types. Default: `EventMap` (loose).
Constructor: `(name: string)` — assigned to public readonly `name`.
#### `process<T>(input: T, options?: TransformOptions): Promise<T>`
Abstract. Subclasses implement the operation. Implementations SHOULD call `this.assertNotAborted(options)` at suitable yield points.
#### `hook<K>(name: K, fn: Hook<THooks[K]>): Unsubscribe`
Registers a payload-chaining hook under `name`. Returns an `Unsubscribe` function that removes this exact registration.
#### `removeHook<K>(name: K, fn: Hook<THooks[K]>): boolean`
Removes a previously registered hook. Returns `true` if the hook was found and removed; `false` if it wasn't registered.
#### `on<K>(name: K, fn: Listener<TEvents[K]>): Unsubscribe`
Registers a fire-and-forget event listener.
#### `off<K>(name: K, fn: Listener<TEvents[K]>): boolean`
Removes a registered listener. Returns `true` on success.
#### `protected applyHooks<K>(name, payload, options?): Promise<THooks[K]>`
Walks the hook chain registered under `name`. Each hook receives the prior hook's return value. Returns the final payload. Honors `options.signal` between hooks.
#### `protected emit<K>(name, payload, options?): void`
Notifies listeners synchronously. Listener errors are caught and forwarded to `options.logger.error` (if present). Never throws.
#### `protected assertNotAborted(options?): void`
Throws `StreamAbortedError` when `options.signal.aborted` is true. The thrown error's `cause` is `options.signal.reason`.
#### `toTransformStream<TIn, TOut>(options?): TransformStream<TIn, TOut>`
Default WHATWG bridge. Each input chunk is passed through `this.process(chunk, options)`. Errors from `process` flow into the readable side via `controller.error`. Subclasses MAY override for streaming semantics (e.g. AEAD frame-counter binding).
#### `dispose(): void`
Clears all hooks and listeners. Idempotent — calling twice is a no-op. Subclasses overriding `dispose` MUST call `super.dispose()`.
### `TransformOptionsSchema`
Strict Zod schema for `TransformOptions`. Rejects unknown keys, validates `signal` as an `AbortSignal`, `clock` as a function, and `logger` as an object with `debug`, `info`, `warn`, `error` methods.
## Implementation Notes
The class stores hooks and listeners in `Map<string, Function[]>` rather than per-name fields so subclass-declared hook/event names can be added without fields conflicting. Hooks fire **in registration order** — this is observable behavior pinned by `transform.test.ts::"hooks fire in registration order with payload chaining"` and is part of the spec contract.
`emit` is synchronous and swallows listener errors so a misbehaving observer never affects `process`. This matches the suaveplan convention — events are _passive observability_, never load-bearing flow control. Hooks, by contrast, are load-bearing: a hook return value flows through the chain, and a hook throw aborts `process`.
`toTransformStream` returns a fresh `TransformStream` per call but all streams share the same hook/listener state on the underlying `BaseTransform` instance — so registering a hook before constructing the stream affects every subsequent stream.
`dispose` is intentionally minimal — it does not cancel pending `process` calls. Cancellation is the caller's responsibility via `AbortSignal`. Subclasses with additional disposable resources (e.g. `CryptoTransform` zeroizing its key buffer) override `dispose` and call `super.dispose()` to chain cleanup.
The design intentionally avoids depending on `@suaveplan/events`, even though that package provides richer event/hook primitives. The streams package is a foundational layer; pulling in events would create an awkward dependency cycle for any consumer that wants to compose streams + events without the indirection. The minimal inline implementation here is sufficient for the spec scenarios and faster on the hot path.

View File

@@ -0,0 +1,101 @@
# stream-error
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Defines the structured error hierarchy for the `@suaveplan/streams` package. Every runtime error emitted by stream operations is an instance of `StreamError` (which extends `SuaveplanError`) and carries a stable, dot-namespaced error code from the `STREAM.*` registry. This design means callers can narrow errors programmatically — `error.code === STREAM_CODES.ABORTED` — without string-matching unstructured messages, and ensures that OTel integrations, log pipelines, and dead-letter queues can route stream errors reliably.
The codes are registered once via `defineCodes("STREAM", [...])` and reused across every throw site. Codes are intentionally coarse-grained: they represent the semantic category of failure, not the specific call stack location, so they remain stable across refactors.
## Features
- Six concrete error subclasses covering all stream failure modes
- Stable `STREAM.*` error code registry via `defineCodes`
- Exported `StreamErrorCode` union type for exhaustive switch statements
- Each subclass propagates `cause` when it is an `Error` instance
- All classes extend `StreamError` which extends `SuaveplanError`, compatible with the project error wire format
## Usage: Basic Example
```ts
import { StreamAbortedError, STREAM_CODES } from "@suaveplan/streams/errors/stream-error";
function checkAborted(signal?: AbortSignal): void {
if (signal?.aborted) {
throw new StreamAbortedError({ cause: signal.reason });
}
}
try {
checkAborted(AbortSignal.abort());
} catch (err) {
if (err instanceof StreamAbortedError) {
console.log(err.code); // "STREAM.ABORTED"
}
}
```
## Usage: Advanced Example
```ts
import {
StreamTransformFailedError,
StreamClosedError,
StreamReadFailedError,
STREAM_CODES,
} from "@suaveplan/streams/errors/stream-error";
function handleStreamError(err: unknown): never {
if (!(err instanceof Error)) {
throw new StreamReadFailedError({ message: "Unknown read failure" });
}
switch ((err as { code?: string }).code) {
case STREAM_CODES.CLOSED:
throw new StreamClosedError({ message: "Cannot read from closed stream" });
case STREAM_CODES.TRANSFORM_FAILED:
throw new StreamTransformFailedError({ cause: err });
default:
throw new StreamReadFailedError({ cause: err });
}
}
```
## API Reference
### `STREAM_CODES`
`Record<string, string>` — the stable code map. Keys: `ABORTED`, `BACKPRESSURE_EXCEEDED`, `CHUNK_INVALID`, `PIPE_FAILED`, `TRANSFORM_FAILED`, `SOURCE_EXHAUSTED`, `READ_FAILED`, `WRITE_FAILED`, `CLOSED`. Values are dot-namespaced strings like `"STREAM.ABORTED"`.
### `StreamError`
Base class. Extends `SuaveplanError<StreamErrorCode>`. All concrete subclasses inherit from this.
### `StreamAbortedError`
Thrown when an `AbortSignal` fires mid-operation. Constructor: `{ cause?: unknown; message?: string }`.
### `StreamTransformFailedError`
Thrown when a user-supplied transform function throws. Constructor: `{ cause?: unknown; message?: string }`.
### `StreamClosedError`
Thrown when an operation is attempted on a closed stream. Constructor: `{ message?: string }`.
### `StreamReadFailedError`
Thrown on unrecoverable read failures. Constructor: `{ cause?: unknown; message?: string }`.
### `StreamWriteFailedError`
Thrown on unrecoverable write failures. Constructor: `{ cause?: unknown; message?: string }`.
### `StreamPipeFailedError`
Thrown when a pipe or pipeline composition fails. Constructor: `{ cause?: unknown; message?: string }`.
## Implementation Notes
`defineCodes` generates stable string codes by joining the namespace prefix and the key with a dot. The generated values are frozen at module-init time and never change, which means error codes survive across serialization round-trips (they appear verbatim in JSON error envelopes). Cause propagation is conditional: we only set `cause` when the original value `instanceof Error` to avoid accidentally serializing non-Error objects into the error chain.

View File

@@ -0,0 +1,133 @@
# TransformFactory
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`TransformFactory` is the open registry that lets consumers construct `BaseTransform` instances by string type rather than direct class import. It exists for two reasons: first, to enable JSON / dataflow-driven pipeline assembly where the transform name is data and not source code (e.g. a saved DAW project that says `[{ type: "crypto", op: "encrypt" }, { type: "compress" }]`); second, to keep the `streams` package tree-shake-friendly — every built-in transform self-registers from its own barrel module, so a consumer that never imports `crypto/crypto.js` never pays for the AEAD engine code.
The registry is global per process (a single static `Map<string, TransformFactoryFn>` shared by the static and instance APIs). Built-in subclasses (the `MapTransform`, `CryptoTransform`, etc. that land in M2/M3) call `TransformFactory.register(name, fn)` at module load. Consumers who want their own discriminated types call the same method.
## Features
- Static API (`TransformFactory.create`, `register`, `unregister`, `has`, `types`) for direct usage
- Instance API (`new TransformFactory().create(...)` etc.) for DI patterns; delegates to the same shared registry
- Generic-narrowed `create<T>(type, ...args): T` so call sites get the precise subclass type without `instanceof` casting
- `StreamUnknownTransformTypeError` (`STREAM.UNKNOWN_TRANSFORM_TYPE`) thrown when a type is not registered, with the offending type recorded in `error.context.transformType`
- `register(type, fn)` replaces any prior entry under the same type (last write wins) — useful for tests and for plugin override scenarios
- `types()` snapshot returned in insertion order so callers can render deterministic UIs
## Usage: Basic Example
```ts
import { TransformFactory } from "@suaveplan/streams/factory/factory/factory";
import { BaseTransform } from "@suaveplan/streams/base/transform/transform";
class UpperTransform extends BaseTransform {
constructor() {
super("upper");
}
async process<T>(input: T): Promise<T> {
return (typeof input === "string"
? input.toUpperCase()
: input) as unknown as T;
}
}
TransformFactory.register("upper", () => new UpperTransform());
const t = TransformFactory.create("upper");
const out = await t.process("hello"); // "HELLO"
```
## Usage: Advanced Example
```ts
import { TransformFactory } from "@suaveplan/streams/factory/factory/factory";
import { StreamManager } from "@suaveplan/streams/manager/manager/manager";
import {
STREAM_CODES,
StreamUnknownTransformTypeError,
} from "@suaveplan/streams/errors/stream-error";
// Saved pipeline definition coming off the wire:
type SavedStage = { type: string; args?: unknown[] };
function build(stages: readonly SavedStage[]): StreamManager {
const transforms = stages.map((stage) => {
try {
return TransformFactory.create(stage.type, ...(stage.args ?? []));
} catch (err) {
if (err instanceof StreamUnknownTransformTypeError) {
console.warn(
`Skipping unknown transform: ${err.context?.transformType}`
);
throw err;
}
throw err;
}
});
return new StreamManager().pipe(...transforms);
}
const m = build([
{ type: "decode-text" },
{ type: "upper" },
{ type: "encode-text" },
]);
// Generic narrowing at the call site
const crypto = TransformFactory.create<CryptoTransform>("crypto", {
engine,
key,
operation: "encrypt",
});
crypto.hook("crypto:before-chunk", (p) => p);
```
## API Reference
### `TransformFactory`
#### `static create<T>(type: string, ...args: unknown[]): T`
Looks up the factory function for `type` and invokes it with `args`. Returns a fresh `BaseTransform` (or narrower subclass `T`).
**Throws:** `StreamUnknownTransformTypeError` (`STREAM.UNKNOWN_TRANSFORM_TYPE`) if `type` is not registered. The error carries `context.transformType` for diagnostics.
#### `static register(type: string, factory: TransformFactoryFn): void`
Adds (or replaces) a factory under `type`. The factory receives the variadic `args` from `create` and must return a `BaseTransform` instance.
#### `static unregister(type: string): boolean`
Removes a registration. Returns `true` on success, `false` if `type` was not present.
#### `static has(type: string): boolean`
Returns whether `type` is currently registered.
#### `static types(): readonly string[]`
Snapshot of registered type names in insertion order.
#### Instance API
`new TransformFactory()` exposes the same five methods as instance methods that delegate to the static registry. Use the instance form when threading the factory through DI containers; use the static form for direct calls.
### `TransformFactoryFn`
```ts
type TransformFactoryFn = (...args: unknown[]) => BaseTransform;
```
## Implementation Notes
The registry is a single static `Map` shared between the static and instance APIs. This is intentional: a process-wide factory means the saved-pipeline pattern works across module boundaries without consumers having to thread a registry instance everywhere. If multiple isolated registries become useful (e.g. per-tenant transform whitelists in a multi-tenant server), a future change can introduce an explicit `TransformRegistry` class — but until that requirement surfaces, a single shared map is the simplest correct design.
`create<T>` accepts a generic type parameter that narrows the return statically without changing the runtime: it's a bare `as T` cast inside `create`, equivalent to `instanceof` checking but at the call site. Callers that want runtime safety should still `instanceof` the result; consumers that trust their registration are free to skip the check.
Built-in subclasses self-register at module-import time via a side-effecting line in their `index.ts` (e.g. `TransformFactory.register("map", () => new MapTransform(...))`). This means importing the `streams` root barrel registers all built-ins; importing a single subclass via subpath registers only that one. The registration call-out is idempotent under last-write-wins semantics, so re-importing is safe.
The factory does **not** validate that the returned instance is actually a `BaseTransform`. It trusts the registered function. If a consumer registers a faulty factory the cast at the call site will surface the type error; this matches the open-registry pattern in the rest of Suaveplan (`@suaveplan/events` `EventBus`, `@suaveplan/feature-flags` `FlagRegistry`, etc.).

View File

@@ -0,0 +1,74 @@
# instrument-transform
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`instrumentTransform` wraps any `TransformStream<TIn, TOut>` with OpenTelemetry instrumentation: a span per chunk (`stream.transform.<name>`), an optional chunk counter (`stream.chunks_total`), and an optional processing duration histogram (`stream.transform_duration_ms`). Callers inject an OTel `Tracer` and `Meter` instance; when these are not provided, the global no-op tracer is used, so the function is always safe to call regardless of whether OTel is configured.
This is the standard way to add observability to any pipeline stage without modifying the stage's own implementation.
## Features
- Per-chunk OTel span with configurable stage name and attributes
- `stream.chunks_total` counter (attribute: `stream.stage`)
- `stream.transform_duration_ms` histogram (attribute: `stream.stage`)
- No-op when tracer/meter are not provided
- Works with any `TransformStream` without modifying it
## Usage: Basic Example
```ts
import { instrumentTransform } from "@suaveplan/streams/instrument/instrument-transform/instrument-transform";
import { mapStream } from "@suaveplan/streams/transforms/map-stream/map-stream";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const result = await toArray(
fromArray([1, 2, 3]).pipeThrough(
instrumentTransform(mapStream((n) => n * 2), "double")
)
);
console.log(result); // [2, 4, 6]
```
## Usage: Advanced Example
```ts
import { instrumentTransform } from "@suaveplan/streams/instrument/instrument-transform/instrument-transform";
import { trace, metrics } from "@opentelemetry/api";
import { mapStream } from "@suaveplan/streams/transforms/map-stream/map-stream";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const tracer = trace.getTracer("my-service");
const meter = metrics.getMeter("my-service");
const transform = instrumentTransform(
mapStream((n: number) => n * 2),
"double",
{ tracer, meter, attributes: { "service.name": "my-service" } }
);
const result = await toArray(fromArray([1, 2, 3]).pipeThrough(transform));
// Spans and metrics recorded for each chunk
```
## API Reference
### `instrumentTransform<TIn, TOut>(transform, name, options?): TransformStream<TIn, TOut>`
**Parameters**:
- `transform` — the transform stream to instrument.
- `name` — stage name; used in span names (`stream.transform.<name>`) and metric attributes.
- `options.tracer` — OTel `Tracer` instance (default: global no-op).
- `options.meter` — OTel `Meter` instance (default: none; counters/histograms omitted).
- `options.attributes` — additional span/metric attributes.
**Returns**: A new `TransformStream<TIn, TOut>` with OTel instrumentation wrapping the original.
## Implementation Notes
The instrumented transform is implemented by creating separate `ReadableStream` and `WritableStream` wrappers. The writable side writes each chunk to the original writable, measuring duration and recording metrics. The readable side pumps from the original readable into a new controller. This approach avoids modifying the transform's internal implementation while still providing full observability.

View File

@@ -0,0 +1,137 @@
# StreamManager
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`StreamManager` is the high-level pipeline orchestrator for `@suaveplan/streams`. It wraps a list of `BaseTransform` instances and threads an input value through their `process` methods sequentially, propagating `ExecuteOptions` (including `AbortSignal`) end-to-end. It is the canonical entry point when callers want a one-shot "transform this value through these N stages" call without managing WHATWG `ReadableStream` plumbing themselves.
The manager is intentionally minimal: it does not own its own event surface, queue, or scheduler. It composes `BaseTransform` instances exactly as constructed, so all observability (hooks, events, OTel spans) lives where it should — on the individual transforms — not bolted on at the orchestrator layer. This keeps `StreamManager` cheap to allocate, predictable, and trivially serializable as a pipeline definition.
## Features
- Chainable `pipe(...transforms)` / `clear()` / `getTransforms()` API
- `execute<T>(input, options?)` runs the pipeline and returns the final value
- Errors thrown by individual transforms are wrapped in `StreamTransformFailedError` so callers see a stable code; existing `StreamError` subclasses (e.g. `StreamAbortedError`) propagate unchanged so abort and source-specific failures keep their semantic identity
- `AbortSignal` is checked between every stage, not just at start
- Empty pipeline returns input unchanged with referential equality
- `pipeline` getter exposes the live list (`readonly BaseTransform[]`) for callers that want to introspect the pipeline shape
## Usage: Basic Example
```ts
import { StreamManager } from "@suaveplan/streams/manager/manager/manager";
import { BaseTransform } from "@suaveplan/streams/base/transform/transform";
class UpperTransform extends BaseTransform {
constructor() {
super("upper");
}
async process<T>(input: T): Promise<T> {
return (typeof input === "string"
? input.toUpperCase()
: input) as unknown as T;
}
}
class ExclaimTransform extends BaseTransform {
constructor() {
super("exclaim");
}
async process<T>(input: T): Promise<T> {
return `${String(input)}!` as unknown as T;
}
}
const out = await new StreamManager()
.pipe(new UpperTransform(), new ExclaimTransform())
.execute("hello");
// "HELLO!"
```
## Usage: Advanced Example
```ts
import { StreamManager } from "@suaveplan/streams/manager/manager/manager";
import { BaseTransform } from "@suaveplan/streams/base/transform/transform";
import { STREAM_CODES } from "@suaveplan/streams/errors/stream-error";
const ctrl = new AbortController();
const m = new StreamManager().pipe(
new SlowTransform(),
new EncryptTransform(key),
new RedactTransform()
);
setTimeout(() => ctrl.abort(new Error("user cancelled")), 50);
try {
const out = await m.execute(payload, {
signal: ctrl.signal,
clock: () => Date.now(),
logger,
});
} catch (err) {
if ((err as { code?: string }).code === STREAM_CODES.ABORTED) {
// User cancelled mid-pipeline; cleanup
} else if ((err as { code?: string }).code === STREAM_CODES.TRANSFORM_FAILED) {
// A transform threw a non-StreamError; cause is on err.cause
}
}
// Re-use the manager for a different input — pipeline persists across calls
const second = await m.execute(otherPayload);
// Or swap the pipeline entirely
m.clear().pipe(new ProbeTransform()).execute("ping");
```
## API Reference
### `StreamManager`
#### `pipe(...transforms: BaseTransform[]): StreamManager`
Replaces the current pipeline with `transforms` in order. Returns `this` for chaining. Calling `pipe` more than once swaps the entire pipeline; use `clear()` followed by `pipe()` if you prefer to be explicit, but both produce the same result.
#### `execute<T>(input: T, options?: ExecuteOptions): Promise<T>`
Threads `input` through every transform's `process(value, options)` in order. Returns the final value. Errors:
- `StreamAbortedError` (`STREAM.ABORTED`) — `options.signal` was already aborted before execution began, or fired between two stages. The signal is also forwarded to each transform's `process` call so subclasses honoring `assertNotAborted` reject mid-stage.
- `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`) — a transform threw a non-`StreamError`. The original throw is preserved on `error.cause`; the failing transform's `name` appears in the message.
- Any `StreamError` subclass thrown by a transform (e.g. `StreamReadFailedError`) propagates verbatim — no double-wrap.
#### `clear(): StreamManager`
Empties the pipeline. Returns `this` for chaining.
#### `getTransforms(): readonly BaseTransform[]`
Returns the current transform list, in order. Useful for testing and introspection.
#### `pipeline: readonly BaseTransform[]`
Read-only getter equivalent to `getTransforms()`. The two coexist because `pipeline` reads cleaner inline (`m.pipeline.length`) while `getTransforms()` is preferred from explicit interface code paths.
### `ExecuteOptions`
```ts
interface ExecuteOptions {
signal?: AbortSignal;
clock?: Clock;
logger?: LoggerLike;
}
```
This shape is identical to `TransformOptions` so the same options object can be threaded straight through every stage without restructuring.
## Implementation Notes
The manager checks `options.signal` three times per stage: once before the loop, once at the top of every iteration, and once after the final stage. The redundancy is intentional — it ensures abort latency is bounded by one transform's runtime, not by the entire pipeline length.
`StreamError` subclasses are caught and re-thrown unwrapped so consumers that already match on specific codes (e.g. `STREAM.ABORTED`, `STREAM.READ_FAILED`) keep their semantic match. Only generic `Error` instances become `StreamTransformFailedError` — that distinction means `try { await m.execute(...) } catch (e) { if (e.code === STREAM.READ_FAILED) ... }` continues to work even if the failing transform is buried mid-pipeline.
The pipeline list is a plain array; `pipe(...transforms)` replaces it via assignment, not mutation, so consumers holding a prior reference (via `getTransforms()` or `pipeline`) still see the snapshot they captured. This is convenient for diagnostics that want to log "pipeline at error time" without racing against `clear()`.
`StreamManager` does not currently own a benchmark suite of its own — its overhead is dominated by the transforms it wraps, so benchmarks live alongside the heavy-lift transforms (`CryptoTransform`, byte-framing) rather than here.

View File

@@ -0,0 +1,61 @@
# node-adapters
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides bidirectional adapters between Node.js streams (`node:stream` `Readable` / `Writable`) and WHATWG streams (`ReadableStream<Uint8Array>` / `WritableStream<Uint8Array>`). These adapters are essential for interoperating with the Node.js ecosystem — reading files via `fs.createReadStream`, writing to HTTP response streams, or integrating with third-party Node.js stream-based libraries — while keeping the rest of the pipeline in standard WHATWG stream primitives.
All adapters handle the three critical lifecycle events: data flow, end-of-stream, and error propagation. Node.js `Buffer` instances are converted to `Uint8Array` views without copying.
## Features
- `fromNodeReadable`: Node.js `Readable` → `ReadableStream<Uint8Array>` (zero-copy Buffer views)
- `toNodeReadable`: `ReadableStream<Uint8Array>` → Node.js `Readable`
- `fromNodeWritable`: Node.js `Writable` → `WritableStream<Uint8Array>`
- Errors wrapped as `StreamPipeFailedError` on both sides
- Stream cancellation propagates to the underlying Node.js stream via `destroy()`
## Usage: Basic Example
```ts
import { fromNodeReadable } from "@suaveplan/streams/node/node-adapters";
import { createReadStream } from "node:fs";
import { toBytes } from "@suaveplan/streams/sinks/to-bytes/to-bytes";
const nodeStream = createReadStream("/path/to/file.bin");
const whatwgStream = fromNodeReadable(nodeStream);
const bytes = await toBytes(whatwgStream);
console.log(bytes.byteLength);
```
## Usage: Advanced Example
```ts
import { fromNodeReadable, fromNodeWritable } from "@suaveplan/streams/node/node-adapters";
import { createReadStream, createWriteStream } from "node:fs";
import { compressStream } from "@suaveplan/streams/transforms/compress/compress";
const readStream = fromNodeReadable(createReadStream("input.txt"));
const writeStream = fromNodeWritable(createWriteStream("output.gz"));
await readStream.pipeThrough(compressStream("gzip")).pipeTo(writeStream);
```
## API Reference
### `fromNodeReadable(readable: Readable): ReadableStream<Uint8Array>`
Wraps a Node.js `Readable` in a WHATWG `ReadableStream`. `Buffer` chunks are converted to zero-copy `Uint8Array` views.
### `toNodeReadable(stream: ReadableStream<Uint8Array>): Readable`
Creates a Node.js `Readable` that pulls from a WHATWG stream. Cancellation propagates via `destroy()`.
### `fromNodeWritable(writable: Writable): WritableStream<Uint8Array>`
Wraps a Node.js `Writable` in a WHATWG `WritableStream`. Write callbacks resolve/reject based on the Node.js callback.
## Implementation Notes
`Buffer` to `Uint8Array` conversion uses `new Uint8Array(chunk.buffer, chunk.byteOffset, chunk.byteLength)` which creates a view into the same underlying `ArrayBuffer` without copying. Error events from the Node.js stream are caught and re-thrown as `StreamPipeFailedError` with the original error as the cause. The `cancel` method calls `readable.destroy()` which immediately terminates the Node.js stream and triggers its `destroy` callback.

View File

@@ -0,0 +1,90 @@
# pipeline-builder
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`PipelineBuilder` is a fluent, type-safe wrapper for composing WHATWG stream pipelines. Starting from a `ReadableStream<T>`, each call to `.pipe()` returns a new `PipelineBuilder<TNext>` typed to the transform's output, making the full pipeline statically typed from source to sink. Terminal methods (`collect`, `drain`, `forEach`, `drainToWritable`) consume the pipeline and return a Promise.
The builder is a convenience layer over `ReadableStream.pipeThrough` — it does not buffer, copy, or re-implement backpressure. Every `pipe()` call delegates directly to the WHATWG `pipeThrough` method.
## Features
- Fluent `.pipe(transform)` and `.pipe(fn)` chaining with full TypeScript inference
- Terminal: `collect()`, `drain()`, `forEach()`, `drainToWritable()`
- `toReadable()` escape hatch for raw stream access
- Optional global `AbortSignal` in `PipelineOptions` applied to all terminals
- Per-terminal signal override
## Usage: Basic Example
```ts
import { pipeline } from "@suaveplan/streams/pipeline/pipeline-builder/pipeline-builder";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { mapStream } from "@suaveplan/streams/transforms/map-stream/map-stream";
import { filterStream } from "@suaveplan/streams/transforms/filter-stream/filter-stream";
const result = await pipeline(fromArray([1, 2, 3, 4, 5]))
.pipe(filterStream((n) => n % 2 === 0))
.pipe(mapStream((n) => n * 10))
.collect();
console.log(result); // [20, 40]
```
## Usage: Advanced Example
```ts
import { pipeline } from "@suaveplan/streams/pipeline/pipeline-builder/pipeline-builder";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
const ac = new AbortController();
const result = await pipeline(fromArray(["hello", "world"]), { signal: ac.signal })
.pipe((s) => s.toUpperCase()) // inline function
.pipe((s) => `[${s}]`) // another inline function
.collect();
console.log(result); // ["[HELLO]", "[WORLD]"]
// Drain to writable
const received: string[] = [];
const writable = new WritableStream<string>({ write(c) { received.push(c); } });
await pipeline(fromArray(["a", "b"])).drainToWritable(writable);
console.log(received); // ["a", "b"]
```
## API Reference
### `pipeline<T>(source: ReadableStream<T>, options?: PipelineOptions): PipelineBuilder<T>`
Creates a new builder from `source`.
### `PipelineBuilder<T>.pipe<TNext>(transform): PipelineBuilder<TNext>`
Applies a `TransformStream<T, TNext>` or a mapping function `(chunk: T) => TNext | Promise<TNext>`.
### `PipelineBuilder<T>.collect(signal?): Promise<T[]>`
Materialises all output chunks into an array.
### `PipelineBuilder<T>.drain(signal?): Promise<void>`
Discards all output chunks.
### `PipelineBuilder<T>.forEach(fn, signal?): Promise<void>`
Calls `fn` for each output chunk.
### `PipelineBuilder<T>.drainToWritable(writable, signal?): Promise<void>`
Pipes all output to a `WritableStream<T>`.
### `PipelineBuilder<T>.toReadable(): ReadableStream<T>`
Returns the underlying `ReadableStream` for raw access.
## Implementation Notes
The inline function overload of `pipe` wraps the function in a `new TransformStream` with a single `transform` callback. This is identical in behaviour to `mapStream` but allows callers to write terse one-liners. Type inference works because the overload signatures have distinct type parameters and TypeScript's overload resolution picks the correct one based on the argument type.

View File

@@ -0,0 +1,113 @@
# concat
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`concat` joins multiple `ReadableStream<T>` inputs end-to-end into a single
`ReadableStream<T>`, draining each source to completion before pulling from the
next. This is the sequential counterpart to `merge`: where `merge` races readers
and emits chunks in arrival order, `concat` preserves the input ordering
deterministically. Use it when downstream consumers depend on the relative
order of upstream chunks — for example, replaying a header stream followed by a
body stream, or stitching paginated results into one contiguous flow.
## Features
- Deterministic, sequential ordering across stream boundaries.
- Lazy: the next source is only acquired (its reader pulled) once the previous
one has reported `done`.
- Backpressure-aware: pulls from the current source only when the output is
being read.
- Error propagation: any input error halts the concatenation immediately and is
surfaced as a `StreamPipeFailedError` (`STREAM.PIPE_FAILED`) with the original
error attached as `cause`. Unread later sources are never touched.
- Cancellation: cancelling the output cancels the currently-reading source.
Sources that have not yet been acquired are left untouched (the caller still
owns them).
- Fast paths for empty input (`[]` returns an immediately-closed stream) and
single input (returned verbatim, identity).
## Usage: Basic Example
```ts
import { concat } from "@suaveplan/streams/pipeline/concat/concat";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const head = fromArray([1, 2, 3]);
const tail = fromArray([4, 5, 6]);
const all = await toArray(concat([head, tail]));
// → [1, 2, 3, 4, 5, 6]
```
## Usage: Advanced Example
Concatenating paginated fetch results, surfacing transport errors:
```ts
import { concat } from "@suaveplan/streams/pipeline/concat/concat";
import { StreamPipeFailedError } from "@suaveplan/streams/errors/stream-error";
function pageStream(pageNumber: number): ReadableStream<Uint8Array> {
return new ReadableStream({
async pull(controller) {
const res = await fetch(`/api/items?page=${pageNumber}`);
if (!res.ok) throw new Error(`page ${pageNumber} failed`);
const buf = new Uint8Array(await res.arrayBuffer());
controller.enqueue(buf);
controller.close();
},
});
}
const stitched = concat([pageStream(1), pageStream(2), pageStream(3)]);
try {
const reader = stitched.getReader();
while (true) {
const { done, value } = await reader.read();
if (done) break;
process(value);
}
} catch (err) {
if (err instanceof StreamPipeFailedError) {
// err.cause holds the original transport / page error
report(err);
}
throw err;
}
```
## API Reference
### `concat<T>(streams: ReadableStream<T>[]): ReadableStream<T>`
- **Parameters**
- `streams` — an ordered array of `ReadableStream<T>` to drain in sequence.
- **Returns** — a new `ReadableStream<T>` that emits every chunk from
`streams[0]`, then every chunk from `streams[1]`, and so on, closing once the
final source closes. For `streams.length === 0`, returns an
immediately-closed stream. For `streams.length === 1`, returns
`streams[0]` unchanged (identity).
- **Throws (downstream)** — `StreamPipeFailedError` with stable code
`STREAM.PIPE_FAILED` whenever any input source errors. The original error is
attached as `cause`.
## Implementation Notes
- The implementation owns at most one reader at a time. After a source reports
`done`, its reader is `releaseLock`-ed before the next source's reader is
acquired. This avoids holding multiple locks open and keeps memory bounded.
- Sources are acquired lazily inside `pull`. Streams that the consumer never
reaches (e.g., because of an early error or cancellation) are never read,
which is intentional — concat must not eagerly drain inputs.
- Cancellation only cancels the currently-acquired source. Un-acquired sources
are still owned by the caller and may be reused or cancelled independently.
- Errors are wrapped, never re-thrown raw. This guarantees that consumers can
reliably pattern-match on `StreamPipeFailedError` / `STREAM.PIPE_FAILED` for
any concat-mediated failure, regardless of the underlying cause.
- The single-source identity fast path is observable via reference equality —
consumers can detect it with `concat([s]) === s`.

View File

@@ -0,0 +1,68 @@
# fan-out
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`fanOut` splits a single `ReadableStream<T>` into N independent readable streams that each receive every chunk in the same order. It is built on top of the WHATWG `ReadableStream.tee()` method, applied recursively to produce N branches. All branches are synchronised: the source stream is only pulled when the slowest consumer is ready, so backpressure from any branch applies to all.
Fan-out is the foundation for broadcast patterns: send the same data to a logger, a metrics collector, and a storage writer simultaneously, without running the source pipeline multiple times.
## Features
- N-way broadcast of a single stream
- Built on native `tee()` — no custom buffering
- Backpressure from slowest consumer propagates to source
- Throws `RangeError` for `n < 1`
- `n = 1` returns the original stream unchanged
## Usage: Basic Example
```ts
import { fanOut } from "@suaveplan/streams/pipeline/fan-out/fan-out";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const [a, b] = fanOut(fromArray([1, 2, 3]), 2);
const [ra, rb] = await Promise.all([toArray(a), toArray(b)]);
console.log(ra); // [1, 2, 3]
console.log(rb); // [1, 2, 3]
```
## Usage: Advanced Example
```ts
import { fanOut } from "@suaveplan/streams/pipeline/fan-out/fan-out";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { mapStream } from "@suaveplan/streams/transforms/map-stream/map-stream";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const [forStorage, forLogging, forMetrics] = fanOut(
fromArray([{ id: 1, value: 42 }, { id: 2, value: 99 }]),
3
);
const [stored, logged, counted] = await Promise.all([
toArray(forStorage),
toArray(forLogging.pipeThrough(mapStream((e) => `[LOG] id=${e.id}`))),
toArray(forMetrics.pipeThrough(mapStream((e) => e.value))),
]);
```
## API Reference
### `fanOut<T>(stream: ReadableStream<T>, n: number): ReadableStream<T>[]`
**Parameters**:
- `stream` — the source stream to fan out.
- `n` — number of output branches; must be `>= 1`.
**Returns**: Array of `n` `ReadableStream<T>` instances.
**Throws** (immediate): `RangeError` if `n < 1`.
## Implementation Notes
`tee()` is called `n - 1` times, each time on the last branch, building a chain: `[a1, (tee b1 into [a2, (tee b2 into [a3, ...])])]`. Each `tee` call creates internal queues that buffer chunks until both branches have consumed them. For large `n`, memory usage grows proportionally to `n × backlog_size`. If consumers process at very different rates, the internal queues can grow unboundedly — combine with `limitStream` or external flow control if this is a concern.

View File

@@ -0,0 +1,119 @@
# mergeOrdered
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`mergeOrdered` performs a k-way merge of pre-sorted `ReadableStream`s
into a single ordered output stream. It is the streaming analogue of
the merge step in merge-sort: when every input is sorted under the
same comparator, the output is sorted under that comparator. This is
the right tool whenever a pipeline needs to consume multiple already-
ordered sources — sharded query results, time-keyed event logs,
externally-sorted partitions — without re-sorting the union in memory.
Unlike the unordered `merge`, which races inputs and emits in arrival
order, `mergeOrdered` emits the global minimum at every step using a
binary min-heap of `(value, sourceIdx)` entries. Each pull dequeues
the heap root, enqueues it on the output, and replaces the slot with
the next chunk from the same source. Throughput is dominated by the
`O(log k)` heap operations where `k` is the number of inputs.
## Features
- K-way merge over `ReadableStream<T>` arrays of any cardinality
- Caller-supplied comparator (ascending, descending, or domain-specific)
- Single-stream and empty-array fast paths
- Streams of differing lengths handled without truncation
- `cancel` propagates to every input stream
- Errors on any input cancel the remaining inputs and surface on the
merged stream
- Backpressure-respecting — only pulls one chunk per source per pull cycle
## Usage: Basic Example
```ts
import { mergeOrdered } from "@suaveplan/streams/pipeline/merge-ordered/merge-ordered";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const s1 = fromArray([1, 4, 7]);
const s2 = fromArray([2, 5, 8]);
const s3 = fromArray([3, 6, 9]);
const merged = mergeOrdered([s1, s2, s3], (a, b) => a - b);
const out = await toArray(merged);
// out === [1, 2, 3, 4, 5, 6, 7, 8, 9]
```
## Usage: Advanced Example
Merge sharded events keyed by timestamp, in descending order, and
forward the result into a downstream consumer that may cancel mid-flight:
```ts
import { mergeOrdered } from "@suaveplan/streams/pipeline/merge-ordered/merge-ordered";
interface Event {
ts: number;
payload: unknown;
}
declare const shardA: ReadableStream<Event>;
declare const shardB: ReadableStream<Event>;
declare const shardC: ReadableStream<Event>;
const merged = mergeOrdered<Event>(
[shardA, shardB, shardC],
(a, b) => b.ts - a.ts, // newest-first
);
const reader = merged.getReader();
try {
const first = await reader.read();
if (!first.done) {
// process newest event
}
} finally {
// Cancellation propagates to every shard.
await reader.cancel("consumer-done");
}
```
## API Reference
### `mergeOrdered<T>(streams, compare): ReadableStream<T>`
- `streams: ReadableStream<T>[]` — input streams, each pre-sorted
under `compare`. May be empty (returns an immediately-closed stream)
or singleton (returns a passthrough merge with the same heap path).
- `compare: (a: T, b: T) => number` — comparator. Negative when `a`
precedes `b`, zero when equal, positive otherwise. Identical contract
to `Array.prototype.sort`.
- **Returns** a `ReadableStream<T>` that emits values from the union of
all inputs in `compare` order, then closes.
- **Throws / errors** — if any input reader rejects, the remaining
inputs are cancelled and the rejection is forwarded as a stream error.
### `MergeOrderedCompare<T>`
Type alias for the comparator argument; exported for downstream typing
of comparator-producing factories.
## Implementation Notes
- The heap stores at most `k` entries (one per active source) so peak
memory is `O(k)`, independent of the total chunk count.
- Heap operations (`siftUp` / `siftDown`) are iterative and avoid
recursion to keep call stacks shallow on large `k`.
- The first `pull` primes every reader in parallel via `Promise.all`;
subsequent pulls only read from the source whose value was just
emitted, which preserves backpressure on the other inputs.
- `cancel` on the output uses `Promise.allSettled` to ensure every
input is cancelled even if one rejects.
- Output sortedness is conditional on input sortedness — passing
unsorted inputs is a caller bug, not a runtime check, because
validating sortedness would defeat the streaming guarantees.
- The single-stream path still goes through the heap for behavioural
parity with k≥2 (cancellation, error propagation).

View File

@@ -0,0 +1,66 @@
# merge
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`merge` combines multiple `ReadableStream<T>` instances into a single `ReadableStream<T>` that emits chunks in arrival order. It is a fan-in operation: the merged stream races all input readers simultaneously and yields whichever chunk becomes available first, interleaving outputs from all sources as they produce data. The merged stream closes when every input stream has closed.
This is useful for consuming multiple independent data sources concurrently — for example, merging results from several parallel API calls, aggregating events from multiple sensors, or combining log streams from different servers.
## Features
- Race-based interleaving — faster streams yield chunks sooner
- Merged stream closes when all inputs close
- Error from any input stream errors the merged stream
- Handles empty input (returns immediately-closed stream)
- Single-stream input returns the stream unchanged
## Usage: Basic Example
```ts
import { merge } from "@suaveplan/streams/pipeline/merge/merge";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const [a, b] = [fromArray([1, 3, 5]), fromArray([2, 4, 6])];
const merged = merge([a, b]);
const result = await toArray(merged);
// result contains 1,2,3,4,5,6 in some interleaved order
console.log(result.length); // 6
```
## Usage: Advanced Example
```ts
import { merge } from "@suaveplan/streams/pipeline/merge/merge";
import { fromAsyncGenerator } from "@suaveplan/streams/readable/from-generator/from-generator";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
async function* slow() { yield "slow-1"; await new Promise(r => setTimeout(r, 10)); yield "slow-2"; }
async function* fast() { yield "fast-1"; yield "fast-2"; }
const result = await toArray(merge([
fromAsyncGenerator(slow),
fromAsyncGenerator(fast),
]));
console.log(result); // fast items likely come before slow items
```
## API Reference
### `merge<T>(streams: ReadableStream<T>[]): ReadableStream<T>`
**Parameters**:
- `streams` — array of readable streams to merge.
**Returns**: `ReadableStream<T>` that emits all chunks from all inputs.
**Returns immediately-closed stream** if `streams` is empty.
**Returns `streams[0]` unchanged** if `streams.length === 1`.
## Implementation Notes
Each input reader has an in-flight `read()` promise maintained in a `Map`. On each `pull`, `Promise.race` is called over all active in-flight promises to get the next available chunk. When a reader is done, it is removed from the active set. When all readers are done, the controller is closed. Errors from any reader propagate immediately to the merged stream's error channel, and all other readers are cancelled via `Promise.allSettled` in the `cancel` callback.

View File

@@ -0,0 +1,105 @@
# `partition`
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`partition` splits a single `ReadableStream<T>` into two output streams using a
boolean predicate. Unlike `tee` or `fanOut`, every input chunk is routed to
_exactly one_ output — the truthy stream or the falsy stream — so the union of
the two outputs equals the source with per-side ordering preserved. This is
the right tool when you want to fork a stream by classification (errors vs
successes, even vs odd, hot vs cold) without paying the buffering cost of a
full duplication.
## Features
- Single-pass split via one underlying reader on the source
- Synchronous or asynchronous predicate (`(chunk: T) => boolean | Promise<boolean>`)
- Backpressure flows from either output back to the source — pulls on each
side drive the reader on demand
- Predicate exceptions error _both_ output streams and cancel the source
- Per-side `cancel()` cooperates: the source is cancelled only when both
sides have been cancelled
- One-slot per-side buffer absorbs the off-routing case (a pull on truthy
yielding a falsy chunk and vice versa)
- Zero workspace runtime dependencies — uses only WHATWG Streams primitives
## Usage: Basic Example
```ts
import { partition } from "@suaveplan/streams/pipeline/partition/partition";
declare const numbers: ReadableStream<number>;
const { truthy: evens, falsy: odds } = partition(
numbers,
(n) => n % 2 === 0
);
```
## Usage: Advanced Example
```ts
import { partition } from "@suaveplan/streams/pipeline/partition/partition";
interface Event {
level: "info" | "warn" | "error";
payload: unknown;
}
declare const events: ReadableStream<Event>;
const { truthy: errors, falsy: rest } = partition(events, async (e) => {
// async predicates are awaited in routing order
await Promise.resolve();
return e.level === "error";
});
// Each side is an independent ReadableStream — pipe through unrelated
// pipelines without worrying about duplication or fan-out cost.
const errorsToSink = errors.pipeTo(errorSink);
const restToSink = rest.pipeTo(analyticsSink);
await Promise.all([errorsToSink, restToSink]);
```
## API Reference
### `partition<T>(source, predicate): { truthy, falsy }`
- **`source: ReadableStream<T>`** — the input stream. `partition` acquires a
single default reader and drives it from the output pulls.
- **`predicate: (chunk: T) => boolean | Promise<boolean>`** — routes each
chunk: `true` enqueues to `truthy`, `false` enqueues to `falsy`. Predicate
exceptions are caught, both outputs are errored with the original cause,
and the source reader is cancelled.
- **Returns `{ truthy: ReadableStream<T>; falsy: ReadableStream<T> }`** — two
independent readable streams. Both close when the source is exhausted.
Cancelling one side does _not_ cancel the source; cancelling both does.
### Throws
- Synchronous: none — invalid arguments propagate as TypeErrors from the
underlying `ReadableStream` machinery on first use.
- Asynchronous: predicate exceptions surface as stream errors on both
outputs; their `.cause` is the original thrown value.
## Implementation Notes
The implementation owns a single `ReadableStreamDefaultReader` on the source
and exposes two consumer streams whose `pull()` callbacks both drain from
that reader. Each `pull` loops until it produces a chunk _for its own side_;
chunks for the other side are stashed in a one-slot buffer that the other
side claims on its next pull. Because pulls only fire under backpressure
(when the consumer side has space), the source is read at most as fast as
the _currently-pulling_ consumer demands — backpressure on either side is
preserved end-to-end.
Cancellation is cooperative. A `cancel()` on a single output marks that side
closed; the source reader is only cancelled once both sides are closed. This
mirrors `tee` semantics and lets the live side continue draining the source.
Predicate exceptions are routed once: the first throw errors both outputs
and cancels the reader, after which `pullFor` short-circuits via the
`errored` flag.

View File

@@ -0,0 +1,115 @@
# route / routeBy
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`route` and `routeBy` split a single `ReadableStream<T>` into N independent
output branches, but unlike `fanOut` each chunk is delivered to **exactly one**
branch — the one selected by a caller-supplied `keyFn`. They are the
single-pass partitioning primitive of the streams pipeline: a chunk is read
from the source once and dispatched, never duplicated.
This is the idiomatic way to model demultiplexing: a tagged event stream that
should be processed differently per kind (writes → DB, audits → log,
notifications → fan-out), without paying the buffering/backpressure-coupling
cost of `fanOut`. The two ops differ only in container: `route` returns a
`Record<K, ReadableStream<T>>` for `string` keys, `routeBy` returns a
`Map<K, ReadableStream<T>>` for arbitrary key types (numbers, symbols,
objects).
## Features
- Single-pass: each source chunk is dispatched to exactly one branch.
- Per-key ordering is preserved (FIFO within each branch).
- Configurable unknown-key policy: `'drop'` (silent, default) or `'error'`
(every branch errors with `StreamRouteUnknownKeyError`).
- Source errors propagate to every branch.
- `routeBy` accepts non-string keys (number, symbol, object identity).
- Zero external dependencies; pure WHATWG Streams.
## Usage: Basic Example
```ts
import { route } from "@suaveplan/streams/pipeline/route/route";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const events = fromArray([
{ kind: "a" as const, n: 1 },
{ kind: "b" as const, n: 2 },
{ kind: "a" as const, n: 3 },
]);
const branches = route(events, (e) => e.kind, ["a", "b"] as const);
const [as, bs] = await Promise.all([
toArray(branches.a),
toArray(branches.b),
]);
// as -> [{ kind: "a", n: 1 }, { kind: "a", n: 3 }]
// bs -> [{ kind: "b", n: 2 }]
```
## Usage: Advanced Example
```ts
import { routeBy } from "@suaveplan/streams/pipeline/route/route";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// Symbol-keyed routing with strict unknown-key handling.
const PRIORITY_HIGH = Symbol("high");
const PRIORITY_LOW = Symbol("low");
const m = routeBy<{ pri: symbol; payload: string }, symbol>(
fromArray([
{ pri: PRIORITY_HIGH, payload: "urgent" },
{ pri: PRIORITY_LOW, payload: "later" },
]),
(e) => e.pri,
[PRIORITY_HIGH, PRIORITY_LOW],
{ onUnknownKey: "error" },
);
const high = m.get(PRIORITY_HIGH)!;
const low = m.get(PRIORITY_LOW)!;
const [hi, lo] = await Promise.all([toArray(high), toArray(low)]);
```
## API Reference
### `route<T, K extends string>(source, keyFn, keys, options?)`
- `source: ReadableStream<T>` — input stream to dispatch.
- `keyFn: (chunk: T) => K` — extracts the routing key from each chunk.
- `keys: readonly K[]` — declared key set; one branch is created per key.
- `options.onUnknownKey?: 'drop' | 'error'` — default `'drop'`.
**Returns**: `Record<K, ReadableStream<T>>` — one branch per declared key.
**Throws** (per branch, asynchronously): `StreamRouteUnknownKeyError`
(`STREAM.ROUTE_UNKNOWN_KEY`) when a chunk's key is not in `keys` and
`onUnknownKey === 'error'`.
### `routeBy<T, K>(source, keyFn, keys, options?)`
Same contract as `route`, but returns `Map<K, ReadableStream<T>>` so the key
type is unconstrained (numbers, symbols, objects).
## Implementation Notes
The implementation creates one `ReadableStream<T>` per declared key whose
controller is captured during `start()`. A single reader loop pulls from the
source on the next microtask after construction and dispatches each chunk to
the controller selected by `keyFn`. On unknown key:
- `'drop'` — the chunk is consumed and discarded.
- `'error'` — every branch's controller is errored with
`StreamRouteUnknownKeyError`, the source is cancelled, and the dispatch loop
terminates.
Branches use the default WHATWG queuing strategy (CountQueuingStrategy with
`highWaterMark: 1`); if a downstream consumer falls far behind, the branch's
internal queue grows. Compose with `limit` or `throttle` if an unbounded
producer must be guarded.

View File

@@ -0,0 +1,92 @@
# safeBranch
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`safeBranch` is a pipeline operator that wraps a sub-pipeline factory in a fault boundary. It runs the supplied `factory(input)` to construct an inner `ReadableStream`, then forwards every chunk it produces to the wrapper's output stream. If the inner pipeline throws — synchronously while the factory is being invoked, or asynchronously while its stream is being read — the wrapper's output stream closes cleanly (no further chunks emitted) and the error is forwarded to `options.onError` as a `StreamBranchFailedError` whose `cause` is the original error.
The outer pipeline never observes the inner error. This is the difference between a normal `pipeThrough` (which propagates errors and tears down the pipeline) and `safeBranch` (which contains them). Use it for opportunistic side branches — analytics, mirroring, telemetry — where a failure must not bring down the primary data path.
## Features
- Isolates errors from inner sub-pipelines so the outer pipeline keeps running
- Forwards wrapped errors to `options.onError` as `StreamBranchFailedError`
- Preserves the original error via the `cause` chain
- Output stream closes cleanly after a branch error — no half-emitted state
- Silent mode when `onError` is omitted
- Synchronous factory throws are caught and reported the same way
## Usage: Basic Example
```ts
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { safeBranch } from "@suaveplan/streams/pipeline/safe-branch/safe-branch";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const wrapped = safeBranch<number, number>(
(input) =>
input.pipeThrough(
new TransformStream<number, number>({
transform(chunk, c) {
c.enqueue(chunk * 2);
},
})
)
);
const out = wrapped(fromArray([1, 2, 3]));
const result = await toArray(out); // [2, 4, 6]
```
## Usage: Advanced Example
```ts
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { safeBranch } from "@suaveplan/streams/pipeline/safe-branch/safe-branch";
import { StreamBranchFailedError } from "@suaveplan/streams/errors/stream-error";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
let lastError: StreamBranchFailedError | undefined;
const analyticsBranch = safeBranch<number, number>(
(input) =>
input.pipeThrough(
new TransformStream<number, number>({
transform(chunk, c) {
if (chunk < 0) throw new Error("negative not allowed");
c.enqueue(chunk);
},
})
),
{
onError: (err) => {
lastError = err as StreamBranchFailedError;
},
}
);
const collected = await toArray(analyticsBranch(fromArray([1, -2, 3])));
// collected captures whatever was emitted before the error;
// the outer pipeline does not throw.
console.log(lastError?.code); // "STREAM.BRANCH_FAILED"
console.log(lastError?.cause); // Error: negative not allowed
```
## API Reference
### `safeBranch<TIn, TOut>(factory, options?)`
- `factory: (input: ReadableStream<TIn>) => ReadableStream<TOut>` — constructs the inner sub-pipeline. May throw synchronously; failures are isolated.
- `options.onError?: (err: unknown) => void` — invoked with a `StreamBranchFailedError` when the inner pipeline errors. The original error is exposed via `err.cause`.
- **Returns** `(input: ReadableStream<TIn>) => ReadableStream<TOut>` — a wrapper function suitable for use anywhere a stream-shape transform is accepted.
- **Throws** Never. All errors are converted to `StreamBranchFailedError` and either delivered to `onError` or silently swallowed.
## Implementation Notes
- The wrapper acquires a single reader against the inner stream and pulls chunk-by-chunk inside the outer stream's `pull` callback. This preserves backpressure end-to-end: the outer consumer's pull cadence drives the inner sub-pipeline's read cadence.
- On error, the wrapper sets a `closed` flag, closes the outer controller, and calls `onError`. Subsequent pulls return `done` without re-invoking the inner reader.
- Cancellation of the wrapper propagates to the inner reader via `reader.cancel(reason)`. Cancellation errors from the inner stream are swallowed — they are part of the isolated branch by definition.
- The error code is `STREAM.BRANCH_FAILED` (registered in `errors/stream-error.ts`). Inspect `err.code` to detect a branch failure programmatically; inspect `err.cause` to recover the original error.
- `safeBranch` does not retry or restart the inner pipeline. After failure, the wrapper is terminal. To retry, pair `safeBranch` with `@suaveplan/retry` upstream of the factory.

View File

@@ -0,0 +1,107 @@
# zip
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`zip` performs a sample-aligned merge of N input `ReadableStream`s into a single
stream of tuples. Unlike `merge`, which interleaves values in arrival order,
`zip` waits for every input to produce a chunk before emitting one tuple per
round. This makes it the canonical primitive for combining parallel time-series,
correlated event streams, or any pair of producers whose values must be paired
position-by-position rather than raced.
The operation is fully back-pressured — readers are pulled exactly once per
output tuple — and is preserved end-to-end by propagating `cancel()` from the
output stream to every input.
## Features
- Tuple output with full TypeScript inference: `zip([s1, s2] as const)` is
typed as `ReadableStream<[T1, T2]>`.
- Three short-input policies: `close` (default), `error`, `pad`.
- `StreamZipLengthMismatchError` (code `STREAM.ZIP_LENGTH_MISMATCH`) under
`error` policy carries the index of the input that ended early.
- `pad` policy substitutes the configured `padValue` for any input that has
already ended, continuing until the longest input closes.
- Cancellation propagates to every input reader.
- Empty input list returns an immediately-closed stream.
## Usage: Basic Example
```ts
import { fromArray, toArray, zip } from "@suaveplan/streams";
const numbers = fromArray([1, 2, 3]);
const letters = fromArray(["a", "b", "c"]);
const tuples = await toArray(zip([numbers, letters] as const));
// tuples === [[1, "a"], [2, "b"], [3, "c"]]
```
## Usage: Advanced Example
```ts
import { fromArray, toArray, zip } from "@suaveplan/streams";
import { StreamZipLengthMismatchError } from "@suaveplan/streams/errors/stream-error";
// Pad short inputs so every tuple has both slots filled.
const padded = await toArray(
zip(
[fromArray([1, 2, 3, 4]), fromArray(["a", "b"])] as const,
{ onShortInput: "pad", padValue: null }
)
);
// padded === [[1, "a"], [2, "b"], [3, null], [4, null]]
// Strict mode — any length mismatch is a hard error.
try {
await toArray(
zip(
[fromArray([1, 2, 3]), fromArray(["a", "b"])] as const,
{ onShortInput: "error" }
)
);
} catch (err) {
if (err instanceof StreamZipLengthMismatchError) {
// err.code === "STREAM.ZIP_LENGTH_MISMATCH"
}
}
```
## API Reference
### `zip<T extends readonly unknown[]>(streams, options?): ReadableStream<T>`
- `streams: { readonly [K in keyof T]: ReadableStream<T[K]> }` — tuple of input
streams. Pass with `as const` for tuple inference.
- `options.onShortInput?: 'close' | 'error' | 'pad'` — short-input behaviour.
Defaults to `'close'`.
- `options.padValue?: unknown` — value substituted for ended inputs when
`onShortInput === 'pad'`. Defaults to `undefined`.
- **Returns:** `ReadableStream<T>` emitting one tuple per round.
- **Throws:** the output errors with `StreamZipLengthMismatchError` when
`onShortInput === 'error'` and any input ends before the others.
### `ZipOptions<TValue = unknown>`
Options bag passed to `zip`. See above for fields.
### `ZipShortInputPolicy`
String-literal union: `'close' | 'error' | 'pad'`.
## Implementation Notes
- Each pull round issues `read()` on every still-open reader in parallel via
`Promise.all`. This keeps tuple emission strictly ordered while letting the
runtime pipeline reads.
- The `pad` policy tracks per-input "ended" flags so closed readers are not
re-read on subsequent rounds — only `padValue` is substituted in their slot.
- `cancel()` on the output settles cancels on every input via
`Promise.allSettled`, so a single failing cancel cannot leak others.
- Read errors propagate verbatim via `controller.error(err)`; only the
`'error'` short-input policy wraps in `StreamZipLengthMismatchError`.
- An empty input list short-circuits to an immediately-closed stream rather
than starting any readers.

View File

@@ -0,0 +1,151 @@
# BaseDuplexStream
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`BaseDuplexStream` is the primitive for bidirectional WHATWG streams. It bundles a `ReadableStream` and a `WritableStream` whose `source` and `sink` are supplied by the caller, with sensible queuing-strategy defaults for byte vs. object mode. It exists as the foundation for higher-level primitives (`NetworkStreamImpl` for WebRTC/WebSocket peers, future `WorkerStream` for transferable-buffer transport, etc.) and as a clean boundary for any consumer that needs "write to one end, read from the other" semantics without re-implementing the WHATWG plumbing.
The two sides are independent by design — a write to `writable` does NOT echo back on `readable` unless the supplied `source` is wired to do so. This decoupling is what makes the primitive reusable for echoing flows (test doubles), proxy flows (network bridges), and asymmetric flows (encode-on-write, decode-on-read).
## Features
- WHATWG-native `ReadableStream` + `WritableStream` exposed as `readable` / `writable` properties
- Caller-supplied `StreamSource<R>` (start / pull / cancel) and `StreamSink<W>` (start / write / abort / close)
- Default queuing strategy chooses between count-based (`objectMode: true`) and byte-size based (default) automatically
- `highWaterMark` is forwarded to both sides
- `cancel(reason)` on `readable` propagates `reason` to the source's `cancel` callback verbatim
- `abort(reason)` on `writable` propagates to the sink's `abort` callback
- All four sink hooks (`start`, `write`, `abort`, `close`) and three source hooks (`start`, `pull`, `cancel`) are optional — callers supply only what they need
- `createDuplexStream(source, sink, options?)` factory for inline usage
## Usage: Basic Example
```ts
import { createDuplexStream } from "@suaveplan/streams/primitives/duplex/duplex";
let pending: number | null = null;
const stream = createDuplexStream<number, number>(
{
pull(controller) {
if (pending !== null) {
controller.enqueue(pending);
pending = null;
}
},
},
{
write(chunk) {
// Echo the doubled value back to the readable side
pending = chunk * 2;
},
}
);
const writer = stream.writable.getWriter();
await writer.write(7);
await writer.close();
const reader = stream.readable.getReader();
console.log((await reader.read()).value); // 14
```
## Usage: Advanced Example
```ts
import { BaseDuplexStream } from "@suaveplan/streams/primitives/duplex/duplex";
import type {
StreamSink,
StreamSource,
DuplexStreamOptions,
} from "@suaveplan/streams/primitives/duplex/duplex.types";
interface Message {
seq: number;
payload: Uint8Array;
}
function createPeerStream(
onIncoming: (m: Message) => void
): BaseDuplexStream<Message, Message> {
const outgoing: Message[] = [];
let pullController:
| ReadableStreamDefaultController<Message>
| undefined;
const source: StreamSource<Message> = {
start(controller) {
pullController = controller;
},
pull(controller) {
const next = outgoing.shift();
if (next !== undefined) controller.enqueue(next);
},
cancel(reason) {
outgoing.length = 0;
pullController = undefined;
console.warn("peer stream cancelled", reason);
},
};
const sink: StreamSink<Message> = {
write(chunk) {
onIncoming(chunk);
},
close() {
pullController?.close();
},
abort(reason) {
pullController?.error(reason);
},
};
const options: DuplexStreamOptions = {
objectMode: true,
highWaterMark: 32,
};
return new BaseDuplexStream(source, sink, options);
}
```
## API Reference
### `BaseDuplexStream<R, W>`
#### `constructor(source: StreamSource<R>, sink: StreamSink<W>, options?: DuplexStreamOptions)`
Builds the bidirectional stream. Every callback on `source` and `sink` is optional.
- `options.objectMode` — `true` selects count-based queuing; `false` (default) selects byte-size queuing where each chunk's `byteLength` (or 1 if absent) counts toward `highWaterMark`.
- `options.highWaterMark` — applied to both sides when set; left to WHATWG defaults when omitted.
#### `readable: ReadableStream<R>`
The readable side, suitable for `pipeThrough` / `pipeTo` composition.
#### `writable: WritableStream<W>`
The writable side. Use `.getWriter()` for direct control or `.pipeTo` from another readable.
### `createDuplexStream<R, W>(source, sink, options?): DuplexStream<R, W>`
Inline factory equivalent to `new BaseDuplexStream(...)`.
### Types
- `StreamSource<R>` — `{ start?, pull?, cancel? }` with WHATWG controller signatures
- `StreamSink<W>` — `{ start?, write?, abort?, close? }` with WHATWG controller signatures
- `DuplexStreamOptions` — `{ objectMode?, highWaterMark? }`
- `DuplexStream<R, W>` — `{ readonly readable, readonly writable }` interface
## Implementation Notes
The constructor builds the `UnderlyingSource` and `UnderlyingSink` shapes by **conditionally** copying only the callbacks that exist on the caller-supplied objects. This avoids the WHATWG quirk where supplying `start: undefined` is materially different from omitting `start`, and it keeps the underlying-source object minimal so spec-compliant runtimes can apply their fast-path optimisations.
The `size` function lives outside the conditional copy because it's strategy-level, not source-level: a `byteLength`-aware function is installed when `objectMode` is false, and a default count-based strategy is installed when it is true. The chunk-size lookup uses optional chaining and falls back to 1 for non-byte chunks (`{ byteLength?: number } ?? 1`), which is the same approach used by `node:stream` and the WHATWG reference implementation.
`createDuplexStream` is a thin alias rather than a method on the class because the JS-style "create-some-stream" naming reads more idiomatically inside pipelines than `new BaseDuplexStream(...)`. The two have identical behaviour and runtime cost.
This primitive is intentionally minimal — it does not own a benchmark suite, an abort surface beyond the standard WHATWG plumbing, or an event surface. Higher-level primitives (`NetworkStreamImpl`) layer those concerns on top via composition.

View File

@@ -0,0 +1,148 @@
# NetworkStreamImpl
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`NetworkStreamImpl` is the bidirectional byte-stream primitive intended for network transports — WebRTC data channels, WebSockets, plain TCP, or any other peer-to-peer flow that needs an addressable, observable, resettable byte pipe. It extends the `BaseDuplexStream` shape with three things every real network stream needs and that a generic duplex doesn't carry: a stable `id`, an optional `protocol` discriminator, and a mutable `metadata` map so callers can attach session-level state (user ids, connection timestamps, peer fingerprints) to the stream itself.
It also adds two lifecycle methods — `close()` for graceful shutdown and `reset()` for abort-with-structured-error — that cleanly distinguish the "we're done" case from the "something went wrong, tear it down" case. Both are idempotent so that retry logic and cleanup hooks can call them defensively without managing state machines.
## Features
- WHATWG `readable: ReadableStream<Uint8Array>` + `writable: WritableStream<Uint8Array>`
- Stable `id`, optional `protocol` (e.g. `"webrtc"`, `"websocket"`, `"tcp"`)
- Mutable `metadata: Map<string, unknown>` — caller-supplied or empty by default
- `closed: boolean` flag observable at any time
- `close()` — idempotent graceful shutdown of both sides
- `reset()` — idempotent abort with `StreamNetworkResetError` (`STREAM.NETWORK_RESET`); the stream's `id` surfaces in `error.context.networkId`
- Writing after close or reset rejects with `StreamNetworkResetError`
- Pending readers are signalled with the same structured error on `reset()`
## Usage: Basic Example
```ts
import { createNetworkStream } from "@suaveplan/streams/primitives/network/network";
const stream = createNetworkStream("peer-1", { protocol: "webrtc" });
// Write a frame
const writer = stream.writable.getWriter();
await writer.write(new TextEncoder().encode("hello"));
// Read it on the other side
const reader = stream.readable.getReader();
const { value } = await reader.read();
console.log(new TextDecoder().decode(value!)); // "hello"
await stream.close();
```
## Usage: Advanced Example
```ts
import { NetworkStreamImpl } from "@suaveplan/streams/primitives/network/network";
import {
STREAM_CODES,
StreamNetworkResetError,
} from "@suaveplan/streams/errors/stream-error";
class WebRtcAdapter {
private readonly streams = new Map<string, NetworkStreamImpl>();
createPeer(peerId: string, userId: string): NetworkStreamImpl {
const meta = new Map<string, unknown>([
["userId", userId],
["openedAt", Date.now()],
]);
const stream = new NetworkStreamImpl(peerId, {
protocol: "webrtc",
metadata: meta,
});
this.streams.set(peerId, stream);
return stream;
}
async dropPeer(peerId: string, reason: "close" | "reset"): Promise<void> {
const stream = this.streams.get(peerId);
if (!stream) return;
await (reason === "close" ? stream.close() : stream.reset());
this.streams.delete(peerId);
}
}
// Distinguish reset from close at consumption time:
const stream = new NetworkStreamImpl("peer-x");
const reader = stream.readable.getReader();
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
process(value);
}
} catch (err) {
if ((err as { code?: string }).code === STREAM_CODES.NETWORK_RESET) {
const peerId = (err as StreamNetworkResetError).context?.["networkId"];
console.warn(`peer ${peerId} reset; reconnecting`);
} else {
throw err;
}
}
```
## API Reference
### `NetworkStreamImpl`
#### `constructor(id: string, options?: NetworkStreamOptions)`
- `id` — required. Stable identifier for this stream; surfaced verbatim in `StreamNetworkResetError` context on `reset()`.
- `options.protocol` — optional transport label.
- `options.metadata` — optional initial map. **The Map is owned by the stream after construction** — callers may keep a reference and mutate it, but the stream and the caller share the same instance.
#### `id: string`, `protocol?: string`, `metadata: Map<string, unknown>`
Read-only addressing and metadata fields.
#### `closed: boolean`
`true` once `close()` or `reset()` has run, or after the readable side has been cancelled.
#### `readable: ReadableStream<Uint8Array>`, `writable: WritableStream<Uint8Array>`
The two halves of the duplex pair. Writes to `writable` enqueue on `readable`.
#### `close(): Promise<void>`
Cleanly closes both sides. Idempotent. After `close()`, writes reject and `closed === true`.
#### `reset(): Promise<void>`
Aborts both sides with a `StreamNetworkResetError`. Pending readers reject with the same. Idempotent. The error carries the stream's `id` in `context.networkId`.
### `createNetworkStream(id, options?): NetworkStream`
Factory function — equivalent to `new NetworkStreamImpl(id, options)` but reads more naturally inline.
### `NetworkStreamOptions`
```ts
interface NetworkStreamOptions {
protocol?: string;
metadata?: Map<string, unknown>;
}
```
## Implementation Notes
The internal model is the simplest correct shape: writes call `controller.enqueue` on the readable controller. There's no internal queue beyond what WHATWG provides — backpressure is handled by the standard queuing strategy. This means a slow reader naturally applies backpressure to writes through the `getWriter().ready` promise, and consumers don't have to wire any custom flow control.
`close()` and `reset()` are both idempotent. The first call sets `isClosed = true` and triggers the matching cleanup on both sides; the second call is a no-op short-circuit. This matters because real-world transports often invoke shutdown handlers more than once (e.g. WebRTC `oniceconnectionstatechange` firing both "disconnected" and "closed"), and the consumer should be free to call `close` defensively without writing state-machine wrappers.
`reset()` sources its error class from `@suaveplan/streams` rather than throwing a generic `Error` so consumers downstream can match on `STREAM.NETWORK_RESET` without string-matching. The `context.networkId` field gives observability tools the exact stream that reset, which matters when a process holds many concurrent peer streams.
The metadata `Map` is intentionally mutable and intentionally shared between caller and stream. This is a deliberate design choice: callers often need to update connection state ("last seen", "auth token rotation") after construction, and copying the map at construction time would force them to re-read it through the stream every time. If a future use case demands isolation, a `new Map(initial)` copy can be added without breaking compatibility.
`NetworkStreamImpl` does not extend `BaseDuplexStream` even though both expose `readable` + `writable`. The two have different lifecycle semantics (network has reset/close, duplex doesn't), and inheritance would force the duplex base to grow methods it shouldn't have. They share `DuplexStream` as a structural interface instead, which is the right level of coupling.

View File

@@ -0,0 +1,69 @@
# from-array
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Converts a JavaScript array into a WHATWG `ReadableStream`. Each array element is enqueued as a separate chunk in index order. The stream is closed automatically after all elements are pushed. This factory is the simplest entry point for creating a stream from in-memory data and is the building block for test fixtures and benchmark pipelines throughout the package.
The `highWaterMark` option controls the WHATWG backpressure queue depth. Keeping it at the default (`1`) applies maximum backpressure, ensuring the consumer controls the pull rate. Higher values pre-buffer more chunks, which is useful when the consumer is consistently slower than the source.
## Features
- Converts any `T[]` to `ReadableStream<T>`
- `AbortSignal` support: errors the stream immediately when the signal fires
- Configurable `highWaterMark` via `ReadableOptions`
- Zero external runtime dependencies
## Usage: Basic Example
```ts
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
const stream = fromArray([1, 2, 3]);
const reader = stream.getReader();
let result;
while (!(result = await reader.read()).done) {
console.log(result.value); // 1, 2, 3
}
```
## Usage: Advanced Example
```ts
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { mapStream } from "@suaveplan/streams/transforms/map-stream/map-stream";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const ac = new AbortController();
const stream = fromArray(["a", "b", "c"], { signal: ac.signal, highWaterMark: 4 });
const result = await toArray(stream.pipeThrough(mapStream((x) => x.toUpperCase())));
console.log(result); // ["A", "B", "C"]
// Cancellation: abort mid-flight
const longStream = fromArray(new Array(1_000_000).fill(0), { signal: ac.signal });
ac.abort();
const reader = longStream.getReader();
const { done } = await reader.read().catch(() => ({ done: true }));
console.log(done); // true (stream was aborted)
```
## API Reference
### `fromArray<T>(items: T[], options?: ReadableOptions): ReadableStream<T>`
**Parameters**:
- `items` — the source array; elements are enqueued in insertion order.
- `options.signal` — optional `AbortSignal`; errors the stream on abort.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns**: `ReadableStream<T>` that emits each element and then closes.
**Throws**: never directly; the stream's reader will throw the abort reason when the signal fires.
## Implementation Notes
The `pull` callback checks `signal.aborted` on every pull cycle rather than using an `abort` event listener. This avoids the need to clean up listeners and keeps the implementation leak-free. When the signal fires between two pull cycles the worst case is one additional chunk being enqueued before the abort is detected — this is acceptable because WHATWG streams do not guarantee atomic cancellation at the chunk boundary.

View File

@@ -0,0 +1,73 @@
# from-bytes
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides three factories and utilities for working with binary `Uint8Array` streams: `fromBytes` slices a buffer into a chunked `ReadableStream<Uint8Array>`, `concatByteStreams` sequentially reads multiple streams into one, and `sliceByteStream` lazily emits a byte range without buffering the full stream. All operations use `Uint8Array.subarray` for zero-copy views wherever the platform permits.
These utilities are the foundation for file-processing pipelines, compression/decompression, and any feature that needs to move binary data through a backpressure-aware pipeline without unnecessary copying.
## Features
- `fromBytes`: splits a `Uint8Array` into configurable chunk sizes (default 64 KiB)
- `concatByteStreams`: sequential fan-in of multiple byte streams without buffering
- `sliceByteStream`: range-based lazy slice of a byte stream (`start`/`end` offsets)
- Zero-copy `subarray` views — no `slice` allocations unless the platform requires it
- AbortSignal support in `fromBytes`
- Configurable `highWaterMark` for WHATWG backpressure
## Usage: Basic Example
```ts
import { fromBytes } from "@suaveplan/streams/readable/from-bytes/from-bytes";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const data = new Uint8Array([0, 1, 2, 3, 4, 5, 6, 7]);
const stream = fromBytes(data, 3); // chunks: [0,1,2], [3,4,5], [6,7]
const chunks = await toArray(stream);
console.log(chunks.map(c => Array.from(c)));
// [[0,1,2], [3,4,5], [6,7]]
```
## Usage: Advanced Example
```ts
import {
fromBytes,
concatByteStreams,
sliceByteStream,
} from "@suaveplan/streams/readable/from-bytes/from-bytes";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const a = fromBytes(new Uint8Array([1, 2, 3]));
const b = fromBytes(new Uint8Array([4, 5, 6]));
const combined = concatByteStreams([a, b]);
// Slice bytes 1 to 4 (exclusive) from the combined stream
const sliced = sliceByteStream(combined, 1, 4);
const result = await toArray(sliced);
// result: Uint8Array([2, 3, 4])
```
## API Reference
### `fromBytes(data: Uint8Array, chunkSize?: number, options?: ReadableOptions): ReadableStream<Uint8Array>`
- `data` — source buffer; no copy is made.
- `chunkSize` — bytes per chunk (default: 65536).
- `options.signal` — AbortSignal, errors the stream on abort.
- `options.highWaterMark` — WHATWG queue depth (default: 1).
### `concatByteStreams(streams: ReadableStream<Uint8Array>[]): ReadableStream<Uint8Array>`
Reads each stream to completion before starting the next. The returned stream closes after all input streams close.
### `sliceByteStream(stream: ReadableStream<Uint8Array>, start: number, end?: number): ReadableStream<Uint8Array>`
Emits only bytes whose position falls within `[start, end)`. Cancels the underlying reader once `end` is reached.
## Implementation Notes
`fromBytes` uses `data.subarray(offset, end)` which returns a view sharing the same underlying `ArrayBuffer` — no allocation. `concatByteStreams` maintains a single active reader reference, releasing it before advancing to the next stream. `sliceByteStream` reads chunks from the underlying stream, computing per-chunk sub-ranges in a single loop without materialising the full stream.

View File

@@ -0,0 +1,75 @@
# from-generator
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Converts an async generator factory function into a WHATWG `ReadableStream`. The factory is invoked lazily — only when the first consumer starts pulling — which avoids eager resource acquisition. Generator teardown on cancellation is handled via `iterator.return()`, ensuring that `finally` blocks in the generator are executed and file handles, network sockets, or other resources are released.
This factory is the preferred entry point for wrapping any asynchronous data source that is naturally expressed as an `async function*` — database cursors, paginated API clients, file-line readers, or infinite event sequences.
## Features
- Lazy generator construction (factory called on first pull)
- Clean teardown via `iterator.return()` on stream cancellation
- AbortSignal support: errors the stream when the signal fires
- Configurable `highWaterMark` for WHATWG backpressure control
- Propagates generator `throw` errors through the stream's error channel
## Usage: Basic Example
```ts
import { fromAsyncGenerator } from "@suaveplan/streams/readable/from-generator/from-generator";
async function* counter(limit: number) {
for (let i = 0; i < limit; i++) yield i;
}
const stream = fromAsyncGenerator(() => counter(3));
const reader = stream.getReader();
console.log((await reader.read()).value); // 0
console.log((await reader.read()).value); // 1
console.log((await reader.read()).value); // 2
console.log((await reader.read()).done); // true
```
## Usage: Advanced Example
```ts
import { fromAsyncGenerator } from "@suaveplan/streams/readable/from-generator/from-generator";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
async function* fetchPages(baseUrl: string): AsyncGenerator<string> {
let cursor: string | undefined;
do {
const url = cursor ? `${baseUrl}?cursor=${cursor}` : baseUrl;
const res = await fetch(url);
const data = await res.json();
yield* data.items;
cursor = data.nextCursor;
} while (cursor);
}
const ac = new AbortController();
const stream = fromAsyncGenerator(() => fetchPages("/api/items"), { signal: ac.signal });
const items = await toArray(stream);
```
## API Reference
### `fromAsyncGenerator<T>(gen: () => AsyncGenerator<T>, options?: ReadableOptions): ReadableStream<T>`
**Parameters**:
- `gen` — factory function that returns an `AsyncGenerator<T>`; called once on first pull.
- `options.signal` — optional `AbortSignal`; errors the stream on abort.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns**: `ReadableStream<T>` that yields each value from the generator in order.
**Throws**: never directly. Generator errors propagate through the stream's error channel and surface as rejected `reader.read()` promises.
## Implementation Notes
The generator instance is created lazily inside `pull` rather than in `start`. This means that if a `ReadableStream` is constructed but never read, the generator factory is never invoked and no resources are acquired. Cancellation calls `iterator.return(undefined)` which triggers any `try/finally` cleanup in the generator. The `cancel` method is `async` to await the return value in case the generator performs async cleanup.

View File

@@ -0,0 +1,66 @@
# for-each
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides three stream sinks: `forEach` calls a user function for each chunk without accumulating results in memory, `toWritable` pipes a readable stream to a writable stream using the WHATWG `pipeTo` API, and `drain` reads and discards all chunks from a stream to completion.
These are the primary sinks for side-effectful consumption and plumbing: writing chunks to a file, sending them over a network, logging, or simply clearing a stream. They complement `toArray` and `toBytes` which materialise stream contents into memory.
## Features
- `forEach`: sequential, low-memory chunk processing
- `toWritable`: native WHATWG `pipeTo` with optional abort signal
- `drain`: discard all chunks efficiently
- AbortSignal support in `forEach` and `drain`
- Lock-safe: always releases reader lock
## Usage: Basic Example
```ts
import { forEach } from "@suaveplan/streams/sinks/for-each/for-each";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
let sum = 0;
await forEach(fromArray([1, 2, 3, 4, 5]), (n) => { sum += n; });
console.log(sum); // 15
```
## Usage: Advanced Example
```ts
import { forEach, toWritable, drain } from "@suaveplan/streams/sinks/for-each/for-each";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
// Pipe to a writable
const received: string[] = [];
const writable = new WritableStream<string>({
write(chunk) { received.push(chunk); }
});
await toWritable(fromArray(["a", "b", "c"]), writable);
console.log(received); // ["a", "b", "c"]
// Drain a stream (discard all chunks)
await drain(fromArray([1, 2, 3, 4, 5]));
```
## API Reference
### `forEach<T>(stream: ReadableStream<T>, fn: (chunk: T) => void | Promise<void>, signal?: AbortSignal): Promise<void>`
- `fn` — called for each chunk; may be async.
- `signal` — rejects with `StreamAbortedError` on abort.
### `toWritable<T>(stream: ReadableStream<T>, writable: WritableStream<T>, signal?: AbortSignal): Promise<void>`
Pipes `stream` to `writable` using `stream.pipeTo(writable)`. Propagates errors from both sides.
### `drain(stream: ReadableStream<unknown>, signal?: AbortSignal): Promise<void>`
Reads all chunks and discards them. Useful for exhausting a stream whose side effects are in the transforms.
## Implementation Notes
`forEach` uses the same abort-signal pattern as `toArray`: an event listener calls `reader.cancel()` to interrupt a pending `read()`, and the main loop checks `signal.aborted` before each read for pre-aborted signals. `toWritable` delegates entirely to `ReadableStream.pipeTo` which has native error propagation and abort support in the WHATWG spec. `drain` is a thin wrapper over `forEach` with a no-op callback — it exists as a named concept rather than requiring callers to know the pattern.

View File

@@ -0,0 +1,73 @@
# to-array
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`toArray` (aliased as `collect`) is the primary sink for materialising a `ReadableStream<T>` into an in-memory array. It reads chunks sequentially until the stream closes and returns all chunks in order. An optional `AbortSignal` allows the consumer to cancel the operation mid-flight: when the signal fires, the reader is cancelled and the promise rejects with `StreamAbortedError`.
This is the most common way to consume a stream in tests and batch pipelines. For large streams, prefer `forEach` (which avoids accumulating the full result) or streaming into a `WritableStream`.
## Features
- Materialises a stream into a `T[]`
- AbortSignal support with `StreamAbortedError` on cancellation
- Lock-safe: always releases the reader lock in `finally`
- `collect` is an alias for `toArray` (both are exported)
## Usage: Basic Example
```ts
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const result = await toArray(fromArray([1, 2, 3]));
console.log(result); // [1, 2, 3]
```
## Usage: Advanced Example
```ts
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { mapStream } from "@suaveplan/streams/transforms/map-stream/map-stream";
import { toArray, collect } from "@suaveplan/streams/sinks/to-array/to-array";
// With abort
const ac = new AbortController();
setTimeout(() => ac.abort(), 50);
try {
const result = await toArray(
fromArray(new Array(1_000_000).fill(0)),
ac.signal
);
console.log(result.length);
} catch (err) {
// StreamAbortedError
console.log("cancelled");
}
// collect alias
const result2 = await collect(fromArray(["a", "b", "c"]));
console.log(result2); // ["a", "b", "c"]
```
## API Reference
### `toArray<T>(stream: ReadableStream<T>, signal?: AbortSignal): Promise<T[]>`
**Parameters**:
- `stream` — the readable stream to consume.
- `signal` — optional `AbortSignal`; rejects with `StreamAbortedError` on abort.
**Returns**: `Promise<T[]>` resolving to all chunks in order.
**Throws**: `StreamAbortedError` when the signal fires; re-throws stream errors unchanged.
### `collect<T>` — alias for `toArray`
## Implementation Notes
The abort signal listener calls `reader.cancel()` asynchronously to interrupt a pending `reader.read()`. The main loop checks `signal.aborted` before each read call to handle pre-aborted signals. The `finally` block always calls `reader.releaseLock()` — even if the stream errors or the abort fires — to prevent the stream from becoming permanently locked and unusable.

View File

@@ -0,0 +1,94 @@
# toAsyncIterable
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`toAsyncIterable` adapts a `ReadableStream<T>` into a standard
`AsyncIterable<T>`, letting consumers use ergonomic `for await ... of` loops
to drain a stream rather than driving the WHATWG reader API by hand. While
modern Node and recent browser engines expose `ReadableStream[Symbol.asyncIterator]`
natively, the streams package ships its own implementation so that:
- behaviour is identical across runtimes regardless of native support;
- abort semantics are pinned — aborts always surface as `StreamAbortedError`;
- early-break cleanup is guaranteed (the underlying reader is cancelled, the
lock is released, and abort listeners are removed) so upstream producers
can promptly free resources.
## Features
- `for await ... of` support for any `ReadableStream<T>`.
- Optional `AbortSignal` input — abort throws `StreamAbortedError`.
- Cancels the underlying reader on early-break (via the iterator's
`return()` hook).
- Idempotent cleanup — releasing an already-released lock is swallowed.
- Generic over `T`; works equally for `Uint8Array` byte streams and
arbitrary object streams.
## Usage: Basic Example
```ts
import { toAsyncIterable } from "@suaveplan/streams/sinks/to-async-iterable/to-async-iterable";
for await (const chunk of toAsyncIterable(stream)) {
console.log(chunk);
}
```
## Usage: Advanced Example
```ts
import { toAsyncIterable } from "@suaveplan/streams/sinks/to-async-iterable/to-async-iterable";
import { StreamAbortedError } from "@suaveplan/streams/errors/stream-error";
async function consumeWithDeadline<T>(
stream: ReadableStream<T>,
deadlineMs: number,
onItem: (item: T) => Promise<void>
): Promise<void> {
const ac = new AbortController();
const timer = setTimeout(() => ac.abort(new Error("deadline")), deadlineMs);
try {
for await (const item of toAsyncIterable(stream, ac.signal)) {
await onItem(item);
}
} catch (err) {
if (err instanceof StreamAbortedError) {
console.warn("hit deadline before stream completed");
return;
}
throw err;
} finally {
clearTimeout(timer);
}
}
```
## API Reference
### `toAsyncIterable<T>(stream, signal?): AsyncIterable<T>`
- **`stream: ReadableStream<T>`** — source stream to adapt.
- **`signal?: AbortSignal`** — optional cancellation token. Aborting causes
the active or next `next()` call to reject with `StreamAbortedError` and
cancels the underlying reader.
- **Returns** — an `AsyncIterable<T>` that yields chunks until the stream
closes or aborts.
- **Throws** — `StreamAbortedError` on abort; otherwise propagates upstream
stream errors verbatim.
## Implementation Notes
- The iterator implements `return()` so `break`/`throw` inside the
`for await` loop both trigger reader cancel + lock release. Without this,
upstream producers would keep filling buffers for chunks no consumer will
ever read.
- Both pre-iteration abort and mid-`read()` abort paths produce
`StreamAbortedError`. The mid-read path uses the same `aborted` flag plus
a defensive re-check after `read()` resolves to handle the race where the
signal fires after the read promise was already in flight.
- Abort listeners are registered with `{ once: true }` and explicitly
removed in cleanup — long-lived signals shared across many iterators do
not accumulate listeners.

View File

@@ -0,0 +1,67 @@
# to-bytes
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`toBytes` and `bytesToString` are binary sinks for `ReadableStream<Uint8Array>`. `toBytes` reads all chunks from the stream and concatenates them into a single `Uint8Array` using a single allocation (collect chunks, compute total size, write all at once). `bytesToString` is a convenience wrapper that decodes the result via `TextDecoder`.
These sinks are the counterparts to `fromBytes` and are used when a pipeline produces binary data that needs to be materialised into memory — for example, the final step after `compressStream` or `encodeTextStream`.
## Features
- `toBytes`: single-allocation concatenation of all stream chunks
- `bytesToString`: `toBytes` followed by `TextDecoder` with configurable encoding
- AbortSignal support with `StreamAbortedError` on cancellation
- Lock-safe: always releases reader lock in `finally`
## Usage: Basic Example
```ts
import { fromBytes } from "@suaveplan/streams/readable/from-bytes/from-bytes";
import { toBytes } from "@suaveplan/streams/sinks/to-bytes/to-bytes";
const data = new Uint8Array([1, 2, 3, 4, 5]);
const result = await toBytes(fromBytes(data, 2));
console.log(Array.from(result)); // [1, 2, 3, 4, 5]
```
## Usage: Advanced Example
```ts
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { encodeTextStream } from "@suaveplan/streams/transforms/text-codec/text-codec";
import { toBytes, bytesToString } from "@suaveplan/streams/sinks/to-bytes/to-bytes";
const bytes = await toBytes(
fromArray(["hello", " world"]).pipeThrough(encodeTextStream())
);
console.log(bytes.byteLength); // 11
// Or directly as string
const text = await bytesToString(
fromArray(["hello", " world"]).pipeThrough(encodeTextStream())
);
console.log(text); // "hello world"
```
## API Reference
### `toBytes(stream: ReadableStream<Uint8Array>, signal?: AbortSignal): Promise<Uint8Array>`
- `stream` — byte stream to consume.
- `signal` — optional `AbortSignal`; rejects with `StreamAbortedError` on abort.
**Returns**: `Promise<Uint8Array>` — all bytes concatenated into a single buffer.
### `bytesToString(stream: ReadableStream<Uint8Array>, encoding?: string): Promise<string>`
- `stream` — byte stream to decode.
- `encoding` — IANA encoding label (default `"utf-8"`).
**Returns**: `Promise<string>`.
## Implementation Notes
Chunks are collected into an array first, then the total byte length is computed, then a single `Uint8Array` is allocated and each chunk is copied in with `set`. This is more efficient than repeatedly concatenating buffers (which would require O(N²) total copy work). The lock is always released in `finally`, even on error.

View File

@@ -0,0 +1,100 @@
# batch
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `batch` feature provides two coordinated APIs for grouping a stream of individual chunks into fixed-size arrays before passing them downstream: the functional factory `batchStream` and the plugin-OO wrapper class `BatchTransform`. Both accumulate chunks into arrays of a configured `size` and flush a partial trailing batch when the upstream closes. This is the stream equivalent of chunking an array with `lodash.chunk` — it groups a continuous flow of items into discrete pages for bulk operations like database bulk-inserts, API batch requests, or log aggregation.
The internal buffer is reset between batches, so there is no risk of the same chunk appearing in two batches. `BatchTransform` is a thin shell around `batchStream`: its streaming bridge delegates to the factory directly so semantics are identical between the two call paths. The class exists so consumers using the `BaseTransform` plugin contract (hooks, events, `StreamManager`, `TransformFactory`) can express a batching stage alongside other class-form transforms (`MapTransform`, `FilterTransform`, …) without mixing function-form and class-form APIs.
## Features
- Fixed-size batching with partial-final-batch flush on stream close
- Synchronous processing — no async overhead per chunk
- `AbortSignal` support on both call paths
- Configurable `highWaterMark` (functional API)
- Throws `RangeError` immediately if `size < 1` (both APIs)
- Plugin-OO `BatchTransform` integrates with `BaseTransform` hooks/events
- Single-shot `BatchTransform.process(input)` returns `[input]` — a one-element batch
- Type-parameterised on `T` so callers get `T[]` output inference
## Usage: Basic Example
```ts
import { batchStream } from "@suaveplan/streams/transforms/batch/batch";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const result = await toArray(
fromArray([1, 2, 3, 4, 5]).pipeThrough(batchStream(2))
);
console.log(result); // [[1, 2], [3, 4], [5]]
```
## Usage: Advanced Example
```ts
import { BatchTransform } from "@suaveplan/streams/transforms/batch/batch";
const t = new BatchTransform<{ id: string }>({ size: 100 });
// Hook to add timing metadata before downstream consumers
t.hook("before-emit", (batch) => batch);
const ctrl = new AbortController();
setTimeout(() => ctrl.abort(new Error("shutdown")), 5_000);
await source
.pipeThrough(t.toTransformStream({ signal: ctrl.signal }))
.pipeTo(sinkWritable, { signal: ctrl.signal });
```
## API Reference
### `batchStream<T>(size: number, options?: TransformOptions): TransformStream<T, T[]>`
**Parameters**:
- `size` — number of chunks per batch; must be `>= 1` (throws `RangeError` otherwise).
- `options.signal` — AbortSignal; errors the stream on abort.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns**: `TransformStream<T, T[]>`.
**Throws** (immediate): `RangeError` if `size < 1`.
**Throws** (via error channel): `StreamAbortedError` when the signal fires.
### `class BatchTransform<T = unknown>`
#### `constructor(config: BatchTransformConfig)`
Throws `RangeError` if `config.size < 1`.
#### `process<U>(input: U, options?: TransformOptions): Promise<U>`
Returns `[input]` (a one-element batch) cast to `U`. Honors `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired.
#### `toTransformStream<A, B>(options?): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `batchStream<T>(size, options)`. The readable side emits `T[]` arrays of length `config.size`, with a possibly-shorter final array when the upstream closes.
### `interface BatchTransformConfig`
```ts
interface BatchTransformConfig {
readonly size: number;
readonly flushOnClose?: boolean; // reserved; currently always true
}
```
## Implementation Notes
Batches are emitted as fresh arrays; the internal buffer is replaced on each flush rather than cleared so the emitted batch slice is not mutated by subsequent chunk accumulation. The `flush` callback handles the trailing partial batch. Signal checking happens in `transform` only; if the signal fires between chunks, the next pull from the source will trigger the abort check.
**Batching is stateful — single-shot `process` is a degenerate case.** A batch requires multiple inputs to fill, but `process(input)` is called once per pipeline execution and cannot accumulate across executions without leaking buffer state between unrelated callers. We chose the most useful single-shot interpretation: `process(x)` returns `[x]`, treating each call as "batch this one item." Real batching happens on the streaming side via `toTransformStream`, where `batchStream` owns the buffer and emits arrays as they fill.
This trade-off is intentional. The alternative (instance-scoped buffer state used by `process`) would couple every caller to a hidden lifecycle they would have to flush manually, and would diverge from the `BaseTransform` contract (`process` is "operate on this input now"). Consumers that want true batching reach for `toTransformStream`; consumers that want per-item transformation under a uniform plugin surface still get a sensible `process` shape.
The `flushOnClose` flag on `BatchTransformConfig` is reserved for forward compatibility — `batchStream` currently always flushes its partial buffer on close, so the option has no behaviour to gate. It is documented now to avoid a breaking config change later. Type erasure on the streaming bridge follows the same pattern as `MapTransform`: callers supply method-level generics (`t.toTransformStream<number, number[]>()`) for strict typing; default generics fall through to `unknown` from the abstract base.

View File

@@ -0,0 +1,104 @@
# chunk-bytes
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`chunkBytes` re-frames an arbitrary `Uint8Array` byte stream into fixed-size
frames of exactly `frameSize` bytes. Upstream producers rarely emit byte
chunks that align to a downstream protocol's frame boundary — file readers,
sockets, and decompression streams all hand out variable-length buffers. This
transform absorbs that variability so consumers can rely on stable frame
geometry, which is the precondition for length-prefixed framing, fixed-block
ciphers, and any sliding-window decoder.
## Features
- Fixed-size output frames with caller-supplied `frameSize`.
- Three tail policies for the residual: `emit` (default), `pad`, `drop`.
- Zero-copy frame emission via `Uint8Array.subarray` for full frames.
- Synchronous `RangeError` when `frameSize <= 0`.
- Plugin-OO wrapper (`ChunkBytesTransform`) for use with `BaseTransform`-aware
pipelines, with a `process()` that returns the full concatenated output.
- AbortSignal honoured by `process()` — rejects with `StreamAbortedError`.
## Usage: Basic Example
```ts
import { chunkBytes } from "@suaveplan/streams/transforms/chunk-bytes/chunk-bytes";
import { fromBytes } from "@suaveplan/streams/readable/from-bytes/from-bytes";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const input = new Uint8Array(16);
for (let i = 0; i < 16; i++) input[i] = i + 1;
const frames = await toArray(
fromBytes(input, 7).pipeThrough(chunkBytes(4))
);
// frames.length === 4, every frame is exactly 4 bytes
```
## Usage: Advanced Example
Pad the trailing partial frame with zeroes (useful when feeding a fixed-block
cipher or a fixed-frame radio modem):
```ts
import { ChunkBytesTransform } from "@suaveplan/streams/transforms/chunk-bytes/chunk-bytes";
const t = new ChunkBytesTransform(8, { tail: "pad" });
const padded = await t.process(new Uint8Array([1, 2, 3, 4, 5]));
// padded === Uint8Array([1, 2, 3, 4, 5, 0, 0, 0])
// Or use it in a pipeline alongside any other transform:
readable.pipeThrough(t.toTransformStream()).pipeTo(downstream);
t.dispose();
```
Drop the trailing partial frame when downstream cannot handle short reads:
```ts
chunkBytes(1024, { tail: "drop" });
```
## API Reference
### `chunkBytes(frameSize, options?)`
- `frameSize: number` — exact byte count per output frame; must be `> 0`.
- `options.tail?: 'emit' | 'pad' | 'drop'` — residual policy. Defaults to
`'emit'`.
- Returns: `TransformStream<Uint8Array, Uint8Array>`.
- Throws: `RangeError` if `frameSize <= 0`. The stream errors with `TypeError`
on non-`Uint8Array` chunks.
### `class ChunkBytesTransform extends BaseTransform`
- `new ChunkBytesTransform(frameSize, options?)` — same parameters; throws
`RangeError` synchronously.
- `process(input: Uint8Array, options?): Promise<Uint8Array>` — pushes `input`
through an internal `chunkBytes` stream and returns the concatenated frames
as a single `Uint8Array`. Honours `options.signal`.
- `toTransformStream<A, B>()` — returns `chunkBytes(frameSize, options)`.
- `dispose()` — clears hooks/listeners (calls `super.dispose()`).
## Implementation Notes
The transform keeps a `carry` `Uint8Array` of bytes received but not yet
consumed by a complete frame. On each upstream chunk it concatenates `carry +
chunk`, emits as many `frameSize`-byte subarrays as fit, and copies the
residual into a fresh allocation so the upstream chunk can be released. The
`flush` callback resolves the residual against the configured tail policy.
Performance: full frames are emitted as `subarray` views (no copy). Only the
residual carry is copied, bounded by `frameSize - 1` bytes per upstream
chunk, so steady-state allocations are O(frameSize) regardless of upstream
chunk size. The `'pad'` policy allocates exactly one `frameSize` buffer at
end-of-stream.
Known limitations: the transform is byte-oriented; it does not understand
record boundaries beyond fixed length. For length-prefixed or
delimiter-separated framing, layer this on top of a higher-level decoder.

View File

@@ -0,0 +1,93 @@
# coalesce-bytes
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`coalesceBytes` and `CoalesceBytesTransform` accumulate small inbound
`Uint8Array` chunks into larger frames before emitting them downstream. They
exist to bridge byte producers that hand out tiny fragments (network sockets,
serial ports, decoders) with consumers that prefer fewer, larger writes
(framing protocols, hash functions, syscalls), without forcing every producer
to implement its own buffering and flush policy. Coalescing is driven by two
independent triggers: a size threshold (`minBytes`) and a latency cap
(`maxLatencyMs`) measured from the moment the first chunk in the current
buffer arrived. On stream close any remainder is always emitted, so no bytes
are silently dropped.
## Features
- Size-driven flush when accumulated bytes reach `minBytes`.
- Latency-driven flush when `maxLatencyMs` ms have elapsed since the first
buffered chunk (clock-based, evaluated on the next chunk arrival).
- Injectable `Clock` from `@suaveplan/types` for fully deterministic tests.
- Final-frame emission on stream close — no bytes are ever dropped.
- `AbortSignal` integration: aborted writes reject with `StreamAbortedError`.
- Plugin-OO wrapper `CoalesceBytesTransform` for use with `BaseTransform`,
hooks, listeners, and the `StreamManager` pipeline.
- Strict input validation: `minBytes <= 0` and `maxLatencyMs < 0` both throw
`RangeError` at construction time.
## Usage: Basic Example
```ts
import { coalesceBytes } from "@suaveplan/streams/transforms/coalesce-bytes/coalesce-bytes";
const coalesce = coalesceBytes(1024);
const result = source.pipeThrough(coalesce);
```
## Usage: Advanced Example
```ts
import { CoalesceBytesTransform } from "@suaveplan/streams/transforms/coalesce-bytes/coalesce-bytes";
let now = 0;
const t = new CoalesceBytesTransform({
minBytes: 4096,
maxLatencyMs: 50,
clock: () => now, // deterministic in tests
});
await source.pipeTo(
new WritableStream({
write(chunk) {
// ...
},
}),
{
// pipe through the OO wrapper
preventClose: false,
},
);
```
## API Reference
- `coalesceBytes(minBytes: number, options?: CoalesceBytesOptions):
TransformStream<Uint8Array, Uint8Array>` — factory.
- `CoalesceBytesOptions`:
- `maxLatencyMs?: number` (default `Infinity`)
- `clock?: Clock` (default `() => Date.now()`)
- `signal?: AbortSignal`
- `class CoalesceBytesTransform extends BaseTransform`
- `constructor(config: CoalesceBytesTransformConfig)`
- `process<T>(input: T): Promise<T>` — single-shot no-op pass-through.
- `toTransformStream<A, B>(): TransformStream<A, B>` — returns a fresh
coalescing stream wired to the constructor config.
- Throws `RangeError` for `minBytes <= 0` or `maxLatencyMs < 0`.
- Throws `StreamAbortedError` on aborted writes.
## Implementation Notes
The latency check is evaluated lazily on every chunk arrival rather than
through a real timer. This keeps the implementation deterministic under an
injected clock — tests advance time and write a probe chunk to flush. In
production, byte producers naturally tick frequently enough that this is the
desired behaviour. Buffered chunks are stored as a list of `Uint8Array`
references; only the final emission allocates a contiguous output buffer
(`new Uint8Array(bufferedSize)` + `set()` per chunk), so the steady-state
allocation is one buffer per emitted frame. The single-shot `process()` is a
pass-through because coalescing is fundamentally a streaming operation that
needs multiple inputs — real coalescing happens via `toTransformStream()`.

View File

@@ -0,0 +1,78 @@
# compress
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides `compressStream` and `decompressStream` — thin wrappers over the WHATWG `CompressionStream` and `DecompressionStream` platform APIs — for gzip, deflate, and raw deflate compression of binary streams. The wrappers add a consistent error wrapping contract: if the underlying platform constructor throws (e.g. for an unsupported format), the error is caught and re-thrown as `StreamPipeFailedError` with the original cause.
These transforms are designed for large-scale binary data movement: compressing database dumps, network payloads, or log files before writing to storage, and decompressing them on read.
## Features
- `compressStream`: wraps `CompressionStream` (gzip / deflate / deflate-raw)
- `decompressStream`: wraps `DecompressionStream` (gzip / deflate / deflate-raw)
- Constructor errors wrapped as `StreamPipeFailedError`
- Composable with `fromBytes`, `toBytes`, `encodeTextStream`
## Usage: Basic Example
```ts
import { compressStream, decompressStream } from "@suaveplan/streams/transforms/compress/compress";
import { fromBytes } from "@suaveplan/streams/readable/from-bytes/from-bytes";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const data = new TextEncoder().encode("hello world");
const compressed = fromBytes(data).pipeThrough(compressStream("gzip"));
const decompressed = compressed.pipeThrough(decompressStream("gzip"));
const chunks = await toArray(decompressed);
const text = new TextDecoder().decode(chunks[0]);
console.log(text); // "hello world"
```
## Usage: Advanced Example
```ts
import { compressStream, decompressStream } from "@suaveplan/streams/transforms/compress/compress";
import { fromBytes } from "@suaveplan/streams/readable/from-bytes/from-bytes";
import { encodeTextStream } from "@suaveplan/streams/transforms/text-codec/text-codec";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
// Compress a text stream to gzip
const textStream = fromArray(["line1\n", "line2\n"])
.pipeThrough(encodeTextStream())
.pipeThrough(compressStream("gzip"));
// Accumulate compressed bytes
const reader = textStream.getReader();
const parts: Uint8Array[] = [];
let r = await reader.read();
while (!r.done) { parts.push(r.value); r = await reader.read(); }
const totalBytes = parts.reduce((s, c) => s + c.byteLength, 0);
console.log("compressed size:", totalBytes);
```
## API Reference
### `compressStream(format?: CompressionFormat): TransformStream<Uint8Array, Uint8Array>`
- `format` — `"gzip"` | `"deflate"` | `"deflate-raw"` (default `"gzip"`).
**Throws** (immediate): `StreamPipeFailedError` if the platform `CompressionStream` constructor rejects the format.
### `decompressStream(format?: CompressionFormat): TransformStream<Uint8Array, Uint8Array>`
- `format` — `"gzip"` | `"deflate"` | `"deflate-raw"` (default `"gzip"`).
**Throws** (immediate): `StreamPipeFailedError` if the platform `DecompressionStream` constructor rejects the format.
### `CompressionFormat`
```ts
type CompressionFormat = "gzip" | "deflate" | "deflate-raw";
```
## Implementation Notes
Both functions are synchronous constructors that create and return the built-in stream objects. No additional buffering or chunk manipulation is performed — the platform implementation handles all compression logic. The wrapping `try/catch` is there purely for the error code contract; it should almost never trigger on modern runtimes that support all three formats.

View File

@@ -0,0 +1,157 @@
# crypto
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `crypto` module provides a streaming AEAD transform for `@suaveplan/streams`. It wraps a one-shot AEAD primitive (XChaCha20-Poly1305 or AES-256-GCM today, any custom engine tomorrow) into a `BaseTransform`-subclass plugin that can either encrypt or decrypt a stream of `Uint8Array` chunks. Per-frame nonce binding (`noncePrefix || u64be(counter)`) and per-frame additional-authenticated-data binding (`u64be(counter) || u8(isFinal)`) ensure that reorder, duplication, truncation, and tamper attacks all surface as a decryption tag-mismatch and are rethrown as `StreamCryptoDecryptFailedError` (`STREAM.CRYPTO_DECRYPT_FAILED`).
Encryption emits a one-time HEADER chunk carrying the random nonce-prefix on stream start, then one length-prefixed frame per input chunk, then a terminating empty `isFinal=true` frame on flush. Decryption mirrors that: it reads the header first, then frames, and errors out if the stream ends before a final frame is observed. The transform is engine-agnostic: any `StreamingAeadEngine` that exposes `encryptFrame` / `decryptFrame` and declares its `keyLength` and `noncePrefixLength` plugs in.
## Features
- Streaming AEAD via WHATWG `TransformStream<Uint8Array, Uint8Array>`
- Engine-pluggable: `xchacha20StreamingEngine` and `aesGcmStreamingEngine` ship in-package; custom engines welcome
- Per-frame counter and `isFinal` flag mixed into both nonce and AAD
- Tamper / reorder / duplicate / truncate detection — all surface as `STREAM.CRYPTO_DECRYPT_FAILED`
- Single-input `process(input)` round-trip mode for unit tests / one-off uses
- Type-safe hooks (`crypto:before-chunk`, `crypto:after-chunk`) — hooks may replace the chunk
- Type-safe events (`crypto:start`, `crypto:chunk`, `crypto:complete`, `crypto:error`)
- Constructor validation with descriptive `TypeError` for engine / key / operation mistakes
- `dispose()` zeroizes the private key copy
- AbortSignal-aware via `BaseTransform.assertNotAborted`
## Usage: Basic Example
```ts
import {
CryptoTransform,
xchacha20StreamingEngine,
} from "@suaveplan/streams/transforms/crypto/crypto";
const key = crypto.getRandomValues(new Uint8Array(32));
const enc = new CryptoTransform({
engine: xchacha20StreamingEngine,
key,
operation: "encrypt",
});
const dec = new CryptoTransform({
engine: xchacha20StreamingEngine,
key,
operation: "decrypt",
});
const plaintext = new Uint8Array([1, 2, 3, 4, 5]);
const wire = await enc.process(plaintext);
const recovered = await dec.process(wire);
// recovered == plaintext
```
## Usage: Advanced Example
Full streaming pipeline with hook-based redaction and event observability:
```ts
import {
aesGcmStreamingEngine,
CryptoTransform,
} from "@suaveplan/streams/transforms/crypto/crypto";
const key = crypto.getRandomValues(new Uint8Array(32));
const enc = new CryptoTransform({
engine: aesGcmStreamingEngine,
key,
operation: "encrypt",
});
enc.on("crypto:start", ({ engineId }) =>
console.log(`encrypt start (engine=${engineId})`)
);
enc.on("crypto:chunk", ({ counter, inBytes, outBytes }) =>
console.log(`frame ${counter}: ${inBytes} -> ${outBytes}`)
);
enc.on("crypto:complete", ({ totalChunks, totalBytes }) =>
console.log(`done: ${totalChunks} frames, ${totalBytes} bytes`)
);
// Mutate plaintext before sealing (redaction example)
enc.hook("crypto:before-chunk", ({ chunk, counter, operation }) => ({
chunk: counter === 0 ? new Uint8Array(chunk.length).fill(0) : chunk,
counter,
operation,
}));
const dec = new CryptoTransform({
engine: aesGcmStreamingEngine,
key,
operation: "decrypt",
});
await source
.pipeThrough(enc.toTransformStream())
.pipeThrough(dec.toTransformStream())
.pipeTo(sink);
enc.dispose(); // zeroize key
dec.dispose();
```
## API Reference
### `class CryptoTransform extends BaseTransform<CryptoHookMap, CryptoEventMap>`
#### `constructor(config: CryptoTransformConfig)`
Validates `config.engine` (must implement `StreamingAeadEngine`), `config.key` (must be a `Uint8Array` of length `engine.keyLength`), and `config.operation` (must be `"encrypt"` or `"decrypt"`). Throws plain `TypeError` with a descriptive message on any mismatch. The key is defensively copied; the original buffer is not retained.
#### `process<T>(input: T, options?: TransformOptions): Promise<T>`
Single-input mode. Treats `input` as one Uint8Array. Encrypt produces `header || final-frame`. Decrypt consumes that exact shape and rejects with `StreamCryptoInvalidFrameError` if the input is not exactly one final frame after the header.
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Streaming mode. On the encrypt side, emits the header chunk on the first input (or on flush if the input was empty), then one length-prefixed frame per input chunk, then a terminating final frame on flush. On the decrypt side, buffers until the header is complete, then drains complete frames, ignoring the final frame's payload but enforcing its presence — flushing without observing a final frame raises `StreamCryptoDecryptFailedError`.
#### `dispose(): void`
Zeroizes the internal key copy via `key.fill(0)`, then calls `super.dispose()` to clear hooks and listeners.
### Engine factories
- `xchacha20StreamingEngine` — `id="xchacha20-poly1305"`, `keyLength=32`, `noncePrefixLength=16`, wraps `@suaveplan/crypto-cipher::xchacha20poly1305`.
- `aesGcmStreamingEngine` — `id="aes-256-gcm"`, `keyLength=32`, `noncePrefixLength=4`, wraps `@suaveplan/crypto-cipher::aes256gcm`.
### Wire format
```text
HEADER := u8(0x01) || noncePrefix(N)
FRAME := u32be(payloadLen) || u8(isFinal) || ciphertext+tag
STREAM := HEADER FRAME* (last FRAME has isFinal=1, payload may be empty)
```
`N = engine.noncePrefixLength`. Per-frame nonce = `noncePrefix || u64be(counter)`. Per-frame AAD = `u64be(counter) || u8(isFinal)`.
### Hooks
- `crypto:before-chunk` — `{ chunk, counter, operation }`. May mutate `chunk` to redact or replace data before sealing/opening.
- `crypto:after-chunk` — fires after sealing/opening; `chunk` contains the post-operation bytes.
### Events
- `crypto:start` — `{ engineId, operation }`. Fires once per stream / once per `process` call.
- `crypto:chunk` — `{ counter, inBytes, outBytes }`. Fires per frame, including the terminating final frame.
- `crypto:complete` — `{ totalChunks, totalBytes }`. Fires on flush / after the last frame on the success path.
- `crypto:error` — `{ error }`. Fires on the failure path.
## Implementation Notes
The single-input `process(input)` path is provided for ergonomic one-shot uses (unit tests, single-message envelopes); streaming via `toTransformStream` is the canonical path. The wire format is identical between the two — `process` just always emits exactly one `isFinal=true` frame at counter 0.
Counter is a JS `number`; the `u64be` packing handles values up to `2^53 - 1`, which is more than enough for any realistic stream. AES-GCM is the harder case here: the AES-GCM nonce is 12 bytes, of which 4 are the prefix and 8 are the counter — operationally equivalent to the standard "fixed | invocation" partition recommended by NIST SP 800-38D.
The decrypt-side stream buffers incoming bytes in a single growing `Uint8Array`. For very large frame sizes, callers can shrink memory pressure by using a chunked source upstream — frames are decoded as soon as they are complete, which bounds the resident buffer at one frame plus a partial second frame.
`StreamCryptoInvalidFrameError` is reserved for structural problems (header tag mismatch, bad `isFinal` byte, premature end of buffer). `StreamCryptoDecryptFailedError` is reserved for cryptographic failures (tag mismatch, truncation, reorder). The two error codes let callers distinguish "this isn't AEAD-format data at all" from "this was tampered with."

View File

@@ -0,0 +1,130 @@
# dedup
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The dedup module ships two complementary deduplication primitives over the same key-extraction contract: `dedupStream`, a functional `TransformStream<T, T>` factory that drops _consecutive_ duplicates, and `DedupTransform`, a `BaseTransform` plugin-OO wrapper that holds a per-instance `Set<key>` for _all-time_ dedup via its single-input `process` path while delegating its streaming bridge to `dedupStream`. Together they cover both window-1 streaming dedup and full-set call-by-call dedup without duplicating key-extraction or abort plumbing.
This is the only dedup surface in `@suaveplan/streams`. For unbounded set-based global deduplication on a stream prefer `filterStream` with an externally-managed `Set` so memory bounds remain the caller's choice.
## Features
- `dedupStream` suppresses _consecutive_ duplicate chunks via strict-equality comparison on the optional `keyFn` result
- First chunk always passes through (sentinel-guarded)
- `DedupTransform.process` rejects any _all-time_ duplicate with `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`)
- `DedupTransform.toTransformStream()` delegates to `dedupStream(keyFn)` — identical wire-level semantics
- Default key function on the class is `JSON.stringify` (works for any structurally-comparable value); `dedupStream` uses identity
- `DedupKeyFn<T> = (chunk: T) => string | number` lets callers supply a stable, cheap key
- `AbortSignal` honoured before and after key extraction
- Configurable `highWaterMark` on the streaming form
- `dispose()` clears the per-instance `seen` set so reused instances do not leak memory
## Usage: Basic Example
```ts
import { dedupStream } from "@suaveplan/streams/transforms/dedup/dedup";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const result = await toArray(
fromArray([1, 1, 2, 3, 3, 3, 2]).pipeThrough(dedupStream())
);
console.log(result); // [1, 2, 3, 2] — non-consecutive 2s both pass
import { DedupTransform } from "@suaveplan/streams/transforms/dedup/dedup";
const t = new DedupTransform<number>();
await t.process(1); // 1
await t.process(2); // 2
await t.process(1); // throws StreamTransformFailedError("duplicate input")
```
## Usage: Advanced Example
```ts
import { DedupTransform, dedupStream } from "@suaveplan/streams/transforms/dedup/dedup";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// Key-based streaming dedup: dedup events by their type
const events = [
{ type: "click", x: 1 }, { type: "click", x: 2 },
{ type: "keydown", key: "a" }, { type: "keydown", key: "b" },
];
const consecutive = await toArray(
fromArray(events).pipeThrough(dedupStream((e) => e.type))
);
console.log(consecutive);
// [{ type: "click", x: 1 }, { type: "keydown", key: "a" }]
interface Event { id: string; payload: unknown }
// Class form — dedup by id only, ignore payload, all-time set
const t = new DedupTransform<Event>((e) => e.id);
await t.process({ id: "a", payload: 1 }); // ok
await t.process({ id: "a", payload: 2 }); // throws — same id
// Streaming bridge from the same instance (consecutive-only via dedupStream)
const source = new ReadableStream<Event>({
start(c) {
c.enqueue({ id: "x", payload: 1 });
c.enqueue({ id: "x", payload: 2 });
c.enqueue({ id: "y", payload: 3 });
c.close();
},
});
await source
.pipeThrough(t.toTransformStream<Event, Event>())
.pipeTo(
new WritableStream<Event>({
write(e) { console.log(e.id); },
})
);
// Logs: "x", "y"
t.dispose(); // clears the in-memory seen set
```
## API Reference
### `dedupStream<T>(keyFn?: (item: T) => unknown, options?: TransformOptions): TransformStream<T, T>`
**Parameters**
- `keyFn` — optional function to compute a comparison key; defaults to identity comparison.
- `options.signal` — `AbortSignal`.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns** `TransformStream<T, T>`.
**Throws** (via error channel) `StreamAbortedError` when the signal fires.
### `class DedupTransform<T = unknown>`
#### `constructor(keyFn?: DedupKeyFn<T>)`
`keyFn` extracts the dedup key. Defaults to `JSON.stringify(chunk)`.
#### `process<U>(input: U, options?: TransformOptions): Promise<U>`
Returns `input` if its key has not been seen on this instance; otherwise throws `StreamTransformFailedError` with code `STREAM.TRANSFORM_FAILED` and message `"duplicate input"`. Honours `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired.
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `dedupStream(keyFn)`. Drops only _consecutive_ duplicates — independent of the all-time `seen` set held for `process`.
#### `dispose(): void`
Clears the private `seen` set, then calls `super.dispose()` to release hooks and listeners. Idempotent.
### `type DedupKeyFn<T>`
`(chunk: T) => string | number`
## Implementation Notes
The sentinel `Symbol("EMPTY")` ensures the first chunk always passes through `dedupStream` — there is no real "previous key" yet. Comparison uses strict equality (`!==`) on the key, so objects are compared by reference unless `keyFn` returns a primitive. The last key is retained in the transform closure; state is tied to the stream instance lifetime.
State is per-`DedupTransform` instance: each instance owns its own `Set<string | number>`. Two instances do not share dedup state, even if they were constructed with structurally-identical key functions. Construct one instance per logical dedup window. The class deliberately diverges from `dedupStream` in the single-input call path — `dedupStream` is consecutive-only because it cannot assume safe memory bounds for an unbounded incoming stream, whereas `process` is invoked with caller-controlled cardinality and can hold the full key set. `dispose()` clears the seen set first, then delegates to `BaseTransform.dispose()` to drop hooks and listeners; this ordering keeps the seen-set cleanup observable to subclasses that wrap the method.

View File

@@ -0,0 +1,101 @@
# filter
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `filter` feature provides two surfaces over the same predicate-based filtering behaviour: `filterStream`, a functional factory that returns a `TransformStream<T, T>` passing only chunks for which a predicate returns `true`, and `FilterTransform`, the plugin-OO wrapper around it that satisfies the `BaseTransform` contract (hooks, events, `StreamManager`, `TransformFactory`). Chunks that fail the predicate are silently dropped in streaming mode; in single-input mode (`FilterTransform.process`) rejection surfaces as a `STREAM.TRANSFORM_FAILED` error because there is no natural absent-chunk return value across an `async` boundary.
Both forms accept synchronous or asynchronous predicates and honour `AbortSignal` before invoking the predicate. Errors thrown by the predicate are caught and re-wrapped as `StreamTransformFailedError` so they carry stable codes. This is the stream equivalent of `Array.prototype.filter` and is most useful combined with `mapStream` and `batchStream`: filter early to reduce downstream work.
## Features
- Sync and async predicates
- Failing chunks are dropped in streaming mode (not errored)
- `FilterTransform.process` throws `STREAM.TRANSFORM_FAILED` on rejection
- Predicate errors re-wrapped as `StreamTransformFailedError`
- AbortSignal support (checked before and after predicate in OO mode)
- Configurable `highWaterMark`
- Type-parameterised on `T` for full inference
## Usage: Basic Example
```ts
import { filterStream, FilterTransform } from "@suaveplan/streams/transforms/filter/filter";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// Functional form
const result = await toArray(
fromArray([1, 2, 3, 4, 5]).pipeThrough(filterStream((n) => n % 2 === 0))
);
console.log(result); // [2, 4]
// OO form
const positive = new FilterTransform<number>((n) => n > 0);
console.log(await positive.process(5)); // 5
await positive.process(-1); // throws STREAM.TRANSFORM_FAILED
```
## Usage: Advanced Example
```ts
import { filterStream, FilterTransform } from "@suaveplan/streams/transforms/filter/filter";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
import { StreamManager } from "@suaveplan/streams/manager/manager/manager";
// Async predicate (e.g. database lookup)
const validIds = new Set([1, 3, 5]);
const result = await toArray(
fromArray([1, 2, 3, 4, 5]).pipeThrough(
filterStream(async (id) => {
await Promise.resolve();
return validIds.has(id);
})
)
);
console.log(result); // [1, 3, 5]
// FilterTransform as a stage in a StreamManager pipeline
const even = new FilterTransform<number>(async (n) => {
await Promise.resolve();
return n % 2 === 0;
});
const out = await new StreamManager().pipe(even).execute(4); // 4
```
## API Reference
### `filterStream<T>(predicate, options?): TransformStream<T, T>`
**Parameters**:
- `predicate` — `(chunk: T) => boolean | Promise<boolean>`. Returning a falsy value drops the chunk.
- `options.signal` — AbortSignal; errors the stream on abort.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Throws** (via error channel): `StreamTransformFailedError` when the predicate throws, `StreamAbortedError` when the signal fires.
### `class FilterTransform<T = unknown>`
#### `constructor(predicate: FilterPredicate<T>)`
#### `process<U>(input: U, options?: TransformOptions): Promise<U>`
Awaits `predicate(input)`. Returns `input` unchanged on `true`. Throws `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`, message `"filter rejected input"`) on `false`. Honours `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired, both before and after the predicate runs.
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `filterStream(predicate)`. Rejected chunks are silently dropped; predicate errors are wrapped in `StreamTransformFailedError` at the readable side.
### `type FilterPredicate<T>`
`(chunk: T) => boolean | Promise<boolean>`
## Implementation Notes
Dropped chunks never reach `controller.enqueue`, so downstream backpressure is not applied for them. A filter-heavy pipeline may not apply upstream backpressure as quickly as expected — consider pairing with `limitStream` if you need a hard cap on output. The abort check runs before the predicate, so a signal that fires between chunks aborts cleanly without running the user function.
The OO class is a thin shell — the real filtering logic lives in `filterStream`. The class only provides the plugin surface (`name`, `process`, abort assertion) and a streaming bridge that delegates straight back to `filterStream(predicate)` so there is no parallel implementation and no drift surface. A filter has no natural single-input return value when the predicate rejects, so the OO contract uses the error path as the spec for rejection; callers that need silent drops should use `toTransformStream()` instead.

View File

@@ -0,0 +1,119 @@
# flat-map
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `flat-map` module ships two co-located APIs that share a single behavioural core: the `flatMapStream` factory function and the `FlatMapTransform` plugin-OO class. Both map each input chunk to an array of output chunks and enqueue every element individually, providing the streaming equivalent of `Array.prototype.flatMap`. They are useful for tokenising lines into words, expanding a page of results into individual records, splitting delimited chunks, or fanning a single batch event into many downstream chunks.
The function form is the canonical implementation; the class form is a thin shell that delegates to it through `toTransformStream()`. Because all flattening logic lives in one place, abort and transform-error semantics are identical between the two call paths and there is no parallel implementation drift.
## Features
- One-to-many chunk expansion (zero, one, or many outputs per input)
- Sync and async expansion functions
- Empty arrays silently produce no output chunks
- Errors re-wrapped as `StreamTransformFailedError`
- AbortSignal support — fires `StreamAbortedError` on the error channel
- Configurable `highWaterMark`
- Plugin-OO `FlatMapTransform` integrates with `BaseTransform`, `StreamManager`, and `TransformFactory`
- Single-input `process()` returns the array as-is (flattening is the streaming form's job)
## Usage: Basic Example
```ts
import { flatMapStream } from "@suaveplan/streams/transforms/flat-map/flat-map";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// Split sentences into words
const result = await toArray(
fromArray(["hello world", "foo bar"]).pipeThrough(
flatMapStream((sentence) => sentence.split(" "))
)
);
console.log(result); // ["hello", "world", "foo", "bar"]
```
## Usage: Advanced Example
```ts
import { FlatMapTransform } from "@suaveplan/streams/transforms/flat-map/flat-map";
// Tokenizer pipeline: split lines into words, then stream them out one at a time.
const tokenizer = new FlatMapTransform<string, string>(async (line) => {
await Promise.resolve();
return line
.split(/\s+/)
.map((w) => w.trim())
.filter((w) => w.length > 0);
});
// Single-shot — returns the array as-is, no flattening
console.log(await tokenizer.process("the quick brown fox"));
// ["the", "quick", "brown", "fox"]
// Streaming — each word emitted as its own chunk
const lines = new ReadableStream<string>({
start(c) {
c.enqueue("the quick brown fox");
c.enqueue("jumps over the lazy dog");
c.close();
},
});
const words: string[] = [];
await lines
.pipeThrough(tokenizer.toTransformStream<string, string>())
.pipeTo(
new WritableStream<string>({
write(w) {
words.push(w);
},
})
);
// words = ["the","quick","brown","fox","jumps","over","the","lazy","dog"]
```
## API Reference
### `flatMapStream<TIn, TOut>(fn, options?): TransformStream<TIn, TOut>`
**Parameters**:
- `fn` — `(chunk: TIn) => TOut[] | Promise<TOut[]>`. Expansion function; may be sync or async. Empty arrays produce no output.
- `options.signal` — AbortSignal; errors the stream on abort with `StreamAbortedError`.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns**: `TransformStream<TIn, TOut>`.
**Throws** (via error channel): `StreamTransformFailedError` when `fn` throws, `StreamAbortedError` when the signal fires.
### `class FlatMapTransform<TIn = unknown, TOut = unknown>`
#### `constructor(fn: FlatMapFn<TIn, TOut>)`
`fn` is `(chunk: TIn) => readonly TOut[] | Promise<readonly TOut[]>`.
#### `process<T>(input: T, options?: TransformOptions): Promise<T>`
Runs `fn(input)` and returns the resulting array as-is. `process` deliberately does NOT flatten — there is nothing to flatten into when there is no downstream consumer. Honors `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired, both before and after the work.
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `flatMapStream(fn)`. Each element of every array returned by `fn` is enqueued as its own chunk. Errors thrown by `fn` are wrapped in `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`).
### `type FlatMapFn<TIn, TOut>`
`(chunk: TIn) => readonly TOut[] | Promise<readonly TOut[]>`
## Implementation Notes
This module follows the canonical M2b `BaseTransform`-subclass pattern established by `MapTransform`:
1. The functional factory (`flatMapStream`) is the single source of truth for behaviour.
2. The OO subclass holds configuration in private readonly fields and exposes `name` from the constructor.
3. `process(input, options)` runs the operation single-shot, calling `assertNotAborted` before and after the work. It does NOT flatten — flattening only makes sense in a stream where downstream chunks have a natural ordering.
4. `toTransformStream()` returns the result of the functional factory directly — no parallel implementation, no drift surface.
All items in the returned array are enqueued synchronously before returning from `transform`, so a single input chunk may produce many downstream chunks without additional backpressure. If output arrays are large, downstream consumers may see bursty throughput. For smooth backpressure over variable-length expansions, split the source into smaller chunks before applying `flatMap`. Type erasure on the streaming bridge mirrors `MapTransform`: `BaseTransform.toTransformStream` declares method-level generics with loose `unknown` defaults, so the `as unknown as TransformStream<A, B>` cast is the right shape.

View File

@@ -0,0 +1,115 @@
# length-prefix-decode
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`lengthPrefixDecode` decodes a length-prefixed binary framing protocol back into
raw payload `Uint8Array` chunks. It is the inverse of a length-prefix encoder
and is the canonical primitive for parsing wire formats such as TCP message
frames, SLIP-style records, BSON / MessagePack blocks delimited by an explicit
size header, and any custom protocol that prefixes each payload with a
fixed-width little-endian or big-endian length field.
The transform is fully streaming: a single logical frame may span any number of
upstream chunks, and an upstream chunk may carry any number of complete frames
plus a residual partial frame. A small carry buffer holds bytes between calls
so that no frame boundary is ever lost.
## Features
- 1, 2, 4, and 8-byte prefix widths.
- Big-endian (default — network order) and little-endian byte orders.
- Configurable `maxFrameBytes` guard against memory-exhaustion attacks.
- Strict truncation detection: closing the upstream mid-prefix or mid-payload
raises `STREAM.TRUNCATED_FRAME` with the expected and received byte counts in
the error context.
- Plugin-OO `LengthPrefixDecodeTransform` wrapper with an
`await transform.process(input)` shorthand for one-shot decode of a single
`Uint8Array` containing one or more concatenated frames.
- Zero-copy on the prefix scan; payloads are copied once into stable
`Uint8Array` instances so consumers may retain references safely.
## Usage: Basic Example
```ts
import { lengthPrefixDecode } from "@suaveplan/streams/transforms/length-prefix-decode/length-prefix-decode";
// Wire bytes: 4-byte BE length prefix per frame.
const framed = new Uint8Array([
0x00, 0x00, 0x00, 0x03, 0xaa, 0xbb, 0xcc, // frame: [aa bb cc]
0x00, 0x00, 0x00, 0x02, 0xde, 0xad, // frame: [de ad]
]);
const stream = new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(framed);
controller.close();
},
});
const reader = stream.pipeThrough(lengthPrefixDecode(4)).getReader();
for (;;) {
const { value, done } = await reader.read();
if (done) break;
console.log(value); // Uint8Array(3) [aa bb cc], Uint8Array(2) [de ad]
}
```
## Usage: Advanced Example
Decoding a little-endian, 2-byte prefix protocol with a 64 KiB cap and a
plugin-OO wrapper so the same transform can be wired into a `StreamManager`
pipeline or invoked directly:
```ts
import { LengthPrefixDecodeTransform } from "@suaveplan/streams/transforms/length-prefix-decode/length-prefix-decode";
const decoder = new LengthPrefixDecodeTransform(2, {
byteOrder: "le",
maxFrameBytes: 64 * 1024,
});
// One-shot path: a single Uint8Array carrying many concatenated frames.
const payloadsConcat = await decoder.process(framedBytes);
// Streaming path: same options, fresh stream per call.
upstream.pipeThrough(decoder.toTransformStream());
```
If a frame's prefix value exceeds `maxFrameBytes` the stream errors with
`StreamFrameTooLargeError` (`STREAM.FRAME_TOO_LARGE`) carrying
`{ framePrefixValue, maxFrameBytes }` in its context. If the upstream closes
mid-frame the stream errors with `StreamTruncatedFrameError`
(`STREAM.TRUNCATED_FRAME`) carrying `{ expectedBytes, receivedBytes }`.
## API Reference
### `lengthPrefixDecode(prefixBytes, options?)`
- `prefixBytes: 1 | 2 | 4 | 8` — width of the length prefix.
- `options.byteOrder?: "le" | "be"` — defaults to `"be"`.
- `options.maxFrameBytes?: number` — defaults to `1 << 24` (16 MiB).
- Returns `TransformStream<Uint8Array, Uint8Array>`.
- Errors the stream with `StreamFrameTooLargeError` or `StreamTruncatedFrameError`.
### `class LengthPrefixDecodeTransform extends BaseTransform`
- `constructor(prefixBytes, options?)` — same surface as the factory.
- `process(input: Uint8Array): Promise<Uint8Array>` — decodes one input buffer
containing one or more frames into the concatenated payloads.
- `toTransformStream(): TransformStream<Uint8Array, Uint8Array>` — returns the
factory form for pipeline use.
## Implementation Notes
The carry buffer is allocated freshly on each yielded boundary so subarray
views never pin the entire upstream chunk in memory. The prefix is read with
multiplicative accumulation rather than bit-shifts so that 5–8 byte big-endian
prefixes do not silently wrap to 32-bit on V8. `maxFrameBytes` is checked
against the decoded prefix value before any payload bytes are buffered, so a
hostile peer cannot exhaust memory by announcing a huge frame they never
deliver. Truncation is detected in the WHATWG `flush` callback by inspecting
the carry buffer; this preserves the invariant that a clean stream close
always yields zero pending bytes.

View File

@@ -0,0 +1,98 @@
# `length-prefix-encode`
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `length-prefix-encode` transform prepends each `Uint8Array` chunk with a
binary length prefix, producing a self-delimiting wire format suitable for
framing messages over byte streams (TCP sockets, WebSockets in binary mode,
Node `Duplex`, or any pipe that does not preserve message boundaries). Pair it
with the matching `length-prefix-decode` transform on the receiving side to
recover the original chunk boundaries.
The encoder is the symmetric counterpart of the decoder used elsewhere in the
streams package and is intentionally allocation-light: each chunk produces a
single fresh `Uint8Array` of `prefixBytes + chunk.length` bytes, and the prefix
is written in place via direct byte arithmetic — no `DataView`, no scratch
buffer, no copy of the original chunk beyond `set()` into the output.
## Features
- Functional form (`lengthPrefixEncode`) returning a WHATWG `TransformStream`.
- Plugin-OO form (`LengthPrefixEncodeTransform`) extending `BaseTransform` for
use with `StreamManager`, `TransformFactory`, hooks, and listeners.
- Configurable prefix width: `1`, `2`, `4`, or `8` bytes. The 8-byte form
encodes the length as a `u64` via `BigInt`.
- Configurable byte order: `'be'` (default) or `'le'`.
- `AbortSignal` integration on the streaming form: aborted signals reject the
next read with `StreamAbortedError`.
- Overflow detection: chunks larger than the prefix can represent surface as
`StreamFrameTooLargeError` (which extends `RangeError` semantics by carrying
the frame value and configured maximum on its `context`).
## Usage: Basic Example
```ts
import { lengthPrefixEncode } from "@suaveplan/streams/transforms/length-prefix-encode/length-prefix-encode";
const stream = source.pipeThrough(lengthPrefixEncode(4));
// Each chunk is now prefixed with a 4-byte big-endian length.
```
## Usage: Advanced Example
```ts
import { LengthPrefixEncodeTransform } from "@suaveplan/streams/transforms/length-prefix-encode/length-prefix-encode";
const encoder = new LengthPrefixEncodeTransform(2, { byteOrder: "le" });
// Plugin-OO single-shot encode:
const framed = await encoder.process(new Uint8Array([1, 2, 3]));
// framed -> Uint8Array([0x03, 0x00, 1, 2, 3])
// Streaming form, sharing the same configuration:
await source.pipeThrough(encoder.toTransformStream<Uint8Array, Uint8Array>())
.pipeTo(sink);
```
## API Reference
### `lengthPrefixEncode(prefixBytes, options?)`
- `prefixBytes`: `1 | 2 | 4 | 8` — width of the length prefix in bytes.
- `options.byteOrder`: `'le' | 'be'` (default `'be'`).
- `options.signal`: `AbortSignal` — when aborted, the next `read()` rejects
with `StreamAbortedError`.
- Returns: `TransformStream<Uint8Array, Uint8Array>`.
- Throws: per-chunk `StreamFrameTooLargeError` into the stream when
`chunk.length` exceeds the maximum representable in `prefixBytes`.
### `class LengthPrefixEncodeTransform extends BaseTransform`
- `new LengthPrefixEncodeTransform(prefixBytes, options?)` — same arguments as
the function form (without `signal`, which is supplied per call).
- `process(input, options?)` — encodes a single `Uint8Array`. Honors
`options.signal` via `assertNotAborted`. Returns the prefixed bytes.
- `toTransformStream<A, B>()` — returns the function form, preserving the
configured `prefixBytes` and `byteOrder`.
- `name` — `'length-prefix-encode'`.
## Implementation Notes
- The 8-byte branch uses `BigInt` arithmetic so it can represent any chunk
length the runtime can allocate. The 1/2/4-byte branches likewise share the
`BigInt` shift loop for code economy; benchmarks show the cost is a single-
digit nanosecond-per-chunk overhead for the small widths and is dominated
by the `Uint8Array` allocation in every realistic workload.
- Overflow checking compares `BigInt(length) > MAX_BY_PREFIX[prefixBytes]`;
the maximum is `2^(prefixBytes*8) - 1`.
- Empty chunks are encoded as a bare prefix (`prefixBytes` zero bytes), which
the decoder MUST accept and emit as an empty `Uint8Array`. This preserves
exact one-to-one chunk boundaries through the framing layer.
- The `toTransformStream` override delegates to `lengthPrefixEncode`, so the
streaming and OO forms share a single tested implementation path. Hooks and
listeners registered on the `LengthPrefixEncodeTransform` instance are not
walked by the streaming form — this matches the precedent set by
`MapTransform` and is documented in the `BaseTransform` contract.

View File

@@ -0,0 +1,137 @@
# limit
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `limit` module exports both a functional factory (`limitStream`) and a plugin-OO wrapper (`LimitTransform`) that cap how many chunks may flow through a stream. `limitStream` creates a `TransformStream<T, T>` that passes at most `max` chunks downstream and then terminates the stream by calling `controller.terminate()`. `LimitTransform` is the `BaseTransform` plugin form: in single-input mode (`process(chunk)`) it returns the input unchanged for the first `max` invocations and throws `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`) on every subsequent call; in streaming mode (`toTransformStream()`) it delegates to `limitStream(max)`.
This is the stream equivalent of `Array.prototype.slice(0, max)` and is useful when you only need the first N results from an infinite or very large stream — for example, paginated previews, search result limits, or safety caps on resource-intensive pipelines. Once the limit is reached, the writable side is closed and any further chunks from the source are not pulled, allowing backpressure to propagate naturally to the upstream producer.
## Features
- Hard cap on output chunk count (`limitStream`)
- Stream is terminated (not errored) once the limit is reached
- `max = 0` produces an empty stream immediately
- Throws `RangeError` for `max < 0` (both forms)
- Configurable `highWaterMark` on the functional factory
- Plugin-OO `LimitTransform.process(chunk)` returns input unchanged within the limit, throws `StreamTransformFailedError` once exceeded
- `LimitTransform.toTransformStream()` delegates to `limitStream(max)` so abort + termination semantics are identical to the functional factory
- Honors `options.signal` before and after the count check
- `dispose()` resets the per-instance `process` counter so the contract is well-defined for `StreamManager`-managed lifecycles
- Type-parameterised on `T` so callers get full inference
## Usage: Basic Example
```ts
import { limitStream } from "@suaveplan/streams/transforms/limit/limit";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const result = await toArray(
fromArray([1, 2, 3, 4, 5]).pipeThrough(limitStream(3))
);
console.log(result); // [1, 2, 3]
```
```ts
import { LimitTransform } from "@suaveplan/streams/transforms/limit/limit";
const t = new LimitTransform<number>(2);
await t.process(1); // 1
await t.process(2); // 2
await t.process(3); // throws StreamTransformFailedError
```
## Usage: Advanced Example
```ts
import { limitStream } from "@suaveplan/streams/transforms/limit/limit";
import { fromAsyncGenerator } from "@suaveplan/streams/readable/from-generator/from-generator";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// Safely cap an infinite generator
async function* infiniteCounter() {
let n = 0;
while (true) yield n++;
}
const first10 = await toArray(
fromAsyncGenerator(infiniteCounter).pipeThrough(limitStream(10))
);
console.log(first10); // [0, 1, 2, ..., 9]
```
```ts
import { LimitTransform } from "@suaveplan/streams/transforms/limit/limit";
const t = new LimitTransform<number>(3);
// Streaming form: emits at most 3 chunks then closes the readable side.
const source = new ReadableStream<number>({
start(c) {
for (let i = 0; i < 100; i++) c.enqueue(i);
c.close();
},
});
const out: number[] = [];
await source
.pipeThrough(t.toTransformStream<number, number>())
.pipeTo(
new WritableStream<number>({
write(v) {
out.push(v);
},
})
);
console.log(out); // [0, 1, 2]
```
## API Reference
### `limitStream<T>(max: number, options?: TransformOptions): TransformStream<T, T>`
**Parameters**:
- `max` — maximum number of chunks to emit; must be `>= 0`.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns**: `TransformStream<T, T>`.
**Throws** (immediate): `RangeError` if `max < 0`.
### `class LimitTransform<T = unknown>`
#### `constructor(max: number)`
`max` is the maximum number of chunks the transform will admit. Must be `>= 0`; otherwise the constructor throws `RangeError`.
#### `process<U>(input: U, options?: TransformOptions): Promise<U>`
Honors `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired. Increments the per-instance counter and returns `input` unchanged while the counter is `<= max`. Once the counter has reached `max`, every further call rejects with `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`).
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `limitStream(max)`. The streaming form **does not throw** when the limit is reached — it silently terminates the readable side after enqueuing `max` chunks. This matches the historical `limitStream` contract.
#### `dispose(): void`
Resets the `process` counter to zero, clears all hooks/listeners, and calls `super.dispose()`. Idempotent.
### `interface LimitTransformConfig`
`{ max: number }` — config record kept under `limit.types.ts` for `TransformFactory` integration.
## Implementation Notes
`controller.terminate()` is called immediately after enqueuing the `max`-th chunk in `limitStream`. This closes the readable side of the transform stream while also signalling the writable side to stop accepting chunks. The WHATWG Streams specification defines `terminate()` as the correct way to close a transform from within the transformer — it is distinct from `close()` (which would only close the readable side and could leave the writable side open).
`LimitTransform` is one of the rare stateful transforms in this package. Its state is a single `count` integer that tracks single-input `process` invocations.
**Per-instance counter, not shared with the streaming form.** Each call to `toTransformStream()` constructs a fresh `TransformStream` via `limitStream(max)` which keeps its own internal counter inside the closure. That counter is independent from the class's `process` counter — running the streaming form to completion does not advance the `process` counter and vice versa. This is intentional: the two call paths represent different consumption models (one-at-a-time vs. pipe-through) and bleeding state between them would surprise callers.
**Reset via `dispose()`.** The streaming form's counter is collected with the `TransformStream` itself when its readable side is consumed, so it needs no explicit reset. The `process` counter is per-instance and persists for the life of the `LimitTransform` instance. `dispose()` zeroes it before delegating to `super.dispose()` so a `StreamManager` reusing a transform instance after teardown gets a clean slate.
**Single source of truth for streaming behaviour.** `toTransformStream()` delegates straight to `limitStream(max)` — no parallel implementation, no drift surface. Single-input semantics (throw on exceed) deliberately differ from streaming semantics (silently terminate) because the WHATWG `TransformStream` cannot signal "limit reached" without erroring the readable side, which is the wrong default for take-N composition.

View File

@@ -0,0 +1,150 @@
# map
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `map` module provides the fundamental value-transformation primitive for `@suaveplan/streams` in two complementary call shapes: `mapStream`, a functional `TransformStream` factory, and `MapTransform`, a `BaseTransform`-subclass plugin wrapper. Both apply a synchronous or asynchronous mapping function to every chunk passing through, with identical abort and error semantics. Errors thrown inside the mapping function are caught and re-wrapped as `StreamTransformFailedError` so they carry a stable error code and are distinguishable from infrastructure errors. AbortSignal support lets callers cancel the pipeline mid-flight without leaking resources.
The function form is the single source of truth for behaviour; the class is a thin shell that adds the `BaseTransform` plugin surface (`name`, `process`, `toTransformStream`) so consumers using `StreamManager`, `TransformFactory`, or hooks/events can express a map stage uniformly. Composition with `filterStream`, `batchStream`, or `tapStream` via `pipeThrough` covers every production use case.
## Features
- Synchronous and async mapping functions
- Per-chunk abort check via `AbortSignal`
- Errors re-wrapped as `StreamTransformFailedError` with original cause
- Configurable `highWaterMark` for downstream backpressure
- Zero buffering — one-to-one chunk throughput
- Single-input `process(chunk)` returns the mapped value, awaiting async `fn`
- Streaming `toTransformStream()` delegates to `mapStream(fn)` so behaviour never drifts between call paths
- Honors `options.signal` before and after applying the mapping function (class form)
- Type-parameterised on `TIn` / `TOut` so callers get full inference
## Usage: Basic Example
Functional form:
```ts
import { mapStream } from "@suaveplan/streams/transforms/map/map";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const result = await toArray(
fromArray([1, 2, 3]).pipeThrough(mapStream((n) => n * 2))
);
console.log(result); // [2, 4, 6]
```
Class form:
```ts
import { MapTransform } from "@suaveplan/streams/transforms/map/map";
const upper = new MapTransform<string, string>((s) => s.toUpperCase());
console.log(await upper.process("hello")); // "HELLO"
```
## Usage: Advanced Example
Functional form — async mapping with abort and chained composition:
```ts
import { mapStream } from "@suaveplan/streams/transforms/map/map";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const ac = new AbortController();
const stream = fromArray(["https://example.com/a", "https://example.com/b"])
.pipeThrough(
mapStream(async (url) => {
const res = await fetch(url, { signal: ac.signal });
return res.text();
}, { signal: ac.signal })
);
const doubled = fromArray([1, 2, 3])
.pipeThrough(mapStream((x) => x * 2))
.pipeThrough(mapStream((x) => x + 1));
const result = await toArray(doubled);
console.log(result); // [3, 5, 7]
```
Class form — single-input plus streaming bridge:
```ts
import { MapTransform } from "@suaveplan/streams/transforms/map/map";
import { StreamManager } from "@suaveplan/streams/manager/manager/manager";
const t = new MapTransform<number, number>(async (n) => {
await Promise.resolve();
return n * 2;
});
const out = await new StreamManager().pipe(t).execute(7); // 14
const source = new ReadableStream<number>({
start(c) {
c.enqueue(1);
c.enqueue(2);
c.close();
},
});
const ctrl = new AbortController();
await source
.pipeThrough(t.toTransformStream<number, number>())
.pipeTo(
new WritableStream<number>({
write(v) {
console.log(v);
},
}),
{ signal: ctrl.signal }
);
```
## API Reference
### `mapStream<TIn, TOut>(fn: (chunk: TIn) => TOut | Promise<TOut>, options?: TransformOptions): TransformStream<TIn, TOut>`
**Parameters**:
- `fn` — mapping function; may be sync or async.
- `options.signal` — AbortSignal; errors the transform on abort.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns**: `TransformStream<TIn, TOut>`.
**Throws** (via stream error channel): `StreamTransformFailedError` when `fn` throws, `StreamAbortedError` when `signal` fires.
### `class MapTransform<TIn = unknown, TOut = unknown>`
#### `constructor(fn: MapFn<TIn, TOut>)`
`fn` is `(chunk: TIn) => TOut | Promise<TOut>`.
#### `process<T>(input: T, options?: TransformOptions): Promise<T>`
Applies `fn(input)` and returns the result. Honors `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired. The generic `T` lets the caller narrow `input`/return at the call site without the class needing two generic parameters at the method level.
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `mapStream(fn)`. Errors thrown by `fn` are wrapped in `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`) at the readable side.
### `type MapFn<TIn, TOut>`
`(chunk: TIn) => TOut | Promise<TOut>`
## Implementation Notes
The abort check happens at the start of each `transform` call, not via an event listener, to avoid listener cleanup complexity. When `fn` throws synchronously, the error is caught by the surrounding `try/catch` and re-wrapped — the stream controller is errored, which cancels both the readable and writable sides of the transform. There is no retry logic; error recovery is the caller's responsibility.
`MapTransform` is the canonical reference for the `BaseTransform`-subclass pattern. Every other ported transform in the streams package follows the same shape:
1. The functional factory (`mapStream`, `filterStream`, …) stays untouched — single source of truth for behaviour.
2. The OO subclass holds the configuration in private readonly fields and exposes `name` from the constructor.
3. `process(input, options)` runs the operation single-shot, calling `assertNotAborted` before and after the work.
4. `toTransformStream()` returns the result of the functional factory directly — no parallel implementation, no drift surface.
Type erasure on the streaming bridge is intentional: `BaseTransform.toTransformStream` declares `<TIn, TOut>` as method-level generics with the loose `unknown` defaults from the abstract class, so the `as unknown as TransformStream<A, B>` cast is the right shape. Callers that want strict typing supply the generics at the call site (`t.toTransformStream<number, string>()`); callers that don't get the loose defaults.

View File

@@ -0,0 +1,146 @@
# sliding-bytes
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `sliding-bytes` transform re-frames a `Uint8Array` byte stream into
fixed-length overlapping (or gapped) byte windows, advancing by a
caller-controlled stride per emission. It is the byte-oriented counterpart of
the generic `window` transform: where `window` operates on item streams and
emits `T[]` arrays, `slidingBytes` operates directly on bytes and emits
zero-copy `Uint8Array` windows of length `size`.
This is the right primitive when you need fixed-size frames with overlap
semantics — for example, computing rolling hashes (Rabin-Karp,
content-defined chunking previews), running a sliding FFT or convolution,
batching audio samples for a vocoder, or feeding a signature-detection state
machine that must straddle frame boundaries. Unlike `chunkBytes`, sliding
windows can revisit the same byte across multiple emissions, which is
load-bearing for any algorithm whose context window exceeds its hop size.
## Features
- Functional factory `slidingBytes(size, stride)` returning a
`TransformStream<Uint8Array, Uint8Array>`.
- Plugin-OO wrapper `SlidingBytesTransform` extending `BaseTransform` with
hook/event/dispose semantics inherited from the base class.
- Configurable `size` (window length) and `stride` (emission step).
- `stride < size` produces overlapping windows (overlap = `size - stride`).
- `stride === size` produces non-overlapping windows (equivalent to a
drop-tail `chunkBytes`).
- `stride > size` produces gapped windows that discard `stride - size` bytes
between consecutive emissions.
- Carry buffer that correctly accumulates and re-emits across upstream chunk
boundaries, regardless of how the producer happens to slice its writes.
- Trailing residual smaller than `size` is dropped on flush — sliding windows
are fixed-length by definition.
- Synchronous `RangeError` for invalid `size` or `stride` (`<= 0`,
non-finite).
## Usage: Basic Example
```ts
import { fromBytes } from "@suaveplan/streams/readable/from-bytes/from-bytes";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
import { slidingBytes } from "@suaveplan/streams/transforms/sliding-bytes/sliding-bytes";
const input = new Uint8Array(16);
for (let i = 0; i < 16; i++) input[i] = i;
const windows = await toArray(
fromBytes(input, 16).pipeThrough(slidingBytes(8, 4))
);
// windows.length === 3
// windows[0] = bytes 0..7
// windows[1] = bytes 4..11 (overlaps windows[0] by 4 bytes)
// windows[2] = bytes 8..15 (overlaps windows[1] by 4 bytes)
```
## Usage: Advanced Example
```ts
import { SlidingBytesTransform } from "@suaveplan/streams/transforms/sliding-bytes/sliding-bytes";
// Gapped sampling: read 4 bytes, skip 4, read 4, skip 4, ...
const sampler = new SlidingBytesTransform({ size: 4, stride: 8 });
// Compose with the rest of the pipeline via the standard plugin surface.
const stream = sampler.toTransformStream<Uint8Array, Uint8Array>();
// Plug into any ReadableStream<Uint8Array> source. The transform is stateful
// across chunk boundaries — the upstream is free to deliver bytes in any
// chunk granularity (1 byte at a time, mega-chunks, anything in between)
// and the windowing semantics are identical.
const out: Uint8Array[] = [];
await stream.readable.pipeTo(
new WritableStream({
write(chunk) {
out.push(chunk);
},
})
);
```
## API Reference
### `slidingBytes(size: number, stride: number): TransformStream<Uint8Array, Uint8Array>`
Returns a WHATWG `TransformStream` that emits `Uint8Array` windows of length
`size`, advancing by `stride` bytes per emission.
- `size` — window length in bytes. Must be a finite number `> 0`.
- `stride` — emission step in bytes. Must be a finite number `> 0`.
Throws `RangeError` synchronously when either argument is non-finite or
`<= 0`. Errors the stream with `TypeError` when an upstream chunk is not a
`Uint8Array`.
### `class SlidingBytesTransform extends BaseTransform`
Plugin-OO wrapper. Constructor takes a `SlidingBytesTransformConfig` with
`size` and `stride` fields. Same validation as the functional factory.
- `name` — `"sliding-bytes"`.
- `process<T>(input: T): Promise<T>` — degenerate single-shot: returns
`input` unchanged. Real windowing requires stream-mode use because the
operator is stateful across chunks.
- `toTransformStream<A, B>(): TransformStream<A, B>` — delegates to
`slidingBytes(size, stride)`. Each call returns a fresh stream.
### `interface SlidingBytesTransformConfig`
```ts
interface SlidingBytesTransformConfig {
readonly size: number;
readonly stride: number;
}
```
## Implementation Notes
- **Carry buffer.** Each upstream chunk is concatenated onto a residual carry
buffer. The transform then emits every complete `size`-byte window starting
at offsets `0, stride, 2*stride, ...` within that combined buffer. Once no
further window fits, the bytes from the next would-be start offset onward
are retained as the new carry. When `stride > size` and the offset
overshoots the buffer (the "gap" case), the carry is reset to empty so the
inter-window bytes are correctly discarded.
- **Zero-copy emission.** Each emitted window is copied into a fresh
`Uint8Array` rather than a `subarray` view. This is required because the
carry buffer is replaced on every transform call and a `subarray` would
alias memory whose lifetime ends at the next chunk. The cost is one
`size`-byte copy per emission; the win is no aliasing-induced memory pin.
- **Flush policy.** The trailing residual that cannot fill a complete window
is dropped. Unlike `chunkBytes` (which exposes `tail: emit | pad | drop`),
sliding windows are only meaningful at exact length, so there is no
configurable tail policy here. Callers needing partial-tail behaviour
should compose with `chunkBytes` instead.
- **Single-shot degeneracy.** `SlidingBytesTransform.process(input)` returns
the input unchanged. The operator is fundamentally a streaming construct —
a single buffer has only one "frame," and the most useful semantics for a
one-call API is the identity. Use `toTransformStream()` for real work.
- **Validation.** Both the factory and the class throw `RangeError` for
non-finite or non-positive `size`/`stride`. Validation runs synchronously
at construction so misuse fails before any data flows.

View File

@@ -0,0 +1,102 @@
# SplitJoinTransform
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`SplitJoinTransform` is a plugin-OO `BaseTransform` that breaks a single value (string, `ArrayBuffer`, or array) into a sequence of chunks of fixed size with optional overlap, and reassembles such a sequence back into the original value. It is the workhorse used upstream of windowed processors (chunked LLM context, large-blob upload pipelines, paged compression) and downstream of those same pipelines when chunked output must be glued back together.
A single instance is bound to one operation — `"split"` or `"join"` — chosen at construction time. Both single-shot (`process`) and chunk-at-a-time (`toTransformStream`) semantics are implemented: `process` operates on one whole input, while the transform stream emits each sub-chunk individually for split and buffers-then-emits-once-on-close for join. AbortSignal cancellation is honoured at every yield point and surfaced via `StreamAbortedError` (`STREAM.ABORTED`); shape mismatches surface as `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`).
## Features
- Three input families: `string`, `ArrayBuffer`, and `T[]`.
- Configurable `chunkSize` and `overlap` (overlap must satisfy `0 <= overlap < chunkSize`).
- Round-trip-preserving join with `removeOverlap` for sliding-window splits.
- Configurable `separator` for plain (no-overlap) string joins.
- Type guards (`isSplittable`, `isJoinable`) for safe runtime branching.
- Stand-alone helpers: `splitString`, `splitArrayBuffer`, `splitArray`, `joinChunks`.
- Signal-aware `process` and stream constructors — both reject with `StreamAbortedError` once the supplied `AbortSignal` fires.
## Usage: Basic Example
```ts
import { SplitJoinTransform } from "@suaveplan/streams/transforms/split-join/split-join";
const splitter = new SplitJoinTransform("split", { chunkSize: 4 });
const result = await splitter.process("abcdefghij");
// result === { chunks: ["abcd", "efgh", "ij"], metadata: { chunkSize: 4, overlap: 0, totalChunks: 3, originalLength: 10 } }
const joiner = new SplitJoinTransform("join");
const original = await joiner.process(result);
// original === "abcdefghij"
```
## Usage: Advanced Example
Sliding-window split over a UTF-8-encoded buffer with overlap, piped through a stream:
```ts
import { SplitJoinTransform } from "@suaveplan/streams/transforms/split-join/split-join";
const splitter = new SplitJoinTransform("split", { chunkSize: 1024, overlap: 64 });
const joiner = new SplitJoinTransform("join", { removeOverlap: true });
const inStream = new ReadableStream<string>({
start(c) {
c.enqueue(longInputString);
c.close();
},
});
const subChunks: string[] = [];
await inStream
.pipeThrough(splitter.toTransformStream<string, string>())
.pipeTo(
new WritableStream<string>({ write: (v) => { subChunks.push(v); } })
);
// Reassemble with overlap stripped:
const rejoined = await joiner.process({
chunks: subChunks,
metadata: { chunkSize: 1024, overlap: 64, totalChunks: subChunks.length, originalLength: longInputString.length },
});
// rejoined === longInputString
```
## API Reference
### `new SplitJoinTransform(operation, config?)`
- `operation: "split" | "join"` — required. Throws `TypeError` for any other value.
- `config?: SplitJoinConfig` — `{ chunkSize?: number; overlap?: number; separator?: string; removeOverlap?: boolean }`. Defaults: `chunkSize = 1024`, `overlap = 0`, `separator = ""`, `removeOverlap = true`.
### `process<T>(input, options?): Promise<T>`
- Split mode: validates `isSplittable(input)`. Returns `{ chunks, metadata }`.
- Join mode: validates `isJoinable(input)`. Returns the reassembled value (string, `ArrayBuffer`, or `unknown[]`).
- Throws `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`) on shape mismatch.
- Throws `StreamAbortedError` (`STREAM.ABORTED`) when `options.signal` has fired.
### `toTransformStream<A, B>(options?): TransformStream<A, B>`
- Split: every input value is split, and each sub-chunk is enqueued individually.
- Join: every input chunk is buffered; the joined value is enqueued exactly once on `flush`.
### Helpers
- `splitString(s, chunkSize, overlap?): string[]`
- `splitArrayBuffer(buf, chunkSize, overlap?): ArrayBuffer[]`
- `splitArray<T>(arr, chunkSize, overlap?): T[][]`
- `joinChunks(chunks, { separator?, removeOverlap?, overlap? }): string | ArrayBuffer | unknown[]`
- `isSplittable(value): value is string | ArrayBuffer | unknown[]`
- `isJoinable(value): value is { chunks: unknown[]; metadata?: object }`
## Implementation Notes
- ArrayBuffer chunks are independent copies (`view.slice(...)`); the source buffer can be detached without invalidating chunks.
- Overlap-aware joins are applied per element type: strings are concatenated minus the leading `overlap` chars on every chunk after the first; `ArrayBuffer`s skip the leading `overlap` bytes; arrays skip the leading `overlap` items. With `removeOverlap: false`, overlap is preserved verbatim.
- Type guards are pure structural checks — they do not run zod validation. Both are exposed for callers to branch safely without try/catch.
- The constructor rejects unknown operations with a plain `TypeError` rather than `StreamTransformFailedError` because the failure is a programming bug, not a runtime stream condition.
- The default chunk size is 1024, matching the legacy package's heuristic for "small enough to stream, large enough to amortise per-call overhead." Tune via `chunkSize` for tighter sliding windows.

View File

@@ -0,0 +1,120 @@
# tap
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `tap` module exposes two complementary forms of the same side-effect probe: `tapStream`, the WHATWG `TransformStream` factory, and `TapTransform`, the plugin-OO wrapper used by `BaseTransform`-aware pipelines (`StreamManager`, `TransformFactory`, hook/event consumers). Both forms invoke a caller-supplied function for every chunk and forward the chunk downstream unchanged — the stream equivalent of a debug probe attached to logging, metrics, tracing, or test assertions.
Errors thrown by the side-effect are caught and re-wrapped as `StreamTransformFailedError`, erroring the stream and propagating upstream cancellation. For error-tolerant tapping (fire-and-forget) wrap the function in a try/catch before passing it in. The class form additionally honours `options.signal` before and after the side-effect via `assertNotAborted`, rejecting with `StreamAbortedError` when fired.
## Features
- Identity passthrough — chunks are never modified
- Sync and async side-effect functions
- Errors in `fn` error the stream (`StreamTransformFailedError`)
- AbortSignal support (`StreamAbortedError`)
- Configurable `highWaterMark`
- OO form (`TapTransform`) plugs into `BaseTransform`-driven pipelines
- Streaming form (`tapStream`) is the single source of truth — the OO form delegates to it
## Usage: Basic Example
```ts
import { tapStream } from "@suaveplan/streams/transforms/tap/tap";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const seen: number[] = [];
const result = await toArray(
fromArray([1, 2, 3]).pipeThrough(tapStream((n) => seen.push(n)))
);
console.log(result); // [1, 2, 3]
console.log(seen); // [1, 2, 3]
```
```ts
import { TapTransform } from "@suaveplan/streams/transforms/tap/tap";
const log = new TapTransform<string>((s) => console.log("[tap]", s));
await log.process("hello"); // logs "[tap] hello", returns "hello"
```
## Usage: Advanced Example
```ts
import { TapTransform, tapStream } from "@suaveplan/streams/transforms/tap/tap";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { mapStream } from "@suaveplan/streams/transforms/map/map";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
import { StreamManager } from "@suaveplan/streams/manager/manager/manager";
let chunkCount = 0;
const result = await toArray(
fromArray([10, 20, 30])
.pipeThrough(
tapStream((n) => {
chunkCount++;
console.log("before:", n);
})
)
.pipeThrough(mapStream((n) => n / 10))
.pipeThrough(tapStream((n) => console.log("after:", n)))
);
// logs: before: 10, after: 1, before: 20, after: 2, before: 30, after: 3
console.log(chunkCount); // 3
let total = 0;
const meter = new TapTransform<number>(async (n) => {
await Promise.resolve();
total += n;
});
await new StreamManager().pipe(meter).execute(5); // process(5) returns 5, total === 5
```
## API Reference
### `tapStream<T>(fn: (chunk: T) => void | Promise<void>, options?: TransformOptions): TransformStream<T, T>`
**Parameters**:
- `fn` — side-effect function; return value is ignored. May be async.
- `options.signal` — AbortSignal; errors the stream on abort.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns**: `TransformStream<T, T>`.
**Throws** (via error channel): `StreamTransformFailedError` when `fn` throws, `StreamAbortedError` when the signal fires.
### `class TapTransform<T = unknown>`
#### `constructor(fn: TapFn<T>)`
`fn` is `(chunk: T) => void | Promise<void>`. Any value returned by the function is ignored — only the original chunk is forwarded.
#### `process<U>(input: U, options?: TransformOptions): Promise<U>`
Awaits `fn(input)` and returns `input` unchanged. Honours `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired before or after the side-effect runs.
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `tapStream(fn)`. Errors thrown by `fn` are wrapped in `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`) at the readable side.
### `type TapFn<T>`
`(chunk: T) => void | Promise<void>` — sync or async side-effect callback.
## Implementation Notes
`fn` is awaited before the chunk is enqueued, so an async tap delays throughput by the duration of the async operation. For fire-and-forget semantics (e.g. non-blocking analytics calls), pass a function that starts the async work but does not `await` it. The chunk passthrough (`controller.enqueue(chunk)`) executes only after `fn` resolves, ensuring ordering is preserved.
The OO class follows the canonical M2b `BaseTransform`-subclass pattern established by `MapTransform`:
1. The functional factory (`tapStream`) stays untouched — single source of truth for behaviour.
2. The OO subclass holds the configuration in a private readonly field and exposes `name` from the constructor.
3. `process(input, options)` runs the side-effect single-shot, calling `assertNotAborted` before and after the work, then returns the original input.
4. `toTransformStream()` returns the result of the functional factory directly — no parallel implementation, no drift surface.
Because tap is a stateless pass-through, there is no buffering, no batching, and no internal state to reset between invocations. Callers who need rate limiting on the side-effect should wrap `fn` themselves; this module deliberately stays unopinionated.

View File

@@ -0,0 +1,63 @@
# text-codec
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`text-codec` provides two thin wrapper factories over the WHATWG `TextEncoderStream` and `TextDecoderStream` platform APIs. `encodeTextStream` converts a stream of JavaScript strings to a stream of UTF-8 `Uint8Array` chunks, and `decodeTextStream` does the reverse. These are essential building blocks for any binary protocol, file write, or network transmission pipeline that bridges the text and binary worlds.
The implementations delegate entirely to the browser/Bun/Node built-in streaming codecs, which are well-tested, spec-compliant, and handle multi-byte character boundaries correctly across chunk splits. This wrapper exists to provide a consistent import path and API surface that matches the rest of the `@suaveplan/streams` transform catalogue.
## Features
- `encodeTextStream`: string → `Uint8Array` (UTF-8)
- `decodeTextStream`: `Uint8Array` → string with configurable encoding (default `utf-8`)
- Zero overhead — delegates to the platform `TextEncoderStream`/`TextDecoderStream`
- Handles multi-byte character splits across chunk boundaries automatically
- Composable with `batchStream`, `compressStream`, and other transforms
## Usage: Basic Example
```ts
import { encodeTextStream, decodeTextStream } from "@suaveplan/streams/transforms/text-codec/text-codec";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
const chunks = await toArray(
fromArray(["hello", " world"]).pipeThrough(encodeTextStream())
);
// chunks[0] is Uint8Array([104, 101, 108, 108, 111])
// chunks[1] is Uint8Array([32, 119, 111, 114, 108, 100])
```
## Usage: Advanced Example
```ts
import { encodeTextStream, decodeTextStream } from "@suaveplan/streams/transforms/text-codec/text-codec";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// Round-trip: encode then decode
const original = ["Hello", " World"];
const encoded = fromArray(original).pipeThrough(encodeTextStream());
const decoded = encoded.pipeThrough(decodeTextStream());
const result = await toArray(decoded);
console.log(result.join("")); // "Hello World"
```
## API Reference
### `encodeTextStream(encoding?: string): TransformStream<string, Uint8Array>`
- `encoding` — accepted for API consistency; the WHATWG spec mandates UTF-8 output so this parameter has no effect.
### `decodeTextStream(encoding?: string): TransformStream<Uint8Array, string>`
- `encoding` — character encoding label (default `"utf-8"`). Any IANA encoding label supported by `TextDecoderStream` is accepted.
**Throws** (immediate): if `encoding` is not recognised by the platform's `TextDecoderStream` constructor.
## Implementation Notes
`TextEncoderStream` always produces UTF-8 per the WHATWG Encoding specification; the `encoding` parameter on `encodeTextStream` is present for API symmetry only. The `_encoding` parameter is prefixed with an underscore to signal intentional non-use without disabling lint checks. `TextDecoderStream` handles split multi-byte sequences internally — it buffers incomplete sequences across chunk boundaries, so no manual buffering is needed here.

View File

@@ -0,0 +1,116 @@
# throttle
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `throttle` feature ships a drop-based rate limiter for WHATWG streams in two interchangeable forms: the functional factory `throttleStream` and the plugin-OO class `ThrottleTransform`. Both enforce the same invariant — at most one chunk passes through per `ms` milliseconds, and any chunks arriving inside the suppression window are silently dropped, not buffered. The class form is a thin shell over the factory so abort, clock, and drop semantics stay bit-identical between the two.
The clock is injectable on both forms, enabling fully deterministic unit tests without real timers or `vi.useFakeTimers`. This is intended for high-frequency streams (sensor readings, UI events, polling results) where you want to reduce the emission rate without paying for an excess buffer.
## Features
- Drop-based rate limiting (excess chunks discarded, not buffered)
- Functional factory `throttleStream` and plugin-OO `ThrottleTransform`, sharing one implementation
- Injectable `StreamClock` for deterministic tests
- `ms = 0` passes all chunks through
- AbortSignal support on both single-shot `process` and the streaming bridge
- Configurable `highWaterMark`
- Immediate `RangeError` for negative `ms`
- `name = "throttle"` on the class, for telemetry, factory keying, and StreamManager stage identification
## Usage: Basic Example
```ts
import { throttleStream } from "@suaveplan/streams/transforms/throttle/throttle";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// With a real clock, all items arrive "simultaneously" so only the first passes
const result = await toArray(
fromArray([1, 2, 3]).pipeThrough(throttleStream(100))
);
console.log(result); // [1] — items 2 and 3 dropped (within 100ms window)
```
## Usage: Advanced Example
```ts
import { ThrottleTransform } from "@suaveplan/streams/transforms/throttle/throttle";
// Drive time deterministically with a manually advanced clock
let now = 0;
const clock = { now: () => now };
const ctrl = new AbortController();
const t = new ThrottleTransform<{ value: number }>({
ms: 50,
clock,
streamOptions: { signal: ctrl.signal, highWaterMark: 4 },
});
const source = new ReadableStream<{ value: number }>({
start(c) {
c.enqueue({ value: 1 }); // now=0 -> emitted
now = 20;
c.enqueue({ value: 2 }); // now=20 -> dropped (20 < 50)
now = 60;
c.enqueue({ value: 3 }); // now=60 -> emitted
c.close();
},
});
const out: number[] = [];
await source
.pipeThrough(t.toTransformStream<{ value: number }, { value: number }>())
.pipeTo(
new WritableStream({
write(chunk) {
out.push(chunk.value);
},
})
);
// out === [1, 3]
```
## API Reference
### `throttleStream<T>(ms: number, clock?: StreamClock, options?: TransformOptions): TransformStream<T, T>`
**Parameters**:
- `ms` — minimum interval between emitted chunks in milliseconds; `>= 0`.
- `clock` — injectable clock with `now(): number` (default: `Date.now`).
- `options.signal` — AbortSignal.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Returns**: `TransformStream<T, T>`.
**Throws** (immediate): `RangeError` if `ms < 0`.
### `class ThrottleTransform<T = unknown>`
#### `constructor(config: ThrottleTransformConfig)`
`config.ms` is the minimum interval between emitted chunks. Must be `>= 0`; `throttleStream` throws `RangeError` for negative values when the stream is constructed. `config.clock` is an optional `StreamClock` (`{ now(): number }`) for deterministic testing — defaults to `Date.now`. `config.streamOptions` forwards `signal` and `highWaterMark` to the underlying `throttleStream`.
#### `process<U>(input: U, options?: TransformOptions): Promise<U>`
Single-shot pass-through. Returns `input` unchanged after asserting `options.signal` is not aborted. Throttling is meaningless for a single value, so the streaming form via `toTransformStream` is the natural shape — `process` exists only to satisfy the `BaseTransform` plugin contract.
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `throttleStream(ms, clock, streamOptions)`. Each call returns a new stream with its own `lastEmit` cursor — call sites that need parallel pipelines get independent rate limiters.
### `interface ThrottleTransformConfig`
`{ ms: number; clock?: StreamClock; streamOptions?: TransformOptions }`.
## Implementation Notes
`lastEmit` is initialised to `-Infinity` so the first chunk always passes regardless of the current time. The check `now - lastEmit >= ms` means `ms = 0` allows every chunk through. Dropped chunks produce no downstream backpressure; the source may push chunks faster than the throttle allows, so this transform does not regulate memory usage — it only limits the output rate.
`ThrottleTransform` is a stateful transform written against the M2b `BaseTransform`-subclass pattern. The single-shot `process` is intentionally a pass-through identity: the `lastEmit` cursor that drives drop decisions only makes sense across a sequence of chunks. Folding it into `process` would require either persisting state across `process` calls (turning the class into a singleton-per-pipeline that breaks under concurrent reuse) or always returning the input unchanged. The latter — pass-through — is the honest answer and is what we ship.
Streaming is therefore the canonical call path for the class. Every byte of throttling behaviour lives in `throttleStream`; the class never re-implements drop logic, never reads the clock itself, and never duplicates the `RangeError` ms-validation. This keeps the class form and functional form bit-identical and means there is exactly one place to fix any bug. The `as unknown as TransformStream<A, B>` cast on the streaming bridge is intentional: `BaseTransform.toTransformStream` declares loose `<TIn, TOut>` generics with `unknown` defaults, and the underlying `throttleStream<T>` is single-typed (chunks pass through unchanged). Callers who want strict typing supply `<A, B>` at the call site.

View File

@@ -0,0 +1,129 @@
# window
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The `window` module provides both a functional factory (`windowStream`) and a plugin-OO wrapper (`WindowTransform`) for grouping a stream of items into sliding (or non-overlapping) windows of a fixed `size`. The `overlap` parameter controls how many items from the previous window are retained in the next one. When `overlap = 0` (the default), windows are non-overlapping (tumbling). When `overlap > 0`, consecutive windows share `overlap` items, advancing by `size - overlap` chunks each step.
Sliding windows are essential for time-series analysis, moving averages, n-gram extraction, and any algorithm that needs to process a fixed-width view of a sequence. The class form exists so consumers using the `BaseTransform` plugin contract (hooks, events, `StreamManager`, `TransformFactory`) can express a windowing stage without mixing function-form and class-form APIs. The class is a thin shell — every byte of windowing state lives in `windowStream` — so abort and emission semantics are identical between the two call paths.
## Features
- Tumbling windows (overlap = 0) and sliding windows (overlap > 0)
- Final incomplete window flushed on stream close when input count < size
- AbortSignal support
- Configurable `highWaterMark`
- Immediate `RangeError` for invalid `size` or `overlap`
- Plugin-OO `WindowTransform` honoring `BaseTransform` contract (`name`, `process`, hooks)
- `toTransformStream()` delegates to `windowStream` for a single source of truth
## Usage: Basic Example
```ts
import { windowStream } from "@suaveplan/streams/transforms/window/window";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// Tumbling window of size 3
const result = await toArray(
fromArray([1, 2, 3, 4, 5, 6]).pipeThrough(windowStream(3))
);
console.log(result); // [[1, 2, 3], [4, 5, 6]]
// Sliding window of size 3, overlap 1 (step = 2)
const sliding = await toArray(
fromArray([1, 2, 3, 4, 5]).pipeThrough(windowStream(3, 1))
);
console.log(sliding); // [[1, 2, 3], [3, 4, 5]]
```
## Usage: Advanced Example
```ts
import {
WindowTransform,
windowStream,
} from "@suaveplan/streams/transforms/window/window";
import { mapStream } from "@suaveplan/streams/transforms/map-stream/map-stream";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
// Functional: moving average of window size 3
const averages = await toArray(
fromArray([2, 4, 6, 8, 10])
.pipeThrough(windowStream(3, 2))
.pipeThrough(
mapStream((window) =>
window.reduce((a, b) => a + b, 0) / window.length
)
)
);
console.log(averages); // [4, 6, 8]
// Class form with abort + custom queue depth
const ctrl = new AbortController();
const t = new WindowTransform<number>({
size: 4,
overlap: 2,
streamOptions: { signal: ctrl.signal, highWaterMark: 4 },
});
// Sliding window: step = size - overlap = 2.
// Input [1..8] -> [[1,2,3,4],[3,4,5,6],[5,6,7,8]]
const source = new ReadableStream<number>({
start(c) {
for (const n of [1, 2, 3, 4, 5, 6, 7, 8]) c.enqueue(n);
c.close();
},
});
await source.pipeThrough(t.toTransformStream<number, number[]>()).pipeTo(
new WritableStream<number[]>({
write(w) {
console.log(w);
},
})
);
```
## API Reference
### `windowStream<T>(size: number, overlap?: number, options?: TransformOptions): TransformStream<T, T[]>`
**Parameters**:
- `size` — window width; must be `>= 1`.
- `overlap` — items shared between adjacent windows (default: `0`); must be `>= 0` and `< size`.
- `options.signal` — AbortSignal.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).
**Throws** (immediate): `RangeError` for invalid `size` or `overlap`.
### `class WindowTransform<T = unknown>`
#### `constructor(config: WindowTransformConfig)`
`config.size` is the window length; `config.overlap` defaults to `0`. `config.streamOptions` is forwarded to `windowStream` (used only by `toTransformStream`).
#### `process<U>(input: U, options?: TransformOptions): Promise<U>`
Returns `[input]` (a single-element window). Honors `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired. This shape exists to satisfy `BaseTransform.process`; meaningful windowing requires `toTransformStream`.
#### `toTransformStream<A, B>(): TransformStream<A, B>`
Returns a fresh WHATWG `TransformStream` produced by `windowStream(size, overlap, streamOptions)`. State (the partial buffer) is per-stream; each call yields a new state.
### `interface WindowTransformConfig`
`{ size: number; overlap?: number; streamOptions?: TransformOptions }`
## Implementation Notes
The internal buffer retains the last `overlap` items after each emission, achieved with `buffer.slice(step)`. When the stream closes, any partial window still in the buffer is emitted as-is, provided the stream had fewer total items than `size` (otherwise the last full window was already emitted). A `seen` counter tracks total inputs to disambiguate an already-flushed final window from a genuinely partial one.
**Single-shot vs streaming.** Sliding-window is inherently stateful — its semantics live across many input chunks. A `process(input)` that takes a single value cannot express windowing meaningfully, so the class returns `[input]` (a length-1 window) on that path purely to honor the `BaseTransform.process` contract. Callers needing real windowing MUST use `toTransformStream()`, which delegates to `windowStream` and accumulates state across chunks.
This asymmetry is deliberate. Hiding state in the class (e.g. an internal buffer mutated by `process`) would create a second, divergent implementation that drifts from the functional factory and surprises pipeline consumers who expect `process` to be re-entrant. Keeping `process` stateless and pushing all windowing through the streaming bridge preserves the single-source-of-truth invariant the M2b refactor was designed around.
Type erasure on the streaming bridge mirrors `MapTransform`: `<A, B>` are method-level generics with `unknown` defaults; the `as unknown as TransformStream<A, B>` cast is the right shape. Callers supply concrete generics (`t.toTransformStream<number, number[]>()`) to recover strict typing.

View File

@@ -0,0 +1,85 @@
# common (types)
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Centralises all shared TypeScript interfaces, option types, and Zod schemas used across the `@suaveplan/streams` package. Keeping these in one place avoids circular imports between the readable, transform, sink, and pipeline modules and gives consumers a single import location for type-only usage.
The Zod schemas exported here (`ReadableOptionsSchema`, `TransformOptionsSchema`, `InstrumentOptionsSchema`) serve a dual purpose: they validate runtime options objects passed by callers, and they document the accepted shapes in a machine-readable, self-describing format.
## Features
- `ReadableOptions` — high-water mark and abort signal for readable factories
- `TransformOptions` — high-water mark and abort signal for transform factories
- `InstrumentOptions` — OTel tracer, meter, and attribute bag for instrumented stages
- `StreamClock` — minimal clock interface (`now(): number`) for injectable time
- Zod schemas for all option types, reusable for validation and documentation
## Usage: Basic Example
```ts
import type { ReadableOptions, TransformOptions } from "@suaveplan/streams";
function createMyStream(opts: ReadableOptions = {}) {
const hwm = opts.highWaterMark ?? 1;
return new ReadableStream({ /* ... */ }, new CountQueuingStrategy({ highWaterMark: hwm }));
}
```
## Usage: Advanced Example
```ts
import { ReadableOptionsSchema, InstrumentOptionsSchema } from "@suaveplan/streams";
import type { InstrumentOptions } from "@suaveplan/streams";
function validateAndInstrument(rawOpts: unknown): InstrumentOptions {
return InstrumentOptionsSchema.parse(rawOpts);
}
// Runtime validation of user-supplied options
const opts = validateAndInstrument({ attributes: { "stream.stage": "parse" } });
```
## API Reference
### `ReadableOptions`
```ts
interface ReadableOptions {
signal?: AbortSignal;
highWaterMark?: number;
}
```
### `TransformOptions`
```ts
interface TransformOptions {
signal?: AbortSignal;
highWaterMark?: number;
}
```
### `InstrumentOptions`
```ts
interface InstrumentOptions {
tracer?: Tracer;
meter?: Meter;
attributes?: Record<string, string>;
}
```
### `StreamClock`
Minimal injectable clock: `{ now(): number }`. Defaults to `Date.now`.
### `ReadableOptionsSchema` / `TransformOptionsSchema` / `InstrumentOptionsSchema`
Zod schemas validating the corresponding option types. Use for runtime assertion of user-supplied configurations.
## Implementation Notes
`StreamClock` is a structural subtype of `@suaveplan/types`' `Clock` — any valid `Clock` implementation satisfies it. The OTel types in `InstrumentOptions` are imported as type-only to avoid a hard runtime dependency; consumers who do not use OTel pay zero cost.

View File

@@ -0,0 +1,71 @@
# bufferToUint8Array
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`bufferToUint8Array` normalises the two byte-container shapes that show up in
mixed Node/browser code — `Buffer` and `ArrayBuffer` — into the single
canonical `Uint8Array` form used throughout `@suaveplan/streams`. The streams
package treats `Uint8Array` as the universal byte chunk; this helper lets call
sites accept either input without sprinkling `instanceof` checks across every
adapter.
The conversion is zero-copy. For `Buffer` inputs we construct a view that
respects the source's `byteOffset` / `byteLength`, which matters because Node
typically slices new `Buffer`s out of a shared pool. For `ArrayBuffer` inputs
we wrap the entire buffer.
## Features
- Accepts both Node `Buffer` and standard `ArrayBuffer` inputs.
- Zero-copy: never allocates a new backing buffer.
- Honours `Buffer.subarray` offsets correctly (avoids a common foot-gun).
- Pure synchronous function with no dependencies.
## Usage: Basic Example
```ts
import { bufferToUint8Array } from "@suaveplan/streams/utils/buffer/buffer-to-uint8/buffer-to-uint8";
const u8 = bufferToUint8Array(Buffer.from("hello"));
console.log(u8.byteLength); // 5
```
## Usage: Advanced Example
```ts
import { bufferToUint8Array } from "@suaveplan/streams/utils/buffer/buffer-to-uint8/buffer-to-uint8";
// Coming from `node:fs/promises#readFile` (returns Buffer in Node)
async function loadAsU8(path: string): Promise<Uint8Array> {
const { readFile } = await import("node:fs/promises");
return bufferToUint8Array(await readFile(path));
}
// Coming from a browser `Response.arrayBuffer()`
async function fetchAsU8(url: string): Promise<Uint8Array> {
const ab = await (await fetch(url)).arrayBuffer();
return bufferToUint8Array(ab);
}
```
## API Reference
### `bufferToUint8Array(buffer: Buffer | ArrayBuffer): Uint8Array`
- **`buffer`** — either a Node `Buffer` (a `Uint8Array` subclass with possibly
non-zero `byteOffset`) or a standard DOM/ES `ArrayBuffer`.
- **Returns** — a `Uint8Array` view over the same memory as the input.
- **Throws** — never; type narrowing is performed via `instanceof ArrayBuffer`.
## Implementation Notes
- The `instanceof ArrayBuffer` check intentionally precedes the `Buffer` branch
because `Buffer instanceof ArrayBuffer` is `false` in Node (Buffer extends
`Uint8Array`, not `ArrayBuffer`).
- The returned view shares memory with the source. If you need an independent
copy, follow up with `.slice()` or pair this helper with
`createUint8ArraySlice`.
- Universal: this module has no Node-only or DOM-only imports.

View File

@@ -0,0 +1,75 @@
# getDataFromFile
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`getDataFromFile` reads a file from the local filesystem and returns its contents
as a plain `Uint8Array`. It is a thin Node-only convenience for the streams
package: the streams package is environment-agnostic, but many byte-oriented
helpers (chunking, framing, hashing) need a way to load test fixtures and
real-world inputs from disk without each call site re-implementing the dynamic
import + error-mapping dance.
The function is browser-safe in the sense that importing it does not pull in
`node:fs` eagerly. The Node module is loaded lazily inside the function body and
calling it in a non-Node environment surfaces a structured
`StreamFileBrowserNotSupportedError` rather than an opaque module-resolution
crash.
## Features
- Lazy `node:fs/promises` import — zero overhead on the browser bundle path.
- Runtime environment detection via `globalThis.process?.versions?.node`.
- Returns a fresh `Uint8Array` decoupled from Node's pooled `Buffer` allocator.
- Structured errors with the original `cause` and the offending `filePath`.
- Stable `STREAM.FILE_BROWSER_NOT_SUPPORTED` and `STREAM.FILE_READ_FAILED` codes.
## Usage: Basic Example
```ts
import { getDataFromFile } from "@suaveplan/streams/utils/buffer/get-data-from-file/get-data-from-file";
const bytes = await getDataFromFile("/etc/hostname");
console.log(bytes.byteLength);
```
## Usage: Advanced Example
```ts
import { getDataFromFile } from "@suaveplan/streams/utils/buffer/get-data-from-file/get-data-from-file";
import { StreamFileReadFailedError } from "@suaveplan/streams/errors/stream-error";
try {
const data = await getDataFromFile("/no/such/file");
process.stdout.write(data);
} catch (err) {
if (err instanceof StreamFileReadFailedError) {
console.error("read failed", err.context, err.cause);
} else {
throw err;
}
}
```
## API Reference
### `getDataFromFile(filePath: string): Promise<Uint8Array>`
- **`filePath`** — absolute or relative path resolved against `process.cwd()`.
- **Returns** — `Promise<Uint8Array>` containing the file's exact bytes.
- **Throws** — `StreamFileBrowserNotSupportedError` outside Node;
`StreamFileReadFailedError` (with `cause` and `context.filePath`) on any
underlying I/O failure (ENOENT, EACCES, EISDIR, …).
## Implementation Notes
- The dynamic `import("node:fs/promises")` keeps the universal entry-point free
of Node-only deps; bundlers that target the browser tree-shake this branch
away, and the runtime guard prevents accidental invocation.
- The returned array is sliced off the underlying `ArrayBuffer` so consumers
never receive a `Buffer` subclass — useful when feeding the result into APIs
that strictly type-check `Uint8Array`.
- All thrown errors extend `SuaveplanError` and round-trip through the standard
redaction / wire-format adapters supplied by `@suaveplan/error`.

View File

@@ -0,0 +1,68 @@
# createUint8ArraySlice
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`createUint8ArraySlice` returns an independent copy of a sub-range of a
`Uint8Array`. The streams package favours explicit copy-vs-view semantics over
the conventional `subarray` (view) / `slice` (copy) ambiguity that has bitten
many consumers writing both browser and Node code. This helper makes the
intent obvious at the call site: when you call `createUint8ArraySlice`, you
get back memory you fully own.
It is functionally a thin wrapper around `Uint8Array.prototype.slice`, but
named to communicate the copy semantics, and centralised so that future
optimisations (e.g. fast-path detection of zero-length, large-buffer pooling)
have a single place to live.
## Features
- Always copies — never returns a view aliasing the source buffer.
- Clamps `end` past `length` per the standard `slice` algorithm.
- Returns an empty `Uint8Array` when `start === end`.
- Pure, synchronous, dependency-free.
## Usage: Basic Example
```ts
import { createUint8ArraySlice } from "@suaveplan/streams/utils/buffer/uint8-slice/uint8-slice";
const src = new Uint8Array([1, 2, 3, 4, 5]);
const middle = createUint8ArraySlice(src, 1, 4); // Uint8Array(3) [2, 3, 4]
```
## Usage: Advanced Example
```ts
import { createUint8ArraySlice } from "@suaveplan/streams/utils/buffer/uint8-slice/uint8-slice";
// Frame parser: each frame owns its bytes so the surrounding ring buffer can
// be recycled without stalling downstream consumers.
function parseFrames(buf: Uint8Array, frameSize: number): Uint8Array[] {
const out: Uint8Array[] = [];
for (let i = 0; i + frameSize <= buf.byteLength; i += frameSize) {
out.push(createUint8ArraySlice(buf, i, i + frameSize));
}
return out;
}
```
## API Reference
### `createUint8ArraySlice(array, start, end): Uint8Array`
- **`array: Uint8Array`** — source buffer.
- **`start: number`** — inclusive start index.
- **`end: number`** — exclusive end index; clamped to `array.length`.
- **Returns** — a fresh `Uint8Array` of length `max(0, end - start)`.
- **Throws** — never.
## Implementation Notes
- Implemented as `array.slice(start, end)`. The function exists because the
copy-vs-view distinction matters for streams correctness (subarray would
hold the entire upstream buffer alive) and a named helper documents intent
better than an inline `.slice` call would.
- Universal: zero environment dependencies.

View File

@@ -0,0 +1,81 @@
# createDataStream
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`createDataStream` is the canonical bridge from "I have a typed in-memory
value" to "I have a `ReadableStream<Uint8Array>`". It is the read-side counter
to `processDataStream`: each accepts a transformer that bridges between the
domain type `T` and the byte representation, and they compose cleanly when a
pipeline needs to enter and leave the byte world without callers spelling out
the encode/decode boilerplate.
The transformer runs once when the function is called. The function returns a
`Promise<ReadableStream<Uint8Array>>` so that asynchronous serialisation
(JSON.stringify of a complex value, msgpack encoding, gzip pre-compression)
fits naturally into the call signature without forcing every consumer to wrap
the producer in a `start()` callback.
## Features
- Sync and async transformer functions both supported.
- Pluggable chunk size via the underlying `uint8ArrayToStream`.
- Type-safe generic input — the compiler infers `T` from the value supplied.
- Propagates transformer errors as the rejection of the returned promise.
## Usage: Basic Example
```ts
import { createDataStream } from "@suaveplan/streams/utils/stream/create-data-stream/create-data-stream";
const stream = await createDataStream(
{ hello: "world" },
(obj) => new TextEncoder().encode(JSON.stringify(obj))
);
```
## Usage: Advanced Example
```ts
import { createDataStream } from "@suaveplan/streams/utils/stream/create-data-stream/create-data-stream";
import { encode } from "@msgpack/msgpack";
interface Telemetry {
ts: number;
metrics: Record<string, number>;
}
async function telemetryStream(t: Telemetry) {
return createDataStream<Telemetry>(t, (v) => encode(v), {
chunkSize: 4096,
});
}
const stream = await telemetryStream({
ts: Date.now(),
metrics: { rps: 42, p99: 18.7 },
});
```
## API Reference
### `createDataStream<T>(data, transformer, options?)`
- **`data: T`** — value to serialise.
- **`transformer: (data: T) => Uint8Array | Promise<Uint8Array>`** — encoder.
- **`options?: { chunkSize?: number }`** — emission chunk size, forwarded to
`uint8ArrayToStream`.
- **Returns** — `Promise<ReadableStream<Uint8Array>>`.
- **Throws** — anything the transformer throws (rejected promise).
## Implementation Notes
- The transformer runs exactly once before the stream is constructed, so the
byte count is known up-front; this lets downstream sinks pre-allocate when
they want to.
- The function does not accept an `AbortSignal` — cancellation belongs to the
drain side (`streamToUint8Array`, `processDataStream`) where it can actually
influence work in flight.
- Universal: zero environment dependencies.

View File

@@ -0,0 +1,71 @@
# processDataStream
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`processDataStream` is the drain-side complement to `createDataStream`. It
collects every byte emitted by a `ReadableStream<Uint8Array>` into a single
`Uint8Array` and hands the result to a processor function that maps the bytes
into a domain value `T`. It exists so consumers can write straight-line code —
"give me the parsed result" — without explicitly looping over the reader.
The function is intentionally small but centralising it pays off: it standardises
how upstream errors propagate, how an empty stream is represented (as an empty
`Uint8Array`, not `undefined`), and gives observability/instrumentation a
single chokepoint to wrap when needed.
## Features
- Accepts sync and async processor functions.
- Returns the processor's resolved value with full type inference.
- Concatenates multiple chunks transparently before processing.
- Propagates upstream stream errors and processor throws as a rejected promise.
## Usage: Basic Example
```ts
import { processDataStream } from "@suaveplan/streams/utils/stream/process-data-stream/process-data-stream";
const text = await processDataStream(stream, (b) =>
new TextDecoder().decode(b)
);
```
## Usage: Advanced Example
```ts
import { processDataStream } from "@suaveplan/streams/utils/stream/process-data-stream/process-data-stream";
import { decode } from "@msgpack/msgpack";
interface Record {
id: string;
payload: unknown;
}
async function readMsgpack<T = Record>(stream: ReadableStream<Uint8Array>) {
return processDataStream(stream, async (bytes) => {
// hand off to a worker thread for large payloads
return decode(bytes) as T;
});
}
```
## API Reference
### `processDataStream<T>(stream, processor): Promise<T>`
- **`stream: ReadableStream<Uint8Array>`** — source of bytes to drain.
- **`processor: (data: Uint8Array) => T | Promise<T>`** — function that maps
the concatenated bytes into a domain value.
- **Returns** — `Promise<T>` resolving with `processor`'s return value.
- **Throws** — rejects with whatever the underlying stream or processor throws.
## Implementation Notes
- Built on top of `streamToUint8Array`; the same single-allocation behaviour
applies (chunks pre-summed before allocation).
- The processor is awaited even when its return type is synchronous to keep a
single promise contract for callers.
- Universal — no Node or DOM globals beyond the WHATWG Streams API.

View File

@@ -0,0 +1,91 @@
# round-trip (streamToUint8Array / uint8ArrayToStream)
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
The round-trip module provides the two halves of byte ↔ stream conversion that
underpin most file/network adapters in `@suaveplan/streams`. `uint8ArrayToStream`
turns an in-memory buffer into a backpressure-respecting `ReadableStream`, and
`streamToUint8Array` drains an arbitrary readable stream back into a single
`Uint8Array`. They are deliberately co-located because most consumers reach for
both in the same call path (e.g. test fixtures that load a buffer, push it
through a transform pipeline, then re-collect the output for assertion).
The drain side is `AbortSignal`-aware and rejects with `StreamAbortedError`
when cancellation is requested, cancelling the underlying reader so upstream
producers can release resources promptly.
## Features
- `uint8ArrayToStream`: configurable chunk size; defaults to a single emission.
- Per-chunk copy emission so consumers can safely retain references.
- `streamToUint8Array`: total-length pre-sum, single allocation for output.
- Honours `AbortSignal` both before and during read; cancels the reader.
- Pure WHATWG Streams API — no Node-only or DOM-only deps.
## Usage: Basic Example
```ts
import {
streamToUint8Array,
uint8ArrayToStream,
} from "@suaveplan/streams/utils/stream/round-trip/round-trip";
const original = new Uint8Array([1, 2, 3, 4]);
const drained = await streamToUint8Array(uint8ArrayToStream(original));
// drained equals original
```
## Usage: Advanced Example
```ts
import {
streamToUint8Array,
uint8ArrayToStream,
} from "@suaveplan/streams/utils/stream/round-trip/round-trip";
async function pipeWithTimeout(
data: Uint8Array,
transform: TransformStream<Uint8Array, Uint8Array>,
ms: number
): Promise<Uint8Array> {
const ac = new AbortController();
const timer = setTimeout(() => ac.abort(new Error("timeout")), ms);
try {
const piped = uint8ArrayToStream(data, { chunkSize: 4096 }).pipeThrough(
transform
);
return await streamToUint8Array(piped, ac.signal);
} finally {
clearTimeout(timer);
}
}
```
## API Reference
### `uint8ArrayToStream(data, options?)`
- **`data: Uint8Array`** — bytes to emit.
- **`options.chunkSize?: number`** — bytes per chunk; defaults to entire array.
- **Returns** — `ReadableStream<Uint8Array>` that closes after the last chunk.
### `streamToUint8Array(stream, signal?)`
- **`stream: ReadableStream<Uint8Array>`** — source stream.
- **`signal?: AbortSignal`** — cancellation token; cancelling rejects the call
with `StreamAbortedError` and cancels the reader.
- **Returns** — `Promise<Uint8Array>` of the concatenated bytes.
- **Throws** — `StreamAbortedError` on cancellation.
## Implementation Notes
- The drain pre-collects chunks and sums lengths so the final `Uint8Array` is
allocated exactly once — avoids the quadratic-copy trap of repeatedly
appending into a growing buffer.
- Each emission of `uint8ArrayToStream` is a copy via `data.slice`, so the
source `Uint8Array` may be mutated independently after the stream is built.
- The abort listener is removed in `finally` to keep the signal listener count
bounded across long-lived signals.

View File

@@ -0,0 +1,77 @@
# validateStreamIntegrity
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`validateStreamIntegrity` walks a `ReadableStream<Uint8Array>` chunk by chunk
and applies a caller-supplied predicate to each chunk. It returns `true` if
every chunk satisfies the predicate and the stream closes cleanly, or `false`
the first time the predicate rejects a chunk. On a rejection it cancels the
underlying reader so upstream producers can release resources rather than
continuing to fill a buffer that will be discarded.
The function is the streams-package equivalent of "fold a boolean over the
chunks". It is small but worth centralising because the short-circuit + reader
cancel + lock-release dance is easy to get wrong, especially when combined
with WHATWG Streams' lock semantics around early teardown.
## Features
- Short-circuits on first failure — does not consume the rest of the stream.
- Cancels the underlying reader on failure to free upstream resources.
- Always releases the reader lock via `finally`, even on validator throws.
- Returns `true` for an empty stream (vacuous truth).
- Synchronous predicate keeps the per-chunk cost minimal.
## Usage: Basic Example
```ts
import { validateStreamIntegrity } from "@suaveplan/streams/utils/stream/validate-integrity/validate-integrity";
const allEven = await validateStreamIntegrity(stream, (chunk) =>
chunk.every((b) => b % 2 === 0)
);
```
## Usage: Advanced Example
```ts
import { validateStreamIntegrity } from "@suaveplan/streams/utils/stream/validate-integrity/validate-integrity";
// Each frame: 4-byte big-endian length || N payload bytes || 1-byte checksum.
// Verify checksum without buffering the whole stream.
function checksummed(chunk: Uint8Array): boolean {
if (chunk.byteLength < 5) return false;
const dv = new DataView(chunk.buffer, chunk.byteOffset, chunk.byteLength);
const len = dv.getUint32(0);
if (chunk.byteLength !== 4 + len + 1) return false;
let sum = 0;
for (let i = 0; i < 4 + len; i++) sum = (sum + chunk[i]!) & 0xff;
return sum === chunk[chunk.byteLength - 1];
}
const ok = await validateStreamIntegrity(stream, checksummed);
```
## API Reference
### `validateStreamIntegrity(stream, validate): Promise<boolean>`
- **`stream: ReadableStream<Uint8Array>`** — source stream.
- **`validate: (chunk: Uint8Array) => boolean`** — predicate run per chunk.
- **Returns** — `Promise<boolean>` resolving to `true` if all chunks pass,
`false` on first failure.
- **Throws** — rejects with whatever the underlying stream throws.
## Implementation Notes
- The reader is acquired via `getReader()` and released in `finally`. On the
failure path we additionally `await reader.cancel()` to ensure the upstream
source observes the cancellation before we resolve `false`.
- The predicate is invoked exactly once per non-empty chunk; chunks of size 0
still trigger a call (a zero-byte chunk is observable in the WHATWG model
and a strict validator may want to reject it).
- A predicate `throw` propagates as a rejection rather than a `false`; callers
who wish otherwise can wrap their predicate in a try/catch.