first commit
This commit is contained in:
164
plugins/library-navigator/skills/sp-streams/SKILL.md
Normal file
164
plugins/library-navigator/skills/sp-streams/SKILL.md
Normal file
@@ -0,0 +1,164 @@
|
||||
---
|
||||
name: sp-streams
|
||||
description: WHATWG-Streams primitives with backpressure-first transforms, fluent pipelines, OTel instrumentation, and streaming AEAD crypto.
|
||||
---
|
||||
|
||||
# Streams for Suaveplan
|
||||
|
||||
`@suaveplan/streams` provides WHATWG-Streams-based primitives for backpressure-first transforms, zero-copy `Uint8Array` operations, end-to-end `AbortSignal` integration, structured errors, and OTel instrumentation. It offers both functional factory APIs and an extensible plugin substrate via `BaseTransform`.
|
||||
|
||||
## Architecture
|
||||
|
||||
| Component | Responsibility |
|
||||
|-----------|---------------|
|
||||
| `BaseTransform` | Abstract plugin substrate with typed hooks, fire-and-forget events, `AbortSignal` checks, and a default `toTransformStream()` bridge to WHATWG streams. |
|
||||
| `StreamManager` | Chainable orchestrator that threads a single value through a pipeline of `BaseTransform.process()` calls, wrapping non-`StreamError` throws in `STREAM.TRANSFORM_FAILED`. |
|
||||
| `TransformFactory` | Open registry for transform plugins; built-in transforms self-register at module load so tree-shaking drops unused entries. |
|
||||
| `PipelineBuilder` | Fluent, type-safe builder over native `ReadableStream.pipeThrough()` with `.collect()`, `.drain()`, `.forEach()`, and `.drainToWritable()` terminators. |
|
||||
| `instrumentTransform()` | OTel wrapper that adds per-chunk spans, counters, and duration histograms around any `TransformStream`. |
|
||||
| `CryptoTransform` | Streaming AEAD encrypt/decrypt with pluggable engines (`xchacha20StreamingEngine`, `aesGcmStreamingEngine`), per-frame nonces, and length-prefixed wire framing. |
|
||||
| Node adapters | `fromNodeReadable`, `toNodeReadable`, `fromNodeWritable` — bridge between Node.js streams and WHATWG streams without extra copies. |
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
foundation/flow/streams/
|
||||
src/base/transform/ # BaseTransform abstract class and types
|
||||
src/errors/ # StreamError hierarchy + STREAM_CODES via defineCodes
|
||||
src/factory/factory/ # TransformFactory open registry
|
||||
src/instrument/transform/ # OTel instrumentTransform wrapper
|
||||
src/manager/manager/ # StreamManager sequential orchestrator
|
||||
src/node/ # Node.js ↔ WHATWG stream adapters
|
||||
src/pipeline/ # builder, concat, fan-out, merge, merge-ordered, partition, route, safe-branch, zip
|
||||
src/primitives/ # BaseDuplexStream, NetworkStreamImpl
|
||||
src/readable/ # fromArray, fromBytes, fromAsyncGenerator
|
||||
src/sinks/ # toArray, toBytes, forEach, toAsyncIterable
|
||||
src/transforms/ # map, filter, flatMap, batch, window, throttle, dedup, limit, tap, text-codec, compress, crypto, chunk-bytes, coalesce-bytes, sliding-bytes, split-join, length-prefix-encode/decode
|
||||
src/types/ # Common schemas (ReadableOptions, TransformOptions, InstrumentOptions)
|
||||
src/utils/ # Buffer helpers (buffer-to-uint8, uint8-slice, get-data-from-file) and stream helpers (create-data-stream, process-data-stream, round-trip, validate-integrity)
|
||||
```
|
||||
|
||||
Sibling packages:
|
||||
- `@suaveplan/retry` — backoff, jitter, circuit breaker, and timeout helpers
|
||||
- `@suaveplan/fsm` — typed finite state machines with guards, actions, and hierarchical states
|
||||
- `@suaveplan/rate-limiter` — TokenBucket, SlidingWindow, and FixedWindow algorithms with tenant routing
|
||||
- `@suaveplan/queue` — memory/priority/delayed queues with concurrent workers, retry, and dead-letter
|
||||
- `@suaveplan/workflow` — sequential + parallel + conditional step orchestration with saga compensation
|
||||
- `@suaveplan/rules` — async rule engine with fluent builder, 30+ operators, and Zod boundaries
|
||||
- `@suaveplan/cbor` — RFC 8949 CBOR encoder/decoder with zero browser/Node deps
|
||||
- `@suaveplan/crdt` — conflict-free replicated data types (G-Counter, OR-Set, LWW-Register, RGA)
|
||||
- `@suaveplan/idempotency` — idempotency key guard, dedup adapters, and distributed locking
|
||||
- `@suaveplan/job-scheduler` — in-process cron + queued runs with retries and concurrency control
|
||||
- `@suaveplan/undo` — undo/redo engine with branching history and async commands
|
||||
|
||||
## Patterns
|
||||
|
||||
### Quick Start
|
||||
|
||||
```typescript
|
||||
import { fromArray, mapStream, filterStream, batchStream, toArray, pipeline } from "@suaveplan/streams";
|
||||
|
||||
const result = await pipeline(fromArray([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]))
|
||||
.pipe(filterStream((n) => n % 2 === 0))
|
||||
.pipe(mapStream((n) => n * n))
|
||||
.pipe(batchStream(3))
|
||||
.collect();
|
||||
|
||||
console.log(result); // [[4, 16, 36], [64, 100]]
|
||||
```
|
||||
|
||||
### Byte Stream Pipeline
|
||||
|
||||
```typescript
|
||||
import { fromBytes, compressStream, toBytes } from "@suaveplan/streams";
|
||||
|
||||
const data = new TextEncoder().encode("hello world");
|
||||
const compressed = await toBytes(
|
||||
fromBytes(data).pipeThrough(compressStream("gzip"))
|
||||
);
|
||||
```
|
||||
|
||||
### Streaming AEAD Crypto
|
||||
|
||||
```typescript
|
||||
import { CryptoTransform, xchacha20StreamingEngine } from "@suaveplan/streams";
|
||||
|
||||
const key = crypto.getRandomValues(new Uint8Array(32));
|
||||
const enc = new CryptoTransform({ engine: xchacha20StreamingEngine, key, operation: "encrypt" });
|
||||
const dec = new CryptoTransform({ engine: xchacha20StreamingEngine, key, operation: "decrypt" });
|
||||
|
||||
const plaintext = new Uint8Array([1, 2, 3]);
|
||||
const writer = enc.toTransformStream().writable.getWriter();
|
||||
await writer.write(plaintext);
|
||||
await writer.close();
|
||||
|
||||
const reader = dec.toTransformStream().readable.getReader();
|
||||
const chunks: Uint8Array[] = [];
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
chunks.push(value);
|
||||
}
|
||||
```
|
||||
|
||||
### BaseTransform Plugin with Hooks and Events
|
||||
|
||||
```typescript
|
||||
import { BaseTransform } from "@suaveplan/streams";
|
||||
|
||||
class AppendTransform extends BaseTransform {
|
||||
constructor(private readonly suffix: string) {
|
||||
super(`append:${suffix}`);
|
||||
}
|
||||
async process<T = unknown>(input: T): Promise<T> {
|
||||
return `${String(input)}${this.suffix}` as unknown as T;
|
||||
}
|
||||
}
|
||||
|
||||
const t = new AppendTransform("-done");
|
||||
t.hook("before", (payload) => payload); // typed hook chain
|
||||
t.on("complete", ({ result }) => console.log(result)); // fire-and-forget event
|
||||
```
|
||||
|
||||
### OTel Instrumentation
|
||||
|
||||
```typescript
|
||||
import { instrumentTransform, mapStream, fromArray, toArray } from "@suaveplan/streams";
|
||||
|
||||
const result = await toArray(
|
||||
fromArray([1, 2, 3]).pipeThrough(
|
||||
instrumentTransform(mapStream((n: number) => n * 2), "double", {
|
||||
attributes: { "service.name": "ingest" },
|
||||
})
|
||||
)
|
||||
);
|
||||
```
|
||||
|
||||
### Node.js Stream Interop
|
||||
|
||||
```typescript
|
||||
import { fromNodeReadable, toNodeReadable, fromNodeWritable } from "@suaveplan/streams";
|
||||
import { Readable, Writable } from "node:stream";
|
||||
|
||||
const whatwg = fromNodeReadable(Readable.from([Buffer.from("hello")]));
|
||||
const node = toNodeReadable(whatwg);
|
||||
```
|
||||
|
||||
## Constraints
|
||||
|
||||
- Every factory function and `BaseTransform` subclass accepts an optional `AbortSignal`; abortion always surfaces as `StreamAbortedError` with code `STREAM.ABORTED`.
|
||||
- All transform factory functions return native `TransformStream` instances — they can be used with `ReadableStream.pipeThrough()` directly without wrapper objects.
|
||||
- `BaseTransform.toTransformStream()` creates a fresh `TransformStream` on each call while sharing the same hook and listener state.
|
||||
- `BatchTransform.process()` returns a one-element array because batching is inherently streaming and cannot safely accumulate state across single-shot calls.
|
||||
- `CryptoTransform.dispose()` zeroizes the internal key copy with `.fill(0)` before clearing hooks and listeners.
|
||||
- `fromBytes` emits zero-copy `subarray` views; mutating emitted chunks mutates the original buffer.
|
||||
- `TransformFactory` uses a static shared registry; both static and instance APIs delegate to the same underlying map.
|
||||
- `partition()` drives a single source reader and serializes pulls via a chained promise so concurrent consumers cannot race for chunks.
|
||||
|
||||
## Checklist
|
||||
|
||||
1. Verify that `BaseTransform` subclasses call `this.assertNotAborted(options)` at yield points inside `process()` to honor the AbortSignal contract.
|
||||
2. When adding a new transform, provide both a functional factory (`fooStream()`) and a `BaseTransform` subclass (`FooTransform`) with `toTransformStream()` delegating to the factory.
|
||||
3. Ensure error codes are declared via `defineCodes("STREAM", [...])` and thrown through the `StreamError` hierarchy — never use raw string literals.
|
||||
4. Confirm subpath exports are added to `package.json` `exports` for any new public module.
|
||||
5. Run `bunx turbo lint typecheck test build --filter=@suaveplan/streams` and `test:coverage` before declaring changes complete.
|
||||
Reference in New Issue
Block a user