first commit

docs/foundation/flow/streams/src/transforms/dedup/dedup.md

@@ -0,0 +1,130 @@
+# dedup
+
+<!-- begin:auto-docs -->
+_(no submodules)_
+<!-- end:auto-docs -->
+## Purpose
+
+The dedup module ships two complementary deduplication primitives over the same key-extraction contract: `dedupStream`, a functional `TransformStream<T, T>` factory that drops _consecutive_ duplicates, and `DedupTransform`, a `BaseTransform` plugin-OO wrapper that holds a per-instance `Set<key>` for _all-time_ dedup via its single-input `process` path while delegating its streaming bridge to `dedupStream`. Together they cover both window-1 streaming dedup and full-set call-by-call dedup without duplicating key-extraction or abort plumbing.
+
+This is the only dedup surface in `@suaveplan/streams`. For unbounded set-based global deduplication on a stream prefer `filterStream` with an externally-managed `Set` so memory bounds remain the caller's choice.
+
+## Features
+
+- `dedupStream` suppresses _consecutive_ duplicate chunks via strict-equality comparison on the optional `keyFn` result
+- First chunk always passes through (sentinel-guarded)
+- `DedupTransform.process` rejects any _all-time_ duplicate with `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`)
+- `DedupTransform.toTransformStream()` delegates to `dedupStream(keyFn)` — identical wire-level semantics
+- Default key function on the class is `JSON.stringify` (works for any structurally-comparable value); `dedupStream` uses identity
+- `DedupKeyFn<T> = (chunk: T) => string | number` lets callers supply a stable, cheap key
+- `AbortSignal` honoured before and after key extraction
+- Configurable `highWaterMark` on the streaming form
+- `dispose()` clears the per-instance `seen` set so reused instances do not leak memory
+
+## Usage: Basic Example
+
+```ts
+import { dedupStream } from "@suaveplan/streams/transforms/dedup/dedup";
+import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
+import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
+
+const result = await toArray(
+  fromArray([1, 1, 2, 3, 3, 3, 2]).pipeThrough(dedupStream())
+);
+console.log(result); // [1, 2, 3, 2] — non-consecutive 2s both pass
+
+import { DedupTransform } from "@suaveplan/streams/transforms/dedup/dedup";
+
+const t = new DedupTransform<number>();
+await t.process(1); // 1
+await t.process(2); // 2
+await t.process(1); // throws StreamTransformFailedError("duplicate input")
+```
+
+## Usage: Advanced Example
+
+```ts
+import { DedupTransform, dedupStream } from "@suaveplan/streams/transforms/dedup/dedup";
+import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
+import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
+
+// Key-based streaming dedup: dedup events by their type
+const events = [
+  { type: "click", x: 1 }, { type: "click", x: 2 },
+  { type: "keydown", key: "a" }, { type: "keydown", key: "b" },
+];
+const consecutive = await toArray(
+  fromArray(events).pipeThrough(dedupStream((e) => e.type))
+);
+console.log(consecutive);
+// [{ type: "click", x: 1 }, { type: "keydown", key: "a" }]
+
+interface Event { id: string; payload: unknown }
+
+// Class form — dedup by id only, ignore payload, all-time set
+const t = new DedupTransform<Event>((e) => e.id);
+await t.process({ id: "a", payload: 1 }); // ok
+await t.process({ id: "a", payload: 2 }); // throws — same id
+
+// Streaming bridge from the same instance (consecutive-only via dedupStream)
+const source = new ReadableStream<Event>({
+  start(c) {
+    c.enqueue({ id: "x", payload: 1 });
+    c.enqueue({ id: "x", payload: 2 });
+    c.enqueue({ id: "y", payload: 3 });
+    c.close();
+  },
+});
+await source
+  .pipeThrough(t.toTransformStream<Event, Event>())
+  .pipeTo(
+    new WritableStream<Event>({
+      write(e) { console.log(e.id); },
+    })
+  );
+// Logs: "x", "y"
+
+t.dispose(); // clears the in-memory seen set
+```
+
+## API Reference
+
+### `dedupStream<T>(keyFn?: (item: T) => unknown, options?: TransformOptions): TransformStream<T, T>`
+
+**Parameters**
+
+- `keyFn` — optional function to compute a comparison key; defaults to identity comparison.
+- `options.signal` — `AbortSignal`.
+- `options.highWaterMark` — WHATWG queue depth (default: `1`).
+
+**Returns** `TransformStream<T, T>`.
+
+**Throws** (via error channel) `StreamAbortedError` when the signal fires.
+
+### `class DedupTransform<T = unknown>`
+
+#### `constructor(keyFn?: DedupKeyFn<T>)`
+
+`keyFn` extracts the dedup key. Defaults to `JSON.stringify(chunk)`.
+
+#### `process<U>(input: U, options?: TransformOptions): Promise<U>`
+
+Returns `input` if its key has not been seen on this instance; otherwise throws `StreamTransformFailedError` with code `STREAM.TRANSFORM_FAILED` and message `"duplicate input"`. Honours `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired.
+
+#### `toTransformStream<A, B>(): TransformStream<A, B>`
+
+Returns a fresh WHATWG `TransformStream` produced by `dedupStream(keyFn)`. Drops only _consecutive_ duplicates — independent of the all-time `seen` set held for `process`.
+
+#### `dispose(): void`
+
+Clears the private `seen` set, then calls `super.dispose()` to release hooks and listeners. Idempotent.
+
+### `type DedupKeyFn<T>`
+
+`(chunk: T) => string | number`
+
+## Implementation Notes
+
+The sentinel `Symbol("EMPTY")` ensures the first chunk always passes through `dedupStream` — there is no real "previous key" yet. Comparison uses strict equality (`!==`) on the key, so objects are compared by reference unless `keyFn` returns a primitive. The last key is retained in the transform closure; state is tied to the stream instance lifetime.
+
+State is per-`DedupTransform` instance: each instance owns its own `Set<string | number>`. Two instances do not share dedup state, even if they were constructed with structurally-identical key functions. Construct one instance per logical dedup window. The class deliberately diverges from `dedupStream` in the single-input call path — `dedupStream` is consecutive-only because it cannot assume safe memory bounds for an unbounded incoming stream, whereas `process` is invoked with caller-controlled cardinality and can hold the full key set. `dispose()` clears the seen set first, then delegates to `BaseTransform.dispose()` to drop hooks and listeners; this ordering keeps the seen-set cleanup observable to subclasses that wrap the method.