skills/docs/foundation/flow/streams/src/transforms/dedup/dedup.md

dedup

(no submodules)

Purpose

The dedup module ships two complementary deduplication primitives over the same key-extraction contract: dedupStream, a functional TransformStream<T, T> factory that drops consecutive duplicates, and DedupTransform, a BaseTransform plugin-OO wrapper that holds a per-instance Set<key> for all-time dedup via its single-input process path while delegating its streaming bridge to dedupStream. Together they cover both window-1 streaming dedup and full-set call-by-call dedup without duplicating key-extraction or abort plumbing.

This is the only dedup surface in @suaveplan/streams. For unbounded set-based global deduplication on a stream prefer filterStream with an externally-managed Set so memory bounds remain the caller's choice.

Features

• dedupStream suppresses consecutive duplicate chunks via strict-equality comparison on the optional keyFn result
• First chunk always passes through (sentinel-guarded)
• DedupTransform.process rejects any all-time duplicate with StreamTransformFailedError (STREAM.TRANSFORM_FAILED)
• DedupTransform.toTransformStream() delegates to dedupStream(keyFn) — identical wire-level semantics
• Default key function on the class is JSON.stringify (works for any structurally-comparable value); dedupStream uses identity
• DedupKeyFn<T> = (chunk: T) => string | number lets callers supply a stable, cheap key
• AbortSignal honoured before and after key extraction
• Configurable highWaterMark on the streaming form
• dispose() clears the per-instance seen set so reused instances do not leak memory

Usage: Basic Example

import { dedupStream } from "@suaveplan/streams/transforms/dedup/dedup";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";

const result = await toArray(
  fromArray([1, 1, 2, 3, 3, 3, 2]).pipeThrough(dedupStream())
);
console.log(result); // [1, 2, 3, 2] — non-consecutive 2s both pass

import { DedupTransform } from "@suaveplan/streams/transforms/dedup/dedup";

const t = new DedupTransform<number>();
await t.process(1); // 1
await t.process(2); // 2
await t.process(1); // throws StreamTransformFailedError("duplicate input")

Usage: Advanced Example

import { DedupTransform, dedupStream } from "@suaveplan/streams/transforms/dedup/dedup";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";

// Key-based streaming dedup: dedup events by their type
const events = [
  { type: "click", x: 1 }, { type: "click", x: 2 },
  { type: "keydown", key: "a" }, { type: "keydown", key: "b" },
];
const consecutive = await toArray(
  fromArray(events).pipeThrough(dedupStream((e) => e.type))
);
console.log(consecutive);
// [{ type: "click", x: 1 }, { type: "keydown", key: "a" }]

interface Event { id: string; payload: unknown }

// Class form — dedup by id only, ignore payload, all-time set
const t = new DedupTransform<Event>((e) => e.id);
await t.process({ id: "a", payload: 1 }); // ok
await t.process({ id: "a", payload: 2 }); // throws — same id

// Streaming bridge from the same instance (consecutive-only via dedupStream)
const source = new ReadableStream<Event>({
  start(c) {
    c.enqueue({ id: "x", payload: 1 });
    c.enqueue({ id: "x", payload: 2 });
    c.enqueue({ id: "y", payload: 3 });
    c.close();
  },
});
await source
  .pipeThrough(t.toTransformStream<Event, Event>())
  .pipeTo(
    new WritableStream<Event>({
      write(e) { console.log(e.id); },
    })
  );
// Logs: "x", "y"

t.dispose(); // clears the in-memory seen set

API Reference

dedupStream<T>(keyFn?: (item: T) => unknown, options?: TransformOptions): TransformStream<T, T>

Parameters

• keyFn — optional function to compute a comparison key; defaults to identity comparison.
• options.signal — AbortSignal.
• options.highWaterMark — WHATWG queue depth (default: 1).

Returns TransformStream<T, T>.

Throws (via error channel) StreamAbortedError when the signal fires.

class DedupTransform<T = unknown>

constructor(keyFn?: DedupKeyFn<T>)

keyFn extracts the dedup key. Defaults to JSON.stringify(chunk).

process<U>(input: U, options?: TransformOptions): Promise<U>

Returns input if its key has not been seen on this instance; otherwise throws StreamTransformFailedError with code STREAM.TRANSFORM_FAILED and message "duplicate input". Honours options.signal — rejects with StreamAbortedError (STREAM.ABORTED) when fired.

toTransformStream<A, B>(): TransformStream<A, B>

Returns a fresh WHATWG TransformStream produced by dedupStream(keyFn). Drops only consecutive duplicates — independent of the all-time seen set held for process.

dispose(): void

Clears the private seen set, then calls super.dispose() to release hooks and listeners. Idempotent.

type DedupKeyFn<T>

(chunk: T) => string | number

Implementation Notes

The sentinel Symbol("EMPTY") ensures the first chunk always passes through dedupStream — there is no real "previous key" yet. Comparison uses strict equality (!==) on the key, so objects are compared by reference unless keyFn returns a primitive. The last key is retained in the transform closure; state is tied to the stream instance lifetime.

State is per-DedupTransform instance: each instance owns its own Set<string | number>. Two instances do not share dedup state, even if they were constructed with structurally-identical key functions. Construct one instance per logical dedup window. The class deliberately diverges from dedupStream in the single-input call path — dedupStream is consecutive-only because it cannot assume safe memory bounds for an unbounded incoming stream, whereas process is invoked with caller-controlled cardinality and can hold the full key set. dispose() clears the seen set first, then delegates to BaseTransform.dispose() to drop hooks and listeners; this ordering keeps the seen-set cleanup observable to subclasses that wrap the method.