# dedup

<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose

The dedup module ships two complementary deduplication primitives over the same key-extraction contract: `dedupStream`, a functional `TransformStream<T, T>` factory that drops _consecutive_ duplicates, and `DedupTransform`, a `BaseTransform` plugin-OO wrapper that holds a per-instance `Set<key>` for _all-time_ dedup via its single-input `process` path while delegating its streaming bridge to `dedupStream`. Together they cover both window-1 streaming dedup and full-set call-by-call dedup without duplicating key-extraction or abort plumbing.

This is the only dedup surface in `@suaveplan/streams`. For unbounded set-based global deduplication on a stream prefer `filterStream` with an externally-managed `Set` so memory bounds remain the caller's choice.

## Features

- `dedupStream` suppresses _consecutive_ duplicate chunks via strict-equality comparison on the optional `keyFn` result
- First chunk always passes through (sentinel-guarded)
- `DedupTransform.process` rejects any _all-time_ duplicate with `StreamTransformFailedError` (`STREAM.TRANSFORM_FAILED`)
- `DedupTransform.toTransformStream()` delegates to `dedupStream(keyFn)` — identical wire-level semantics
- Default key function on the class is `JSON.stringify` (works for any structurally-comparable value); `dedupStream` uses identity
- `DedupKeyFn<T> = (chunk: T) => string | number` lets callers supply a stable, cheap key
- `AbortSignal` honoured before and after key extraction
- Configurable `highWaterMark` on the streaming form
- `dispose()` clears the per-instance `seen` set so reused instances do not leak memory

## Usage: Basic Example

```ts
import { dedupStream } from "@suaveplan/streams/transforms/dedup/dedup";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";

const result = await toArray(
  fromArray([1, 1, 2, 3, 3, 3, 2]).pipeThrough(dedupStream())
);
console.log(result); // [1, 2, 3, 2] — non-consecutive 2s both pass

import { DedupTransform } from "@suaveplan/streams/transforms/dedup/dedup";

const t = new DedupTransform<number>();
await t.process(1); // 1
await t.process(2); // 2
await t.process(1); // throws StreamTransformFailedError("duplicate input")
```

## Usage: Advanced Example

```ts
import { DedupTransform, dedupStream } from "@suaveplan/streams/transforms/dedup/dedup";
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";

// Key-based streaming dedup: dedup events by their type
const events = [
  { type: "click", x: 1 }, { type: "click", x: 2 },
  { type: "keydown", key: "a" }, { type: "keydown", key: "b" },
];
const consecutive = await toArray(
  fromArray(events).pipeThrough(dedupStream((e) => e.type))
);
console.log(consecutive);
// [{ type: "click", x: 1 }, { type: "keydown", key: "a" }]

interface Event { id: string; payload: unknown }

// Class form — dedup by id only, ignore payload, all-time set
const t = new DedupTransform<Event>((e) => e.id);
await t.process({ id: "a", payload: 1 }); // ok
await t.process({ id: "a", payload: 2 }); // throws — same id

// Streaming bridge from the same instance (consecutive-only via dedupStream)
const source = new ReadableStream<Event>({
  start(c) {
    c.enqueue({ id: "x", payload: 1 });
    c.enqueue({ id: "x", payload: 2 });
    c.enqueue({ id: "y", payload: 3 });
    c.close();
  },
});
await source
  .pipeThrough(t.toTransformStream<Event, Event>())
  .pipeTo(
    new WritableStream<Event>({
      write(e) { console.log(e.id); },
    })
  );
// Logs: "x", "y"

t.dispose(); // clears the in-memory seen set
```

## API Reference

### `dedupStream<T>(keyFn?: (item: T) => unknown, options?: TransformOptions): TransformStream<T, T>`

**Parameters**

- `keyFn` — optional function to compute a comparison key; defaults to identity comparison.
- `options.signal` — `AbortSignal`.
- `options.highWaterMark` — WHATWG queue depth (default: `1`).

**Returns** `TransformStream<T, T>`.

**Throws** (via error channel) `StreamAbortedError` when the signal fires.

### `class DedupTransform<T = unknown>`

#### `constructor(keyFn?: DedupKeyFn<T>)`

`keyFn` extracts the dedup key. Defaults to `JSON.stringify(chunk)`.

#### `process<U>(input: U, options?: TransformOptions): Promise<U>`

Returns `input` if its key has not been seen on this instance; otherwise throws `StreamTransformFailedError` with code `STREAM.TRANSFORM_FAILED` and message `"duplicate input"`. Honours `options.signal` — rejects with `StreamAbortedError` (`STREAM.ABORTED`) when fired.

#### `toTransformStream<A, B>(): TransformStream<A, B>`

Returns a fresh WHATWG `TransformStream` produced by `dedupStream(keyFn)`. Drops only _consecutive_ duplicates — independent of the all-time `seen` set held for `process`.

#### `dispose(): void`

Clears the private `seen` set, then calls `super.dispose()` to release hooks and listeners. Idempotent.

### `type DedupKeyFn<T>`

`(chunk: T) => string | number`

## Implementation Notes

The sentinel `Symbol("EMPTY")` ensures the first chunk always passes through `dedupStream` — there is no real "previous key" yet. Comparison uses strict equality (`!==`) on the key, so objects are compared by reference unless `keyFn` returns a primitive. The last key is retained in the transform closure; state is tied to the stream instance lifetime.

State is per-`DedupTransform` instance: each instance owns its own `Set<string | number>`. Two instances do not share dedup state, even if they were constructed with structurally-identical key functions. Construct one instance per logical dedup window. The class deliberately diverges from `dedupStream` in the single-input call path — `dedupStream` is consecutive-only because it cannot assume safe memory bounds for an unbounded incoming stream, whereas `process` is invoked with caller-controlled cardinality and can hold the full key set. `dispose()` clears the seen set first, then delegates to `BaseTransform.dispose()` to drop hooks and listeners; this ordering keeps the seen-set cleanup observable to subclasses that wrap the method.