first commit
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
# mergeOrdered
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
`mergeOrdered` performs a k-way merge of pre-sorted `ReadableStream`s
|
||||
into a single ordered output stream. It is the streaming analogue of
|
||||
the merge step in merge-sort: when every input is sorted under the
|
||||
same comparator, the output is sorted under that comparator. This is
|
||||
the right tool whenever a pipeline needs to consume multiple already-
|
||||
ordered sources — sharded query results, time-keyed event logs,
|
||||
externally-sorted partitions — without re-sorting the union in memory.
|
||||
|
||||
Unlike the unordered `merge`, which races inputs and emits in arrival
|
||||
order, `mergeOrdered` emits the global minimum at every step using a
|
||||
binary min-heap of `(value, sourceIdx)` entries. Each pull dequeues
|
||||
the heap root, enqueues it on the output, and replaces the slot with
|
||||
the next chunk from the same source. Throughput is dominated by the
|
||||
`O(log k)` heap operations where `k` is the number of inputs.
|
||||
|
||||
## Features
|
||||
|
||||
- K-way merge over `ReadableStream<T>` arrays of any cardinality
|
||||
- Caller-supplied comparator (ascending, descending, or domain-specific)
|
||||
- Single-stream and empty-array fast paths
|
||||
- Streams of differing lengths handled without truncation
|
||||
- `cancel` propagates to every input stream
|
||||
- Errors on any input cancel the remaining inputs and surface on the
|
||||
merged stream
|
||||
- Backpressure-respecting — only pulls one chunk per source per pull cycle
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { mergeOrdered } from "@suaveplan/streams/pipeline/merge-ordered/merge-ordered";
|
||||
import { fromArray } from "@suaveplan/streams/readable/from-array/from-array";
|
||||
import { toArray } from "@suaveplan/streams/sinks/to-array/to-array";
|
||||
|
||||
const s1 = fromArray([1, 4, 7]);
|
||||
const s2 = fromArray([2, 5, 8]);
|
||||
const s3 = fromArray([3, 6, 9]);
|
||||
|
||||
const merged = mergeOrdered([s1, s2, s3], (a, b) => a - b);
|
||||
const out = await toArray(merged);
|
||||
// out === [1, 2, 3, 4, 5, 6, 7, 8, 9]
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Merge sharded events keyed by timestamp, in descending order, and
|
||||
forward the result into a downstream consumer that may cancel mid-flight:
|
||||
|
||||
```ts
|
||||
import { mergeOrdered } from "@suaveplan/streams/pipeline/merge-ordered/merge-ordered";
|
||||
|
||||
interface Event {
|
||||
ts: number;
|
||||
payload: unknown;
|
||||
}
|
||||
|
||||
declare const shardA: ReadableStream<Event>;
|
||||
declare const shardB: ReadableStream<Event>;
|
||||
declare const shardC: ReadableStream<Event>;
|
||||
|
||||
const merged = mergeOrdered<Event>(
|
||||
[shardA, shardB, shardC],
|
||||
(a, b) => b.ts - a.ts, // newest-first
|
||||
);
|
||||
|
||||
const reader = merged.getReader();
|
||||
try {
|
||||
const first = await reader.read();
|
||||
if (!first.done) {
|
||||
// process newest event
|
||||
}
|
||||
} finally {
|
||||
// Cancellation propagates to every shard.
|
||||
await reader.cancel("consumer-done");
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `mergeOrdered<T>(streams, compare): ReadableStream<T>`
|
||||
|
||||
- `streams: ReadableStream<T>[]` — input streams, each pre-sorted
|
||||
under `compare`. May be empty (returns an immediately-closed stream)
|
||||
or singleton (returns a passthrough merge with the same heap path).
|
||||
- `compare: (a: T, b: T) => number` — comparator. Negative when `a`
|
||||
precedes `b`, zero when equal, positive otherwise. Identical contract
|
||||
to `Array.prototype.sort`.
|
||||
- **Returns** a `ReadableStream<T>` that emits values from the union of
|
||||
all inputs in `compare` order, then closes.
|
||||
- **Throws / errors** — if any input reader rejects, the remaining
|
||||
inputs are cancelled and the rejection is forwarded as a stream error.
|
||||
|
||||
### `MergeOrderedCompare<T>`
|
||||
|
||||
Type alias for the comparator argument; exported for downstream typing
|
||||
of comparator-producing factories.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The heap stores at most `k` entries (one per active source) so peak
|
||||
memory is `O(k)`, independent of the total chunk count.
|
||||
- Heap operations (`siftUp` / `siftDown`) are iterative and avoid
|
||||
recursion to keep call stacks shallow on large `k`.
|
||||
- The first `pull` primes every reader in parallel via `Promise.all`;
|
||||
subsequent pulls only read from the source whose value was just
|
||||
emitted, which preserves backpressure on the other inputs.
|
||||
- `cancel` on the output uses `Promise.allSettled` to ensure every
|
||||
input is cancelled even if one rejects.
|
||||
- Output sortedness is conditional on input sortedness — passing
|
||||
unsorted inputs is a caller bug, not a runtime check, because
|
||||
validating sortedness would defeat the streaming guarantees.
|
||||
- The single-stream path still goes through the heap for behavioural
|
||||
parity with k≥2 (cancellation, error propagation).
|
||||
Reference in New Issue
Block a user