suaveplan/openspec/changes/pagination-streaming/proposal.md

# Pagination streaming

**Package**: `@suaveplan/pagination`
**Status**: PROPOSED
**Type**: feature

## Why

The genesis port of `@suaveplan/pagination` ships every cursor / offset / keyset primitive plus iterator helpers (`createCursorIterator`, `createOffsetIterator`, `collectAll`, `collectUpTo`, `mapPaginated`), but it omits the upstream streaming helper that drives an `AsyncIterable<T>` directly off a `PaginationAdapter`. Upstream's `streaming/stream-pagination.ts` (164 L) ships:

- `paginate(adapter, options)` — drains pages from a `PaginationAdapter<T>`, yielding individual items, fetching the next page only when the current page is exhausted (one-page-in-memory invariant).
- `collectStream(iterable)` — drains a streaming paginator into an array.
- `countStream(iterable)` — counts items without retaining them.
- `takeFromStream(iterable, count)` — bounded take that stops fetching once enough items are collected.
- `filterStream(iterable, predicate)` — lazy filter passthrough.
- `mapStream(iterable, mapper)` — lazy map passthrough.

Genesis already has `MemoryPaginationAdapter` matching the upstream `PaginationAdapter` shape (`fetchPage(offset, pageSize, sort?, filter?)`, `count(filter?)`), so the streaming layer slots in cleanly without altering the adapter contract. The gap costs every consumer that wants to drain a paginator — they currently hand-roll the offset-loop or accept loading the entire dataset via `collectAll(createOffsetIterator(...))` (which holds `pageSize` records at a time but exposes pages, not items). The streaming surface is the canonical one upstream consumers reach for.

This change adds the streaming module, wires `PaginationError` codes for any new error paths (no expected new codes in the happy path), and ships the four operator helpers (`collect`, `count`, `take`, `filter`, `map`) so the surface matches upstream byte-for-byte.

## What changes

- ADDED: `streaming/stream-pagination` module — exports `streamPaginated<T>(adapter: PaginationAdapter<T>, options?: StreamPaginationOptions): AsyncIterable<T>` (renamed from upstream `paginate` to avoid collision with `paginateArray`).
- ADDED: `collectStream`, `countStream`, `takeFromStream`, `filterStream`, `mapStream` operator helpers — match upstream signatures, exported from the same module.
- ADDED: `StreamPaginationOptions` type — `{ pageSize?: number; filter?: FilterConfig; sort?: SortConfig; signal?: AbortSignal }`. Upstream omitted `signal`; genesis adds it because every other iterator in the package honours `AbortSignal` (PG-12).
- ADDED: `streamPaginated` honours `options.signal` — checks `signal.aborted` before each `adapter.fetchPage` call AND between yielding items; throws `PaginationError(PAGINATION.ABORTED)` on cancellation.
- ADDED: `streamPaginated` defaults `pageSize` to 50 (matches upstream constant `DEFAULT_PAGE_SIZE`).
- ADDED: subpath export `./streaming/stream-pagination` in `package.json` for tree-shaking.
- ADDED: `streaming/stream-pagination.md` sibling doc with basic + advanced examples.

## Out of scope

- **Web-Streams `ReadableStream` adapter** — wrapping an `AsyncIterable<T>` as a `ReadableStream<T>` is a separate change once this base iterable lands. It needs `ReadableStreamDefaultController` semantics and a backpressure strategy that's worth its own design pass.
- **Reverse iteration** — descending traversal from the end of the dataset is a separate concern; upstream doesn't ship it either.
- **Cursor / keyset streaming** — `streamPaginated` here is offset-driven (matches upstream). Cursor-streaming is already covered by `createCursorIterator` + `collectAll`; keyset-streaming is separately tracked by an existing PG-09 requirement.
- **Multi-adapter parallel streaming** — fanning out across multiple adapters (e.g. shard-aware streaming) needs an orchestrator; not in scope.
- **Backpressure-aware push streaming** — the iterable here yields on `next()` (pull model). Push-style streaming (with consumer rate-limit) is its own change.

## Source mapping

This change derives from the upstream package at `../suaveplan/packages/universal/pagination/src/streaming/stream-pagination.ts` and the upstream enterprise audit at `../suaveplan/enterprise/pagination.md`.

| Requirement | Source file | Source lines | Notes |
|---|---|---|---|
| `streamPaginated(adapter, options)` AsyncIterable | `packages/universal/pagination/src/streaming/stream-pagination.ts` | L17-L78 | Renamed from upstream `paginate` to `streamPaginated` to avoid collision with the existing `paginateArray` in `@suaveplan/pagination`; behavior preserved byte-for-byte (default `pageSize=50`, single-page-in-memory invariant, lazy-init on first `next()`) |
| `collectStream(iterable)` | `packages/universal/pagination/src/streaming/stream-pagination.ts` | L84-L92 | Verbatim |
| `countStream(iterable)` | `packages/universal/pagination/src/streaming/stream-pagination.ts` | L97-L106 | Verbatim |
| `takeFromStream(iterable, count)` | `packages/universal/pagination/src/streaming/stream-pagination.ts` | L112-L122 | Verbatim |
| `filterStream(iterable, predicate)` | `packages/universal/pagination/src/streaming/stream-pagination.ts` | L128-L148 | Verbatim |
| `mapStream(iterable, mapper)` | `packages/universal/pagination/src/streaming/stream-pagination.ts` | L154-L164 | Verbatim |
| `StreamPaginationOptions` (`pageSize`, `filter`, `sort`) | `packages/universal/pagination/src/types/pagination.types.ts` | L166-L170 | Verbatim port of three fields; genesis adds `signal?: AbortSignal` (see "no source line") |

### Requirements with no source line

| Requirement | Justification |
|---|---|
| `StreamPaginationOptions.signal?: AbortSignal` | Genesis-wide AbortSignal rule applied across the package (existing PG-12 already mandates `signal` on `createCursorIterator` / `createOffsetIterator` / `collectAll` / `collectUpTo`). The streaming helper inherits the same contract; not adding it would create the only iterator in the package that ignores cancellation. |
| `streamPaginated` throws `PaginationError(PAGINATION.ABORTED)` on signal abort | Same justification — matches PG-12 behavior on the other iterators. |
| `streamPaginated` named (not `paginate`) | The existing genesis surface already exports `paginateArray` from `./offset/offset` and `cursorPaginateArray` from `./cursor/cursor`. A bare `paginate` would clash with consumer expectations (which import name is the "main" one?). `streamPaginated` is unambiguous — verb-prefix matches the symmetry with `createCursorIterator` / `createOffsetIterator`. |

## Verification approach

Each spec delta requirement carries a `**Verified by:**` line. The test files that will exist after this change ships:

- `packages/data/pagination/src/streaming/stream-pagination.test.ts` — drains `MemoryPaginationAdapter` with various `pageSize` settings; one-page-in-memory invariant verified via instrumentation; default `pageSize=50` honoured; `filter` / `sort` forwarded; lazy-init pattern (no fetch before first `next()`)
- `packages/data/pagination/src/streaming/stream-pagination.spec-conformance.test.ts` — `signal.aborted` before first `fetchPage` throws `PaginationError(PAGINATION.ABORTED)` without calling `fetchPage`; abort during in-flight `fetchPage` propagates rejection; abort between item yields throws on next `next()`; `expectTypeOf<StreamPaginationOptions>` pinned
- `packages/data/pagination/src/streaming/stream-pagination.test.ts` (operators) — `collectStream`, `countStream`, `takeFromStream`, `filterStream`, `mapStream` happy paths AND short-circuit / lazy-fetch behaviors (`takeFromStream(iter, 5)` does NOT fetch beyond the page that contains item 5)

## Impact

### Affected dependents

Run `bunx turbo ls --filter=...@suaveplan/pagination` to enumerate. Today `grep -rl "@suaveplan/pagination" packages/ --include="package.json"` returns:

- `packages/platform/search/package.json`
- `packages/data/pagination/package.json` (self)

So `@suaveplan/search` is the only in-repo dependent.

- `@suaveplan/pagination` (self) — [x] tests run / [x] docs reviewed / [ ] no action needed
- `@suaveplan/search` — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: `streamPaginated` is purely additive — `@suaveplan/search` consumes `cursorPaginateArray` and `createCursorResponse` only; no public-surface change touches its imports.)

If `tests run` is unchecked for any dependent, the change MUST NOT be folded into the canonical spec — finish that work first or split the proposal.

## Migration impact

- **All existing pagination consumers** — no break. `streamPaginated` and the operator helpers are purely additive new exports. No existing surface is renamed, removed, or has its semantics changed.
- **Existing iterator consumers** (`createCursorIterator`, `createOffsetIterator`, `collectAll`, `collectUpTo`, `mapPaginated`) — no break. They continue to expose pages-of-T whereas `streamPaginated` exposes individual T. Both styles co-exist; consumers pick whichever fits their iteration model.