suaveplan/openspec/changes/pagination-streaming/proposal.md

Pagination streaming

Package: @suaveplan/pagination Status: PROPOSED Type: feature

Why

The genesis port of @suaveplan/pagination ships every cursor / offset / keyset primitive plus iterator helpers (createCursorIterator, createOffsetIterator, collectAll, collectUpTo, mapPaginated), but it omits the upstream streaming helper that drives an AsyncIterable<T> directly off a PaginationAdapter. Upstream's streaming/stream-pagination.ts (164 L) ships:

• paginate(adapter, options) — drains pages from a PaginationAdapter<T>, yielding individual items, fetching the next page only when the current page is exhausted (one-page-in-memory invariant).
• collectStream(iterable) — drains a streaming paginator into an array.
• countStream(iterable) — counts items without retaining them.
• takeFromStream(iterable, count) — bounded take that stops fetching once enough items are collected.
• filterStream(iterable, predicate) — lazy filter passthrough.
• mapStream(iterable, mapper) — lazy map passthrough.

Genesis already has MemoryPaginationAdapter matching the upstream PaginationAdapter shape (fetchPage(offset, pageSize, sort?, filter?), count(filter?)), so the streaming layer slots in cleanly without altering the adapter contract. The gap costs every consumer that wants to drain a paginator — they currently hand-roll the offset-loop or accept loading the entire dataset via collectAll(createOffsetIterator(...)) (which holds pageSize records at a time but exposes pages, not items). The streaming surface is the canonical one upstream consumers reach for.

This change adds the streaming module, wires PaginationError codes for any new error paths (no expected new codes in the happy path), and ships the four operator helpers (collect, count, take, filter, map) so the surface matches upstream byte-for-byte.

What changes

• ADDED: streaming/stream-pagination module — exports streamPaginated<T>(adapter: PaginationAdapter<T>, options?: StreamPaginationOptions): AsyncIterable<T> (renamed from upstream paginate to avoid collision with paginateArray).
• ADDED: collectStream, countStream, takeFromStream, filterStream, mapStream operator helpers — match upstream signatures, exported from the same module.
• ADDED: StreamPaginationOptions type — { pageSize?: number; filter?: FilterConfig; sort?: SortConfig; signal?: AbortSignal }. Upstream omitted signal; genesis adds it because every other iterator in the package honours AbortSignal (PG-12).
• ADDED: streamPaginated honours options.signal — checks signal.aborted before each adapter.fetchPage call AND between yielding items; throws PaginationError(PAGINATION.ABORTED) on cancellation.
• ADDED: streamPaginated defaults pageSize to 50 (matches upstream constant DEFAULT_PAGE_SIZE).
• ADDED: subpath export ./streaming/stream-pagination in package.json for tree-shaking.
• ADDED: streaming/stream-pagination.md sibling doc with basic + advanced examples.

Out of scope

• Web-Streams ReadableStream adapter — wrapping an AsyncIterable<T> as a ReadableStream<T> is a separate change once this base iterable lands. It needs ReadableStreamDefaultController semantics and a backpressure strategy that's worth its own design pass.
• Reverse iteration — descending traversal from the end of the dataset is a separate concern; upstream doesn't ship it either.
• Cursor / keyset streaming — streamPaginated here is offset-driven (matches upstream). Cursor-streaming is already covered by createCursorIterator + collectAll; keyset-streaming is separately tracked by an existing PG-09 requirement.
• Multi-adapter parallel streaming — fanning out across multiple adapters (e.g. shard-aware streaming) needs an orchestrator; not in scope.
• Backpressure-aware push streaming — the iterable here yields on next() (pull model). Push-style streaming (with consumer rate-limit) is its own change.

Source mapping

This change derives from the upstream package at ../suaveplan/packages/universal/pagination/src/streaming/stream-pagination.ts and the upstream enterprise audit at ../suaveplan/enterprise/pagination.md.

Requirement	Source file	Source lines	Notes
streamPaginated(adapter, options) AsyncIterable	packages/universal/pagination/src/streaming/stream-pagination.ts	L17-L78	Renamed from upstream paginate to streamPaginated to avoid collision with the existing paginateArray in @suaveplan/pagination; behavior preserved byte-for-byte (default pageSize=50, single-page-in-memory invariant, lazy-init on first next())
collectStream(iterable)	packages/universal/pagination/src/streaming/stream-pagination.ts	L84-L92	Verbatim
countStream(iterable)	packages/universal/pagination/src/streaming/stream-pagination.ts	L97-L106	Verbatim
takeFromStream(iterable, count)	packages/universal/pagination/src/streaming/stream-pagination.ts	L112-L122	Verbatim
filterStream(iterable, predicate)	packages/universal/pagination/src/streaming/stream-pagination.ts	L128-L148	Verbatim
mapStream(iterable, mapper)	packages/universal/pagination/src/streaming/stream-pagination.ts	L154-L164	Verbatim
StreamPaginationOptions (pageSize, filter, sort)	packages/universal/pagination/src/types/pagination.types.ts	L166-L170	Verbatim port of three fields; genesis adds signal?: AbortSignal (see "no source line")

Requirements with no source line

Requirement	Justification
StreamPaginationOptions.signal?: AbortSignal	Genesis-wide AbortSignal rule applied across the package (existing PG-12 already mandates signal on createCursorIterator / createOffsetIterator / collectAll / collectUpTo). The streaming helper inherits the same contract; not adding it would create the only iterator in the package that ignores cancellation.
streamPaginated throws PaginationError(PAGINATION.ABORTED) on signal abort	Same justification — matches PG-12 behavior on the other iterators.
streamPaginated named (not paginate)	The existing genesis surface already exports paginateArray from ./offset/offset and cursorPaginateArray from ./cursor/cursor. A bare paginate would clash with consumer expectations (which import name is the "main" one?). streamPaginated is unambiguous — verb-prefix matches the symmetry with createCursorIterator / createOffsetIterator.

Verification approach

Each spec delta requirement carries a **Verified by:** line. The test files that will exist after this change ships:

• packages/data/pagination/src/streaming/stream-pagination.test.ts — drains MemoryPaginationAdapter with various pageSize settings; one-page-in-memory invariant verified via instrumentation; default pageSize=50 honoured; filter / sort forwarded; lazy-init pattern (no fetch before first next())
• packages/data/pagination/src/streaming/stream-pagination.spec-conformance.test.ts — signal.aborted before first fetchPage throws PaginationError(PAGINATION.ABORTED) without calling fetchPage; abort during in-flight fetchPage propagates rejection; abort between item yields throws on next next(); expectTypeOf<StreamPaginationOptions> pinned
• packages/data/pagination/src/streaming/stream-pagination.test.ts (operators) — collectStream, countStream, takeFromStream, filterStream, mapStream happy paths AND short-circuit / lazy-fetch behaviors (takeFromStream(iter, 5) does NOT fetch beyond the page that contains item 5)

Impact

Affected dependents

Run bunx turbo ls --filter=...@suaveplan/pagination to enumerate. Today grep -rl "@suaveplan/pagination" packages/ --include="package.json" returns:

• packages/platform/search/package.json
• packages/data/pagination/package.json (self)

So @suaveplan/search is the only in-repo dependent.

• @suaveplan/pagination (self) — [x] tests run / [x] docs reviewed / [ ] no action needed
• @suaveplan/search — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: streamPaginated is purely additive — @suaveplan/search consumes cursorPaginateArray and createCursorResponse only; no public-surface change touches its imports.)

If tests run is unchecked for any dependent, the change MUST NOT be folded into the canonical spec — finish that work first or split the proposal.

Migration impact

• All existing pagination consumers — no break. streamPaginated and the operator helpers are purely additive new exports. No existing surface is renamed, removed, or has its semantics changed.
• Existing iterator consumers (createCursorIterator, createOffsetIterator, collectAll, collectUpTo, mapPaginated) — no break. They continue to expose pages-of-T whereas streamPaginated exposes individual T. Both styles co-exist; consumers pick whichever fits their iteration model.