suaveplan/openspec/changes/etl-completion/tasks.md

Tasks — ETL completion

Every checkbox MUST reflect work actually done in this change. Do not retroactively tick boxes for work that was already in main when this change opened.

0. Pre-flight (non-negotiable)

• 0.1 Confirm clean working tree (git status reports nothing) and no in-flight worktrees with uncommitted work (git worktree list then git status in each)
• 0.2 If any worktrees exist under .claude/worktrees/, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (git worktree remove) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
• 0.3 Confirm the package set in MANIFEST.md matches find packages -maxdepth 3 -name package.json count. Drift here means a prior change archived without regenerating the manifest.

1. Spec authoring

• 1.1 Read ../suaveplan/enterprise/etl.md end-to-end
• 1.2 Read openspec/specs/etl/spec.md (does not exist yet — confirm and start fresh)
• 1.3 Read openspec/AGENTS.md Sections 13 and 14
• 1.4 Draft proposal.md from openspec/templates/proposal.md
• 1.5 Fill in ## Source mapping table — every requirement cites a line range OR appears in the "no source line" table with justification
• 1.6 Draft specs/etl/spec.md from openspec/templates/spec-delta.md
• 1.7 Every requirement in the spec delta carries a **Verified by:** line naming concrete it(...) titles

2. Implementation — core/context/context

• 2.1 Port PipelineContext class from upstream src/core/context/context.ts into packages/platform/etl/src/core/context/context.ts
• 2.2 Port createContext(options) factory; options: { pipelineName, initialData?, clock?, logger?, abortSignal? }
• 2.3 Replace upstream Date.now() with the injectable Clock from @suaveplan/types; default to Clock.system() from existing genesis setup
• 2.4 Wire context through every step — Pipeline.run() instantiates a PipelineContext per run and passes it to each stage's run(ctx, ...) method
• 2.5 Add context.md sibling doc (≥200 words, basic + advanced examples)

3. Implementation — BatchConfig

• 3.1 Add BatchConfig interface (size, concurrency, perItemTimeoutMs, batchTimeoutMs, continueOnError) to types/etl.types.ts
• 3.2 Extend PipelineOptions with optional batch?: BatchConfig
• 3.3 Update Pipeline.run to honour batch.concurrency via parallelMap (from §5)
• 3.4 Update Pipeline.run to honour batch.perItemTimeoutMs via withTimeout (from §5)
• 3.5 Update Pipeline.run to honour batch.batchTimeoutMs via withTimeout around the whole-batch promise
• 3.6 Update Pipeline.run to honour batch.continueOnError (skip failed batch, record error in context)
• 3.7 Add BATCH_ITEM_TIMEOUT, BATCH_TIMEOUT, BATCH_CONCURRENCY_INVALID codes to errors/etl-error.ts
• 3.8 Update pipeline.md sibling doc to cover batch config

4. Implementation — types/etl-hooks

• 4.1 Create packages/platform/etl/src/types/etl-hooks/etl-hooks.ts exporting EtlHookMap extending HookMap from @suaveplan/events/hooks.types with all eight lifecycle keys
• 4.2 Define payloads: EtlPipelineHookData, EtlStageHookData, EtlBatchHookData, EtlTransformHookData
• 4.3 Export named handler aliases (EtlPipelineStartedHandler, EtlStageFailedHandler, etc.) for consumer ergonomics
• 4.4 Update Pipeline to accept hooks?: HookBus<EtlHookMap> in options; default to a fresh internal bus
• 4.5 Emit each lifecycle event in the right place inside Pipeline.run and RunnablePipeline.run
• 4.6 Add etl-hooks.md sibling doc (≥200 words, basic + advanced examples)

5. Implementation — utils/utils

• 5.1 Create packages/platform/etl/src/utils/utils.ts with applyMappings, getNestedValue, setNestedValue, parallelMap, toBatches, withTimeout
• 5.2 withTimeout rejects with EtlError(TIMEOUT) (not a plain Error); pass through the original promise's rejection reason in cause
• 5.3 parallelMap rejects with EtlError(BATCH_CONCURRENCY_INVALID) when concurrency < 1
• 5.4 toBatches returns [] for empty input; rejects on size < 1 with EtlError(BATCH_CONCURRENCY_INVALID)
• 5.5 Add utils.md sibling doc (≥200 words, basic + advanced examples)
• 5.6 Re-export every symbol from index.ts

6. Implementation — FieldMapping + mapTransform integration

• 6.1 Add FieldMapping interface to types/etl.types.ts
• 6.2 Update TransformConfig to accept optional mappings?: readonly FieldMapping[]
• 6.3 Update mapTransform to apply mappings (via applyMappings) BEFORE invoking the transform function
• 6.4 Re-export FieldMapping from index.ts
• 6.5 Update transform.md sibling doc

7. Tests

• 7.1 core/context/context.test.ts — set / get / has / delete; hook register / off; errors / stepResults appended; AbortSignal propagates to abortable steps
• 7.2 core/context/context.spec-conformance.test.ts — expectTypeOf<PipelineContextOptions> pinned; injected Clock deterministic in stamping metadata.startedAt
• 7.3 core/pipeline/pipeline.test.ts (extended) — batch.size chunks input correctly; batch.concurrency runs N in parallel (timing assertion); batch.perItemTimeoutMs rejects slow items with BATCH_ITEM_TIMEOUT; batch.batchTimeoutMs rejects slow batches; batch.continueOnError: true skips failure
• 7.4 types/etl-hooks/etl-hooks.test.ts — all 8 lifecycle keys fire once per transition with correct payloads
• 7.5 types/etl-hooks/etl-hooks.spec-conformance.test.ts — expectTypeOf pins each key string; pins payload shapes
• 7.6 utils/utils.test.ts — applyMappings (basic, with transform, missing source), getNestedValue (nested, missing, null guard), setNestedValue (creates intermediates), parallelMap (concurrency 1, 2, equal-to-input), toBatches (empty, exact, remainder), withTimeout (resolves under, rejects over)
• 7.7 utils/utils.spec-conformance.test.ts — parallelMap honours concurrency empirically (max-in-flight assertion via shared counter); withTimeout rejects with EtlError(TIMEOUT); parallelMap rejects with EtlError(BATCH_CONCURRENCY_INVALID) for concurrency < 1
• 7.8 errors/etl-error.spec-conformance.test.ts — ETL_CODES registered after import; idempotent re-registration; new codes (BATCH_ITEM_TIMEOUT, BATCH_TIMEOUT, BATCH_CONCURRENCY_INVALID, TIMEOUT) thrown in production paths
• 7.9 Every it(...) in 7.1–7.8 carries { timeout: 5000 } (or 30000 for genuinely async)

8. Documentation

• 8.1 Sibling .md for every new implementation file (context.md, etl-hooks.md, utils.md); each ≥200 words with basic + advanced examples
• 8.2 Update existing sibling .mds touched in §3, §4, §6 (pipeline.md, transform.md)
• 8.3 Update packages/platform/etl/README.md — add Modules links for the three new files; add Quick-start snippet showing context + hooks subscription
• 8.4 Update CLAUDE.md package listing line for @suaveplan/etl to reflect the expanded surface

9. Pre-archive verification (the 5-gate triple)

• 9.1 bun run check — Biome 0 errors / 0 warnings / 0 info across the whole monorepo
• 9.2 bunx turbo lint typecheck test --filter=@suaveplan/etl — green
• 9.3 bunx turbo test:coverage --filter=@suaveplan/etl — 100/100/100/100
• 9.4 bun scripts/openspec-conformance.ts @suaveplan/etl — exit 0, no error findings
• 9.5 bunx turbo build --filter=@suaveplan/etl — green

10. Pre-archive 7-row checklist (AGENTS.md Section 13)

• 10.1 Generics threaded through PipelineContext<TData>, parallelMap<T, U>, applyMappings<TIn, TOut>
• 10.2 Zod schemas wired at the BatchConfig and PipelineOptions boundaries (already partially in etl.schemas.ts — extend)
• 10.3 Every error code in errors/etl-error.ts thrown by at least one production path
• 10.4 Strict-mode behaviors tested both on and off (continueOnError: true vs false)
• 10.5 OTel instruments emitted from production paths and asserted with a real in-memory MeterProvider
• 10.6 Lifecycle dedup tested where claimed (idempotent registerEtlErrorCodes)
• 10.7 No retroactive tick — every box above reflects work done in this change

11. Archive

• 11.1 Fold spec delta into openspec/specs/etl/spec.md (will be the first time this canonical spec exists)
• 11.2 Move openspec/changes/etl-completion/ → openspec/archive/<yyyy-mm-dd>-etl-completion/
• 11.3 Refresh MANIFEST.md if package set or deps changed (no expected change here)

Audit Note (2026-05-02)

Audit verdict: DEFERRED — 0/65 tasks evidenced in code.

Evidence reviewed:

• packages/platform/etl/src/ contains core/pipeline/, stages/{transform,source,sink}/, errors/, schemas/, types/, constants/. No core/context/, no utils/, no types/etl-hooks/.
• errors/etl-error.ts does not declare BATCH_ITEM_TIMEOUT, BATCH_TIMEOUT, BATCH_CONCURRENCY_INVALID, or TIMEOUT codes.
• No BatchConfig shape in types/etl.types.ts (no batch concurrency / per-item timeout wired into Pipeline.run).

Missing pieces:

1. core/context/context.ts PipelineContext with injectable Clock + AbortSignal.
2. BatchConfig (size / concurrency / perItemTimeoutMs / batchTimeoutMs / continueOnError) wired into Pipeline.run via the new utils/utils.ts (parallelMap, withTimeout, toBatches).
3. types/etl-hooks/etl-hooks.ts EtlHookMap + payloads + emission inside Pipeline.run, plus FieldMapping integration into mapTransform.

Decision: leave change open. The three new modules are interlocking (context threads through hooks, batch config relies on the new utils) and need fresh tests + sibling docs.