suaveplan/openspec/changes/etl-completion/tasks.md

# Tasks — ETL completion

> Every checkbox MUST reflect work actually done in this change. Do not retroactively tick boxes for work that was already in main when this change opened.

## 0. Pre-flight (non-negotiable)

- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.

## 1. Spec authoring

- [ ] 1.1 Read `../suaveplan/enterprise/etl.md` end-to-end
- [ ] 1.2 Read `openspec/specs/etl/spec.md` (does not exist yet — confirm and start fresh)
- [ ] 1.3 Read `openspec/AGENTS.md` Sections 13 and 14
- [ ] 1.4 Draft `proposal.md` from `openspec/templates/proposal.md`
- [ ] 1.5 Fill in `## Source mapping` table — every requirement cites a line range OR appears in the "no source line" table with justification
- [ ] 1.6 Draft `specs/etl/spec.md` from `openspec/templates/spec-delta.md`
- [ ] 1.7 Every requirement in the spec delta carries a `**Verified by:**` line naming concrete `it(...)` titles

## 2. Implementation — `core/context/context`

- [ ] 2.1 Port `PipelineContext` class from upstream `src/core/context/context.ts` into `packages/platform/etl/src/core/context/context.ts`
- [ ] 2.2 Port `createContext(options)` factory; options: `{ pipelineName, initialData?, clock?, logger?, abortSignal? }`
- [ ] 2.3 Replace upstream `Date.now()` with the injectable `Clock` from `@suaveplan/types`; default to `Clock.system()` from existing genesis setup
- [ ] 2.4 Wire context through every step — `Pipeline.run()` instantiates a `PipelineContext` per run and passes it to each stage's `run(ctx, ...)` method
- [ ] 2.5 Add `context.md` sibling doc (≥200 words, basic + advanced examples)

## 3. Implementation — `BatchConfig`

- [ ] 3.1 Add `BatchConfig` interface (`size`, `concurrency`, `perItemTimeoutMs`, `batchTimeoutMs`, `continueOnError`) to `types/etl.types.ts`
- [ ] 3.2 Extend `PipelineOptions` with optional `batch?: BatchConfig`
- [ ] 3.3 Update `Pipeline.run` to honour `batch.concurrency` via `parallelMap` (from §5)
- [ ] 3.4 Update `Pipeline.run` to honour `batch.perItemTimeoutMs` via `withTimeout` (from §5)
- [ ] 3.5 Update `Pipeline.run` to honour `batch.batchTimeoutMs` via `withTimeout` around the whole-batch promise
- [ ] 3.6 Update `Pipeline.run` to honour `batch.continueOnError` (skip failed batch, record error in context)
- [ ] 3.7 Add `BATCH_ITEM_TIMEOUT`, `BATCH_TIMEOUT`, `BATCH_CONCURRENCY_INVALID` codes to `errors/etl-error.ts`
- [ ] 3.8 Update `pipeline.md` sibling doc to cover batch config

## 4. Implementation — `types/etl-hooks`

- [ ] 4.1 Create `packages/platform/etl/src/types/etl-hooks/etl-hooks.ts` exporting `EtlHookMap` extending `HookMap` from `@suaveplan/events/hooks.types` with all eight lifecycle keys
- [ ] 4.2 Define payloads: `EtlPipelineHookData`, `EtlStageHookData`, `EtlBatchHookData`, `EtlTransformHookData`
- [ ] 4.3 Export named handler aliases (`EtlPipelineStartedHandler`, `EtlStageFailedHandler`, etc.) for consumer ergonomics
- [ ] 4.4 Update `Pipeline` to accept `hooks?: HookBus<EtlHookMap>` in options; default to a fresh internal bus
- [ ] 4.5 Emit each lifecycle event in the right place inside `Pipeline.run` and `RunnablePipeline.run`
- [ ] 4.6 Add `etl-hooks.md` sibling doc (≥200 words, basic + advanced examples)

## 5. Implementation — `utils/utils`

- [ ] 5.1 Create `packages/platform/etl/src/utils/utils.ts` with `applyMappings`, `getNestedValue`, `setNestedValue`, `parallelMap`, `toBatches`, `withTimeout`
- [ ] 5.2 `withTimeout` rejects with `EtlError(TIMEOUT)` (not a plain `Error`); pass through the original promise's rejection reason in `cause`
- [ ] 5.3 `parallelMap` rejects with `EtlError(BATCH_CONCURRENCY_INVALID)` when `concurrency < 1`
- [ ] 5.4 `toBatches` returns `[]` for empty input; rejects on `size < 1` with `EtlError(BATCH_CONCURRENCY_INVALID)`
- [ ] 5.5 Add `utils.md` sibling doc (≥200 words, basic + advanced examples)
- [ ] 5.6 Re-export every symbol from `index.ts`

## 6. Implementation — `FieldMapping` + `mapTransform` integration

- [ ] 6.1 Add `FieldMapping` interface to `types/etl.types.ts`
- [ ] 6.2 Update `TransformConfig` to accept optional `mappings?: readonly FieldMapping[]`
- [ ] 6.3 Update `mapTransform` to apply `mappings` (via `applyMappings`) BEFORE invoking the transform function
- [ ] 6.4 Re-export `FieldMapping` from `index.ts`
- [ ] 6.5 Update `transform.md` sibling doc

## 7. Tests

- [ ] 7.1 `core/context/context.test.ts` — set / get / has / delete; hook register / off; errors / stepResults appended; AbortSignal propagates to abortable steps
- [ ] 7.2 `core/context/context.spec-conformance.test.ts` — `expectTypeOf<PipelineContextOptions>` pinned; injected `Clock` deterministic in stamping `metadata.startedAt`
- [ ] 7.3 `core/pipeline/pipeline.test.ts` (extended) — `batch.size` chunks input correctly; `batch.concurrency` runs N in parallel (timing assertion); `batch.perItemTimeoutMs` rejects slow items with `BATCH_ITEM_TIMEOUT`; `batch.batchTimeoutMs` rejects slow batches; `batch.continueOnError: true` skips failure
- [ ] 7.4 `types/etl-hooks/etl-hooks.test.ts` — all 8 lifecycle keys fire once per transition with correct payloads
- [ ] 7.5 `types/etl-hooks/etl-hooks.spec-conformance.test.ts` — `expectTypeOf` pins each key string; pins payload shapes
- [ ] 7.6 `utils/utils.test.ts` — `applyMappings` (basic, with transform, missing source), `getNestedValue` (nested, missing, null guard), `setNestedValue` (creates intermediates), `parallelMap` (concurrency 1, 2, equal-to-input), `toBatches` (empty, exact, remainder), `withTimeout` (resolves under, rejects over)
- [ ] 7.7 `utils/utils.spec-conformance.test.ts` — `parallelMap` honours concurrency empirically (max-in-flight assertion via shared counter); `withTimeout` rejects with `EtlError(TIMEOUT)`; `parallelMap` rejects with `EtlError(BATCH_CONCURRENCY_INVALID)` for `concurrency < 1`
- [ ] 7.8 `errors/etl-error.spec-conformance.test.ts` — `ETL_CODES` registered after import; idempotent re-registration; new codes (`BATCH_ITEM_TIMEOUT`, `BATCH_TIMEOUT`, `BATCH_CONCURRENCY_INVALID`, `TIMEOUT`) thrown in production paths
- [ ] 7.9 Every `it(...)` in 7.1–7.8 carries `{ timeout: 5000 }` (or 30000 for genuinely async)

## 8. Documentation

- [ ] 8.1 Sibling `.md` for every new implementation file (`context.md`, `etl-hooks.md`, `utils.md`); each ≥200 words with basic + advanced examples
- [ ] 8.2 Update existing sibling `.md`s touched in §3, §4, §6 (`pipeline.md`, `transform.md`)
- [ ] 8.3 Update `packages/platform/etl/README.md` — add Modules links for the three new files; add Quick-start snippet showing context + hooks subscription
- [ ] 8.4 Update `CLAUDE.md` package listing line for `@suaveplan/etl` to reflect the expanded surface

## 9. Pre-archive verification (the 5-gate triple)

- [ ] 9.1 `bun run check` — Biome 0 errors / 0 warnings / 0 info across the whole monorepo
- [ ] 9.2 `bunx turbo lint typecheck test --filter=@suaveplan/etl` — green
- [ ] 9.3 `bunx turbo test:coverage --filter=@suaveplan/etl` — 100/100/100/100
- [ ] 9.4 `bun scripts/openspec-conformance.ts @suaveplan/etl` — exit 0, no `error` findings
- [ ] 9.5 `bunx turbo build --filter=@suaveplan/etl` — green

## 10. Pre-archive 7-row checklist (AGENTS.md Section 13)

- [ ] 10.1 Generics threaded through `PipelineContext<TData>`, `parallelMap<T, U>`, `applyMappings<TIn, TOut>`
- [ ] 10.2 Zod schemas wired at the `BatchConfig` and `PipelineOptions` boundaries (already partially in `etl.schemas.ts` — extend)
- [ ] 10.3 Every error code in `errors/etl-error.ts` thrown by at least one production path
- [ ] 10.4 Strict-mode behaviors tested both on and off (`continueOnError: true` vs `false`)
- [ ] 10.5 OTel instruments emitted from production paths and asserted with a real in-memory `MeterProvider`
- [ ] 10.6 Lifecycle dedup tested where claimed (idempotent `registerEtlErrorCodes`)
- [ ] 10.7 No retroactive tick — every box above reflects work done in this change

## 11. Archive

- [ ] 11.1 Fold spec delta into `openspec/specs/etl/spec.md` (will be the first time this canonical spec exists)
- [ ] 11.2 Move `openspec/changes/etl-completion/` → `openspec/archive/<yyyy-mm-dd>-etl-completion/`
- [ ] 11.3 Refresh `MANIFEST.md` if package set or deps changed (no expected change here)

## Audit Note (2026-05-02)

Audit verdict: **DEFERRED — 0/65 tasks evidenced in code**.

Evidence reviewed:

- `packages/platform/etl/src/` contains `core/pipeline/`, `stages/{transform,source,sink}/`, `errors/`, `schemas/`, `types/`, `constants/`. No `core/context/`, no `utils/`, no `types/etl-hooks/`.
- `errors/etl-error.ts` does not declare `BATCH_ITEM_TIMEOUT`, `BATCH_TIMEOUT`, `BATCH_CONCURRENCY_INVALID`, or `TIMEOUT` codes.
- No `BatchConfig` shape in `types/etl.types.ts` (no batch concurrency / per-item timeout wired into `Pipeline.run`).

Missing pieces:

1. `core/context/context.ts` `PipelineContext` with injectable `Clock` + `AbortSignal`.
2. `BatchConfig` (size / concurrency / perItemTimeoutMs / batchTimeoutMs / continueOnError) wired into `Pipeline.run` via the new `utils/utils.ts` (`parallelMap`, `withTimeout`, `toBatches`).
3. `types/etl-hooks/etl-hooks.ts` `EtlHookMap` + payloads + emission inside `Pipeline.run`, plus `FieldMapping` integration into `mapTransform`.

Decision: leave change open. The three new modules are interlocking (context threads through hooks, batch config relies on the new utils) and need fresh tests + sibling docs.