Initial Commit

openspec/changes/etl-completion/proposal.md

@@ -0,0 +1,95 @@
+# ETL completion
+
+**Package**: `@suaveplan/etl`
+**Status**: PROPOSED
+**Type**: spec-conformance + feature
+
+## Why
+
+The genesis port of `@suaveplan/etl` ships the core stage primitives (`arraySource`, `mapTransform`, `BatchSink`, etc.) and a `Pipeline` builder, but the surface around it is materially thinner than the upstream package at `../suaveplan/packages/universal/etl`. Five concrete gaps make consumers either rebuild missing pieces in-app or abandon the package altogether:
+
+1. **No `PipelineContext`** — upstream exposes a mutable execution context (data, errors, stepResults, hooks, metadata K/V) that travels through every step. Genesis stages get isolated `TransformContext` snapshots, so cross-step state requires bolting on a separate map.
+2. **No batch concurrency / timeout config** — upstream's `BatchConfig` ships `size`, `concurrency`, and (via pipeline) per-batch timeouts. Genesis only honours `size` via `BatchSink`, so consumers can't opt into parallel batches or per-item timeout enforcement.
+3. **No `EtlHookMap` lifecycle bus** — upstream emits `pipeline:start`, `pipeline:end`, `step:start`, `step:end`, `step:error`, `batch:start`, `batch:end`, `batch:error` through a typed hook map. Genesis has no observability surface beyond raw error returns.
+4. **Missing utility primitives** — `applyMappings`, `getNestedValue`, `setNestedValue`, `parallelMap`, `toBatches`, `withTimeout` are all absent. These are the shared building blocks every non-trivial ETL pipeline needs and they belong in the package, not the call site.
+5. **Missing `FieldMapping` type** — upstream ships `FieldMapping` (`{ from, to, transform? }`) plus `applyMappings(record, mappings)`. Genesis has no canonical mapping shape, so each consumer hand-rolls one.
+
+This change closes the five gaps in one coordinated pass so downstream services (`@suaveplan/packer` import pipelines, future analytics ETLs) can rely on a single coherent ETL surface.
+
+## What changes
+
+- ADDED: `core/context/context` module — exports `class PipelineContext` and `function createContext(options: PipelineContextOptions)` returning a typed shared-state object with `data`, `errors`, `stepResults`, `metadata` map, hook registration, and an injectable `LoggerLike` / `Clock` / `AbortSignal`. The context threads through every step.
+- ADDED: `BatchConfig` interface — `{ size: number; concurrency?: number; perItemTimeoutMs?: number; batchTimeoutMs?: number; continueOnError?: boolean }`. Pipeline config grows `batch?: BatchConfig`. The pipeline runner SHALL honour all four fields.
+- ADDED: `types/etl-hooks` module — exports `EtlHookMap` extending `HookMap` from `@suaveplan/events::hooks.types`, with keys `"pipeline:started"`, `"pipeline:completed"`, `"stage:started"`, `"stage:completed"`, `"stage:failed"`, `"batch:started"`, `"batch:completed"`, `"batch:failed"`. Plus `EtlTransformHookData` and `EtlTransformingHandler` aliases for the most common subscriber shape.
+- ADDED: `utils/utils` module — exports `applyMappings(record, mappings)`, `getNestedValue(obj, path)`, `setNestedValue(obj, path, value)`, `parallelMap(items, fn, concurrency)`, `toBatches(items, size)`, `withTimeout(promise, ms)`.
+- ADDED: `FieldMapping` type — `{ from: string; to: string; transform?: (value: unknown) => unknown }`. `mapTransform` and any `TransformConfig` SHALL accept `mappings?: readonly FieldMapping[]` applied before the transform function.
+- MODIFIED: `Pipeline` and `RunnablePipeline` — accept and emit `EtlHookMap` events through an injectable `HookBus` (defaults to a fresh internal bus when omitted). Existing `errorHandler` API preserved.
+- ADDED: per-module sibling `.md` for every new file.
+
+## Out of scope
+
+- **Connectors for specific databases** (e.g. SurrealDB source, Postgres sink, S3 source) — separate adapter packages so each can pull in its native client and integration test fixture independently. This proposal only stabilises the runner contract.
+- **CDC stream support** — change-data-capture / event-source ingestion belongs to a streaming-etl change paired with `@suaveplan/events` interceptors; this proposal stays batch-pipeline-only.
+- **Schema inference / auto-mapping** — synthesising `FieldMapping[]` from a sample record is a separate research item.
+- **Web Streams runner** — the existing pipeline is in-memory and uses async iterators; a `ReadableStream` runner is its own change.
+- **Distributed pipeline orchestration** — multi-worker pipelines (queue-backed work items) require an `@suaveplan/idempotency` integration that isn't in scope here.
+
+## Source mapping
+
+This change derives from the upstream package at `../suaveplan/packages/universal/etl/src` and the upstream enterprise audit at `../suaveplan/enterprise/etl.md`.
+
+| Requirement | Source file | Source lines | Notes |
+|---|---|---|---|
+| `PipelineContext` class with mutable shared state | `packages/universal/etl/src/core/context/context.ts` | L1-L120 | Verbatim port; thread through `Clock` / `LoggerLike` from `@suaveplan/types` per genesis convention |
+| `createContext(options)` factory | `packages/universal/etl/src/core/context/context.ts` | (export at file end) | Verbatim |
+| `BatchConfig` interface (`size`, `concurrency`, `continueOnError`) | `packages/universal/etl/src/types/etl.types.ts` | L98-L106 | Verbatim port; extend with `perItemTimeoutMs` and `batchTimeoutMs` (genesis additions, see "no source line" below) |
+| `EtlHookMap` typed event bus | `packages/universal/etl/src/types/etl-hooks.ts` | L1-L29 | Verbatim port; expand from one upstream key (`etl:transforming`) to the eight lifecycle keys upstream emits via `PipelineEventPayload` |
+| `EtlTransformHookData` / `EtlTransformingHandler` | `packages/universal/etl/src/types/etl-hooks.ts` | L9-L29 | Verbatim |
+| `applyMappings(record, mappings)` utility | `packages/universal/etl/src/utils/utils.ts` | L48-L80 | Verbatim |
+| `getNestedValue(obj, path)` utility | `packages/universal/etl/src/utils/utils.ts` | L7-L23 | Verbatim |
+| `setNestedValue(obj, path, value)` utility | `packages/universal/etl/src/utils/utils.ts` | L29-L47 | Verbatim |
+| `parallelMap(items, fn, concurrency)` utility | `packages/universal/etl/src/utils/utils.ts` | L82-L120 | Verbatim |
+| `toBatches(items, size)` utility | `packages/universal/etl/src/utils/utils.ts` | L122-L140 | Verbatim |
+| `withTimeout(promise, ms)` utility | `packages/universal/etl/src/utils/utils.ts` | L142-L151 | Verbatim |
+| `FieldMapping` type | `packages/universal/etl/src/types/etl.types.ts` | L64-L72 | Verbatim |
+| `pipeline:started` / `pipeline:completed` / `stage:*` / `batch:*` lifecycle events | `packages/universal/etl/src/types/etl.types.ts` | L106-L130 | Renamed from upstream `pipeline:start` / `step:start` etc. to lifecycle-named (started/completed/failed) per genesis convention; payload shapes preserved |
+
+### Requirements with no source line
+
+| Requirement | Justification |
+|---|---|
+| `BatchConfig.perItemTimeoutMs` and `batchTimeoutMs` fields | Upstream models pipeline-wide `timeoutMs` only. Genesis enforces granular timeouts because `@suaveplan/retry::withTimeout` is already in the dep graph and per-item enforcement is a common production requirement (slow rows shouldn't stall the whole batch). |
+| Hook keys renamed to lifecycle-named pattern (`*:started` / `*:completed` / `*:failed`) | Genesis-wide naming convention applied across `@suaveplan/migration`, `@suaveplan/network`, `@suaveplan/cache` (see lifecycle keys in those packages). Upstream's `*:start` / `*:end` / `*:error` style was itself inconsistent across packages. |
+| `EtlHookMap` built on `HookMap` from `@suaveplan/events::hooks.types` rather than upstream's bespoke listener registry | Genesis-wide event bus convention (CLAUDE.md `@suaveplan/events` listing). |
+| `ETL_CODES` registered idempotently via `registerEtlErrorCodes()` | Genesis-wide error registry rule (CLAUDE.md §Workflow rule 2). |
+
+## Verification approach
+
+Each spec delta requirement carries a `**Verified by:**` line. The test files that will exist after this change ships:
+
+- `packages/platform/etl/src/core/context/context.test.ts` — set / get / delete metadata, hook register / off, errors / stepResults appended, AbortSignal propagation
+- `packages/platform/etl/src/core/context/context.spec-conformance.test.ts` — pins type-level `PipelineContextOptions` shape; injected `Clock` deterministic
+- `packages/platform/etl/src/core/pipeline/pipeline.test.ts` (extended) — `BatchConfig` honoured: `size` chunks input, `concurrency` runs N batches in parallel, `perItemTimeoutMs` rejects slow items with `EtlError(BATCH_ITEM_TIMEOUT)`, `batchTimeoutMs` rejects slow batches, `continueOnError` skips failed batch
+- `packages/platform/etl/src/types/etl-hooks/etl-hooks.test.ts` — every key in `EtlHookMap` fires once per lifecycle transition with the correct payload
+- `packages/platform/etl/src/utils/utils.test.ts` — `applyMappings` / `getNestedValue` / `setNestedValue` / `parallelMap` / `toBatches` / `withTimeout` happy-path + edge cases (deep path missing, concurrency 0, batch size 0, timeout 0)
+- `packages/platform/etl/src/utils/utils.spec-conformance.test.ts` — `parallelMap` honours concurrency limit empirically; `withTimeout` rejects with `EtlError(TIMEOUT)`
+- `packages/platform/etl/src/types/etl-hooks/etl-hooks.spec-conformance.test.ts` — pins lifecycle key strings via `expectTypeOf`; pins payload shapes
+- `packages/platform/etl/src/errors/etl-error.spec-conformance.test.ts` — `ETL_CODES` registered after import; `registerEtlErrorCodes` idempotent
+
+## Impact
+
+### Affected dependents
+
+Run `bunx turbo ls --filter=...@suaveplan/etl` to enumerate. Today `grep -rl "@suaveplan/etl" packages/ --include="package.json"` returns zero matches outside `packages/platform/etl/package.json` itself, so this change is purely additive.
+
+- `@suaveplan/etl` (self) — [x] tests run / [x] docs reviewed / [ ] no action needed
+- (no in-repo dependents) — verified by `bunx turbo ls --filter=...@suaveplan/etl` returning a single package.
+
+If `tests run` is unchecked for any dependent, the change MUST NOT be folded into the canonical spec — finish that work first or split the proposal.
+
+## Migration impact
+
+- **`Pipeline` consumers** — no break. Existing constructors continue to work; `PipelineContext`, `BatchConfig`, and the hook bus are all opt-in additive.
+- **`mapTransform` consumers** — no break. New `mappings?: readonly FieldMapping[]` argument is optional.
+- **`PipelineOptions` consumers** — no break. New `batch`, `hooks`, and `context` fields are optional and default to no-op behavior.
+- **`SKIP` consumers** — no break. The `SKIP` symbol export is preserved.

openspec/changes/etl-completion/specs/etl/spec.md

@@ -0,0 +1,167 @@
+<!-- markdownlint-disable MD024 -->
+
+# Capability Spec Delta: etl (etl-completion)
+
+**Package**: `@suaveplan/etl`
+**Status**: PROPOSED
+
+> Every requirement below carries a `**Verified by:**` line at authoring time. No exceptions. (See `openspec/AGENTS.md` Section 13.)
+
+## ADDED Requirements
+
+### Requirement: PipelineContext shared mutable execution state
+
+The package SHALL expose a `PipelineContext<TData>` class and a `createContext<TData>(options)` factory. The context SHALL hold mutable `data: TData`, accumulate `errors: EtlError[]` and `stepResults: StepResult[]` in execution order, expose a typed metadata key/value store (`set`, `get`, `has`, `delete`), accept an injectable `Clock`, `LoggerLike`, and `AbortSignal`, and SHALL be threaded through every stage of `Pipeline.run()` so cross-step state is observable. The context SHALL expose `on(event, listener)` / `off(event, listener)` / `emit(event, payload)` for the same lifecycle events as `EtlHookMap`.
+
+#### Scenario: metadata round-trip
+
+- **WHEN** `ctx.set("ingestKey", 42)` is called and then `ctx.get<number>("ingestKey")` is called
+- **THEN** the result is `42` AND `ctx.has("ingestKey")` returns `true`
+
+#### Scenario: errors and stepResults accumulate per stage
+
+- **WHEN** a pipeline runs three stages and the second one throws
+- **THEN** `ctx.stepResults` contains one entry for stage 1, `ctx.errors` contains one entry whose `cause` is the original throw, AND no entry exists for stage 3 unless `batch.continueOnError` is true
+
+#### Scenario: AbortSignal threads to stages
+
+- **WHEN** the supplied `abortSignal` is aborted mid-run
+- **THEN** the next stage receiving the context SHALL observe `ctx.abortSignal.aborted === true` and abortable stages SHALL bail out
+
+#### Scenario: injected Clock stamps metadata
+
+- **WHEN** the context is constructed with `clock: { now: () => 1000 }`
+- **THEN** `ctx.get<number>("startedAt")` returns `1000` (the runner stamps `startedAt` from the injected clock at run start)
+
+**Verified by:** `packages/platform/etl/src/core/context/context.test.ts::"round-trips metadata via set / get / has"` AND `packages/platform/etl/src/core/context/context.test.ts::"accumulates errors and stepResults across stages"` AND `packages/platform/etl/src/core/context/context.test.ts::"propagates AbortSignal to downstream stages"` AND `packages/platform/etl/src/core/context/context.spec-conformance.test.ts::"stamps startedAt from injected Clock"`
+
+---
+
+### Requirement: BatchConfig with concurrency, per-item timeout, batch timeout
+
+The package SHALL accept a `BatchConfig` interface (`size`, `concurrency`, `perItemTimeoutMs`, `batchTimeoutMs`, `continueOnError`). `Pipeline.run()` SHALL honour all five fields. `concurrency < 1` SHALL throw `EtlError(BATCH_CONCURRENCY_INVALID)`. `size < 1` SHALL throw the same code. Per-item timeout exceedance SHALL throw `EtlError(BATCH_ITEM_TIMEOUT)`. Batch timeout exceedance SHALL throw `EtlError(BATCH_TIMEOUT)`. `continueOnError: true` SHALL allow the pipeline to skip a failed batch and continue with the next.
+
+#### Scenario: size chunks input correctly
+
+- **WHEN** input has 100 items and `batch.size` is 25
+- **THEN** the runner processes exactly 4 batches of 25
+
+#### Scenario: concurrency runs N batches in parallel
+
+- **WHEN** `batch.concurrency` is 3 and 6 batches are submitted
+- **THEN** at most 3 batches are in-flight simultaneously (verified via shared counter assertion)
+
+#### Scenario: per-item timeout rejects slow items
+
+- **WHEN** `batch.perItemTimeoutMs` is 50 and a stage's item handler takes 200ms
+- **THEN** the runner throws `EtlError` with code `BATCH_ITEM_TIMEOUT` and `context.itemIndex` set
+
+#### Scenario: batch timeout rejects slow batches
+
+- **WHEN** `batch.batchTimeoutMs` is 100 and a whole batch takes 500ms
+- **THEN** the runner throws `EtlError` with code `BATCH_TIMEOUT` and `context.batchIndex` set
+
+#### Scenario: continueOnError skips failed batch
+
+- **WHEN** batch 2 of 5 throws and `batch.continueOnError` is `true`
+- **THEN** the runner records the error in `ctx.errors`, emits `batch:failed`, and continues with batches 3, 4, 5
+
+**Verified by:** `packages/platform/etl/src/core/pipeline/pipeline.test.ts::"size chunks input correctly"` AND `packages/platform/etl/src/core/pipeline/pipeline.test.ts::"concurrency runs N batches in parallel"` AND `packages/platform/etl/src/core/pipeline/pipeline.test.ts::"perItemTimeoutMs rejects slow items with BATCH_ITEM_TIMEOUT"` AND `packages/platform/etl/src/core/pipeline/pipeline.test.ts::"batchTimeoutMs rejects slow batches with BATCH_TIMEOUT"` AND `packages/platform/etl/src/core/pipeline/pipeline.test.ts::"continueOnError skips a failed batch and continues"`
+
+---
+
+### Requirement: EtlHookMap typed lifecycle event bus
+
+The package SHALL expose `EtlHookMap` extending `HookMap` from `@suaveplan/events::hooks.types` with the keys `"pipeline:started"`, `"pipeline:completed"`, `"stage:started"`, `"stage:completed"`, `"stage:failed"`, `"batch:started"`, `"batch:completed"`, `"batch:failed"`. `Pipeline` SHALL accept an optional `hooks?: HookBus<EtlHookMap>` and emit each event in the corresponding lifecycle position. Subscriber throws SHALL NOT abort the pipeline.
+
+#### Scenario: pipeline:started fires before first stage
+
+- **WHEN** the pipeline begins a `run()`
+- **THEN** subscribers to `"pipeline:started"` receive `{ pipelineName, startedAt }` BEFORE the first stage's `run` is called
+
+#### Scenario: stage:started / stage:completed bracket each stage
+
+- **WHEN** a stage runs successfully
+- **THEN** `"stage:started"` fires before its handler and `"stage:completed"` fires after, with `{ stageName, durationMs }`
+
+#### Scenario: stage:failed fires on stage error
+
+- **WHEN** a stage's handler throws
+- **THEN** `"stage:failed"` fires with `{ stageName, error }` (where `error` is an `EtlError` wrapping the cause)
+
+#### Scenario: batch:* fires only when batching is enabled
+
+- **WHEN** `PipelineOptions.batch` is supplied
+- **THEN** `"batch:started"`, `"batch:completed"`, and `"batch:failed"` fire per batch transition; when `batch` is omitted, none of the batch-* events fire
+
+#### Scenario: subscriber throw does not abort pipeline
+
+- **WHEN** a subscriber to `"pipeline:started"` throws synchronously
+- **THEN** the pipeline continues to `"stage:started"` and the rest of the run completes
+
+**Verified by:** `packages/platform/etl/src/types/etl-hooks/etl-hooks.test.ts::"pipeline:started fires before first stage"` AND `packages/platform/etl/src/types/etl-hooks/etl-hooks.test.ts::"stage:started and stage:completed bracket each stage"` AND `packages/platform/etl/src/types/etl-hooks/etl-hooks.test.ts::"stage:failed fires on stage error with EtlError"` AND `packages/platform/etl/src/types/etl-hooks/etl-hooks.test.ts::"batch:* fires only when batching is enabled"` AND `packages/platform/etl/src/types/etl-hooks/etl-hooks.spec-conformance.test.ts::"subscriber throw does not abort pipeline"`
+
+---
+
+### Requirement: utils — applyMappings, getNestedValue, setNestedValue, parallelMap, toBatches, withTimeout
+
+The package SHALL export six pure utilities under `utils/utils`. Each SHALL have the signatures and behaviors below.
+
+- `applyMappings<TIn, TOut>(record: TIn, mappings: readonly FieldMapping[]): TOut` — produces a new object by reading each mapping's `from` path from `record`, optionally running `transform`, and writing the result at the `to` path.
+- `getNestedValue(obj: unknown, path: string): unknown` — splits `path` on `.` and walks `obj`; returns `undefined` for any missing or non-object segment.
+- `setNestedValue(obj: Record<string, unknown>, path: string, value: unknown): void` — splits `path` on `.`, creating intermediate objects as needed, and assigns `value` at the leaf.
+- `parallelMap<T, U>(items: readonly T[], fn: (item: T, index: number) => Promise<U>, concurrency: number): Promise<U[]>` — runs `fn` over `items` with at most `concurrency` in-flight; preserves index order in the result.
+- `toBatches<T>(items: readonly T[], size: number): T[][]` — partitions `items` into chunks of `size`; final chunk may be shorter.
+- `withTimeout<T>(promise: Promise<T>, ms: number): Promise<T>` — rejects with `EtlError(TIMEOUT)` if `promise` does not settle within `ms`.
+
+#### Scenario: applyMappings copies and transforms
+
+- **WHEN** `applyMappings({ user: { name: "ada" } }, [{ from: "user.name", to: "displayName", transform: (v) => String(v).toUpperCase() }])` is called
+- **THEN** the result is `{ displayName: "ADA" }`
+
+#### Scenario: getNestedValue returns undefined for missing path
+
+- **WHEN** `getNestedValue({ a: { b: 1 } }, "a.c.d")` is called
+- **THEN** the result is `undefined` (no throw)
+
+#### Scenario: parallelMap honours concurrency limit
+
+- **WHEN** `parallelMap([1,2,3,4,5,6,7,8,9,10], slowAdd, 3)` is called
+- **THEN** at most 3 calls to `slowAdd` are in-flight at any moment AND the result preserves input order
+
+#### Scenario: parallelMap rejects on concurrency < 1
+
+- **WHEN** `parallelMap([1,2,3], fn, 0)` is called
+- **THEN** the call rejects with `EtlError(BATCH_CONCURRENCY_INVALID)`
+
+#### Scenario: toBatches partitions exactly
+
+- **WHEN** `toBatches([1,2,3,4,5,6,7], 3)` is called
+- **THEN** the result is `[[1,2,3],[4,5,6],[7]]`
+
+#### Scenario: withTimeout rejects with TIMEOUT code
+
+- **WHEN** `withTimeout(new Promise(() => {}), 50)` is called
+- **THEN** the call rejects with `EtlError` whose `code` is `TIMEOUT`
+
+**Verified by:** `packages/platform/etl/src/utils/utils.test.ts::"applyMappings copies and transforms field"` AND `packages/platform/etl/src/utils/utils.test.ts::"getNestedValue returns undefined for missing path"` AND `packages/platform/etl/src/utils/utils.test.ts::"toBatches partitions input with remainder"` AND `packages/platform/etl/src/utils/utils.spec-conformance.test.ts::"parallelMap honours concurrency limit empirically"` AND `packages/platform/etl/src/utils/utils.spec-conformance.test.ts::"parallelMap rejects on concurrency < 1 with BATCH_CONCURRENCY_INVALID"` AND `packages/platform/etl/src/utils/utils.spec-conformance.test.ts::"withTimeout rejects with EtlError(TIMEOUT)"`
+
+---
+
+### Requirement: FieldMapping type and mapTransform mapping integration
+
+The package SHALL export a `FieldMapping` interface — `{ from: string; to: string; transform?: (value: unknown) => unknown }`. `TransformConfig` SHALL accept `mappings?: readonly FieldMapping[]`. `mapTransform` SHALL apply the mappings (via `applyMappings`) BEFORE invoking the transform function so the function sees the post-mapping shape.
+
+#### Scenario: FieldMapping type shape
+
+- **WHEN** consumer code declares `const m: FieldMapping = { from: "a.b", to: "c", transform: (v) => v }`
+- **THEN** the type-check passes; `from` and `to` are string-only; `transform` is the only optional field
+
+#### Scenario: mapTransform applies mappings before fn
+
+- **WHEN** `mapTransform({ mappings: [{ from: "user.name", to: "name" }], fn: (r: { name: string }) => r.name.toUpperCase() })` is run over `{ user: { name: "ada" } }`
+- **THEN** the output is `"ADA"` (the transform function received `{ name: "ada" }`, not the original `{ user: { name: "ada" } }`)
+
+**Verified by:** `packages/platform/etl/src/types/etl-hooks/etl-hooks.spec-conformance.test.ts::"FieldMapping shape pinned via expectTypeOf"` AND `packages/platform/etl/src/stages/transform/transform.test.ts::"mapTransform applies mappings before fn"`
+
+<!-- markdownlint-enable MD024 -->

openspec/changes/etl-completion/tasks.md

@@ -0,0 +1,125 @@
+# Tasks — ETL completion
+
+> Every checkbox MUST reflect work actually done in this change. Do not retroactively tick boxes for work that was already in main when this change opened.
+
+## 0. Pre-flight (non-negotiable)
+
+- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
+- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
+- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
+
+## 1. Spec authoring
+
+- [ ] 1.1 Read `../suaveplan/enterprise/etl.md` end-to-end
+- [ ] 1.2 Read `openspec/specs/etl/spec.md` (does not exist yet — confirm and start fresh)
+- [ ] 1.3 Read `openspec/AGENTS.md` Sections 13 and 14
+- [ ] 1.4 Draft `proposal.md` from `openspec/templates/proposal.md`
+- [ ] 1.5 Fill in `## Source mapping` table — every requirement cites a line range OR appears in the "no source line" table with justification
+- [ ] 1.6 Draft `specs/etl/spec.md` from `openspec/templates/spec-delta.md`
+- [ ] 1.7 Every requirement in the spec delta carries a `**Verified by:**` line naming concrete `it(...)` titles
+
+## 2. Implementation — `core/context/context`
+
+- [ ] 2.1 Port `PipelineContext` class from upstream `src/core/context/context.ts` into `packages/platform/etl/src/core/context/context.ts`
+- [ ] 2.2 Port `createContext(options)` factory; options: `{ pipelineName, initialData?, clock?, logger?, abortSignal? }`
+- [ ] 2.3 Replace upstream `Date.now()` with the injectable `Clock` from `@suaveplan/types`; default to `Clock.system()` from existing genesis setup
+- [ ] 2.4 Wire context through every step — `Pipeline.run()` instantiates a `PipelineContext` per run and passes it to each stage's `run(ctx, ...)` method
+- [ ] 2.5 Add `context.md` sibling doc (≥200 words, basic + advanced examples)
+
+## 3. Implementation — `BatchConfig`
+
+- [ ] 3.1 Add `BatchConfig` interface (`size`, `concurrency`, `perItemTimeoutMs`, `batchTimeoutMs`, `continueOnError`) to `types/etl.types.ts`
+- [ ] 3.2 Extend `PipelineOptions` with optional `batch?: BatchConfig`
+- [ ] 3.3 Update `Pipeline.run` to honour `batch.concurrency` via `parallelMap` (from §5)
+- [ ] 3.4 Update `Pipeline.run` to honour `batch.perItemTimeoutMs` via `withTimeout` (from §5)
+- [ ] 3.5 Update `Pipeline.run` to honour `batch.batchTimeoutMs` via `withTimeout` around the whole-batch promise
+- [ ] 3.6 Update `Pipeline.run` to honour `batch.continueOnError` (skip failed batch, record error in context)
+- [ ] 3.7 Add `BATCH_ITEM_TIMEOUT`, `BATCH_TIMEOUT`, `BATCH_CONCURRENCY_INVALID` codes to `errors/etl-error.ts`
+- [ ] 3.8 Update `pipeline.md` sibling doc to cover batch config
+
+## 4. Implementation — `types/etl-hooks`
+
+- [ ] 4.1 Create `packages/platform/etl/src/types/etl-hooks/etl-hooks.ts` exporting `EtlHookMap` extending `HookMap` from `@suaveplan/events/hooks.types` with all eight lifecycle keys
+- [ ] 4.2 Define payloads: `EtlPipelineHookData`, `EtlStageHookData`, `EtlBatchHookData`, `EtlTransformHookData`
+- [ ] 4.3 Export named handler aliases (`EtlPipelineStartedHandler`, `EtlStageFailedHandler`, etc.) for consumer ergonomics
+- [ ] 4.4 Update `Pipeline` to accept `hooks?: HookBus<EtlHookMap>` in options; default to a fresh internal bus
+- [ ] 4.5 Emit each lifecycle event in the right place inside `Pipeline.run` and `RunnablePipeline.run`
+- [ ] 4.6 Add `etl-hooks.md` sibling doc (≥200 words, basic + advanced examples)
+
+## 5. Implementation — `utils/utils`
+
+- [ ] 5.1 Create `packages/platform/etl/src/utils/utils.ts` with `applyMappings`, `getNestedValue`, `setNestedValue`, `parallelMap`, `toBatches`, `withTimeout`
+- [ ] 5.2 `withTimeout` rejects with `EtlError(TIMEOUT)` (not a plain `Error`); pass through the original promise's rejection reason in `cause`
+- [ ] 5.3 `parallelMap` rejects with `EtlError(BATCH_CONCURRENCY_INVALID)` when `concurrency < 1`
+- [ ] 5.4 `toBatches` returns `[]` for empty input; rejects on `size < 1` with `EtlError(BATCH_CONCURRENCY_INVALID)`
+- [ ] 5.5 Add `utils.md` sibling doc (≥200 words, basic + advanced examples)
+- [ ] 5.6 Re-export every symbol from `index.ts`
+
+## 6. Implementation — `FieldMapping` + `mapTransform` integration
+
+- [ ] 6.1 Add `FieldMapping` interface to `types/etl.types.ts`
+- [ ] 6.2 Update `TransformConfig` to accept optional `mappings?: readonly FieldMapping[]`
+- [ ] 6.3 Update `mapTransform` to apply `mappings` (via `applyMappings`) BEFORE invoking the transform function
+- [ ] 6.4 Re-export `FieldMapping` from `index.ts`
+- [ ] 6.5 Update `transform.md` sibling doc
+
+## 7. Tests
+
+- [ ] 7.1 `core/context/context.test.ts` — set / get / has / delete; hook register / off; errors / stepResults appended; AbortSignal propagates to abortable steps
+- [ ] 7.2 `core/context/context.spec-conformance.test.ts` — `expectTypeOf<PipelineContextOptions>` pinned; injected `Clock` deterministic in stamping `metadata.startedAt`
+- [ ] 7.3 `core/pipeline/pipeline.test.ts` (extended) — `batch.size` chunks input correctly; `batch.concurrency` runs N in parallel (timing assertion); `batch.perItemTimeoutMs` rejects slow items with `BATCH_ITEM_TIMEOUT`; `batch.batchTimeoutMs` rejects slow batches; `batch.continueOnError: true` skips failure
+- [ ] 7.4 `types/etl-hooks/etl-hooks.test.ts` — all 8 lifecycle keys fire once per transition with correct payloads
+- [ ] 7.5 `types/etl-hooks/etl-hooks.spec-conformance.test.ts` — `expectTypeOf` pins each key string; pins payload shapes
+- [ ] 7.6 `utils/utils.test.ts` — `applyMappings` (basic, with transform, missing source), `getNestedValue` (nested, missing, null guard), `setNestedValue` (creates intermediates), `parallelMap` (concurrency 1, 2, equal-to-input), `toBatches` (empty, exact, remainder), `withTimeout` (resolves under, rejects over)
+- [ ] 7.7 `utils/utils.spec-conformance.test.ts` — `parallelMap` honours concurrency empirically (max-in-flight assertion via shared counter); `withTimeout` rejects with `EtlError(TIMEOUT)`; `parallelMap` rejects with `EtlError(BATCH_CONCURRENCY_INVALID)` for `concurrency < 1`
+- [ ] 7.8 `errors/etl-error.spec-conformance.test.ts` — `ETL_CODES` registered after import; idempotent re-registration; new codes (`BATCH_ITEM_TIMEOUT`, `BATCH_TIMEOUT`, `BATCH_CONCURRENCY_INVALID`, `TIMEOUT`) thrown in production paths
+- [ ] 7.9 Every `it(...)` in 7.1–7.8 carries `{ timeout: 5000 }` (or 30000 for genuinely async)
+
+## 8. Documentation
+
+- [ ] 8.1 Sibling `.md` for every new implementation file (`context.md`, `etl-hooks.md`, `utils.md`); each ≥200 words with basic + advanced examples
+- [ ] 8.2 Update existing sibling `.md`s touched in §3, §4, §6 (`pipeline.md`, `transform.md`)
+- [ ] 8.3 Update `packages/platform/etl/README.md` — add Modules links for the three new files; add Quick-start snippet showing context + hooks subscription
+- [ ] 8.4 Update `CLAUDE.md` package listing line for `@suaveplan/etl` to reflect the expanded surface
+
+## 9. Pre-archive verification (the 5-gate triple)
+
+- [ ] 9.1 `bun run check` — Biome 0 errors / 0 warnings / 0 info across the whole monorepo
+- [ ] 9.2 `bunx turbo lint typecheck test --filter=@suaveplan/etl` — green
+- [ ] 9.3 `bunx turbo test:coverage --filter=@suaveplan/etl` — 100/100/100/100
+- [ ] 9.4 `bun scripts/openspec-conformance.ts @suaveplan/etl` — exit 0, no `error` findings
+- [ ] 9.5 `bunx turbo build --filter=@suaveplan/etl` — green
+
+## 10. Pre-archive 7-row checklist (AGENTS.md Section 13)
+
+- [ ] 10.1 Generics threaded through `PipelineContext<TData>`, `parallelMap<T, U>`, `applyMappings<TIn, TOut>`
+- [ ] 10.2 Zod schemas wired at the `BatchConfig` and `PipelineOptions` boundaries (already partially in `etl.schemas.ts` — extend)
+- [ ] 10.3 Every error code in `errors/etl-error.ts` thrown by at least one production path
+- [ ] 10.4 Strict-mode behaviors tested both on and off (`continueOnError: true` vs `false`)
+- [ ] 10.5 OTel instruments emitted from production paths and asserted with a real in-memory `MeterProvider`
+- [ ] 10.6 Lifecycle dedup tested where claimed (idempotent `registerEtlErrorCodes`)
+- [ ] 10.7 No retroactive tick — every box above reflects work done in this change
+
+## 11. Archive
+
+- [ ] 11.1 Fold spec delta into `openspec/specs/etl/spec.md` (will be the first time this canonical spec exists)
+- [ ] 11.2 Move `openspec/changes/etl-completion/` → `openspec/archive/<yyyy-mm-dd>-etl-completion/`
+- [ ] 11.3 Refresh `MANIFEST.md` if package set or deps changed (no expected change here)
+
+## Audit Note (2026-05-02)
+
+Audit verdict: **DEFERRED — 0/65 tasks evidenced in code**.
+
+Evidence reviewed:
+
+- `packages/platform/etl/src/` contains `core/pipeline/`, `stages/{transform,source,sink}/`, `errors/`, `schemas/`, `types/`, `constants/`. No `core/context/`, no `utils/`, no `types/etl-hooks/`.
+- `errors/etl-error.ts` does not declare `BATCH_ITEM_TIMEOUT`, `BATCH_TIMEOUT`, `BATCH_CONCURRENCY_INVALID`, or `TIMEOUT` codes.
+- No `BatchConfig` shape in `types/etl.types.ts` (no batch concurrency / per-item timeout wired into `Pipeline.run`).
+
+Missing pieces:
+
+1. `core/context/context.ts` `PipelineContext` with injectable `Clock` + `AbortSignal`.
+2. `BatchConfig` (size / concurrency / perItemTimeoutMs / batchTimeoutMs / continueOnError) wired into `Pipeline.run` via the new `utils/utils.ts` (`parallelMap`, `withTimeout`, `toBatches`).
+3. `types/etl-hooks/etl-hooks.ts` `EtlHookMap` + payloads + emission inside `Pipeline.run`, plus `FieldMapping` integration into `mapTransform`.
+
+Decision: leave change open. The three new modules are interlocking (context threads through hooks, batch config relies on the new utils) and need fresh tests + sibling docs.