suaveplan/packages/foundation/etl/etl/README.md

@suaveplan/etl - Extract-Transform-Load pipeline engine with typed stages, error handling, observable progress, and OTel instrumentation

Summary

Extract-Transform-Load pipeline engine with typed stages, error handling, observable progress, and OTel instrumentation.

Documentation tree

• pipeline — (no submodules)
  • pipeline — (no submodules)
• errors — (no submodules)
  • etl-error — (no submodules)
• schemas — (no submodules)
  • etl.schemas — (no submodules)
• sink — (no submodules)
  • sink — (no submodules)
• source — (no submodules)
  • source — (no submodules)
• transform — (no submodules)
  • transform — (no submodules)

- [pipeline](./src/core/pipeline/pipeline.md) — _(no submodules)_- [errors](./src/errors/etl-error.md) — _(no submodules)_- [etl-error](./src/errors/etl-error.md) — _(no submodules)_- [schemas](./src/schemas/etl.schemas.md) — _(no submodules)_- [etl.schemas](./src/schemas/etl.schemas.md) — _(no submodules)_- [sink](./src/stages/sink/sink.md) — _(no submodules)_- [sink](./src/stages/sink/sink.md) — _(no submodules)_- [source](./src/stages/source/source.md) — _(no submodules)_- [source](./src/stages/source/source.md) — _(no submodules)_- [transform](./src/stages/transform/transform.md) — _(no submodules)_- [transform](./src/stages/transform/transform.md) — _(no submodules)_- [pipeline](./src/core/pipeline/pipeline.md) — _(no submodules)_- [errors](./src/errors/etl-error.md) — _(no submodules)_- [etl-error](./src/errors/etl-error.md) — _(no submodules)_- [schemas](./src/schemas/etl.schemas.md) — _(no submodules)_- [etl.schemas](./src/schemas/etl.schemas.md) — _(no submodules)_- [sink](./src/stages/sink/sink.md) — _(no submodules)_- [sink](./src/stages/sink/sink.md) — _(no submodules)_- [source](./src/stages/source/source.md) — _(no submodules)_- [source](./src/stages/source/source.md) — _(no submodules)_- [transform](./src/stages/transform/transform.md) — _(no submodules)_- [transform](./src/stages/transform/transform.md) — _(no submodules)_- [pipeline](./src/core/pipeline/pipeline.md) — _(no submodules)_- [errors](./src/errors/etl-error.md) — _(no submodules)_- [etl-error](./src/errors/etl-error.md) — _(no submodules)_- [schemas](./src/schemas/etl.schemas.md) — _(no submodules)_- [etl.schemas](./src/schemas/etl.schemas.md) — _(no submodules)_- [sink](./src/stages/sink/sink.md) — _(no submodules)_- [sink](./src/stages/sink/sink.md) — _(no submodules)_- [source](./src/stages/source/source.md) — _(no submodules)_- [source](./src/stages/source/source.md) — _(no submodules)_- [transform](./src/stages/transform/transform.md) — _(no submodules)_- [transform](./src/stages/transform/transform.md) — _(no submodules)_- [pipeline](./src/core/pipeline/pipeline.md) — _(no submodules)_- [errors](./src/errors/etl-error.md) — _(no submodules)_ - [etl-error](./src/errors/etl-error.md) — _(no submodules)_- [schemas](./src/schemas/etl.schemas.md) — _(no submodules)_ - [etl.schemas](./src/schemas/etl.schemas.md) — _(no submodules)_- [sink](./src/stages/sink/sink.md) — _(no submodules)_ - [sink](./src/stages/sink/sink.md) — _(no submodules)_- [source](./src/stages/source/source.md) — _(no submodules)_ - [source](./src/stages/source/source.md) — _(no submodules)_- [transform](./src/stages/transform/transform.md) — _(no submodules)_ - [transform](./src/stages/transform/transform.md) — _(no submodules)_

Modules

• Pipeline — pipeline orchestrator with fluent API
• Source — array and generator source factories
• Transform — map, filter, validate, flat-map transforms
• Sink — array, null, callback, and batch sinks
• EtlError — structured errors with stable codes
• Schemas — Zod schemas for pipeline types
• Types — shared TypeScript interfaces

Quick Start

import { createPipeline, arraySource, mapTransform, ArraySink } from "@suaveplan/etl";

const sink = new ArraySink<number>();
const result = await createPipeline({ source: arraySource([1, 2, 3]) })
  .pipe(mapTransform((n) => n * 2))
  .to(sink)
  .run();

console.log(result.stats.loaded); // 3
console.log(sink.records); // [2, 4, 6]