Move docs/ package mirror into document-navigator plugin

Relocate the ~6k-file docs tree from the repo root into
plugins/document-navigator/docs/ so the plugin is self-contained and
portable. docs/superpowers/ (specs and plans) stays at the repo root.

Moved via cp+rm rather than git mv because directory rename() fails with
EPERM on this WSL /mnt DrvFs mount; git rename-detection pairs the
identical-content files at commit time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-30 01:10:00 +01:00
parent fe3fedf42e
commit 9bf56a806b
6022 changed files with 12437 additions and 0 deletions

View File

@@ -0,0 +1,141 @@
# @suaveplan/api-plugin-import - Tabular-data ingest plugin for @suaveplan/api. CSV / JSON / TSV-XLSX parsers, declarative transform pipeline (coerce + validate + computed), real-time progress tracker, async ImportStoreAdapter (in-memory + SurrealDB backends ship), canonical EventEmitter-based ImportService with owner-scoped jobs, 5 auth-gated OpenAPI Hono routes, RFC 7807 problem+json renderer, 9 IMPORT.* error codes, 6 OTel instruments. Server-tier.
Tabular-data ingest plugin for `@suaveplan/api`. Parses CSV / JSON /
TSV-XLSX payloads, applies declarative column mappings + per-field
validation + computed fields, persists owner-scoped import jobs through
a pluggable async store, emits real-time progress via an event bus, and
exposes five auth-gated OpenAPI routes with full Zod request/response
schemas. Ships nine `IMPORT.*` error codes (extending
`SuaveplanError`), four OpenTelemetry instruments (`import.jobs_total`,
`import.rows_total`, `import.row_errors_total`, `import.duration_ms`),
and a hand-rolled RFC 7807 problem+json renderer for the route layer.
<!-- begin:auto-docs -->
## Documentation tree
- [error-codes](./src/error-codes/error-codes.md) — Declares the nine stable `IMPORT.*` error codes used throughout `@suaveplan/api-plugin-import` and exports the `ImportE…
- [error-codes](./src/error-codes/error-codes.md) — Declares the nine stable `IMPORT.*` error codes used throughout `@suaveplan/api-plugin-import` and exports the `ImportE…
- [otel](./src/otel/instruments.md) — Builds the OpenTelemetry instrument bundle emitted by the canonical `ImportService` so consumers can wire production ob…
- [instruments](./src/otel/instruments.md) — Builds the OpenTelemetry instrument bundle emitted by the canonical `ImportService` so consumers can wire production ob…
- [csv](./src/parsers/csv.md) — Provides the CSV-family parsers consumed by the canonical `ImportService.preview` / `startImport` pipeline and by any c…
- [json](./src/parsers/json.md) — Parses JSON-array input documents into the same `Record<string, unknown>[]` row shape produced by the CSV / XLSX parser…
- [xlsx](./src/parsers/xlsx.md) — Provides a lightweight tab-separated-value parser exposed under the `xlsx` format so consumers can ingest spreadsheet-s…
- [plugin](./src/plugin/plugin.md) — Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the `@suaveplan/api-plugin` lifecycle. Valida…
- [plugin](./src/plugin/plugin.md) — Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the `@suaveplan/api-plugin` lifecycle. Valida…
- [progress](./src/progress/tracker.md) — Provides the `ImportProgressTracker` class consumed by the canonical `ImportService` (and exported for direct use by CL…
- [tracker](./src/progress/tracker.md) — Provides the `ImportProgressTracker` class consumed by the canonical `ImportService` (and exported for direct use by CL…
- [progress-sse](./src/routes/progress-sse/progress-sse.md) — Exposes `GET /imports/:id/progress` returning a long-lived `text/event-stream` so clients can subscribe to a job's real…
- [progress-sse](./src/routes/progress-sse/progress-sse.md) — Exposes `GET /imports/:id/progress` returning a long-lived `text/event-stream` so clients can subscribe to a job's real…
- [build-routes](./src/routes/build-routes.md) — Constructs the `OpenAPIHono` router mounted by `createImportPlugin` at the configured `routePrefix`. Five routes are re…
- [problem-json](./src/routes/problem-json.md) — Bridges thrown errors from the route handlers into RFC 7807 `application/problem+json` responses. Reads the HTTP status…
- [schemas](./src/schemas/schemas.md) — Carries the full public Zod surface used by the route handlers and any external client (CLI tool, integration test, bro…
- [options](./src/schemas/options.md) — Declares the Zod schema for the options accepted by `createImportPlugin`. Parsing options at the factory boundary guara…
- [schemas](./src/schemas/schemas.md) — Carries the full public Zod surface used by the route handlers and any external client (CLI tool, integration test, bro…
- [service](./src/service/service.md) — Implements the canonical `ImportService` used by `createImportPlugin` and exported for direct programmatic use. Owns th…
- [service](./src/service/service.md) — Implements the canonical `ImportService` used by `createImportPlugin` and exported for direct programmatic use. Owns th…
- [services](./src/services/registry-augmentation.md) — Augments the `ServiceRegistry` interface from `@suaveplan/api-plugin/service-registry` with the `importService` slot so…
- [registry-augmentation](./src/services/registry-augmentation.md) — Augments the `ServiceRegistry` interface from `@suaveplan/api-plugin/service-registry` with the `importService` slot so…
- [store](./src/store/memory.md) — Declares the async `ImportStoreAdapter` interface and ships the `MemoryImportStore` implementation used by the canonica…
- [surreal](./src/store/surreal/surreal-import-store.md) — Implements the canonical `ImportStoreAdapter` interface against a connected `SurrealAdapter` from `@suaveplan/surrealdb…
- [surreal-import-store](./src/store/surreal/surreal-import-store.md) — Implements the canonical `ImportStoreAdapter` interface against a connected `SurrealAdapter` from `@suaveplan/surrealdb…
- [memory](./src/store/memory.md) — Declares the async `ImportStoreAdapter` interface and ships the `MemoryImportStore` implementation used by the canonica…
- [transform](./src/transform/pipeline.md) — Implements a three-stage row-transform pipeline that any consumer can drive without depending on the canonical `ImportS…
- [pipeline](./src/transform/pipeline.md) — Implements a three-stage row-transform pipeline that any consumer can drive without depending on the canonical `ImportS…
- [types](./src/types/types.md) — Defines the cross-module types and constants consumed by every other feature module in `@suaveplan/api-plugin-import`.…
- [types](./src/types/types.md) — Defines the cross-module types and constants consumed by every other feature module in `@suaveplan/api-plugin-import`.…
<!-- end:auto-docs -->
## Install
```bash
bun add @suaveplan/api-plugin-import
```
## Quick start
```ts
import { createImportPlugin } from "@suaveplan/api-plugin-import";
// Register with @suaveplan/api's plugin registry.
const importPlugin = createImportPlugin({ routePrefix: "/imports" });
// Or use the service directly:
import { ImportService } from "@suaveplan/api-plugin-import";
const service = new ImportService();
const progress = await service.startImport(
{
format: "csv",
data: "name\nAda\nBob",
mappings: [{ sourceColumn: "name", targetField: "displayName" }],
},
{ ownerId: "alice" }
);
console.log(progress.successCount); // 2
```
## Modules
- [error-codes](./src/error-codes/error-codes.md) — `IMPORT_CODES` /
`ImportError` registry, 9 codes.
- [types](./src/types/types.md) — cross-module shared shapes.
- [parsers/csv](./src/parsers/csv.md) — CSV + streaming CSV parsers.
- [parsers/json](./src/parsers/json.md) — JSON parser.
- [parsers/xlsx](./src/parsers/xlsx.md) — TSV-style XLSX parser.
- [transform/pipeline](./src/transform/pipeline.md) — coerce / validate
/ computed-field pipeline.
- [progress/tracker](./src/progress/tracker.md) — `ImportProgressTracker`.
- [store/memory](./src/store/memory.md) — `ImportStoreAdapter` +
`MemoryImportStore`.
- [store/surreal/surreal-import-store](./src/store/surreal/surreal-import-store.md)
— durable `SurrealImportStore` (SurrealDB-backed). Migration shipped at
[`surql/tables/import_job.surql`](./surql/tables/import_job.surql).
- [service/service](./src/service/service.md) — canonical `ImportService`.
- [routes/problem-json](./src/routes/problem-json.md) — RFC 7807
renderer.
- [routes/build-routes](./src/routes/build-routes.md) — 5 auth-gated
OpenAPI routes.
- [otel/instruments](./src/otel/instruments.md) — OTel bundle.
- [schemas/options](./src/schemas/options.md) — plugin options schema.
- [schemas/schemas](./src/schemas/schemas.md) — public request /
response Zod surface.
- [services/registry-augmentation](./src/services/registry-augmentation.md)
— `ServiceRegistry["importService"]` slot.
- [plugin/plugin](./src/plugin/plugin.md) — `createImportPlugin`
factory.
## Error codes
Every failure path raises an `ImportError` carrying one of nine
`IMPORT.*` codes — `NOT_FOUND`, `FORBIDDEN`, `VALIDATION`,
`FILE_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `PROCESSING_FAILED`,
`STORAGE_ERROR`, `TIMEOUT`, `INVALID_OPTIONS`. Anti-enumeration:
foreign-owner lookups return `NOT_FOUND`, never `FORBIDDEN`.
## SurrealDB store
`SurrealImportStore` is the durable alternative to `MemoryImportStore`,
implementing the same `ImportStoreAdapter` interface against a connected
`@suaveplan/surrealdb` `SurrealAdapter`. The matching schema migration
ships co-located at [`surql/tables/import_job.surql`](./surql/tables/import_job.surql);
hosts apply it through `@suaveplan/api-plugin-database`'s migration
runner or via `adapter.query(readFileSync(...))`. The store uses
positional `$p0,$p1,…` SurrealQL placeholders — the canonical pattern
used by every SurrealDB repository.
## Observability
Six OpenTelemetry instruments are emitted on the production hot paths
— call `getImportInstruments()` to reuse the cached bundle, or
`resetImportInstruments()` in tests after installing a fresh
`MeterProvider`. Four cover the service (`jobs_total`, `rows_total`,
`row_errors_total`, `duration_ms`); two cover the store
(`import.store.writes_total{outcome}` and
`import.store.reads_total{outcome}` — emitted by `SurrealImportStore`
on every adapter call so host dashboards can split backend health from
upstream import success/failure).
## License
MIT

View File

@@ -0,0 +1,87 @@
# `error-codes` — IMPORT.* registry
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Declares the nine stable `IMPORT.*` error codes used throughout
`@suaveplan/api-plugin-import` and exports the `ImportError` class that
extends `SuaveplanError` so `instanceof ImportError` narrows the `code`
property to the literal union. Every failure path in the plugin — parser
errors, mapping errors, owner-mismatch lookups, plugin-config validation
failures, store I/O errors, processing timeouts — raises an
`ImportError` carrying one of these codes, so consumers can branch on
the string value without depending on private types.
The registry is built via `defineCodes("IMPORT", […])` from
`@suaveplan/error`, which guarantees every code is namespaced
(`IMPORT.NOT_FOUND`, …) and is mapped 1:1 to a `keyof IMPORT_CODES`
identifier. Renaming a code is a breaking change.
## Features
- Nine canonical codes covering anti-enumeration lookups (NOT_FOUND),
authz (FORBIDDEN), domain validation (VALIDATION), oversized payloads
(FILE_TOO_LARGE), enum mismatches (UNSUPPORTED_FORMAT), runtime
failures (PROCESSING_FAILED), store I/O (STORAGE_ERROR), deadlines
(TIMEOUT), and factory-config gates (INVALID_OPTIONS).
- `ImportError` exposes `code` (the canonical literal) plus a redundant
`importCode` alias so instance-narrowing keeps the literal type when
the variable is widened to `Error`.
- Every code is reachable from a production path; the spec-conformance
test pins the contract that none can be deleted to satisfy a coverage
gate.
## Usage: Basic Example
```ts
import { ImportError, IMPORT_CODES } from "@suaveplan/api-plugin-import";
throw new ImportError(IMPORT_CODES.NOT_FOUND, `Import job "${id}" not found`, {
context: { jobId: id, statusCode: 404 },
});
```
## Usage: Advanced Example
```ts
import { ImportError, IMPORT_CODES } from "@suaveplan/api-plugin-import";
try {
await service.startImport(payload, { ownerId });
} catch (err) {
if (err instanceof ImportError) {
switch (err.code) {
case IMPORT_CODES.PROCESSING_FAILED:
metrics.increment("import.fail.parse");
break;
case IMPORT_CODES.FILE_TOO_LARGE:
return c.json({ error: "payload too large" }, 413);
default:
throw err;
}
}
throw err;
}
```
## API Reference
- `IMPORT_CODES: Readonly<{ NOT_FOUND: "IMPORT.NOT_FOUND"; … }>` — frozen
literal map.
- `type ImportErrorCode = (typeof IMPORT_CODES)[keyof typeof IMPORT_CODES]`
— union of the nine literal values.
- `class ImportError extends SuaveplanError<ImportErrorCode>` — concrete
error; constructor `(code, message, options?: SuaveplanErrorOptions)`.
## Implementation Notes
- Built via `defineCodes("IMPORT", […])` so every key acquires the
`"IMPORT."` prefix automatically.
- The class is `class` not `abstract class` because the plugin throws it
directly (no subclass per failure mode).
- Status-code hints travel via `options.context.statusCode` so the RFC
7807 renderer (`routes/problem-json`) can choose the right HTTP code
without re-importing the registry.

View File

@@ -0,0 +1,91 @@
# `otel/instruments` — OpenTelemetry instruments
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Builds the OpenTelemetry instrument bundle emitted by the canonical
`ImportService` so consumers can wire production observability without
hand-rolling counter names. The bundle is a single cached object keyed
against the global `MeterProvider` — the first `getImportInstruments()`
call resolves the meter; subsequent calls return the same reference.
Tests rebind by installing their own `MeterProvider`, calling
`resetImportInstruments()`, and reading back via an `InMemoryMetricExporter`.
## Features
- Nine instruments: `jobsTotal`, `rowsTotal`, `rowErrorsTotal`,
`durationMs` (histogram), `storeWritesTotal`, `storeReadsTotal`,
`sseConnected`, `sseEventsSent`, `sseDisconnected`.
- `storeWritesTotal{outcome}` and `storeReadsTotal{outcome}` are emitted
by `SurrealImportStore` on every adapter call so dashboards can split
backend health from upstream import success/failure. The in-memory
adapter (`MemoryImportStore`) does not emit; durable backends do.
- `sseConnected`, `sseEventsSent{kind}`, and `sseDisconnected{reason}`
are emitted by `GET /imports/:id/progress` so dashboards can chart
open SSE connections, frame throughput by event kind, and teardown
causes (terminal `complete` / `failed` vs client abort).
- Cached bundle — production code does not re-resolve the meter on
every emit.
- `resetImportInstruments()` drops the cache so a freshly-installed
`MeterProvider` rebinds on the next call.
- Asserted by `instruments.test.ts` against a real in-memory exporter
(no mocks).
## Usage: Basic Example
```ts
import { getImportInstruments } from "@suaveplan/api-plugin-import";
const instruments = getImportInstruments();
instruments.jobsTotal.add(1, { outcome: "success" });
instruments.rowsTotal.add(42, { outcome: "success" });
instruments.durationMs.record(123);
```
## Usage: Advanced Example
```ts
import { metrics } from "@opentelemetry/api";
import {
AggregationTemporality,
InMemoryMetricExporter,
MeterProvider,
PeriodicExportingMetricReader,
} from "@opentelemetry/sdk-metrics";
import { getImportInstruments, resetImportInstruments } from "@suaveplan/api-plugin-import";
const exporter = new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE);
const reader = new PeriodicExportingMetricReader({ exporter, exportIntervalMillis: 60_000 });
const provider = new MeterProvider({ readers: [reader] });
metrics.setGlobalMeterProvider(provider);
resetImportInstruments();
const instruments = getImportInstruments();
instruments.jobsTotal.add(1, { outcome: "success" });
await provider.forceFlush();
const collected = exporter.getMetrics();
// collected[0].scopeMetrics[0].metrics — read back the counter datapoint
```
## API Reference
- `interface ImportInstruments { jobsTotal: Counter; rowsTotal: Counter; rowErrorsTotal: Counter; durationMs: Histogram; storeWritesTotal: Counter; storeReadsTotal: Counter; sseConnected: Counter; sseEventsSent: Counter; sseDisconnected: Counter }`
- `getImportInstruments(): ImportInstruments`
- `resetImportInstruments(): void`
## Implementation Notes
- The meter is named `@suaveplan/api-plugin-import` — matching the
package name keeps cross-package metric reports readable.
- Counters carry the `outcome` label so dashboards can split jobs/rows
by success vs error without re-aggregating.
- The histogram unit is set to `"ms"` so OTel collectors that respect
unit annotation produce labelled charts; consumers MUST emit
millisecond values.
- The cache is module-scoped — every consumer of the package shares the
same instrument set. This is the desired behaviour because the meter
resolves against the global `MeterProvider`, which is a process-wide
singleton.

View File

@@ -0,0 +1,86 @@
# `parsers/csv` — CSV parsers
## Purpose
Provides the CSV-family parsers consumed by the canonical
`ImportService.preview` / `startImport` pipeline and by any caller that
needs to drive imports outside the plugin (CLI tools, batch workers).
Ships both a high-level `parseCsv` returning row objects and a
streaming variant (`streamParseCsv`) that surfaces per-row validation
errors without aborting the run — important when the caller needs to
collect every failed row in one pass rather than restart after each
exception.
The kernel (`parseCsvLine`, `parseStreamLine`) handles double-quote
escaping (`""`), custom delimiters, embedded newlines inside quoted
fields, and trims field whitespace. Empty input or header-only input
returns an empty rows array — not an exception — so route handlers can
distinguish "no data" from "parse failed."
## Features
- `parseCsv` / `parseCsvLine` — happy-path one-shot parsing with
default `,` delimiter.
- `parseStreamLine` / `detectHeaders` — primitives used by
`streamParseCsv`; exported for hosts needing custom-delimited
variants (semicolons, pipes).
- `streamParseCsv` — line-by-line streaming with optional `onRow`,
`onError`, and `validate` callbacks. Validation failures are
collected, not thrown.
- `parseCsvWithHeaders` — header-aware result shape
(`{ headers, rows, totalRows }`) matching `parseJson` and `parseXlsx`
so routes can branch on `format` once.
- `applyColumnMappings` — maps `sourceColumn` → `targetField` with
optional per-mapping transform.
- `detectColumns` — union of keys across heterogeneous rows.
## Usage: Basic Example
```ts
import { parseCsv } from "@suaveplan/api-plugin-import";
const rows = parseCsv("name,city\nAda,London\nBob,Paris");
// [{ name: "Ada", city: "London" }, { name: "Bob", city: "Paris" }]
```
## Usage: Advanced Example
```ts
import { streamParseCsv } from "@suaveplan/api-plugin-import";
const result = streamParseCsv("a,b\n1,2\n,3", {
delimiter: ",",
validate: (row, idx) => (row["a"] === "" ? `row ${idx}: a is required` : undefined),
onError: (err) => logger.warn({ row: err.row, msg: err.message }, "drop"),
});
// result.errors → [{ row: 1, line: ",3", message: "row 1: a is required" }]
// result.rows → [{ a: "1", b: "2" }]
// result.validRows === 1
```
## API Reference
- `parseCsv(data: string): Record<string, string>[]`
- `parseCsvLine(line: string): string[]`
- `detectColumns(rows: Record<string, unknown>[]): string[]`
- `parseStreamLine(line, delimiter, quoteChar): string[]`
- `detectHeaders(firstLine, config: StreamParserConfig): string[]`
- `streamParseCsv(data, config?): StreamParseResult`
- `parseCsvWithHeaders(content, options?: { delimiter?; hasHeaders? }): CsvParseResult`
- `applyColumnMappings(rows, mappings: ColumnMapping[]): Record<string, unknown>[]`
- `interface StreamParserConfig { delimiter?; quoteChar?; hasHeaders?; onRow?; onError?; validate? }`
- `interface StreamParseResult { headers; totalRows; validRows; errors; rows }`
- `interface StreamRowError { row; line; message }`
- `interface CsvParseResult { headers; rows; totalRows }`
## Implementation Notes
- Quote handling is single-state-machine (no regex backtracking) — O(n)
in input length, no exponential pathological inputs.
- `splitLines` keeps an in-quotes counter across newlines so RFC 4180
multi-line records survive the split.
- `applyColumnMappings` only invokes `transform` when the source value
is a string — JSON-sourced numbers / booleans pass through unchanged.
- The high-level `parseCsv` strips the header line by index (line 0);
if the input contains only a header row, the function returns `[]`
rather than `[{ header: undefined }]`.

View File

@@ -0,0 +1,59 @@
# `parsers/json` — JSON parsers
## Purpose
Parses JSON-array input documents into the same `Record<string, unknown>[]`
row shape produced by the CSV / XLSX parsers, so the canonical
`ImportService` can branch on `format` and call a uniform parse function.
The strict `parseJson` rejects non-array input with a deterministic error
string; the high-level `parseJsonWithHeaders` wraps `parseJson` to
produce the `{ headers, rows, totalRows }` shape consumed by the route
preview pipeline.
Scalar elements in the input array (numbers, strings, `null`) are
wrapped as `{ value: scalar }` rather than dropped, so the row count
agrees with the input array length and the resulting rows still satisfy
`Record<string, unknown>`.
## Features
- `parseJson` — strict array parsing; throws on non-array input.
- `parseJsonWithHeaders` — header-aware result shape; tolerant of
single-object inputs (wraps them in a one-element array).
- Header inference unions keys across heterogeneous rows.
## Usage: Basic Example
```ts
import { parseJson } from "@suaveplan/api-plugin-import";
const rows = parseJson('[{"a":1},{"a":2}]'); // [{ a: 1 }, { a: 2 }]
```
## Usage: Advanced Example
```ts
import { parseJsonWithHeaders } from "@suaveplan/api-plugin-import";
const { headers, rows, totalRows } = parseJsonWithHeaders(
'[{"a":1,"b":2},{"a":3,"c":4}]'
);
// headers: ["a","b","c"]
// rows: [{ a: 1, b: 2 }, { a: 3, c: 4 }]
// totalRows: 2
```
## API Reference
- `parseJson(data: string): Record<string, unknown>[]` — throws `Error("JSON data must be an array")` on non-array input.
- `parseJsonWithHeaders(content: string): CsvParseResult` — header-aware variant.
## Implementation Notes
- `parseJson` uses `JSON.parse` directly — invalid JSON syntax surfaces
as the native `SyntaxError` from the runtime, which is what the
service catches and re-throws as `IMPORT.PROCESSING_FAILED`.
- `parseJsonWithHeaders` wraps single-object inputs in a one-element
array so the route's `format: "json"` branch never has to special-case
shape — the field is always `rows[]`.
- Scalar wrapping uses the literal key `value`; consumers that need a
different key can map the result post-parse.

View File

@@ -0,0 +1,59 @@
# `parsers/xlsx` — TSV-style XLSX parser
## Purpose
Provides a lightweight tab-separated-value parser exposed under the
`xlsx` format so consumers can ingest spreadsheet-style exports without
adding a SheetJS dependency. Binary `.xlsx` (the ZIP-packaged
SpreadsheetML format) is explicitly out of scope; hosts that need
binary support should wrap a SheetJS reader, render the cells to TSV,
and call `parseXlsx` on the rendered string. Documenting the limitation
inside the package keeps the dependency footprint small for the
common case (TSV interchange dumps from data warehouses, manual exports
from spreadsheet apps that produce a TSV variant).
## Features
- TSV split with optional header row.
- `column_N` synthesised headers when `hasHeaders: false`.
- Empty-line filtering — blank rows inside the input are dropped before
parsing so consumers don't have to post-filter.
- Per-cell `.trim()` so spreadsheet-pasted content with trailing tab
whitespace produces clean values.
## Usage: Basic Example
```ts
import { parseXlsx } from "@suaveplan/api-plugin-import";
const { headers, rows, totalRows } = parseXlsx("a\tb\n1\t2\n3\t4");
// headers: ["a","b"], rows: [{ a: "1", b: "2" }, { a: "3", b: "4" }], totalRows: 2
```
## Usage: Advanced Example
```ts
import { parseXlsx } from "@suaveplan/api-plugin-import";
const result = parseXlsx("1\tAlice\n2\tBob", { hasHeaders: false });
// headers: ["column_1","column_2"]
// rows: [{ column_1: "1", column_2: "Alice" }, { column_1: "2", column_2: "Bob" }]
```
## API Reference
- `parseXlsx(content: string, options?: { sheetName?: string; hasHeaders?: boolean }): CsvParseResult`
- `sheetName` is accepted for API compatibility with future binary
XLSX support but is ignored by the TSV implementation.
## Implementation Notes
- Uses `content.split(/\r?\n/)` so both LF and CRLF line endings are
handled.
- Empty-line filtering happens before header detection, so a file with
a single blank line followed by a header row still produces the
right header set.
- Trailing-whitespace trim is per-cell, not per-row, so cells whose
body intentionally starts or ends with whitespace are normalised. If
preservation is required, drop in a custom parser via the
`Parser`-shaped interface and skip `parseXlsx`.

View File

@@ -0,0 +1,85 @@
# `plugin/plugin` — createImportPlugin factory
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the
`@suaveplan/api-plugin` lifecycle. Validates the supplied
options at the factory boundary via `ImportPluginOptionsSchema` so
invalid input fails fast with `ImportError(IMPORT_CODES.INVALID_OPTIONS)`
and the `ZodError` as `cause`. The descriptor's three lifecycle hooks
construct the canonical `ImportService`, register it on the
`ServiceRegistry`, mount the routes under the configured prefix, and
clean up listeners on close.
The plugin opts-in to `auth`, `events`, and `database` via
`optionalDependencies` so the topo-sort orders it after siblings that
provide those services without hard-requiring them. Apps that don't
ship auth still register; their `requireAuth()` calls simply throw
`AUTH.UNAUTHENTICATED` per the auth plugin's contract.
## Features
- Zod-validated options at the factory boundary.
- Owner-scoped `ImportService` constructed once in `onInit`.
- `ServiceRegistry["importService"]` slot registration so consumer
plugins can read the service via `ctx.getService("importService")`.
- Routes mounted via `app.route(config.routePrefix, routes)` with a
prefix-collision guard that throws if another plugin already owns
the path.
- `onClose` clears every listener attached to `service.bus`.
## Usage: Basic Example
```ts
import { createImportPlugin } from "@suaveplan/api-plugin-import";
const importPlugin = createImportPlugin();
// register with @suaveplan/api's plugin registry:
// registry.register(importPlugin)
```
## Usage: Advanced Example
```ts
import { createImportPlugin, MemoryImportStore } from "@suaveplan/api-plugin-import";
const store = new MemoryImportStore();
const importPlugin = createImportPlugin({
routePrefix: "/v2/imports",
maxRows: 10_000,
store,
clock: () => Date.now(),
logger: pinoLogger.child({ plugin: "import" }),
});
// store stays in scope so the host can introspect imported jobs
// outside of the plugin lifecycle (e.g. background sweeps)
```
## API Reference
- `createImportPlugin(options?: CreateImportPluginOptions): PluginDescriptor<ImportPluginConfig>`
- `type CreateImportPluginOptions = ImportPluginOptionsInput & Pick<ImportServiceOptions, "store" | "clock" | "idFactory" | "logger">`
## Implementation Notes
- The factory separates the Zod-validated config fields from the
pass-through `ImportService` options (`store`, `clock`, `idFactory`,
`logger`) so option drift between the plugin defaults and the
service constructor stays explicit.
- `mustService` throws if `onReady` runs before `onInit` constructed
the service — this catches plugin-orchestrator regressions where
hooks fire out of order.
- The route-prefix collision guard mirrors the search-plugin
precedent: a second mount under the same prefix silently shadows
the first in Hono, so a hard fail at orchestration time is preferred
to debugging a missing-route mystery in production.
- `service.bus.clear()` removes both listeners and interceptors; the
plugin owns the bus so a hard reset on close is safe.
- `optionalDependencies` lists `"auth"` even though the routes hard-import
`requireAuth` because the topo-sort only uses the dependency list for
ordering — `requireAuth` works correctly even when the auth plugin
hasn't run yet (it simply throws `AUTH.UNAUTHENTICATED`).

View File

@@ -0,0 +1,83 @@
# `progress/tracker` — real-time import progress
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides the `ImportProgressTracker` class consumed by the canonical
`ImportService` (and exported for direct use by CLI tools / batch
workers) to track import jobs in real time. The tracker counts
successes, failures, and overall progress, and exposes derived metrics
(rows per second, percent complete, estimated remaining time) without
requiring the caller to keep track of the wall clock themselves.
The clock is injectable so test suites can pin deterministic ETA /
rows-per-sec assertions without sleeping. Without a clock override the
tracker uses `Date.now`, so the production hot path is allocation-free
after construction.
## Features
- Single-row (`recordSuccess`, `recordFailure`) and batched
(`recordBatchSuccess(count)`, `recordBatchFailure(count)`) updates.
- Listener fan-out via `onProgress(callback)` — returns an unsubscribe
function so consumers can detach cleanly.
- Snapshot shape (`ProgressSnapshot`) carries `rowsPerSecond` rounded to
2 decimal places, `percentComplete` rounded to an integer, and
`estimatedRemainingMs` rounded to an integer.
- Injectable `Clock` from `@suaveplan/types` for deterministic tests.
- `isComplete()` predicate flips once `processedRows >= totalRows` —
handles `totalRows === 0` as immediately complete.
## Usage: Basic Example
```ts
import { ImportProgressTracker } from "@suaveplan/api-plugin-import";
const tracker = new ImportProgressTracker("job-1", 100);
tracker.onProgress((snap) => console.log(snap.percentComplete));
tracker.recordBatchSuccess(50); // logs 50
tracker.recordBatchFailure(10); // logs 60
```
## Usage: Advanced Example
```ts
import { ImportProgressTracker } from "@suaveplan/api-plugin-import";
// Deterministic test with an injected clock — advance time by a known delta
let now = 1_000;
const tracker = new ImportProgressTracker("j", 100, { clock: () => now });
tracker.recordBatchSuccess(50);
now += 1_000;
const snap = tracker.snapshot();
// snap.rowsPerSecond === 50, snap.estimatedRemainingMs === 1_000
```
## API Reference
- `class ImportProgressTracker`
- `constructor(jobId: string, totalRows: number, options?: ImportProgressTrackerOptions)`
- `onProgress(callback: ProgressCallback): () => void`
- `recordSuccess(): void`
- `recordFailure(): void`
- `recordBatchSuccess(count: number): void`
- `recordBatchFailure(count: number): void`
- `snapshot(): ProgressSnapshot`
- `isComplete(): boolean`
- `interface ImportProgressTrackerOptions { clock?: Clock }`
- `interface ProgressSnapshot { jobId; totalRows; processedRows; successCount; failedCount; percentComplete; rowsPerSecond; estimatedRemainingMs; elapsedMs; startedAt }`
- `type ProgressCallback = (snap: ProgressSnapshot) => void`
## Implementation Notes
- `totalRows === 0` short-circuits `percentComplete` to `0` and
`estimatedRemainingMs` to `0` to avoid `NaN` from a zero divisor.
- The listener array is replaced (not spliced) on `off()` so a
listener removing itself from inside its own callback does not
invalidate the in-flight iteration.
- `startedAtIso` is captured once in the constructor and re-used on
every snapshot to avoid `new Date(...)` allocations on the hot path.

View File

@@ -0,0 +1,69 @@
# `routes/build-routes` — auth-gated import routes
## Purpose
Constructs the `OpenAPIHono` router mounted by `createImportPlugin` at
the configured `routePrefix`. Five routes are registered — `POST
/preview`, `POST /`, `GET /`, `GET /{id}`, `DELETE /{id}` — each gated
behind `requireAuth()` from `@suaveplan/api-plugin-auth` so anonymous
requests return 401 problem+json before they ever reach a handler.
Owner-scoped routes read `c.get("user").id` and pass it through to the
canonical `ImportService`, which raises `IMPORT.NOT_FOUND` (mapped to
404 problem+json) on foreign-owner lookups so the surface is
anti-enumeration-safe.
## Features
- Five OpenAPI routes with full Zod request / response schemas.
- Per-route `app.use("/*", requireAuth())` guard.
- `progressToResponse()` shape mapper so the response body always
satisfies `ImportJobResponseSchema`.
- Every handler funnels error paths through `respondProblemJson` so
the response shape is uniform across success and failure.
## Usage: Basic Example
```ts
import { createImportRoutes, ImportService } from "@suaveplan/api-plugin-import";
import { OpenAPIHono } from "@hono/zod-openapi";
const service = new ImportService();
const app = new OpenAPIHono();
app.route("/imports", createImportRoutes({ service }));
```
## Usage: Advanced Example
```ts
import { createImportRoutes, ImportService, MemoryImportStore } from "@suaveplan/api-plugin-import";
import { OpenAPIHono } from "@hono/zod-openapi";
const store = new MemoryImportStore();
const service = new ImportService({ store });
service.bus.on("import.progress", (snap) => sse.broadcast(snap));
const app = new OpenAPIHono();
app.use("/imports/*", customRateLimit({ window: "1m", max: 10 }));
app.route("/imports", createImportRoutes({ service }));
```
## API Reference
- `createImportRoutes(deps: ImportRoutesDeps): OpenAPIHono`
- `interface ImportRoutesDeps { service: ImportService }`
## Implementation Notes
- The `requireAuth()` middleware is mounted via `app.use("/*", ...)` so
every route inherits the guard. Mounting per-route would duplicate
the wiring and miss the OpenAPI catch-all.
- `ownerOf(c)` returns the empty string when `c.get("user")` is null;
the auth middleware throws BEFORE the handler runs, so the empty
string is unreachable from production traffic — but it remains a
safe default if a caller somehow bypasses the middleware in a test
harness.
- The list endpoint returns `{ items, total }` rather than a raw array
so clients can paginate; the schema enforces `total >= 0`.
- The delete endpoint surfaces a `{ deleted: boolean }` payload — a
successful delete returns `{ deleted: true }`; a missing job throws
`IMPORT.NOT_FOUND` from the service layer and is rendered as 404.

View File

@@ -0,0 +1,70 @@
# `routes/problem-json` — RFC 7807 problem+json helper
## Purpose
Bridges thrown errors from the route handlers into RFC 7807
`application/problem+json` responses. Reads the HTTP status from the
error's `context.statusCode` when present so the canonical
`ImportError` raised by the service / plugin layer surfaces with the
correct HTTP code without a code → status lookup table in the router.
Unknown throwables fall back to a generic 500 so the router never
crashes — the renderer is the safety net for non-`SuaveplanError`
exceptions that escape upstream code.
## Features
- `respondProblemJson(c, err)` — writes the `Response` to the Hono
context with `content-type: application/problem+json`.
- `renderProblem(err)` — pure render returning a `ProblemDetails`
object (used by tests + alt transports).
- `PROBLEM_JSON_CONTENT_TYPE` constant for downstream re-use.
- Native `Error` paths preserve the `name` so debugging upstream
failures stays readable.
## Usage: Basic Example
```ts
import { respondProblemJson } from "@suaveplan/api-plugin-import";
app.get("/foo", (c) => {
try {
throw new ImportError(IMPORT_CODES.NOT_FOUND, "missing", {
context: { statusCode: 404 },
});
} catch (err) {
return respondProblemJson(c, err);
}
});
```
## Usage: Advanced Example
```ts
import { renderProblem } from "@suaveplan/api-plugin-import";
// Pure render for use in alternate transports (gRPC adapter, queue worker)
const body = renderProblem(new Error("boom"));
// body === { type: "urn:suaveplan:error:internal", title: "Error", status: 500, detail: "boom" }
const message = JSON.stringify(body);
publishToDeadLetterQueue(message);
```
## API Reference
- `respondProblemJson(c: HonoLike, err: unknown): Response`
- `renderProblem(err: unknown): ProblemDetails`
- `interface HonoLike { header(name, value); body(data, status): Response }`
- `interface ProblemDetails { type; title; status; detail; code?; [k: string]: unknown }`
- `const PROBLEM_JSON_CONTENT_TYPE = "application/problem+json"`
## Implementation Notes
- The `type` field is built as `urn:suaveplan:error:<lowercased-code>`
so consumers can branch on the URN without parsing the body.
- Status hints below 400 (or missing) collapse to 500 — clients should
never receive a problem+json body with a 2xx status code.
- The `Error.name` fallback (`"InternalError"`) protects against
thrown objects whose `name` was wiped by a downstream serializer.
- `renderProblem` returns a `ProblemDetails` literal so consumers that
need to merge additional fields (e.g. `instance`) can spread it.

View File

@@ -0,0 +1,124 @@
# `routes/progress-sse` — Server-Sent Events progress push
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Exposes `GET /imports/:id/progress` returning a long-lived
`text/event-stream` so clients can subscribe to a job's real-time
progress without polling. The handler is driven entirely off the
existing `ImportService.bus` — three subscriptions (`import.progress`,
`import.completed`, `import.failed`) are wired into a
`ReadableStream<Uint8Array>` and detached on terminal events or client
abort. This module was deferred from the original
`add-api-plugin-import` migration's `Excluded` list and ships
independently via the `add-import-sse-progress` change.
## Features
- Three SSE event kinds — `progress` (per `import.progress` tick),
`complete` (terminal, on `import.completed`), `failed` (terminal, on
`import.failed`).
- Initial frame snapshot from `service.getJob` so clients see current
state before the next bus tick.
- Owner-scoped — foreign-owner / missing-job lookups return HTTP 404
RFC 7807 problem+json (`IMPORT.NOT_FOUND`); anonymous requests return
HTTP 401 via the existing `requireAuth()` middleware.
- Per-job filtering on every subscription callback so a long-lived
connection cannot leak progress for other jobs.
- `AbortSignal` lifecycle — client disconnect detaches every listener
and emits `import.sse.disconnected{reason: "client"}`.
- OTel observability — `import.sse.connected`,
`import.sse.events_sent{kind}`, `import.sse.disconnected{reason}`.
- Pure stream — no polling timer, no background interval, no
heartbeat (job runs are short-lived so proxy-keepalive concerns are
handled by upstream LB tuning, not by the handler).
## Usage: Basic Example
```ts
import { createImportRoutes, ImportService } from "@suaveplan/api-plugin-import";
import { Hono } from "hono";
const service = new ImportService();
const host = new Hono();
host.use("*", async (c, next) => {
c.set("user", { id: "alice" });
await next();
});
host.route("/imports", createImportRoutes({ service }));
// Client side:
const es = new EventSource("/imports/job-1/progress");
es.addEventListener("progress", (e) => console.log(JSON.parse(e.data)));
es.addEventListener("complete", () => es.close());
es.addEventListener("failed", () => es.close());
```
## Usage: Advanced Example
```ts
import {
createProgressSseHandler,
ImportService,
} from "@suaveplan/api-plugin-import";
import { Hono } from "hono";
const service = new ImportService();
const app = new Hono();
app.use("*", async (c, next) => {
c.set("user", { id: "alice" });
await next();
});
// Mount stand-alone (without the rest of `createImportRoutes`).
app.get("/jobs/:id/progress", createProgressSseHandler({ service }));
// Drive the service from a worker; every emitted progress event is
// fanned to every connected client whose `:id` matches.
const controller = new AbortController();
const res = await app.request(
new Request("http://t/jobs/job-1/progress", { signal: controller.signal })
);
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buf = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buf += decoder.decode(value, { stream: true });
const frames = buf.split("\n\n");
buf = frames.pop() ?? "";
for (const frame of frames) console.log(frame);
}
```
## API Reference
- `createProgressSseHandler(deps: ProgressSseDeps): (c: Context) => Response`
- `formatProgressFrame(kind: ProgressSseEventKind, seq: number, payload: ImportProgress): string`
- `interface ProgressSseDeps { service: ImportService }`
- `type ProgressSseEventKind = "progress" | "complete" | "failed"`
## Implementation Notes
- The handler is registered via `app.get("/:id/progress", ...)` rather
than `app.openapi(...)` because `c.body(ReadableStream, ...)` does not
conform to a static OpenAPI response schema — the parent router's
`app.use("/*", requireAuth())` and `app.onError(...)` still apply.
- Subscriptions are filtered by `payload.jobId === jobId` so the bus
cannot leak foreign-job progress to a connection. The router's
authoritative ownership check happens once via `service.getJob` at
connection-open time; subsequent leak prevention is structural.
- The terminal `complete` / `failed` branch in `start` handles the
edge case where a job reached terminal state between `getJob` and
the bus subscription — without it, a client would be stuck on the
initial `progress` frame forever.
- The OTel counters are read through `getImportInstruments()` rather
than cached at module scope so `resetImportInstruments()` (used by
test setup) rebinds correctly.
- The response headers (`Cache-Control: no-cache, no-transform`,
`X-Accel-Buffering: no`, `Connection: keep-alive`) match the
`@suaveplan/api-plugin-sse` handler so reverse proxies (nginx,
cloudflare) do not buffer or coalesce the stream.

View File

@@ -0,0 +1,67 @@
# `schemas/options` — plugin options schema
## Purpose
Declares the Zod schema for the options accepted by
`createImportPlugin`. Parsing options at the factory boundary
guarantees the resolved config (`ImportPluginConfig`) carries no
optional fields — every downstream consumer (`onInit`, `onReady`,
routes, service) reads from a fully-resolved object without `?? default`
fallbacks scattered across the call sites.
Defaults are documented inside the schema (`.default(...)`) so the Zod
parse output is the authoritative source of plugin defaults. Renaming a
field is a breaking change.
## Features
- Six options with sensible production defaults: `routePrefix`,
`maxFileSizeBytes`, `maxRows`, `maxErrorsBeforeAbort`, `previewRows`,
`defaultBatchSize`.
- Two TypeScript shapes — `ImportPluginOptionsInput` for caller-supplied
partials, `ImportPluginConfig` for the resolved config.
- Strict positive-integer / non-empty-string constraints surface bad
config as `IMPORT.INVALID_OPTIONS` with the `ZodError` as `cause`.
## Usage: Basic Example
```ts
import { ImportPluginOptionsSchema } from "@suaveplan/api-plugin-import";
const parsed = ImportPluginOptionsSchema.parse({});
// parsed.routePrefix === "/imports"
// parsed.maxFileSizeBytes === 52_428_800
```
## Usage: Advanced Example
```ts
import { ImportPluginOptionsSchema, type ImportPluginConfig } from "@suaveplan/api-plugin-import";
function configureImport(overrides: Partial<ImportPluginConfig>): ImportPluginConfig {
return ImportPluginOptionsSchema.parse({
routePrefix: "/v2/imports",
maxRows: 10_000,
...overrides,
});
}
```
## API Reference
- `ImportPluginOptionsSchema: z.ZodObject<…>` — Zod schema with
defaults for every field.
- `type ImportPluginOptionsInput` — caller-supplied partial input.
- `type ImportPluginConfig` — resolved options (every field
non-optional).
## Implementation Notes
- All numeric bounds are `int().positive()` because the consuming
fields are counts; the schema rejects zero with the standard Zod
error message.
- `previewRows` is capped at 1 000 to prevent a malicious client from
triggering a multi-megabyte preview render.
- `routePrefix` is non-empty; an empty prefix collapses the mount and
causes Hono route shadowing — the Zod gate prevents this footgun at
factory time.

View File

@@ -0,0 +1,81 @@
# `schemas/schemas` — public Zod request/response surface
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Carries the full public Zod surface used by the route handlers and any
external client (CLI tool, integration test, browser app) that wants to
validate or generate import requests / responses without depending on
the route module itself. Every body, query, and response shape that
crosses the HTTP boundary is declared here so the OpenAPI document and
the runtime validators stay in lockstep.
Keeping these schemas in their own module means consumers can import
them without pulling in the Hono/OpenAPI route builder, which is
important for browser clients and shared validation libraries.
## Features
- `ImportFormatSchema` / `ImportStatusSchema` — Zod enums mirroring the
cross-module type unions.
- `ColumnMappingSchema` — wire-form of a column mapping (functions
intentionally omitted; transforms are server-side only).
- `PreviewRequestSchema` / `PreviewResponseSchema` — `POST /preview`.
- `CreateImportJobSchema` — `POST /` body.
- `ImportJobResponseSchema` — every job-shape response.
- `ImportJobListQuerySchema` — `GET /` querystring with `z.coerce`
for the numeric fields so query strings parse without hand-rolling.
## Usage: Basic Example
```ts
import { CreateImportJobSchema } from "@suaveplan/api-plugin-import";
const result = CreateImportJobSchema.safeParse({
format: "csv",
data: "name\nAda",
mappings: [{ sourceColumn: "name", targetField: "displayName" }],
});
if (!result.success) throw new Error(result.error.message);
```
## Usage: Advanced Example
```ts
import { ImportJobResponseSchema, type ImportJobResponseShape } from "@suaveplan/api-plugin-import";
async function fetchJob(id: string): Promise<ImportJobResponseShape> {
const res = await fetch(`/imports/${id}`);
const json = await res.json();
return ImportJobResponseSchema.parse(json);
}
```
## API Reference
- `ImportFormatSchema`, `type ImportFormatInput`
- `ImportStatusSchema`, `type ImportStatusInput`
- `ColumnMappingSchema`, `type ColumnMappingInput`
- `PreviewRequestSchema`, `type PreviewRequestInput`
- `PreviewResponseSchema`, `type PreviewResponseShape`
- `CreateImportJobSchema`, `type CreateImportJobInput`
- `ImportJobResponseSchema`, `type ImportJobResponseShape`
- `ImportJobListQuerySchema`, `type ImportJobListQueryInput`
- `RowErrorSchema`
## Implementation Notes
- `ColumnMappingSchema` omits the `transform` callback — transforms are
authored in TypeScript on the server, not over the wire. Wire-form
mappings are pure renames + required flags.
- `PreviewResponseSchema.sampleRows` uses `z.record(z.string(),
z.unknown())` so heterogeneous row shapes survive validation.
- `ImportJobListQuerySchema` uses `z.coerce.number()` because Hono
delivers query parameters as strings — the coerce ensures `?limit=20`
parses to a number without manual `parseInt`.
- `CreateImportJobSchema.mappings` requires `min(1)` because zero
mappings is the service-layer `IMPORT.VALIDATION` failure; rejecting
it at the Zod boundary saves a service round-trip.

View File

@@ -0,0 +1,115 @@
# `service/service` — canonical ImportService
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Implements the canonical `ImportService` used by `createImportPlugin`
and exported for direct programmatic use. Owns the per-job in-memory
progress map, the optional durable store, the `EventBus<ImportEventMap>`
event surface, and the OTel emit on the production hot path. Every read
and write is owner-scoped: a job belongs to exactly one `ownerId`, and
foreign-owner lookups return `IMPORT.NOT_FOUND` (anti-enumeration) — not
`IMPORT.FORBIDDEN` — so callers cannot probe for the existence of other
users' jobs.
The service is the only place that ties together parsers, transform
pipeline (via `applyMappings`), progress emission, and persistence; the
plugin layer is a thin wrapper that constructs it once per `onInit`
and mounts the routes.
## Features
- `preview(format, data)` — returns column auto-mapping + a 5-row
sample.
- `startImport(config, { ownerId })` — parses, maps, optionally
dedups, optionally runs `onRow`, emits per-row events, batch progress
updates, and OTel counters / histogram.
- `getJob` / `getResults` / `listJobs` / `deleteJob` — owner-scoped.
- `bus: EventBus<ImportEventMap>` — public so consumers can attach
listeners without subclassing.
- Injectable `store`, `clock`, `idFactory`, `logger`, `instruments` for
testability.
- RFC 7807 status hints — every thrown `ImportError` carries
`context.statusCode` so the route renderer doesn't need a code→status
lookup table.
## Usage: Basic Example
```ts
import { ImportService } from "@suaveplan/api-plugin-import";
const service = new ImportService();
const preview = service.preview("csv", "name\nAda");
const progress = await service.startImport(
{
format: "csv",
data: "name\nAda",
mappings: [{ sourceColumn: "name", targetField: "displayName" }],
},
{ ownerId: "alice" }
);
// progress.status === "completed"
```
## Usage: Advanced Example
```ts
import { ImportService, MemoryImportStore } from "@suaveplan/api-plugin-import";
const store = new MemoryImportStore();
const service = new ImportService({
store,
clock: () => 1_700_000_000_000,
idFactory: () => "test-job",
});
service.bus.on("import.progress", (snap) => console.log(snap.percentComplete));
await service.startImport(
{
format: "json",
data: '[{"x":1},{"x":2},{"x":1}]',
mappings: [{ sourceColumn: "x", targetField: "value" }],
skipDuplicates: true,
},
{ ownerId: "alice" }
);
// `test-job` is now in the store; results array dedups to 2 entries
```
## API Reference
- `class ImportService`
- `constructor(options?: ImportServiceOptions)`
- `bus: EventBus<ImportEventMap>`
- `preview(format: ImportFormat, data: string): ImportPreview`
- `startImport(config: ImportJobConfig, owner: ImportOwnerContext): Promise<ImportProgress>`
- `getJob(jobId: string, ownerId: string): ImportProgress`
- `getResults(jobId: string, ownerId: string): Record<string, unknown>[]`
- `listJobs(ownerId: string, filter?: ImportStoreListFilter): ImportProgress[]`
- `deleteJob(jobId: string, ownerId: string): Promise<boolean>`
- `interface ImportServiceOptions { store?; clock?; idFactory?; logger?; instruments? }`
## Implementation Notes
- The service stores progress in two `Map`s (live progress + imported
rows). Persistence is best-effort; a store failure throws
`IMPORT.STORAGE_ERROR` but the in-memory progress is still consistent.
- `onRow` is invoked AFTER `skipDuplicates` so a duplicate row never
reaches the caller's filter. Returning `false` skips the row but does
NOT mark it as an error.
- The batch-progress emit cadence is keyed on `(i + 1) % batchSize === 0`
so a batch size of 100 emits at rows 100, 200, … plus the final
completion emit. This matches the upstream behaviour exactly.
- Foreign-owner reads on `getJob` throw `NOT_FOUND` rather than
`FORBIDDEN` so an attacker cannot distinguish "no such job" from
"your access is denied."
- The XLSX path is rendered inline (not via `parseXlsx`) to keep the
return shape `Record<string, unknown>[]` instead of `CsvParseResult`;
call `parseXlsx` directly if you need headers metadata.
- `parseData` re-raises `ImportError` unchanged but wraps every other
throwable as `IMPORT.PROCESSING_FAILED` so consumers can branch on
one code regardless of which parser failed.

View File

@@ -0,0 +1,83 @@
# `services/registry-augmentation` — ServiceRegistry augmentation
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Augments the `ServiceRegistry` interface from
`@suaveplan/api-plugin/service-registry` with the `importService` slot
so any consumer plugin typed against the augmented registry can call
`ctx.getService("importService")` and receive a typed
`ImportService | undefined` without a cast.
The augmentation also re-declares the host-side `app` slot owned by
`@suaveplan/api` as a cumulative augmentation. TypeScript merges
duplicate declarations, so consumers don't need to explicitly import
`@suaveplan/api`'s augmentation module just to resolve `app` from
inside our plugin's lifecycle hooks.
## Features
- One slot added — `importService: ImportService`.
- Cumulative re-declaration of `app: OpenAPIHono` so the import
plugin's `onReady` can resolve the host app without forcing a
consumer-side import.
- Side-effect import; the `services/index.ts` barrel re-exports it via
`import "./registry-augmentation.js"`.
## Usage: Basic Example
```ts
// Anywhere consuming the import plugin
import "@suaveplan/api-plugin-import"; // pulls in the augmentation
import type { PluginContext } from "@suaveplan/api-plugin";
function readService(ctx: PluginContext): void {
const svc = ctx.getService("importService"); // typed as ImportService | undefined
}
```
## Usage: Advanced Example
```ts
// Direct module-augmentation chain for downstream extension authors
import "@suaveplan/api-plugin-import"; // adds importService
import "@suaveplan/api-plugin-database"; // adds database services
import type { PluginContext } from "@suaveplan/api-plugin";
function bootstrap(ctx: PluginContext): void {
const svc = ctx.getService("importService");
const db = ctx.getService("databasePool");
if (svc && db) {
// wire a custom adapter that writes via the database pool
}
}
```
## API Reference
This module exports nothing at runtime — it is pure TypeScript module
augmentation:
```ts
declare module "@suaveplan/api-plugin/service-registry" {
interface ServiceRegistry {
importService: ImportService;
app: OpenAPIHono;
}
}
```
## Implementation Notes
- The augmentation is a `declare module` statement, not a value export;
treeshaking under `sideEffects: false` keeps it in the bundle because
TypeScript module-augmentation has no runtime correlate.
- The `app` re-declaration intentionally mirrors the host package; the
TypeScript compiler merges identical interface declarations across
modules, so the duplicate is type-safe.
- A consumer that DOES import `@suaveplan/api`'s registry directly will
see only one merged slot — there is no double-declaration error.

View File

@@ -0,0 +1,95 @@
# `store/memory` — in-memory import job store
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Declares the async `ImportStoreAdapter` interface and ships the
`MemoryImportStore` implementation used by the canonical `ImportService`
to persist `ImportJobRecord` entries across the run / lookup boundary.
The interface is the seam for swapping in durable backends (Surreal,
Postgres, Redis-backed cache) without touching the service. The
in-memory implementation is sufficient for tests and single-process
deployments; production hosts implement the same interface against a
real database.
The store is intentionally minimal — five methods, no transactions, no
optimistic concurrency. The service layer is the single writer per
process; cross-process coordination is the database's responsibility
once a real backend lands.
## Features
- Five-method async interface (`create`, `get`, `update`, `delete`,
`list`) matching the upstream surface.
- `update` on a missing id throws with `Job "<id>" not found` so the
service layer can map the failure into `IMPORT.STORAGE_ERROR`.
- `list` filters by `status` and `ownerId` and honours `limit` / `offset`
paging — defaults `limit: 50`, `offset: 0`.
- Injectable `Clock` for deterministic `updatedAt` stamps.
- `size()` / `clear()` test helpers for cheap setup/teardown.
## Usage: Basic Example
```ts
import { MemoryImportStore } from "@suaveplan/api-plugin-import";
const store = new MemoryImportStore();
await store.create({
id: "j1",
ownerId: "alice",
status: "created",
totalRows: 0,
processedRows: 0,
successCount: 0,
errorCount: 0,
errors: [],
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
});
```
## Usage: Advanced Example
```ts
import { MemoryImportStore, type ImportStoreAdapter } from "@suaveplan/api-plugin-import";
// Inject a fixed clock so the updatedAt is reproducible in tests.
const clock = () => 1_700_000_000_000;
const store: ImportStoreAdapter = new MemoryImportStore({ clock });
await store.create(record);
await store.update(record.id, { status: "completed" });
const fresh = await store.get(record.id);
// fresh.updatedAt === new Date(1_700_000_000_000).toISOString()
const recent = await store.list({ status: "completed", ownerId: "alice", limit: 10 });
```
## API Reference
- `interface ImportStoreAdapter`
- `create(job: ImportJobRecord): Promise<void>`
- `get(id: string): Promise<ImportJobRecord | undefined>`
- `update(id: string, patch: Partial<ImportJobRecord>): Promise<void>`
- `delete(id: string): Promise<boolean>`
- `list(filter?: ImportStoreListFilter): Promise<ImportJobRecord[]>`
- `interface ImportStoreListFilter { status?; ownerId?; limit?; offset? }`
- `class MemoryImportStore implements ImportStoreAdapter`
- `interface MemoryImportStoreOptions { clock? }`
## Implementation Notes
- The map keys are job ids — collisions overwrite (mirrors the upstream
surface) so callers MUST generate unique ids per job.
- `delete` returns `false` for missing ids (mirrors `Map.delete`); no
exception is raised because the route handler treats a missing job
as a 404, not a 500.
- `update` re-uses the patch's `updatedAt` when supplied so callers
controlling the clock can stamp deterministic timestamps without
swapping out the entire store.
- `list` does NOT mutate the underlying map; the returned array is a
fresh slice of cloned references, so callers can mutate the result
without poisoning the store.

View File

@@ -0,0 +1,155 @@
# `store/surreal/surreal-import-store` — SurrealDB-backed ImportStoreAdapter
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Implements the canonical `ImportStoreAdapter` interface against a connected
`SurrealAdapter` from `@suaveplan/surrealdb`, providing the durable counterpart
to the in-process `MemoryImportStore`. The store stores every `ImportJobRecord`
in a single `import_job` table (configurable) and uses positional SurrealQL
placeholders (`$p0`, `$p1`, …) so it matches the suaveplan adapter contract —
the same pattern used by `@suaveplan/api-plugin-events`'s `EventRepository`.
The store is a drop-in replacement for `MemoryImportStore`: every public method
preserves the original semantics (including the verbatim
`Job "<id>" not found` rejection on `update` against a missing id) so callers
that selected the in-memory adapter for development can swap to SurrealDB in
production without changing call sites.
## Features
- Full `ImportStoreAdapter` parity — `create`, `get`, `update`, `delete`, `list`.
- Positional SurrealQL placeholders (`$p0`, `$p1`, …) matching the suaveplan
`DatabaseAdapter.query(sql, params: readonly unknown[])` surface.
- Zod-validated constructor config — bad input throws
`ImportError(IMPORT_CODES.INVALID_OPTIONS)` with the `ZodError` as `cause`.
- Adapter failures are wrapped as `ImportError(IMPORT_CODES.STORAGE_ERROR)`
with the original throwable preserved as `cause` and the failing SurQL +
operation captured in `error.context`.
- Configurable table name (`table?: string`, defaults to `"import_job"`).
- Injectable `Clock` for deterministic `updatedAt` stamping in `update`.
- Emits `import.store.writes_total{outcome}` on every write and
`import.store.reads_total{outcome}` on every read; a `get` that returns
`undefined` is still counted as a successful read.
- Snake-case ↔ camel-case mapping built in (SurrealDB columns are snake_case
per the established `api-plugin-events` schema convention; JavaScript
records remain camelCase).
- No mocks — unit tests use a fake `SurrealAdapter` that captures SQL +
params; integration tests run against a real SurrealDB testcontainer.
## Usage: Basic Example
```ts
import { SurrealAdapter } from "@suaveplan/surrealdb";
import {
SurrealImportStore,
type ImportJobRecord,
} from "@suaveplan/api-plugin-import";
const db = new SurrealAdapter({
url: "ws://surreal:8000/rpc",
namespace: "app",
database: "imports",
username: "root",
password: "root",
});
await db.connect();
const store = new SurrealImportStore({ db });
const record: ImportJobRecord = {
id: "j1",
ownerId: "alice",
status: "created",
totalRows: 0,
processedRows: 0,
successCount: 0,
errorCount: 0,
errors: [],
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
};
await store.create(record);
const fetched = await store.get("j1");
```
## Usage: Advanced Example
Swap in a custom table name, inject a deterministic clock for tests, and pull
the OTel bundle eagerly so the store never re-resolves the meter on hot paths:
```ts
import { getImportInstruments, SurrealImportStore } from "@suaveplan/api-plugin-import";
const fixedNow = 1_700_000_000_000;
const store = new SurrealImportStore({
db,
table: "tenant_a_import_job",
clock: () => fixedNow,
instruments: getImportInstruments(),
});
// `update` on a missing id rejects with the canonical message — parity
// with `MemoryImportStore`. Adapter-layer SurQL errors surface as
// `ImportError(IMPORT_CODES.STORAGE_ERROR)`.
try {
await store.update("missing", { status: "completed" });
} catch (err) {
// err.message === 'Job "missing" not found'
}
// A read against an empty backend resolves to `undefined` and still
// records `import.store.reads_total{outcome:"success"}` — the backend
// answered correctly.
const missing = await store.get("never-created");
```
## API Reference
- `class SurrealImportStore implements ImportStoreAdapter`
- `constructor(config: SurrealImportStoreConfig)` — throws
`ImportError(IMPORT_CODES.INVALID_OPTIONS)` on Zod-validation failure
- `create(job: ImportJobRecord): Promise<void>`
- `get(id: string): Promise<ImportJobRecord | undefined>`
- `update(id: string, patch: Partial<ImportJobRecord>): Promise<void>` —
throws `Error('Job "<id>" not found')` on a missing id; wraps adapter
failures as `ImportError(IMPORT_CODES.STORAGE_ERROR)`
- `delete(id: string): Promise<boolean>` — returns `true` if a row was
removed, `false` if no row existed
- `list(filter?: ImportStoreListFilter): Promise<ImportJobRecord[]>` —
`status`, `ownerId`, `limit` (default 50), `offset` (default 0)
- `interface SurrealImportStoreConfig { db: SurrealAdapter; table?: string;
logger?: LoggerLike; clock?: Clock; instruments?: ImportInstruments }`
- `SurrealImportStoreConfigSchema: z.ZodTypeAny` — boundary schema; reuses
the same shape as the constructor argument
- `DEFAULT_IMPORT_JOB_TABLE = "import_job"`
## Implementation Notes
- The SurrealDB schema lives at `surql/tables/import_job.surql` (shipped with
the package). The store does NOT apply migrations — the host must run them
via `@suaveplan/api-plugin-database`'s migration runner or by manually
invoking `adapter.query(readFileSync(...))` in `beforeAll`.
- Columns are snake_case in SurrealDB (`owner_id`, `total_rows`, …) but the
`ImportJobRecord` keeps the suaveplan camelCase convention. The mapping is
fully encapsulated in `rowToRecord` and the `create` / `update` SET
clauses; downstream callers see only `ImportJobRecord`.
- `update` first issues a `SELECT * FROM <id>` to detect missing-id and
preserve the `Job "<id>" not found` contract from `MemoryImportStore`. This
is one extra round-trip per update; if write-throughput is the bottleneck
the host can layer a `Map`-based existence cache on top.
- The constructor accepts an optional pre-resolved `ImportInstruments`
bundle so high-volume hosts avoid the global-`MeterProvider` lookup on
the hot path; when omitted, the bundle is resolved lazily on the first
call (mirrors the `getImportInstruments()` cache).
- All errors thrown by the underlying `SurrealAdapter` are wrapped as
`ImportError(IMPORT_CODES.STORAGE_ERROR)` with `cause` set to the original
throwable and `context = { op, sql }` so observability tooling can surface
the failing query without parsing the message.
- `list` orders by `created_at DESC` to mirror most real-world job-listing
use cases. Add an explicit `ORDER BY` upstream if a different order is
required — the `ImportStoreListFilter` contract does not currently expose
a `sort` field.

View File

@@ -0,0 +1,112 @@
# `transform/pipeline` — declarative transform pipeline
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Implements a three-stage row-transform pipeline that any consumer can
drive without depending on the canonical `ImportService`: per-mapping
coerce + transform, computed fields, then field validation. The
pipeline is pure (no side effects, no I/O) and never throws — every
failure is captured as a `TransformError` and returned in the result
object, so callers can collect every failed row in one pass.
This module powers both the standalone `transformRow` / `transformRows`
helpers exported to consumers and the per-row inner loop of
`ImportService.startImport`. Keeping the pipeline separate from the
service means CLI tools, batch workers, and tests can drive the
transform logic against arbitrary input without spinning up an event
emitter or store adapter.
## Features
- `coerceType` — explicit-result coercion for `number` / `boolean` /
`date` with documented "valid" sets (`1`/`0`/`yes`/`no`/`true`/`false`
for booleans, anything `Date.parse`-able for dates).
- `validateField` — five built-in rules (`required`, `min`, `max`,
`pattern`, `custom`); `custom` is an opt-in extension point that
always returns `undefined` so callers wire their own.
- `transformRow` — applies mappings, computed fields, and validations
in order; pushes `TransformError` for each failure and returns
`{ success, row, errors }`.
- `transformRows` — array fan-out; errored rows are dropped from `rows`
but their errors are still collected in `errors`.
- `toRowError` — bridge to the service-layer `RowError` shape so the
service can re-emit transform failures on the `import.row_error`
event topic without duplicating shapes.
## Usage: Basic Example
```ts
import { transformRow } from "@suaveplan/api-plugin-import";
const result = transformRow(
{ name: "Ada", age: "37" },
0,
{
mappings: [
{ sourceColumn: "name", targetField: "displayName" },
{ sourceColumn: "age", targetField: "age", type: "number" },
],
}
);
// result.row → { displayName: "Ada", age: 37 }
```
## Usage: Advanced Example
```ts
import { transformRow } from "@suaveplan/api-plugin-import";
const result = transformRow(
{ first: "Ada", last: "Lovelace", born: "1815" },
0,
{
mappings: [
{ sourceColumn: "born", targetField: "birthYear", type: "number" },
],
computedFields: [
{
targetField: "fullName",
sourceColumns: ["first", "last"],
compute: (v) => `${String(v.first)} ${String(v.last)}`,
},
],
validations: [
{ field: "birthYear", rule: "min", param: 1800 },
],
}
);
// result.row → { birthYear: 1815, fullName: "Ada Lovelace" }
```
## API Reference
- `coerceType(value: unknown, type: string): CoerceResult`
- `validateField(value, rule: FieldValidation, rowIndex): TransformError | undefined`
- `transformRow(source, rowIndex, config: TransformPipelineConfig): TransformRowResult`
- `transformRows(sources, config): { rows; errors }`
- `toRowError(err: TransformError): RowError`
- `interface TransformMapping { sourceColumn; targetField; type?; defaultValue?; required?; transform? }`
- `interface ComputedField { targetField; sourceColumns; compute }`
- `interface FieldValidation { field; rule; param?; message? }`
- `interface TransformError { row; field; message; value? }`
- `interface TransformRowResult { success; row; errors }`
- `interface TransformPipelineConfig { mappings; computedFields?; validations? }`
- `interface CoerceResult { ok; value }`
## Implementation Notes
- Coercion short-circuits on empty / null / undefined to `{ ok: true,
value: undefined }` so a downstream `defaultValue` can still apply.
- The transform function (`mapping.transform`) is wrapped in `try` so a
thrown library never escapes the pipeline; the row's `success` flag
reports the failure and the rest of the mappings still process.
- `custom` validation rules are intentionally no-op — extension is via
the `transform` callback on the mapping, which has access to the raw
value. Custom rules without a default are documented in the
enterprise audit as "extended by consumers."
- `validateField` runs after both mapping and computed-field stages, so
validations target the post-coercion result, not the raw source row.

View File

@@ -0,0 +1,89 @@
# `types` — cross-module types
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Defines the cross-module types and constants consumed by every other
feature module in `@suaveplan/api-plugin-import`. Parsers, the transform
pipeline, the progress tracker, the store adapter, the canonical
`ImportService`, the routes, and the plugin all import their shared
shapes from here so they stay in lockstep across a refactor. The file
contains zero runtime logic beyond two literal constants (`VALID_FORMATS`,
`DEFAULT_BATCH_SIZE`) — it is effectively a `.d.ts` declaration with
named exports.
The types follow the suaveplan convention of `readonly` fields and
discriminated unions rather than free-form interfaces, so consumers get
type-narrowing on `status` and `format` without runtime casts.
## Features
- `ImportFormat` / `ImportStatus` / `ImportJobStatus` discriminated unions.
- `ColumnMapping`, `RowError`, `ImportPreview`, `ImportProgress`,
`ImportJobConfig`, `ImportOwnerContext` interfaces.
- `ImportJobRecord` — store-tier shape kept distinct from
`ImportProgress` so durable backends do not pay the per-row write cost.
- `ImportEventMap` — strongly-typed event-bus topic map for the
`EventEmitter<ImportEventMap>` extended by `ImportService`.
- `VALID_FORMATS` / `DEFAULT_BATCH_SIZE` constants.
## Usage: Basic Example
```ts
import type { ImportJobConfig, ColumnMapping } from "@suaveplan/api-plugin-import";
const mappings: ColumnMapping[] = [
{ sourceColumn: "name", targetField: "displayName", required: true },
];
const config: ImportJobConfig = {
format: "csv",
data: "name\nDoe",
mappings,
};
```
## Usage: Advanced Example
```ts
import type {
ImportEventMap,
ImportProgress,
} from "@suaveplan/api-plugin-import";
import { EventEmitter } from "@suaveplan/events";
const bus = new EventEmitter<ImportEventMap>();
bus.on("import.progress", (progress: ImportProgress) => {
console.log(progress.jobId, progress.processedRows, progress.totalRows);
});
```
## API Reference
- `ImportFormat = "csv" | "json" | "xlsx"`
- `ImportStatus = "pending" | "validating" | "previewing" | "importing" | "completed" | "failed"`
- `ImportJobStatus = "created" | "validating" | "importing" | "completed" | "failed"`
- `interface ColumnMapping { sourceColumn; targetField; transform?; required?; defaultValue? }`
- `interface RowError { row; column; message; value? }`
- `interface ImportPreview { totalRows; columns; sampleRows; mappings }`
- `interface ImportProgress { jobId; ownerId; status; totalRows; processedRows; successCount; errorCount; errors; startedAt?; completedAt? }`
- `interface ImportJobConfig { format; data; mappings; dryRun?; batchSize?; skipDuplicates?; onRow? }`
- `interface ImportOwnerContext { ownerId }`
- `interface ImportJobRecord { id; ownerId; status; totalRows; processedRows; successCount; errorCount; errors; metadata?; createdAt; updatedAt; completedAt? }`
- `interface ImportEventMap { … }`
- `interface ImportRowProcessedEvent { jobId; rowIndex; success }`
- `const VALID_FORMATS: readonly ImportFormat[]`
- `const DEFAULT_BATCH_SIZE = 100`
## Implementation Notes
- `ImportProgress` and `ImportJobRecord` are deliberately separate so the
durable store stays narrow; the live progress object can fan-out
per-row errors while the store records a digest.
- `ownerId` lives on both the progress object and the record so every
read path can enforce owner scoping without a cross-table join.
- `ImportEventMap` widens via index signatures so consumers can layer
app-specific topics on top without `as any`.