Move docs/ package mirror into document-navigator plugin
Relocate the ~6k-file docs tree from the repo root into plugins/document-navigator/docs/ so the plugin is self-contained and portable. docs/superpowers/ (specs and plans) stays at the repo root. Moved via cp+rm rather than git mv because directory rename() fails with EPERM on this WSL /mnt DrvFs mount; git rename-detection pairs the identical-content files at commit time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,141 @@
|
||||
# @suaveplan/api-plugin-import - Tabular-data ingest plugin for @suaveplan/api. CSV / JSON / TSV-XLSX parsers, declarative transform pipeline (coerce + validate + computed), real-time progress tracker, async ImportStoreAdapter (in-memory + SurrealDB backends ship), canonical EventEmitter-based ImportService with owner-scoped jobs, 5 auth-gated OpenAPI Hono routes, RFC 7807 problem+json renderer, 9 IMPORT.* error codes, 6 OTel instruments. Server-tier.
|
||||
|
||||
Tabular-data ingest plugin for `@suaveplan/api`. Parses CSV / JSON /
|
||||
TSV-XLSX payloads, applies declarative column mappings + per-field
|
||||
validation + computed fields, persists owner-scoped import jobs through
|
||||
a pluggable async store, emits real-time progress via an event bus, and
|
||||
exposes five auth-gated OpenAPI routes with full Zod request/response
|
||||
schemas. Ships nine `IMPORT.*` error codes (extending
|
||||
`SuaveplanError`), four OpenTelemetry instruments (`import.jobs_total`,
|
||||
`import.rows_total`, `import.row_errors_total`, `import.duration_ms`),
|
||||
and a hand-rolled RFC 7807 problem+json renderer for the route layer.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [error-codes](./src/error-codes/error-codes.md) — Declares the nine stable `IMPORT.*` error codes used throughout `@suaveplan/api-plugin-import` and exports the `ImportE…
|
||||
- [error-codes](./src/error-codes/error-codes.md) — Declares the nine stable `IMPORT.*` error codes used throughout `@suaveplan/api-plugin-import` and exports the `ImportE…
|
||||
- [otel](./src/otel/instruments.md) — Builds the OpenTelemetry instrument bundle emitted by the canonical `ImportService` so consumers can wire production ob…
|
||||
- [instruments](./src/otel/instruments.md) — Builds the OpenTelemetry instrument bundle emitted by the canonical `ImportService` so consumers can wire production ob…
|
||||
- [csv](./src/parsers/csv.md) — Provides the CSV-family parsers consumed by the canonical `ImportService.preview` / `startImport` pipeline and by any c…
|
||||
- [json](./src/parsers/json.md) — Parses JSON-array input documents into the same `Record<string, unknown>[]` row shape produced by the CSV / XLSX parser…
|
||||
- [xlsx](./src/parsers/xlsx.md) — Provides a lightweight tab-separated-value parser exposed under the `xlsx` format so consumers can ingest spreadsheet-s…
|
||||
- [plugin](./src/plugin/plugin.md) — Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the `@suaveplan/api-plugin` lifecycle. Valida…
|
||||
- [plugin](./src/plugin/plugin.md) — Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the `@suaveplan/api-plugin` lifecycle. Valida…
|
||||
- [progress](./src/progress/tracker.md) — Provides the `ImportProgressTracker` class consumed by the canonical `ImportService` (and exported for direct use by CL…
|
||||
- [tracker](./src/progress/tracker.md) — Provides the `ImportProgressTracker` class consumed by the canonical `ImportService` (and exported for direct use by CL…
|
||||
- [progress-sse](./src/routes/progress-sse/progress-sse.md) — Exposes `GET /imports/:id/progress` returning a long-lived `text/event-stream` so clients can subscribe to a job's real…
|
||||
- [progress-sse](./src/routes/progress-sse/progress-sse.md) — Exposes `GET /imports/:id/progress` returning a long-lived `text/event-stream` so clients can subscribe to a job's real…
|
||||
- [build-routes](./src/routes/build-routes.md) — Constructs the `OpenAPIHono` router mounted by `createImportPlugin` at the configured `routePrefix`. Five routes are re…
|
||||
- [problem-json](./src/routes/problem-json.md) — Bridges thrown errors from the route handlers into RFC 7807 `application/problem+json` responses. Reads the HTTP status…
|
||||
- [schemas](./src/schemas/schemas.md) — Carries the full public Zod surface used by the route handlers and any external client (CLI tool, integration test, bro…
|
||||
- [options](./src/schemas/options.md) — Declares the Zod schema for the options accepted by `createImportPlugin`. Parsing options at the factory boundary guara…
|
||||
- [schemas](./src/schemas/schemas.md) — Carries the full public Zod surface used by the route handlers and any external client (CLI tool, integration test, bro…
|
||||
- [service](./src/service/service.md) — Implements the canonical `ImportService` used by `createImportPlugin` and exported for direct programmatic use. Owns th…
|
||||
- [service](./src/service/service.md) — Implements the canonical `ImportService` used by `createImportPlugin` and exported for direct programmatic use. Owns th…
|
||||
- [services](./src/services/registry-augmentation.md) — Augments the `ServiceRegistry` interface from `@suaveplan/api-plugin/service-registry` with the `importService` slot so…
|
||||
- [registry-augmentation](./src/services/registry-augmentation.md) — Augments the `ServiceRegistry` interface from `@suaveplan/api-plugin/service-registry` with the `importService` slot so…
|
||||
- [store](./src/store/memory.md) — Declares the async `ImportStoreAdapter` interface and ships the `MemoryImportStore` implementation used by the canonica…
|
||||
- [surreal](./src/store/surreal/surreal-import-store.md) — Implements the canonical `ImportStoreAdapter` interface against a connected `SurrealAdapter` from `@suaveplan/surrealdb…
|
||||
- [surreal-import-store](./src/store/surreal/surreal-import-store.md) — Implements the canonical `ImportStoreAdapter` interface against a connected `SurrealAdapter` from `@suaveplan/surrealdb…
|
||||
- [memory](./src/store/memory.md) — Declares the async `ImportStoreAdapter` interface and ships the `MemoryImportStore` implementation used by the canonica…
|
||||
- [transform](./src/transform/pipeline.md) — Implements a three-stage row-transform pipeline that any consumer can drive without depending on the canonical `ImportS…
|
||||
- [pipeline](./src/transform/pipeline.md) — Implements a three-stage row-transform pipeline that any consumer can drive without depending on the canonical `ImportS…
|
||||
- [types](./src/types/types.md) — Defines the cross-module types and constants consumed by every other feature module in `@suaveplan/api-plugin-import`.…
|
||||
- [types](./src/types/types.md) — Defines the cross-module types and constants consumed by every other feature module in `@suaveplan/api-plugin-import`.…
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
bun add @suaveplan/api-plugin-import
|
||||
```
|
||||
|
||||
## Quick start
|
||||
|
||||
```ts
|
||||
import { createImportPlugin } from "@suaveplan/api-plugin-import";
|
||||
|
||||
// Register with @suaveplan/api's plugin registry.
|
||||
const importPlugin = createImportPlugin({ routePrefix: "/imports" });
|
||||
|
||||
// Or use the service directly:
|
||||
import { ImportService } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const service = new ImportService();
|
||||
const progress = await service.startImport(
|
||||
{
|
||||
format: "csv",
|
||||
data: "name\nAda\nBob",
|
||||
mappings: [{ sourceColumn: "name", targetField: "displayName" }],
|
||||
},
|
||||
{ ownerId: "alice" }
|
||||
);
|
||||
console.log(progress.successCount); // 2
|
||||
```
|
||||
|
||||
## Modules
|
||||
|
||||
- [error-codes](./src/error-codes/error-codes.md) — `IMPORT_CODES` /
|
||||
`ImportError` registry, 9 codes.
|
||||
- [types](./src/types/types.md) — cross-module shared shapes.
|
||||
- [parsers/csv](./src/parsers/csv.md) — CSV + streaming CSV parsers.
|
||||
- [parsers/json](./src/parsers/json.md) — JSON parser.
|
||||
- [parsers/xlsx](./src/parsers/xlsx.md) — TSV-style XLSX parser.
|
||||
- [transform/pipeline](./src/transform/pipeline.md) — coerce / validate
|
||||
/ computed-field pipeline.
|
||||
- [progress/tracker](./src/progress/tracker.md) — `ImportProgressTracker`.
|
||||
- [store/memory](./src/store/memory.md) — `ImportStoreAdapter` +
|
||||
`MemoryImportStore`.
|
||||
- [store/surreal/surreal-import-store](./src/store/surreal/surreal-import-store.md)
|
||||
— durable `SurrealImportStore` (SurrealDB-backed). Migration shipped at
|
||||
[`surql/tables/import_job.surql`](./surql/tables/import_job.surql).
|
||||
- [service/service](./src/service/service.md) — canonical `ImportService`.
|
||||
- [routes/problem-json](./src/routes/problem-json.md) — RFC 7807
|
||||
renderer.
|
||||
- [routes/build-routes](./src/routes/build-routes.md) — 5 auth-gated
|
||||
OpenAPI routes.
|
||||
- [otel/instruments](./src/otel/instruments.md) — OTel bundle.
|
||||
- [schemas/options](./src/schemas/options.md) — plugin options schema.
|
||||
- [schemas/schemas](./src/schemas/schemas.md) — public request /
|
||||
response Zod surface.
|
||||
- [services/registry-augmentation](./src/services/registry-augmentation.md)
|
||||
— `ServiceRegistry["importService"]` slot.
|
||||
- [plugin/plugin](./src/plugin/plugin.md) — `createImportPlugin`
|
||||
factory.
|
||||
|
||||
## Error codes
|
||||
|
||||
Every failure path raises an `ImportError` carrying one of nine
|
||||
`IMPORT.*` codes — `NOT_FOUND`, `FORBIDDEN`, `VALIDATION`,
|
||||
`FILE_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `PROCESSING_FAILED`,
|
||||
`STORAGE_ERROR`, `TIMEOUT`, `INVALID_OPTIONS`. Anti-enumeration:
|
||||
foreign-owner lookups return `NOT_FOUND`, never `FORBIDDEN`.
|
||||
|
||||
## SurrealDB store
|
||||
|
||||
`SurrealImportStore` is the durable alternative to `MemoryImportStore`,
|
||||
implementing the same `ImportStoreAdapter` interface against a connected
|
||||
`@suaveplan/surrealdb` `SurrealAdapter`. The matching schema migration
|
||||
ships co-located at [`surql/tables/import_job.surql`](./surql/tables/import_job.surql);
|
||||
hosts apply it through `@suaveplan/api-plugin-database`'s migration
|
||||
runner or via `adapter.query(readFileSync(...))`. The store uses
|
||||
positional `$p0,$p1,…` SurrealQL placeholders — the canonical pattern
|
||||
used by every SurrealDB repository.
|
||||
|
||||
## Observability
|
||||
|
||||
Six OpenTelemetry instruments are emitted on the production hot paths
|
||||
— call `getImportInstruments()` to reuse the cached bundle, or
|
||||
`resetImportInstruments()` in tests after installing a fresh
|
||||
`MeterProvider`. Four cover the service (`jobs_total`, `rows_total`,
|
||||
`row_errors_total`, `duration_ms`); two cover the store
|
||||
(`import.store.writes_total{outcome}` and
|
||||
`import.store.reads_total{outcome}` — emitted by `SurrealImportStore`
|
||||
on every adapter call so host dashboards can split backend health from
|
||||
upstream import success/failure).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# `error-codes` — IMPORT.* registry
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Declares the nine stable `IMPORT.*` error codes used throughout
|
||||
`@suaveplan/api-plugin-import` and exports the `ImportError` class that
|
||||
extends `SuaveplanError` so `instanceof ImportError` narrows the `code`
|
||||
property to the literal union. Every failure path in the plugin — parser
|
||||
errors, mapping errors, owner-mismatch lookups, plugin-config validation
|
||||
failures, store I/O errors, processing timeouts — raises an
|
||||
`ImportError` carrying one of these codes, so consumers can branch on
|
||||
the string value without depending on private types.
|
||||
|
||||
The registry is built via `defineCodes("IMPORT", […])` from
|
||||
`@suaveplan/error`, which guarantees every code is namespaced
|
||||
(`IMPORT.NOT_FOUND`, …) and is mapped 1:1 to a `keyof IMPORT_CODES`
|
||||
identifier. Renaming a code is a breaking change.
|
||||
|
||||
## Features
|
||||
|
||||
- Nine canonical codes covering anti-enumeration lookups (NOT_FOUND),
|
||||
authz (FORBIDDEN), domain validation (VALIDATION), oversized payloads
|
||||
(FILE_TOO_LARGE), enum mismatches (UNSUPPORTED_FORMAT), runtime
|
||||
failures (PROCESSING_FAILED), store I/O (STORAGE_ERROR), deadlines
|
||||
(TIMEOUT), and factory-config gates (INVALID_OPTIONS).
|
||||
- `ImportError` exposes `code` (the canonical literal) plus a redundant
|
||||
`importCode` alias so instance-narrowing keeps the literal type when
|
||||
the variable is widened to `Error`.
|
||||
- Every code is reachable from a production path; the spec-conformance
|
||||
test pins the contract that none can be deleted to satisfy a coverage
|
||||
gate.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { ImportError, IMPORT_CODES } from "@suaveplan/api-plugin-import";
|
||||
|
||||
throw new ImportError(IMPORT_CODES.NOT_FOUND, `Import job "${id}" not found`, {
|
||||
context: { jobId: id, statusCode: 404 },
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { ImportError, IMPORT_CODES } from "@suaveplan/api-plugin-import";
|
||||
|
||||
try {
|
||||
await service.startImport(payload, { ownerId });
|
||||
} catch (err) {
|
||||
if (err instanceof ImportError) {
|
||||
switch (err.code) {
|
||||
case IMPORT_CODES.PROCESSING_FAILED:
|
||||
metrics.increment("import.fail.parse");
|
||||
break;
|
||||
case IMPORT_CODES.FILE_TOO_LARGE:
|
||||
return c.json({ error: "payload too large" }, 413);
|
||||
default:
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `IMPORT_CODES: Readonly<{ NOT_FOUND: "IMPORT.NOT_FOUND"; … }>` — frozen
|
||||
literal map.
|
||||
- `type ImportErrorCode = (typeof IMPORT_CODES)[keyof typeof IMPORT_CODES]`
|
||||
— union of the nine literal values.
|
||||
- `class ImportError extends SuaveplanError<ImportErrorCode>` — concrete
|
||||
error; constructor `(code, message, options?: SuaveplanErrorOptions)`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Built via `defineCodes("IMPORT", […])` so every key acquires the
|
||||
`"IMPORT."` prefix automatically.
|
||||
- The class is `class` not `abstract class` because the plugin throws it
|
||||
directly (no subclass per failure mode).
|
||||
- Status-code hints travel via `options.context.statusCode` so the RFC
|
||||
7807 renderer (`routes/problem-json`) can choose the right HTTP code
|
||||
without re-importing the registry.
|
||||
@@ -0,0 +1,91 @@
|
||||
# `otel/instruments` — OpenTelemetry instruments
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Builds the OpenTelemetry instrument bundle emitted by the canonical
|
||||
`ImportService` so consumers can wire production observability without
|
||||
hand-rolling counter names. The bundle is a single cached object keyed
|
||||
against the global `MeterProvider` — the first `getImportInstruments()`
|
||||
call resolves the meter; subsequent calls return the same reference.
|
||||
Tests rebind by installing their own `MeterProvider`, calling
|
||||
`resetImportInstruments()`, and reading back via an `InMemoryMetricExporter`.
|
||||
|
||||
## Features
|
||||
|
||||
- Nine instruments: `jobsTotal`, `rowsTotal`, `rowErrorsTotal`,
|
||||
`durationMs` (histogram), `storeWritesTotal`, `storeReadsTotal`,
|
||||
`sseConnected`, `sseEventsSent`, `sseDisconnected`.
|
||||
- `storeWritesTotal{outcome}` and `storeReadsTotal{outcome}` are emitted
|
||||
by `SurrealImportStore` on every adapter call so dashboards can split
|
||||
backend health from upstream import success/failure. The in-memory
|
||||
adapter (`MemoryImportStore`) does not emit; durable backends do.
|
||||
- `sseConnected`, `sseEventsSent{kind}`, and `sseDisconnected{reason}`
|
||||
are emitted by `GET /imports/:id/progress` so dashboards can chart
|
||||
open SSE connections, frame throughput by event kind, and teardown
|
||||
causes (terminal `complete` / `failed` vs client abort).
|
||||
- Cached bundle — production code does not re-resolve the meter on
|
||||
every emit.
|
||||
- `resetImportInstruments()` drops the cache so a freshly-installed
|
||||
`MeterProvider` rebinds on the next call.
|
||||
- Asserted by `instruments.test.ts` against a real in-memory exporter
|
||||
(no mocks).
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { getImportInstruments } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const instruments = getImportInstruments();
|
||||
instruments.jobsTotal.add(1, { outcome: "success" });
|
||||
instruments.rowsTotal.add(42, { outcome: "success" });
|
||||
instruments.durationMs.record(123);
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { metrics } from "@opentelemetry/api";
|
||||
import {
|
||||
AggregationTemporality,
|
||||
InMemoryMetricExporter,
|
||||
MeterProvider,
|
||||
PeriodicExportingMetricReader,
|
||||
} from "@opentelemetry/sdk-metrics";
|
||||
import { getImportInstruments, resetImportInstruments } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const exporter = new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE);
|
||||
const reader = new PeriodicExportingMetricReader({ exporter, exportIntervalMillis: 60_000 });
|
||||
const provider = new MeterProvider({ readers: [reader] });
|
||||
metrics.setGlobalMeterProvider(provider);
|
||||
resetImportInstruments();
|
||||
|
||||
const instruments = getImportInstruments();
|
||||
instruments.jobsTotal.add(1, { outcome: "success" });
|
||||
await provider.forceFlush();
|
||||
const collected = exporter.getMetrics();
|
||||
// collected[0].scopeMetrics[0].metrics — read back the counter datapoint
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `interface ImportInstruments { jobsTotal: Counter; rowsTotal: Counter; rowErrorsTotal: Counter; durationMs: Histogram; storeWritesTotal: Counter; storeReadsTotal: Counter; sseConnected: Counter; sseEventsSent: Counter; sseDisconnected: Counter }`
|
||||
- `getImportInstruments(): ImportInstruments`
|
||||
- `resetImportInstruments(): void`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The meter is named `@suaveplan/api-plugin-import` — matching the
|
||||
package name keeps cross-package metric reports readable.
|
||||
- Counters carry the `outcome` label so dashboards can split jobs/rows
|
||||
by success vs error without re-aggregating.
|
||||
- The histogram unit is set to `"ms"` so OTel collectors that respect
|
||||
unit annotation produce labelled charts; consumers MUST emit
|
||||
millisecond values.
|
||||
- The cache is module-scoped — every consumer of the package shares the
|
||||
same instrument set. This is the desired behaviour because the meter
|
||||
resolves against the global `MeterProvider`, which is a process-wide
|
||||
singleton.
|
||||
@@ -0,0 +1,86 @@
|
||||
# `parsers/csv` — CSV parsers
|
||||
|
||||
## Purpose
|
||||
|
||||
Provides the CSV-family parsers consumed by the canonical
|
||||
`ImportService.preview` / `startImport` pipeline and by any caller that
|
||||
needs to drive imports outside the plugin (CLI tools, batch workers).
|
||||
Ships both a high-level `parseCsv` returning row objects and a
|
||||
streaming variant (`streamParseCsv`) that surfaces per-row validation
|
||||
errors without aborting the run — important when the caller needs to
|
||||
collect every failed row in one pass rather than restart after each
|
||||
exception.
|
||||
|
||||
The kernel (`parseCsvLine`, `parseStreamLine`) handles double-quote
|
||||
escaping (`""`), custom delimiters, embedded newlines inside quoted
|
||||
fields, and trims field whitespace. Empty input or header-only input
|
||||
returns an empty rows array — not an exception — so route handlers can
|
||||
distinguish "no data" from "parse failed."
|
||||
|
||||
## Features
|
||||
|
||||
- `parseCsv` / `parseCsvLine` — happy-path one-shot parsing with
|
||||
default `,` delimiter.
|
||||
- `parseStreamLine` / `detectHeaders` — primitives used by
|
||||
`streamParseCsv`; exported for hosts needing custom-delimited
|
||||
variants (semicolons, pipes).
|
||||
- `streamParseCsv` — line-by-line streaming with optional `onRow`,
|
||||
`onError`, and `validate` callbacks. Validation failures are
|
||||
collected, not thrown.
|
||||
- `parseCsvWithHeaders` — header-aware result shape
|
||||
(`{ headers, rows, totalRows }`) matching `parseJson` and `parseXlsx`
|
||||
so routes can branch on `format` once.
|
||||
- `applyColumnMappings` — maps `sourceColumn` → `targetField` with
|
||||
optional per-mapping transform.
|
||||
- `detectColumns` — union of keys across heterogeneous rows.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { parseCsv } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const rows = parseCsv("name,city\nAda,London\nBob,Paris");
|
||||
// [{ name: "Ada", city: "London" }, { name: "Bob", city: "Paris" }]
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { streamParseCsv } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const result = streamParseCsv("a,b\n1,2\n,3", {
|
||||
delimiter: ",",
|
||||
validate: (row, idx) => (row["a"] === "" ? `row ${idx}: a is required` : undefined),
|
||||
onError: (err) => logger.warn({ row: err.row, msg: err.message }, "drop"),
|
||||
});
|
||||
// result.errors → [{ row: 1, line: ",3", message: "row 1: a is required" }]
|
||||
// result.rows → [{ a: "1", b: "2" }]
|
||||
// result.validRows === 1
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `parseCsv(data: string): Record<string, string>[]`
|
||||
- `parseCsvLine(line: string): string[]`
|
||||
- `detectColumns(rows: Record<string, unknown>[]): string[]`
|
||||
- `parseStreamLine(line, delimiter, quoteChar): string[]`
|
||||
- `detectHeaders(firstLine, config: StreamParserConfig): string[]`
|
||||
- `streamParseCsv(data, config?): StreamParseResult`
|
||||
- `parseCsvWithHeaders(content, options?: { delimiter?; hasHeaders? }): CsvParseResult`
|
||||
- `applyColumnMappings(rows, mappings: ColumnMapping[]): Record<string, unknown>[]`
|
||||
- `interface StreamParserConfig { delimiter?; quoteChar?; hasHeaders?; onRow?; onError?; validate? }`
|
||||
- `interface StreamParseResult { headers; totalRows; validRows; errors; rows }`
|
||||
- `interface StreamRowError { row; line; message }`
|
||||
- `interface CsvParseResult { headers; rows; totalRows }`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Quote handling is single-state-machine (no regex backtracking) — O(n)
|
||||
in input length, no exponential pathological inputs.
|
||||
- `splitLines` keeps an in-quotes counter across newlines so RFC 4180
|
||||
multi-line records survive the split.
|
||||
- `applyColumnMappings` only invokes `transform` when the source value
|
||||
is a string — JSON-sourced numbers / booleans pass through unchanged.
|
||||
- The high-level `parseCsv` strips the header line by index (line 0);
|
||||
if the input contains only a header row, the function returns `[]`
|
||||
rather than `[{ header: undefined }]`.
|
||||
@@ -0,0 +1,59 @@
|
||||
# `parsers/json` — JSON parsers
|
||||
|
||||
## Purpose
|
||||
|
||||
Parses JSON-array input documents into the same `Record<string, unknown>[]`
|
||||
row shape produced by the CSV / XLSX parsers, so the canonical
|
||||
`ImportService` can branch on `format` and call a uniform parse function.
|
||||
The strict `parseJson` rejects non-array input with a deterministic error
|
||||
string; the high-level `parseJsonWithHeaders` wraps `parseJson` to
|
||||
produce the `{ headers, rows, totalRows }` shape consumed by the route
|
||||
preview pipeline.
|
||||
|
||||
Scalar elements in the input array (numbers, strings, `null`) are
|
||||
wrapped as `{ value: scalar }` rather than dropped, so the row count
|
||||
agrees with the input array length and the resulting rows still satisfy
|
||||
`Record<string, unknown>`.
|
||||
|
||||
## Features
|
||||
|
||||
- `parseJson` — strict array parsing; throws on non-array input.
|
||||
- `parseJsonWithHeaders` — header-aware result shape; tolerant of
|
||||
single-object inputs (wraps them in a one-element array).
|
||||
- Header inference unions keys across heterogeneous rows.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { parseJson } from "@suaveplan/api-plugin-import";
|
||||
const rows = parseJson('[{"a":1},{"a":2}]'); // [{ a: 1 }, { a: 2 }]
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { parseJsonWithHeaders } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const { headers, rows, totalRows } = parseJsonWithHeaders(
|
||||
'[{"a":1,"b":2},{"a":3,"c":4}]'
|
||||
);
|
||||
// headers: ["a","b","c"]
|
||||
// rows: [{ a: 1, b: 2 }, { a: 3, c: 4 }]
|
||||
// totalRows: 2
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `parseJson(data: string): Record<string, unknown>[]` — throws `Error("JSON data must be an array")` on non-array input.
|
||||
- `parseJsonWithHeaders(content: string): CsvParseResult` — header-aware variant.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `parseJson` uses `JSON.parse` directly — invalid JSON syntax surfaces
|
||||
as the native `SyntaxError` from the runtime, which is what the
|
||||
service catches and re-throws as `IMPORT.PROCESSING_FAILED`.
|
||||
- `parseJsonWithHeaders` wraps single-object inputs in a one-element
|
||||
array so the route's `format: "json"` branch never has to special-case
|
||||
shape — the field is always `rows[]`.
|
||||
- Scalar wrapping uses the literal key `value`; consumers that need a
|
||||
different key can map the result post-parse.
|
||||
@@ -0,0 +1,59 @@
|
||||
# `parsers/xlsx` — TSV-style XLSX parser
|
||||
|
||||
## Purpose
|
||||
|
||||
Provides a lightweight tab-separated-value parser exposed under the
|
||||
`xlsx` format so consumers can ingest spreadsheet-style exports without
|
||||
adding a SheetJS dependency. Binary `.xlsx` (the ZIP-packaged
|
||||
SpreadsheetML format) is explicitly out of scope; hosts that need
|
||||
binary support should wrap a SheetJS reader, render the cells to TSV,
|
||||
and call `parseXlsx` on the rendered string. Documenting the limitation
|
||||
inside the package keeps the dependency footprint small for the
|
||||
common case (TSV interchange dumps from data warehouses, manual exports
|
||||
from spreadsheet apps that produce a TSV variant).
|
||||
|
||||
## Features
|
||||
|
||||
- TSV split with optional header row.
|
||||
- `column_N` synthesised headers when `hasHeaders: false`.
|
||||
- Empty-line filtering — blank rows inside the input are dropped before
|
||||
parsing so consumers don't have to post-filter.
|
||||
- Per-cell `.trim()` so spreadsheet-pasted content with trailing tab
|
||||
whitespace produces clean values.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { parseXlsx } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const { headers, rows, totalRows } = parseXlsx("a\tb\n1\t2\n3\t4");
|
||||
// headers: ["a","b"], rows: [{ a: "1", b: "2" }, { a: "3", b: "4" }], totalRows: 2
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { parseXlsx } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const result = parseXlsx("1\tAlice\n2\tBob", { hasHeaders: false });
|
||||
// headers: ["column_1","column_2"]
|
||||
// rows: [{ column_1: "1", column_2: "Alice" }, { column_1: "2", column_2: "Bob" }]
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `parseXlsx(content: string, options?: { sheetName?: string; hasHeaders?: boolean }): CsvParseResult`
|
||||
- `sheetName` is accepted for API compatibility with future binary
|
||||
XLSX support but is ignored by the TSV implementation.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Uses `content.split(/\r?\n/)` so both LF and CRLF line endings are
|
||||
handled.
|
||||
- Empty-line filtering happens before header detection, so a file with
|
||||
a single blank line followed by a header row still produces the
|
||||
right header set.
|
||||
- Trailing-whitespace trim is per-cell, not per-row, so cells whose
|
||||
body intentionally starts or ends with whitespace are normalised. If
|
||||
preservation is required, drop in a custom parser via the
|
||||
`Parser`-shaped interface and skip `parseXlsx`.
|
||||
@@ -0,0 +1,85 @@
|
||||
# `plugin/plugin` — createImportPlugin factory
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the
|
||||
`@suaveplan/api-plugin` lifecycle. Validates the supplied
|
||||
options at the factory boundary via `ImportPluginOptionsSchema` so
|
||||
invalid input fails fast with `ImportError(IMPORT_CODES.INVALID_OPTIONS)`
|
||||
and the `ZodError` as `cause`. The descriptor's three lifecycle hooks
|
||||
construct the canonical `ImportService`, register it on the
|
||||
`ServiceRegistry`, mount the routes under the configured prefix, and
|
||||
clean up listeners on close.
|
||||
|
||||
The plugin opts-in to `auth`, `events`, and `database` via
|
||||
`optionalDependencies` so the topo-sort orders it after siblings that
|
||||
provide those services without hard-requiring them. Apps that don't
|
||||
ship auth still register; their `requireAuth()` calls simply throw
|
||||
`AUTH.UNAUTHENTICATED` per the auth plugin's contract.
|
||||
|
||||
## Features
|
||||
|
||||
- Zod-validated options at the factory boundary.
|
||||
- Owner-scoped `ImportService` constructed once in `onInit`.
|
||||
- `ServiceRegistry["importService"]` slot registration so consumer
|
||||
plugins can read the service via `ctx.getService("importService")`.
|
||||
- Routes mounted via `app.route(config.routePrefix, routes)` with a
|
||||
prefix-collision guard that throws if another plugin already owns
|
||||
the path.
|
||||
- `onClose` clears every listener attached to `service.bus`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createImportPlugin } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const importPlugin = createImportPlugin();
|
||||
// register with @suaveplan/api's plugin registry:
|
||||
// registry.register(importPlugin)
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { createImportPlugin, MemoryImportStore } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const store = new MemoryImportStore();
|
||||
const importPlugin = createImportPlugin({
|
||||
routePrefix: "/v2/imports",
|
||||
maxRows: 10_000,
|
||||
store,
|
||||
clock: () => Date.now(),
|
||||
logger: pinoLogger.child({ plugin: "import" }),
|
||||
});
|
||||
// store stays in scope so the host can introspect imported jobs
|
||||
// outside of the plugin lifecycle (e.g. background sweeps)
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `createImportPlugin(options?: CreateImportPluginOptions): PluginDescriptor<ImportPluginConfig>`
|
||||
- `type CreateImportPluginOptions = ImportPluginOptionsInput & Pick<ImportServiceOptions, "store" | "clock" | "idFactory" | "logger">`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The factory separates the Zod-validated config fields from the
|
||||
pass-through `ImportService` options (`store`, `clock`, `idFactory`,
|
||||
`logger`) so option drift between the plugin defaults and the
|
||||
service constructor stays explicit.
|
||||
- `mustService` throws if `onReady` runs before `onInit` constructed
|
||||
the service — this catches plugin-orchestrator regressions where
|
||||
hooks fire out of order.
|
||||
- The route-prefix collision guard mirrors the search-plugin
|
||||
precedent: a second mount under the same prefix silently shadows
|
||||
the first in Hono, so a hard fail at orchestration time is preferred
|
||||
to debugging a missing-route mystery in production.
|
||||
- `service.bus.clear()` removes both listeners and interceptors; the
|
||||
plugin owns the bus so a hard reset on close is safe.
|
||||
- `optionalDependencies` lists `"auth"` even though the routes hard-import
|
||||
`requireAuth` because the topo-sort only uses the dependency list for
|
||||
ordering — `requireAuth` works correctly even when the auth plugin
|
||||
hasn't run yet (it simply throws `AUTH.UNAUTHENTICATED`).
|
||||
@@ -0,0 +1,83 @@
|
||||
# `progress/tracker` — real-time import progress
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Provides the `ImportProgressTracker` class consumed by the canonical
|
||||
`ImportService` (and exported for direct use by CLI tools / batch
|
||||
workers) to track import jobs in real time. The tracker counts
|
||||
successes, failures, and overall progress, and exposes derived metrics
|
||||
(rows per second, percent complete, estimated remaining time) without
|
||||
requiring the caller to keep track of the wall clock themselves.
|
||||
|
||||
The clock is injectable so test suites can pin deterministic ETA /
|
||||
rows-per-sec assertions without sleeping. Without a clock override the
|
||||
tracker uses `Date.now`, so the production hot path is allocation-free
|
||||
after construction.
|
||||
|
||||
## Features
|
||||
|
||||
- Single-row (`recordSuccess`, `recordFailure`) and batched
|
||||
(`recordBatchSuccess(count)`, `recordBatchFailure(count)`) updates.
|
||||
- Listener fan-out via `onProgress(callback)` — returns an unsubscribe
|
||||
function so consumers can detach cleanly.
|
||||
- Snapshot shape (`ProgressSnapshot`) carries `rowsPerSecond` rounded to
|
||||
2 decimal places, `percentComplete` rounded to an integer, and
|
||||
`estimatedRemainingMs` rounded to an integer.
|
||||
- Injectable `Clock` from `@suaveplan/types` for deterministic tests.
|
||||
- `isComplete()` predicate flips once `processedRows >= totalRows` —
|
||||
handles `totalRows === 0` as immediately complete.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { ImportProgressTracker } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const tracker = new ImportProgressTracker("job-1", 100);
|
||||
tracker.onProgress((snap) => console.log(snap.percentComplete));
|
||||
tracker.recordBatchSuccess(50); // logs 50
|
||||
tracker.recordBatchFailure(10); // logs 60
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { ImportProgressTracker } from "@suaveplan/api-plugin-import";
|
||||
|
||||
// Deterministic test with an injected clock — advance time by a known delta
|
||||
let now = 1_000;
|
||||
const tracker = new ImportProgressTracker("j", 100, { clock: () => now });
|
||||
|
||||
tracker.recordBatchSuccess(50);
|
||||
now += 1_000;
|
||||
const snap = tracker.snapshot();
|
||||
// snap.rowsPerSecond === 50, snap.estimatedRemainingMs === 1_000
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `class ImportProgressTracker`
|
||||
- `constructor(jobId: string, totalRows: number, options?: ImportProgressTrackerOptions)`
|
||||
- `onProgress(callback: ProgressCallback): () => void`
|
||||
- `recordSuccess(): void`
|
||||
- `recordFailure(): void`
|
||||
- `recordBatchSuccess(count: number): void`
|
||||
- `recordBatchFailure(count: number): void`
|
||||
- `snapshot(): ProgressSnapshot`
|
||||
- `isComplete(): boolean`
|
||||
- `interface ImportProgressTrackerOptions { clock?: Clock }`
|
||||
- `interface ProgressSnapshot { jobId; totalRows; processedRows; successCount; failedCount; percentComplete; rowsPerSecond; estimatedRemainingMs; elapsedMs; startedAt }`
|
||||
- `type ProgressCallback = (snap: ProgressSnapshot) => void`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `totalRows === 0` short-circuits `percentComplete` to `0` and
|
||||
`estimatedRemainingMs` to `0` to avoid `NaN` from a zero divisor.
|
||||
- The listener array is replaced (not spliced) on `off()` so a
|
||||
listener removing itself from inside its own callback does not
|
||||
invalidate the in-flight iteration.
|
||||
- `startedAtIso` is captured once in the constructor and re-used on
|
||||
every snapshot to avoid `new Date(...)` allocations on the hot path.
|
||||
@@ -0,0 +1,69 @@
|
||||
# `routes/build-routes` — auth-gated import routes
|
||||
|
||||
## Purpose
|
||||
|
||||
Constructs the `OpenAPIHono` router mounted by `createImportPlugin` at
|
||||
the configured `routePrefix`. Five routes are registered — `POST
|
||||
/preview`, `POST /`, `GET /`, `GET /{id}`, `DELETE /{id}` — each gated
|
||||
behind `requireAuth()` from `@suaveplan/api-plugin-auth` so anonymous
|
||||
requests return 401 problem+json before they ever reach a handler.
|
||||
Owner-scoped routes read `c.get("user").id` and pass it through to the
|
||||
canonical `ImportService`, which raises `IMPORT.NOT_FOUND` (mapped to
|
||||
404 problem+json) on foreign-owner lookups so the surface is
|
||||
anti-enumeration-safe.
|
||||
|
||||
## Features
|
||||
|
||||
- Five OpenAPI routes with full Zod request / response schemas.
|
||||
- Per-route `app.use("/*", requireAuth())` guard.
|
||||
- `progressToResponse()` shape mapper so the response body always
|
||||
satisfies `ImportJobResponseSchema`.
|
||||
- Every handler funnels error paths through `respondProblemJson` so
|
||||
the response shape is uniform across success and failure.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createImportRoutes, ImportService } from "@suaveplan/api-plugin-import";
|
||||
import { OpenAPIHono } from "@hono/zod-openapi";
|
||||
|
||||
const service = new ImportService();
|
||||
const app = new OpenAPIHono();
|
||||
app.route("/imports", createImportRoutes({ service }));
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { createImportRoutes, ImportService, MemoryImportStore } from "@suaveplan/api-plugin-import";
|
||||
import { OpenAPIHono } from "@hono/zod-openapi";
|
||||
|
||||
const store = new MemoryImportStore();
|
||||
const service = new ImportService({ store });
|
||||
service.bus.on("import.progress", (snap) => sse.broadcast(snap));
|
||||
|
||||
const app = new OpenAPIHono();
|
||||
app.use("/imports/*", customRateLimit({ window: "1m", max: 10 }));
|
||||
app.route("/imports", createImportRoutes({ service }));
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `createImportRoutes(deps: ImportRoutesDeps): OpenAPIHono`
|
||||
- `interface ImportRoutesDeps { service: ImportService }`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The `requireAuth()` middleware is mounted via `app.use("/*", ...)` so
|
||||
every route inherits the guard. Mounting per-route would duplicate
|
||||
the wiring and miss the OpenAPI catch-all.
|
||||
- `ownerOf(c)` returns the empty string when `c.get("user")` is null;
|
||||
the auth middleware throws BEFORE the handler runs, so the empty
|
||||
string is unreachable from production traffic — but it remains a
|
||||
safe default if a caller somehow bypasses the middleware in a test
|
||||
harness.
|
||||
- The list endpoint returns `{ items, total }` rather than a raw array
|
||||
so clients can paginate; the schema enforces `total >= 0`.
|
||||
- The delete endpoint surfaces a `{ deleted: boolean }` payload — a
|
||||
successful delete returns `{ deleted: true }`; a missing job throws
|
||||
`IMPORT.NOT_FOUND` from the service layer and is rendered as 404.
|
||||
@@ -0,0 +1,70 @@
|
||||
# `routes/problem-json` — RFC 7807 problem+json helper
|
||||
|
||||
## Purpose
|
||||
|
||||
Bridges thrown errors from the route handlers into RFC 7807
|
||||
`application/problem+json` responses. Reads the HTTP status from the
|
||||
error's `context.statusCode` when present so the canonical
|
||||
`ImportError` raised by the service / plugin layer surfaces with the
|
||||
correct HTTP code without a code → status lookup table in the router.
|
||||
|
||||
Unknown throwables fall back to a generic 500 so the router never
|
||||
crashes — the renderer is the safety net for non-`SuaveplanError`
|
||||
exceptions that escape upstream code.
|
||||
|
||||
## Features
|
||||
|
||||
- `respondProblemJson(c, err)` — writes the `Response` to the Hono
|
||||
context with `content-type: application/problem+json`.
|
||||
- `renderProblem(err)` — pure render returning a `ProblemDetails`
|
||||
object (used by tests + alt transports).
|
||||
- `PROBLEM_JSON_CONTENT_TYPE` constant for downstream re-use.
|
||||
- Native `Error` paths preserve the `name` so debugging upstream
|
||||
failures stays readable.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { respondProblemJson } from "@suaveplan/api-plugin-import";
|
||||
|
||||
app.get("/foo", (c) => {
|
||||
try {
|
||||
throw new ImportError(IMPORT_CODES.NOT_FOUND, "missing", {
|
||||
context: { statusCode: 404 },
|
||||
});
|
||||
} catch (err) {
|
||||
return respondProblemJson(c, err);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { renderProblem } from "@suaveplan/api-plugin-import";
|
||||
|
||||
// Pure render for use in alternate transports (gRPC adapter, queue worker)
|
||||
const body = renderProblem(new Error("boom"));
|
||||
// body === { type: "urn:suaveplan:error:internal", title: "Error", status: 500, detail: "boom" }
|
||||
const message = JSON.stringify(body);
|
||||
publishToDeadLetterQueue(message);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `respondProblemJson(c: HonoLike, err: unknown): Response`
|
||||
- `renderProblem(err: unknown): ProblemDetails`
|
||||
- `interface HonoLike { header(name, value); body(data, status): Response }`
|
||||
- `interface ProblemDetails { type; title; status; detail; code?; [k: string]: unknown }`
|
||||
- `const PROBLEM_JSON_CONTENT_TYPE = "application/problem+json"`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The `type` field is built as `urn:suaveplan:error:<lowercased-code>`
|
||||
so consumers can branch on the URN without parsing the body.
|
||||
- Status hints below 400 (or missing) collapse to 500 — clients should
|
||||
never receive a problem+json body with a 2xx status code.
|
||||
- The `Error.name` fallback (`"InternalError"`) protects against
|
||||
thrown objects whose `name` was wiped by a downstream serializer.
|
||||
- `renderProblem` returns a `ProblemDetails` literal so consumers that
|
||||
need to merge additional fields (e.g. `instance`) can spread it.
|
||||
@@ -0,0 +1,124 @@
|
||||
# `routes/progress-sse` — Server-Sent Events progress push
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Exposes `GET /imports/:id/progress` returning a long-lived
|
||||
`text/event-stream` so clients can subscribe to a job's real-time
|
||||
progress without polling. The handler is driven entirely off the
|
||||
existing `ImportService.bus` — three subscriptions (`import.progress`,
|
||||
`import.completed`, `import.failed`) are wired into a
|
||||
`ReadableStream<Uint8Array>` and detached on terminal events or client
|
||||
abort. This module was deferred from the original
|
||||
`add-api-plugin-import` migration's `Excluded` list and ships
|
||||
independently via the `add-import-sse-progress` change.
|
||||
|
||||
## Features
|
||||
|
||||
- Three SSE event kinds — `progress` (per `import.progress` tick),
|
||||
`complete` (terminal, on `import.completed`), `failed` (terminal, on
|
||||
`import.failed`).
|
||||
- Initial frame snapshot from `service.getJob` so clients see current
|
||||
state before the next bus tick.
|
||||
- Owner-scoped — foreign-owner / missing-job lookups return HTTP 404
|
||||
RFC 7807 problem+json (`IMPORT.NOT_FOUND`); anonymous requests return
|
||||
HTTP 401 via the existing `requireAuth()` middleware.
|
||||
- Per-job filtering on every subscription callback so a long-lived
|
||||
connection cannot leak progress for other jobs.
|
||||
- `AbortSignal` lifecycle — client disconnect detaches every listener
|
||||
and emits `import.sse.disconnected{reason: "client"}`.
|
||||
- OTel observability — `import.sse.connected`,
|
||||
`import.sse.events_sent{kind}`, `import.sse.disconnected{reason}`.
|
||||
- Pure stream — no polling timer, no background interval, no
|
||||
heartbeat (job runs are short-lived so proxy-keepalive concerns are
|
||||
handled by upstream LB tuning, not by the handler).
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createImportRoutes, ImportService } from "@suaveplan/api-plugin-import";
|
||||
import { Hono } from "hono";
|
||||
|
||||
const service = new ImportService();
|
||||
const host = new Hono();
|
||||
host.use("*", async (c, next) => {
|
||||
c.set("user", { id: "alice" });
|
||||
await next();
|
||||
});
|
||||
host.route("/imports", createImportRoutes({ service }));
|
||||
|
||||
// Client side:
|
||||
const es = new EventSource("/imports/job-1/progress");
|
||||
es.addEventListener("progress", (e) => console.log(JSON.parse(e.data)));
|
||||
es.addEventListener("complete", () => es.close());
|
||||
es.addEventListener("failed", () => es.close());
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
createProgressSseHandler,
|
||||
ImportService,
|
||||
} from "@suaveplan/api-plugin-import";
|
||||
import { Hono } from "hono";
|
||||
|
||||
const service = new ImportService();
|
||||
const app = new Hono();
|
||||
app.use("*", async (c, next) => {
|
||||
c.set("user", { id: "alice" });
|
||||
await next();
|
||||
});
|
||||
// Mount stand-alone (without the rest of `createImportRoutes`).
|
||||
app.get("/jobs/:id/progress", createProgressSseHandler({ service }));
|
||||
|
||||
// Drive the service from a worker; every emitted progress event is
|
||||
// fanned to every connected client whose `:id` matches.
|
||||
const controller = new AbortController();
|
||||
const res = await app.request(
|
||||
new Request("http://t/jobs/job-1/progress", { signal: controller.signal })
|
||||
);
|
||||
const reader = res.body!.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let buf = "";
|
||||
while (true) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
buf += decoder.decode(value, { stream: true });
|
||||
const frames = buf.split("\n\n");
|
||||
buf = frames.pop() ?? "";
|
||||
for (const frame of frames) console.log(frame);
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `createProgressSseHandler(deps: ProgressSseDeps): (c: Context) => Response`
|
||||
- `formatProgressFrame(kind: ProgressSseEventKind, seq: number, payload: ImportProgress): string`
|
||||
- `interface ProgressSseDeps { service: ImportService }`
|
||||
- `type ProgressSseEventKind = "progress" | "complete" | "failed"`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The handler is registered via `app.get("/:id/progress", ...)` rather
|
||||
than `app.openapi(...)` because `c.body(ReadableStream, ...)` does not
|
||||
conform to a static OpenAPI response schema — the parent router's
|
||||
`app.use("/*", requireAuth())` and `app.onError(...)` still apply.
|
||||
- Subscriptions are filtered by `payload.jobId === jobId` so the bus
|
||||
cannot leak foreign-job progress to a connection. The router's
|
||||
authoritative ownership check happens once via `service.getJob` at
|
||||
connection-open time; subsequent leak prevention is structural.
|
||||
- The terminal `complete` / `failed` branch in `start` handles the
|
||||
edge case where a job reached terminal state between `getJob` and
|
||||
the bus subscription — without it, a client would be stuck on the
|
||||
initial `progress` frame forever.
|
||||
- The OTel counters are read through `getImportInstruments()` rather
|
||||
than cached at module scope so `resetImportInstruments()` (used by
|
||||
test setup) rebinds correctly.
|
||||
- The response headers (`Cache-Control: no-cache, no-transform`,
|
||||
`X-Accel-Buffering: no`, `Connection: keep-alive`) match the
|
||||
`@suaveplan/api-plugin-sse` handler so reverse proxies (nginx,
|
||||
cloudflare) do not buffer or coalesce the stream.
|
||||
@@ -0,0 +1,67 @@
|
||||
# `schemas/options` — plugin options schema
|
||||
|
||||
## Purpose
|
||||
|
||||
Declares the Zod schema for the options accepted by
|
||||
`createImportPlugin`. Parsing options at the factory boundary
|
||||
guarantees the resolved config (`ImportPluginConfig`) carries no
|
||||
optional fields — every downstream consumer (`onInit`, `onReady`,
|
||||
routes, service) reads from a fully-resolved object without `?? default`
|
||||
fallbacks scattered across the call sites.
|
||||
|
||||
Defaults are documented inside the schema (`.default(...)`) so the Zod
|
||||
parse output is the authoritative source of plugin defaults. Renaming a
|
||||
field is a breaking change.
|
||||
|
||||
## Features
|
||||
|
||||
- Six options with sensible production defaults: `routePrefix`,
|
||||
`maxFileSizeBytes`, `maxRows`, `maxErrorsBeforeAbort`, `previewRows`,
|
||||
`defaultBatchSize`.
|
||||
- Two TypeScript shapes — `ImportPluginOptionsInput` for caller-supplied
|
||||
partials, `ImportPluginConfig` for the resolved config.
|
||||
- Strict positive-integer / non-empty-string constraints surface bad
|
||||
config as `IMPORT.INVALID_OPTIONS` with the `ZodError` as `cause`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { ImportPluginOptionsSchema } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const parsed = ImportPluginOptionsSchema.parse({});
|
||||
// parsed.routePrefix === "/imports"
|
||||
// parsed.maxFileSizeBytes === 52_428_800
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { ImportPluginOptionsSchema, type ImportPluginConfig } from "@suaveplan/api-plugin-import";
|
||||
|
||||
function configureImport(overrides: Partial<ImportPluginConfig>): ImportPluginConfig {
|
||||
return ImportPluginOptionsSchema.parse({
|
||||
routePrefix: "/v2/imports",
|
||||
maxRows: 10_000,
|
||||
...overrides,
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `ImportPluginOptionsSchema: z.ZodObject<…>` — Zod schema with
|
||||
defaults for every field.
|
||||
- `type ImportPluginOptionsInput` — caller-supplied partial input.
|
||||
- `type ImportPluginConfig` — resolved options (every field
|
||||
non-optional).
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- All numeric bounds are `int().positive()` because the consuming
|
||||
fields are counts; the schema rejects zero with the standard Zod
|
||||
error message.
|
||||
- `previewRows` is capped at 1 000 to prevent a malicious client from
|
||||
triggering a multi-megabyte preview render.
|
||||
- `routePrefix` is non-empty; an empty prefix collapses the mount and
|
||||
causes Hono route shadowing — the Zod gate prevents this footgun at
|
||||
factory time.
|
||||
@@ -0,0 +1,81 @@
|
||||
# `schemas/schemas` — public Zod request/response surface
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Carries the full public Zod surface used by the route handlers and any
|
||||
external client (CLI tool, integration test, browser app) that wants to
|
||||
validate or generate import requests / responses without depending on
|
||||
the route module itself. Every body, query, and response shape that
|
||||
crosses the HTTP boundary is declared here so the OpenAPI document and
|
||||
the runtime validators stay in lockstep.
|
||||
|
||||
Keeping these schemas in their own module means consumers can import
|
||||
them without pulling in the Hono/OpenAPI route builder, which is
|
||||
important for browser clients and shared validation libraries.
|
||||
|
||||
## Features
|
||||
|
||||
- `ImportFormatSchema` / `ImportStatusSchema` — Zod enums mirroring the
|
||||
cross-module type unions.
|
||||
- `ColumnMappingSchema` — wire-form of a column mapping (functions
|
||||
intentionally omitted; transforms are server-side only).
|
||||
- `PreviewRequestSchema` / `PreviewResponseSchema` — `POST /preview`.
|
||||
- `CreateImportJobSchema` — `POST /` body.
|
||||
- `ImportJobResponseSchema` — every job-shape response.
|
||||
- `ImportJobListQuerySchema` — `GET /` querystring with `z.coerce`
|
||||
for the numeric fields so query strings parse without hand-rolling.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { CreateImportJobSchema } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const result = CreateImportJobSchema.safeParse({
|
||||
format: "csv",
|
||||
data: "name\nAda",
|
||||
mappings: [{ sourceColumn: "name", targetField: "displayName" }],
|
||||
});
|
||||
if (!result.success) throw new Error(result.error.message);
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { ImportJobResponseSchema, type ImportJobResponseShape } from "@suaveplan/api-plugin-import";
|
||||
|
||||
async function fetchJob(id: string): Promise<ImportJobResponseShape> {
|
||||
const res = await fetch(`/imports/${id}`);
|
||||
const json = await res.json();
|
||||
return ImportJobResponseSchema.parse(json);
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `ImportFormatSchema`, `type ImportFormatInput`
|
||||
- `ImportStatusSchema`, `type ImportStatusInput`
|
||||
- `ColumnMappingSchema`, `type ColumnMappingInput`
|
||||
- `PreviewRequestSchema`, `type PreviewRequestInput`
|
||||
- `PreviewResponseSchema`, `type PreviewResponseShape`
|
||||
- `CreateImportJobSchema`, `type CreateImportJobInput`
|
||||
- `ImportJobResponseSchema`, `type ImportJobResponseShape`
|
||||
- `ImportJobListQuerySchema`, `type ImportJobListQueryInput`
|
||||
- `RowErrorSchema`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `ColumnMappingSchema` omits the `transform` callback — transforms are
|
||||
authored in TypeScript on the server, not over the wire. Wire-form
|
||||
mappings are pure renames + required flags.
|
||||
- `PreviewResponseSchema.sampleRows` uses `z.record(z.string(),
|
||||
z.unknown())` so heterogeneous row shapes survive validation.
|
||||
- `ImportJobListQuerySchema` uses `z.coerce.number()` because Hono
|
||||
delivers query parameters as strings — the coerce ensures `?limit=20`
|
||||
parses to a number without manual `parseInt`.
|
||||
- `CreateImportJobSchema.mappings` requires `min(1)` because zero
|
||||
mappings is the service-layer `IMPORT.VALIDATION` failure; rejecting
|
||||
it at the Zod boundary saves a service round-trip.
|
||||
@@ -0,0 +1,115 @@
|
||||
# `service/service` — canonical ImportService
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Implements the canonical `ImportService` used by `createImportPlugin`
|
||||
and exported for direct programmatic use. Owns the per-job in-memory
|
||||
progress map, the optional durable store, the `EventBus<ImportEventMap>`
|
||||
event surface, and the OTel emit on the production hot path. Every read
|
||||
and write is owner-scoped: a job belongs to exactly one `ownerId`, and
|
||||
foreign-owner lookups return `IMPORT.NOT_FOUND` (anti-enumeration) — not
|
||||
`IMPORT.FORBIDDEN` — so callers cannot probe for the existence of other
|
||||
users' jobs.
|
||||
|
||||
The service is the only place that ties together parsers, transform
|
||||
pipeline (via `applyMappings`), progress emission, and persistence; the
|
||||
plugin layer is a thin wrapper that constructs it once per `onInit`
|
||||
and mounts the routes.
|
||||
|
||||
## Features
|
||||
|
||||
- `preview(format, data)` — returns column auto-mapping + a 5-row
|
||||
sample.
|
||||
- `startImport(config, { ownerId })` — parses, maps, optionally
|
||||
dedups, optionally runs `onRow`, emits per-row events, batch progress
|
||||
updates, and OTel counters / histogram.
|
||||
- `getJob` / `getResults` / `listJobs` / `deleteJob` — owner-scoped.
|
||||
- `bus: EventBus<ImportEventMap>` — public so consumers can attach
|
||||
listeners without subclassing.
|
||||
- Injectable `store`, `clock`, `idFactory`, `logger`, `instruments` for
|
||||
testability.
|
||||
- RFC 7807 status hints — every thrown `ImportError` carries
|
||||
`context.statusCode` so the route renderer doesn't need a code→status
|
||||
lookup table.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { ImportService } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const service = new ImportService();
|
||||
const preview = service.preview("csv", "name\nAda");
|
||||
const progress = await service.startImport(
|
||||
{
|
||||
format: "csv",
|
||||
data: "name\nAda",
|
||||
mappings: [{ sourceColumn: "name", targetField: "displayName" }],
|
||||
},
|
||||
{ ownerId: "alice" }
|
||||
);
|
||||
// progress.status === "completed"
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { ImportService, MemoryImportStore } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const store = new MemoryImportStore();
|
||||
const service = new ImportService({
|
||||
store,
|
||||
clock: () => 1_700_000_000_000,
|
||||
idFactory: () => "test-job",
|
||||
});
|
||||
|
||||
service.bus.on("import.progress", (snap) => console.log(snap.percentComplete));
|
||||
|
||||
await service.startImport(
|
||||
{
|
||||
format: "json",
|
||||
data: '[{"x":1},{"x":2},{"x":1}]',
|
||||
mappings: [{ sourceColumn: "x", targetField: "value" }],
|
||||
skipDuplicates: true,
|
||||
},
|
||||
{ ownerId: "alice" }
|
||||
);
|
||||
// `test-job` is now in the store; results array dedups to 2 entries
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `class ImportService`
|
||||
- `constructor(options?: ImportServiceOptions)`
|
||||
- `bus: EventBus<ImportEventMap>`
|
||||
- `preview(format: ImportFormat, data: string): ImportPreview`
|
||||
- `startImport(config: ImportJobConfig, owner: ImportOwnerContext): Promise<ImportProgress>`
|
||||
- `getJob(jobId: string, ownerId: string): ImportProgress`
|
||||
- `getResults(jobId: string, ownerId: string): Record<string, unknown>[]`
|
||||
- `listJobs(ownerId: string, filter?: ImportStoreListFilter): ImportProgress[]`
|
||||
- `deleteJob(jobId: string, ownerId: string): Promise<boolean>`
|
||||
- `interface ImportServiceOptions { store?; clock?; idFactory?; logger?; instruments? }`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The service stores progress in two `Map`s (live progress + imported
|
||||
rows). Persistence is best-effort; a store failure throws
|
||||
`IMPORT.STORAGE_ERROR` but the in-memory progress is still consistent.
|
||||
- `onRow` is invoked AFTER `skipDuplicates` so a duplicate row never
|
||||
reaches the caller's filter. Returning `false` skips the row but does
|
||||
NOT mark it as an error.
|
||||
- The batch-progress emit cadence is keyed on `(i + 1) % batchSize === 0`
|
||||
so a batch size of 100 emits at rows 100, 200, … plus the final
|
||||
completion emit. This matches the upstream behaviour exactly.
|
||||
- Foreign-owner reads on `getJob` throw `NOT_FOUND` rather than
|
||||
`FORBIDDEN` so an attacker cannot distinguish "no such job" from
|
||||
"your access is denied."
|
||||
- The XLSX path is rendered inline (not via `parseXlsx`) to keep the
|
||||
return shape `Record<string, unknown>[]` instead of `CsvParseResult`;
|
||||
call `parseXlsx` directly if you need headers metadata.
|
||||
- `parseData` re-raises `ImportError` unchanged but wraps every other
|
||||
throwable as `IMPORT.PROCESSING_FAILED` so consumers can branch on
|
||||
one code regardless of which parser failed.
|
||||
@@ -0,0 +1,83 @@
|
||||
# `services/registry-augmentation` — ServiceRegistry augmentation
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Augments the `ServiceRegistry` interface from
|
||||
`@suaveplan/api-plugin/service-registry` with the `importService` slot
|
||||
so any consumer plugin typed against the augmented registry can call
|
||||
`ctx.getService("importService")` and receive a typed
|
||||
`ImportService | undefined` without a cast.
|
||||
|
||||
The augmentation also re-declares the host-side `app` slot owned by
|
||||
`@suaveplan/api` as a cumulative augmentation. TypeScript merges
|
||||
duplicate declarations, so consumers don't need to explicitly import
|
||||
`@suaveplan/api`'s augmentation module just to resolve `app` from
|
||||
inside our plugin's lifecycle hooks.
|
||||
|
||||
## Features
|
||||
|
||||
- One slot added — `importService: ImportService`.
|
||||
- Cumulative re-declaration of `app: OpenAPIHono` so the import
|
||||
plugin's `onReady` can resolve the host app without forcing a
|
||||
consumer-side import.
|
||||
- Side-effect import; the `services/index.ts` barrel re-exports it via
|
||||
`import "./registry-augmentation.js"`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
// Anywhere consuming the import plugin
|
||||
import "@suaveplan/api-plugin-import"; // pulls in the augmentation
|
||||
import type { PluginContext } from "@suaveplan/api-plugin";
|
||||
|
||||
function readService(ctx: PluginContext): void {
|
||||
const svc = ctx.getService("importService"); // typed as ImportService | undefined
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
// Direct module-augmentation chain for downstream extension authors
|
||||
import "@suaveplan/api-plugin-import"; // adds importService
|
||||
import "@suaveplan/api-plugin-database"; // adds database services
|
||||
|
||||
import type { PluginContext } from "@suaveplan/api-plugin";
|
||||
|
||||
function bootstrap(ctx: PluginContext): void {
|
||||
const svc = ctx.getService("importService");
|
||||
const db = ctx.getService("databasePool");
|
||||
if (svc && db) {
|
||||
// wire a custom adapter that writes via the database pool
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
This module exports nothing at runtime — it is pure TypeScript module
|
||||
augmentation:
|
||||
|
||||
```ts
|
||||
declare module "@suaveplan/api-plugin/service-registry" {
|
||||
interface ServiceRegistry {
|
||||
importService: ImportService;
|
||||
app: OpenAPIHono;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The augmentation is a `declare module` statement, not a value export;
|
||||
treeshaking under `sideEffects: false` keeps it in the bundle because
|
||||
TypeScript module-augmentation has no runtime correlate.
|
||||
- The `app` re-declaration intentionally mirrors the host package; the
|
||||
TypeScript compiler merges identical interface declarations across
|
||||
modules, so the duplicate is type-safe.
|
||||
- A consumer that DOES import `@suaveplan/api`'s registry directly will
|
||||
see only one merged slot — there is no double-declaration error.
|
||||
@@ -0,0 +1,95 @@
|
||||
# `store/memory` — in-memory import job store
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Declares the async `ImportStoreAdapter` interface and ships the
|
||||
`MemoryImportStore` implementation used by the canonical `ImportService`
|
||||
to persist `ImportJobRecord` entries across the run / lookup boundary.
|
||||
The interface is the seam for swapping in durable backends (Surreal,
|
||||
Postgres, Redis-backed cache) without touching the service. The
|
||||
in-memory implementation is sufficient for tests and single-process
|
||||
deployments; production hosts implement the same interface against a
|
||||
real database.
|
||||
|
||||
The store is intentionally minimal — five methods, no transactions, no
|
||||
optimistic concurrency. The service layer is the single writer per
|
||||
process; cross-process coordination is the database's responsibility
|
||||
once a real backend lands.
|
||||
|
||||
## Features
|
||||
|
||||
- Five-method async interface (`create`, `get`, `update`, `delete`,
|
||||
`list`) matching the upstream surface.
|
||||
- `update` on a missing id throws with `Job "<id>" not found` so the
|
||||
service layer can map the failure into `IMPORT.STORAGE_ERROR`.
|
||||
- `list` filters by `status` and `ownerId` and honours `limit` / `offset`
|
||||
paging — defaults `limit: 50`, `offset: 0`.
|
||||
- Injectable `Clock` for deterministic `updatedAt` stamps.
|
||||
- `size()` / `clear()` test helpers for cheap setup/teardown.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { MemoryImportStore } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const store = new MemoryImportStore();
|
||||
await store.create({
|
||||
id: "j1",
|
||||
ownerId: "alice",
|
||||
status: "created",
|
||||
totalRows: 0,
|
||||
processedRows: 0,
|
||||
successCount: 0,
|
||||
errorCount: 0,
|
||||
errors: [],
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { MemoryImportStore, type ImportStoreAdapter } from "@suaveplan/api-plugin-import";
|
||||
|
||||
// Inject a fixed clock so the updatedAt is reproducible in tests.
|
||||
const clock = () => 1_700_000_000_000;
|
||||
const store: ImportStoreAdapter = new MemoryImportStore({ clock });
|
||||
|
||||
await store.create(record);
|
||||
await store.update(record.id, { status: "completed" });
|
||||
const fresh = await store.get(record.id);
|
||||
// fresh.updatedAt === new Date(1_700_000_000_000).toISOString()
|
||||
|
||||
const recent = await store.list({ status: "completed", ownerId: "alice", limit: 10 });
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `interface ImportStoreAdapter`
|
||||
- `create(job: ImportJobRecord): Promise<void>`
|
||||
- `get(id: string): Promise<ImportJobRecord | undefined>`
|
||||
- `update(id: string, patch: Partial<ImportJobRecord>): Promise<void>`
|
||||
- `delete(id: string): Promise<boolean>`
|
||||
- `list(filter?: ImportStoreListFilter): Promise<ImportJobRecord[]>`
|
||||
- `interface ImportStoreListFilter { status?; ownerId?; limit?; offset? }`
|
||||
- `class MemoryImportStore implements ImportStoreAdapter`
|
||||
- `interface MemoryImportStoreOptions { clock? }`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The map keys are job ids — collisions overwrite (mirrors the upstream
|
||||
surface) so callers MUST generate unique ids per job.
|
||||
- `delete` returns `false` for missing ids (mirrors `Map.delete`); no
|
||||
exception is raised because the route handler treats a missing job
|
||||
as a 404, not a 500.
|
||||
- `update` re-uses the patch's `updatedAt` when supplied so callers
|
||||
controlling the clock can stamp deterministic timestamps without
|
||||
swapping out the entire store.
|
||||
- `list` does NOT mutate the underlying map; the returned array is a
|
||||
fresh slice of cloned references, so callers can mutate the result
|
||||
without poisoning the store.
|
||||
@@ -0,0 +1,155 @@
|
||||
# `store/surreal/surreal-import-store` — SurrealDB-backed ImportStoreAdapter
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Implements the canonical `ImportStoreAdapter` interface against a connected
|
||||
`SurrealAdapter` from `@suaveplan/surrealdb`, providing the durable counterpart
|
||||
to the in-process `MemoryImportStore`. The store stores every `ImportJobRecord`
|
||||
in a single `import_job` table (configurable) and uses positional SurrealQL
|
||||
placeholders (`$p0`, `$p1`, …) so it matches the suaveplan adapter contract —
|
||||
the same pattern used by `@suaveplan/api-plugin-events`'s `EventRepository`.
|
||||
|
||||
The store is a drop-in replacement for `MemoryImportStore`: every public method
|
||||
preserves the original semantics (including the verbatim
|
||||
`Job "<id>" not found` rejection on `update` against a missing id) so callers
|
||||
that selected the in-memory adapter for development can swap to SurrealDB in
|
||||
production without changing call sites.
|
||||
|
||||
## Features
|
||||
|
||||
- Full `ImportStoreAdapter` parity — `create`, `get`, `update`, `delete`, `list`.
|
||||
- Positional SurrealQL placeholders (`$p0`, `$p1`, …) matching the suaveplan
|
||||
`DatabaseAdapter.query(sql, params: readonly unknown[])` surface.
|
||||
- Zod-validated constructor config — bad input throws
|
||||
`ImportError(IMPORT_CODES.INVALID_OPTIONS)` with the `ZodError` as `cause`.
|
||||
- Adapter failures are wrapped as `ImportError(IMPORT_CODES.STORAGE_ERROR)`
|
||||
with the original throwable preserved as `cause` and the failing SurQL +
|
||||
operation captured in `error.context`.
|
||||
- Configurable table name (`table?: string`, defaults to `"import_job"`).
|
||||
- Injectable `Clock` for deterministic `updatedAt` stamping in `update`.
|
||||
- Emits `import.store.writes_total{outcome}` on every write and
|
||||
`import.store.reads_total{outcome}` on every read; a `get` that returns
|
||||
`undefined` is still counted as a successful read.
|
||||
- Snake-case ↔ camel-case mapping built in (SurrealDB columns are snake_case
|
||||
per the established `api-plugin-events` schema convention; JavaScript
|
||||
records remain camelCase).
|
||||
- No mocks — unit tests use a fake `SurrealAdapter` that captures SQL +
|
||||
params; integration tests run against a real SurrealDB testcontainer.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { SurrealAdapter } from "@suaveplan/surrealdb";
|
||||
import {
|
||||
SurrealImportStore,
|
||||
type ImportJobRecord,
|
||||
} from "@suaveplan/api-plugin-import";
|
||||
|
||||
const db = new SurrealAdapter({
|
||||
url: "ws://surreal:8000/rpc",
|
||||
namespace: "app",
|
||||
database: "imports",
|
||||
username: "root",
|
||||
password: "root",
|
||||
});
|
||||
await db.connect();
|
||||
|
||||
const store = new SurrealImportStore({ db });
|
||||
|
||||
const record: ImportJobRecord = {
|
||||
id: "j1",
|
||||
ownerId: "alice",
|
||||
status: "created",
|
||||
totalRows: 0,
|
||||
processedRows: 0,
|
||||
successCount: 0,
|
||||
errorCount: 0,
|
||||
errors: [],
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date().toISOString(),
|
||||
};
|
||||
await store.create(record);
|
||||
const fetched = await store.get("j1");
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Swap in a custom table name, inject a deterministic clock for tests, and pull
|
||||
the OTel bundle eagerly so the store never re-resolves the meter on hot paths:
|
||||
|
||||
```ts
|
||||
import { getImportInstruments, SurrealImportStore } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const fixedNow = 1_700_000_000_000;
|
||||
const store = new SurrealImportStore({
|
||||
db,
|
||||
table: "tenant_a_import_job",
|
||||
clock: () => fixedNow,
|
||||
instruments: getImportInstruments(),
|
||||
});
|
||||
|
||||
// `update` on a missing id rejects with the canonical message — parity
|
||||
// with `MemoryImportStore`. Adapter-layer SurQL errors surface as
|
||||
// `ImportError(IMPORT_CODES.STORAGE_ERROR)`.
|
||||
try {
|
||||
await store.update("missing", { status: "completed" });
|
||||
} catch (err) {
|
||||
// err.message === 'Job "missing" not found'
|
||||
}
|
||||
|
||||
// A read against an empty backend resolves to `undefined` and still
|
||||
// records `import.store.reads_total{outcome:"success"}` — the backend
|
||||
// answered correctly.
|
||||
const missing = await store.get("never-created");
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `class SurrealImportStore implements ImportStoreAdapter`
|
||||
- `constructor(config: SurrealImportStoreConfig)` — throws
|
||||
`ImportError(IMPORT_CODES.INVALID_OPTIONS)` on Zod-validation failure
|
||||
- `create(job: ImportJobRecord): Promise<void>`
|
||||
- `get(id: string): Promise<ImportJobRecord | undefined>`
|
||||
- `update(id: string, patch: Partial<ImportJobRecord>): Promise<void>` —
|
||||
throws `Error('Job "<id>" not found')` on a missing id; wraps adapter
|
||||
failures as `ImportError(IMPORT_CODES.STORAGE_ERROR)`
|
||||
- `delete(id: string): Promise<boolean>` — returns `true` if a row was
|
||||
removed, `false` if no row existed
|
||||
- `list(filter?: ImportStoreListFilter): Promise<ImportJobRecord[]>` —
|
||||
`status`, `ownerId`, `limit` (default 50), `offset` (default 0)
|
||||
- `interface SurrealImportStoreConfig { db: SurrealAdapter; table?: string;
|
||||
logger?: LoggerLike; clock?: Clock; instruments?: ImportInstruments }`
|
||||
- `SurrealImportStoreConfigSchema: z.ZodTypeAny` — boundary schema; reuses
|
||||
the same shape as the constructor argument
|
||||
- `DEFAULT_IMPORT_JOB_TABLE = "import_job"`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The SurrealDB schema lives at `surql/tables/import_job.surql` (shipped with
|
||||
the package). The store does NOT apply migrations — the host must run them
|
||||
via `@suaveplan/api-plugin-database`'s migration runner or by manually
|
||||
invoking `adapter.query(readFileSync(...))` in `beforeAll`.
|
||||
- Columns are snake_case in SurrealDB (`owner_id`, `total_rows`, …) but the
|
||||
`ImportJobRecord` keeps the suaveplan camelCase convention. The mapping is
|
||||
fully encapsulated in `rowToRecord` and the `create` / `update` SET
|
||||
clauses; downstream callers see only `ImportJobRecord`.
|
||||
- `update` first issues a `SELECT * FROM <id>` to detect missing-id and
|
||||
preserve the `Job "<id>" not found` contract from `MemoryImportStore`. This
|
||||
is one extra round-trip per update; if write-throughput is the bottleneck
|
||||
the host can layer a `Map`-based existence cache on top.
|
||||
- The constructor accepts an optional pre-resolved `ImportInstruments`
|
||||
bundle so high-volume hosts avoid the global-`MeterProvider` lookup on
|
||||
the hot path; when omitted, the bundle is resolved lazily on the first
|
||||
call (mirrors the `getImportInstruments()` cache).
|
||||
- All errors thrown by the underlying `SurrealAdapter` are wrapped as
|
||||
`ImportError(IMPORT_CODES.STORAGE_ERROR)` with `cause` set to the original
|
||||
throwable and `context = { op, sql }` so observability tooling can surface
|
||||
the failing query without parsing the message.
|
||||
- `list` orders by `created_at DESC` to mirror most real-world job-listing
|
||||
use cases. Add an explicit `ORDER BY` upstream if a different order is
|
||||
required — the `ImportStoreListFilter` contract does not currently expose
|
||||
a `sort` field.
|
||||
@@ -0,0 +1,112 @@
|
||||
# `transform/pipeline` — declarative transform pipeline
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Implements a three-stage row-transform pipeline that any consumer can
|
||||
drive without depending on the canonical `ImportService`: per-mapping
|
||||
coerce + transform, computed fields, then field validation. The
|
||||
pipeline is pure (no side effects, no I/O) and never throws — every
|
||||
failure is captured as a `TransformError` and returned in the result
|
||||
object, so callers can collect every failed row in one pass.
|
||||
|
||||
This module powers both the standalone `transformRow` / `transformRows`
|
||||
helpers exported to consumers and the per-row inner loop of
|
||||
`ImportService.startImport`. Keeping the pipeline separate from the
|
||||
service means CLI tools, batch workers, and tests can drive the
|
||||
transform logic against arbitrary input without spinning up an event
|
||||
emitter or store adapter.
|
||||
|
||||
## Features
|
||||
|
||||
- `coerceType` — explicit-result coercion for `number` / `boolean` /
|
||||
`date` with documented "valid" sets (`1`/`0`/`yes`/`no`/`true`/`false`
|
||||
for booleans, anything `Date.parse`-able for dates).
|
||||
- `validateField` — five built-in rules (`required`, `min`, `max`,
|
||||
`pattern`, `custom`); `custom` is an opt-in extension point that
|
||||
always returns `undefined` so callers wire their own.
|
||||
- `transformRow` — applies mappings, computed fields, and validations
|
||||
in order; pushes `TransformError` for each failure and returns
|
||||
`{ success, row, errors }`.
|
||||
- `transformRows` — array fan-out; errored rows are dropped from `rows`
|
||||
but their errors are still collected in `errors`.
|
||||
- `toRowError` — bridge to the service-layer `RowError` shape so the
|
||||
service can re-emit transform failures on the `import.row_error`
|
||||
event topic without duplicating shapes.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { transformRow } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const result = transformRow(
|
||||
{ name: "Ada", age: "37" },
|
||||
0,
|
||||
{
|
||||
mappings: [
|
||||
{ sourceColumn: "name", targetField: "displayName" },
|
||||
{ sourceColumn: "age", targetField: "age", type: "number" },
|
||||
],
|
||||
}
|
||||
);
|
||||
// result.row → { displayName: "Ada", age: 37 }
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { transformRow } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const result = transformRow(
|
||||
{ first: "Ada", last: "Lovelace", born: "1815" },
|
||||
0,
|
||||
{
|
||||
mappings: [
|
||||
{ sourceColumn: "born", targetField: "birthYear", type: "number" },
|
||||
],
|
||||
computedFields: [
|
||||
{
|
||||
targetField: "fullName",
|
||||
sourceColumns: ["first", "last"],
|
||||
compute: (v) => `${String(v.first)} ${String(v.last)}`,
|
||||
},
|
||||
],
|
||||
validations: [
|
||||
{ field: "birthYear", rule: "min", param: 1800 },
|
||||
],
|
||||
}
|
||||
);
|
||||
// result.row → { birthYear: 1815, fullName: "Ada Lovelace" }
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `coerceType(value: unknown, type: string): CoerceResult`
|
||||
- `validateField(value, rule: FieldValidation, rowIndex): TransformError | undefined`
|
||||
- `transformRow(source, rowIndex, config: TransformPipelineConfig): TransformRowResult`
|
||||
- `transformRows(sources, config): { rows; errors }`
|
||||
- `toRowError(err: TransformError): RowError`
|
||||
- `interface TransformMapping { sourceColumn; targetField; type?; defaultValue?; required?; transform? }`
|
||||
- `interface ComputedField { targetField; sourceColumns; compute }`
|
||||
- `interface FieldValidation { field; rule; param?; message? }`
|
||||
- `interface TransformError { row; field; message; value? }`
|
||||
- `interface TransformRowResult { success; row; errors }`
|
||||
- `interface TransformPipelineConfig { mappings; computedFields?; validations? }`
|
||||
- `interface CoerceResult { ok; value }`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Coercion short-circuits on empty / null / undefined to `{ ok: true,
|
||||
value: undefined }` so a downstream `defaultValue` can still apply.
|
||||
- The transform function (`mapping.transform`) is wrapped in `try` so a
|
||||
thrown library never escapes the pipeline; the row's `success` flag
|
||||
reports the failure and the rest of the mappings still process.
|
||||
- `custom` validation rules are intentionally no-op — extension is via
|
||||
the `transform` callback on the mapping, which has access to the raw
|
||||
value. Custom rules without a default are documented in the
|
||||
enterprise audit as "extended by consumers."
|
||||
- `validateField` runs after both mapping and computed-field stages, so
|
||||
validations target the post-coercion result, not the raw source row.
|
||||
@@ -0,0 +1,89 @@
|
||||
# `types` — cross-module types
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Defines the cross-module types and constants consumed by every other
|
||||
feature module in `@suaveplan/api-plugin-import`. Parsers, the transform
|
||||
pipeline, the progress tracker, the store adapter, the canonical
|
||||
`ImportService`, the routes, and the plugin all import their shared
|
||||
shapes from here so they stay in lockstep across a refactor. The file
|
||||
contains zero runtime logic beyond two literal constants (`VALID_FORMATS`,
|
||||
`DEFAULT_BATCH_SIZE`) — it is effectively a `.d.ts` declaration with
|
||||
named exports.
|
||||
|
||||
The types follow the suaveplan convention of `readonly` fields and
|
||||
discriminated unions rather than free-form interfaces, so consumers get
|
||||
type-narrowing on `status` and `format` without runtime casts.
|
||||
|
||||
## Features
|
||||
|
||||
- `ImportFormat` / `ImportStatus` / `ImportJobStatus` discriminated unions.
|
||||
- `ColumnMapping`, `RowError`, `ImportPreview`, `ImportProgress`,
|
||||
`ImportJobConfig`, `ImportOwnerContext` interfaces.
|
||||
- `ImportJobRecord` — store-tier shape kept distinct from
|
||||
`ImportProgress` so durable backends do not pay the per-row write cost.
|
||||
- `ImportEventMap` — strongly-typed event-bus topic map for the
|
||||
`EventEmitter<ImportEventMap>` extended by `ImportService`.
|
||||
- `VALID_FORMATS` / `DEFAULT_BATCH_SIZE` constants.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type { ImportJobConfig, ColumnMapping } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const mappings: ColumnMapping[] = [
|
||||
{ sourceColumn: "name", targetField: "displayName", required: true },
|
||||
];
|
||||
|
||||
const config: ImportJobConfig = {
|
||||
format: "csv",
|
||||
data: "name\nDoe",
|
||||
mappings,
|
||||
};
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import type {
|
||||
ImportEventMap,
|
||||
ImportProgress,
|
||||
} from "@suaveplan/api-plugin-import";
|
||||
import { EventEmitter } from "@suaveplan/events";
|
||||
|
||||
const bus = new EventEmitter<ImportEventMap>();
|
||||
bus.on("import.progress", (progress: ImportProgress) => {
|
||||
console.log(progress.jobId, progress.processedRows, progress.totalRows);
|
||||
});
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `ImportFormat = "csv" | "json" | "xlsx"`
|
||||
- `ImportStatus = "pending" | "validating" | "previewing" | "importing" | "completed" | "failed"`
|
||||
- `ImportJobStatus = "created" | "validating" | "importing" | "completed" | "failed"`
|
||||
- `interface ColumnMapping { sourceColumn; targetField; transform?; required?; defaultValue? }`
|
||||
- `interface RowError { row; column; message; value? }`
|
||||
- `interface ImportPreview { totalRows; columns; sampleRows; mappings }`
|
||||
- `interface ImportProgress { jobId; ownerId; status; totalRows; processedRows; successCount; errorCount; errors; startedAt?; completedAt? }`
|
||||
- `interface ImportJobConfig { format; data; mappings; dryRun?; batchSize?; skipDuplicates?; onRow? }`
|
||||
- `interface ImportOwnerContext { ownerId }`
|
||||
- `interface ImportJobRecord { id; ownerId; status; totalRows; processedRows; successCount; errorCount; errors; metadata?; createdAt; updatedAt; completedAt? }`
|
||||
- `interface ImportEventMap { … }`
|
||||
- `interface ImportRowProcessedEvent { jobId; rowIndex; success }`
|
||||
- `const VALID_FORMATS: readonly ImportFormat[]`
|
||||
- `const DEFAULT_BATCH_SIZE = 100`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `ImportProgress` and `ImportJobRecord` are deliberately separate so the
|
||||
durable store stays narrow; the live progress object can fan-out
|
||||
per-row errors while the store records a digest.
|
||||
- `ownerId` lives on both the progress object and the record so every
|
||||
read path can enforce owner scoping without a cross-table join.
|
||||
- `ImportEventMap` widens via index signatures so consumers can layer
|
||||
app-specific topics on top without `as any`.
|
||||
Reference in New Issue
Block a user