142 lines
9.7 KiB
Markdown
142 lines
9.7 KiB
Markdown
# @suaveplan/api-plugin-import - Tabular-data ingest plugin for @suaveplan/api. CSV / JSON / TSV-XLSX parsers, declarative transform pipeline (coerce + validate + computed), real-time progress tracker, async ImportStoreAdapter (in-memory + SurrealDB backends ship), canonical EventEmitter-based ImportService with owner-scoped jobs, 5 auth-gated OpenAPI Hono routes, RFC 7807 problem+json renderer, 9 IMPORT.* error codes, 6 OTel instruments. Server-tier.
|
|
|
|
Tabular-data ingest plugin for `@suaveplan/api`. Parses CSV / JSON /
|
|
TSV-XLSX payloads, applies declarative column mappings + per-field
|
|
validation + computed fields, persists owner-scoped import jobs through
|
|
a pluggable async store, emits real-time progress via an event bus, and
|
|
exposes five auth-gated OpenAPI routes with full Zod request/response
|
|
schemas. Ships nine `IMPORT.*` error codes (extending
|
|
`SuaveplanError`), four OpenTelemetry instruments (`import.jobs_total`,
|
|
`import.rows_total`, `import.row_errors_total`, `import.duration_ms`),
|
|
and a hand-rolled RFC 7807 problem+json renderer for the route layer.
|
|
|
|
<!-- begin:auto-docs -->
|
|
## Documentation tree
|
|
|
|
- [error-codes](./src/error-codes/error-codes.md) — Declares the nine stable `IMPORT.*` error codes used throughout `@suaveplan/api-plugin-import` and exports the `ImportE…
|
|
- [error-codes](./src/error-codes/error-codes.md) — Declares the nine stable `IMPORT.*` error codes used throughout `@suaveplan/api-plugin-import` and exports the `ImportE…
|
|
- [otel](./src/otel/instruments.md) — Builds the OpenTelemetry instrument bundle emitted by the canonical `ImportService` so consumers can wire production ob…
|
|
- [instruments](./src/otel/instruments.md) — Builds the OpenTelemetry instrument bundle emitted by the canonical `ImportService` so consumers can wire production ob…
|
|
- [csv](./src/parsers/csv.md) — Provides the CSV-family parsers consumed by the canonical `ImportService.preview` / `startImport` pipeline and by any c…
|
|
- [json](./src/parsers/json.md) — Parses JSON-array input documents into the same `Record<string, unknown>[]` row shape produced by the CSV / XLSX parser…
|
|
- [xlsx](./src/parsers/xlsx.md) — Provides a lightweight tab-separated-value parser exposed under the `xlsx` format so consumers can ingest spreadsheet-s…
|
|
- [plugin](./src/plugin/plugin.md) — Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the `@suaveplan/api-plugin` lifecycle. Valida…
|
|
- [plugin](./src/plugin/plugin.md) — Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the `@suaveplan/api-plugin` lifecycle. Valida…
|
|
- [progress](./src/progress/tracker.md) — Provides the `ImportProgressTracker` class consumed by the canonical `ImportService` (and exported for direct use by CL…
|
|
- [tracker](./src/progress/tracker.md) — Provides the `ImportProgressTracker` class consumed by the canonical `ImportService` (and exported for direct use by CL…
|
|
- [progress-sse](./src/routes/progress-sse/progress-sse.md) — Exposes `GET /imports/:id/progress` returning a long-lived `text/event-stream` so clients can subscribe to a job's real…
|
|
- [progress-sse](./src/routes/progress-sse/progress-sse.md) — Exposes `GET /imports/:id/progress` returning a long-lived `text/event-stream` so clients can subscribe to a job's real…
|
|
- [build-routes](./src/routes/build-routes.md) — Constructs the `OpenAPIHono` router mounted by `createImportPlugin` at the configured `routePrefix`. Five routes are re…
|
|
- [problem-json](./src/routes/problem-json.md) — Bridges thrown errors from the route handlers into RFC 7807 `application/problem+json` responses. Reads the HTTP status…
|
|
- [schemas](./src/schemas/schemas.md) — Carries the full public Zod surface used by the route handlers and any external client (CLI tool, integration test, bro…
|
|
- [options](./src/schemas/options.md) — Declares the Zod schema for the options accepted by `createImportPlugin`. Parsing options at the factory boundary guara…
|
|
- [schemas](./src/schemas/schemas.md) — Carries the full public Zod surface used by the route handlers and any external client (CLI tool, integration test, bro…
|
|
- [service](./src/service/service.md) — Implements the canonical `ImportService` used by `createImportPlugin` and exported for direct programmatic use. Owns th…
|
|
- [service](./src/service/service.md) — Implements the canonical `ImportService` used by `createImportPlugin` and exported for direct programmatic use. Owns th…
|
|
- [services](./src/services/registry-augmentation.md) — Augments the `ServiceRegistry` interface from `@suaveplan/api-plugin/service-registry` with the `importService` slot so…
|
|
- [registry-augmentation](./src/services/registry-augmentation.md) — Augments the `ServiceRegistry` interface from `@suaveplan/api-plugin/service-registry` with the `importService` slot so…
|
|
- [store](./src/store/memory.md) — Declares the async `ImportStoreAdapter` interface and ships the `MemoryImportStore` implementation used by the canonica…
|
|
- [surreal](./src/store/surreal/surreal-import-store.md) — Implements the canonical `ImportStoreAdapter` interface against a connected `SurrealAdapter` from `@suaveplan/surrealdb…
|
|
- [surreal-import-store](./src/store/surreal/surreal-import-store.md) — Implements the canonical `ImportStoreAdapter` interface against a connected `SurrealAdapter` from `@suaveplan/surrealdb…
|
|
- [memory](./src/store/memory.md) — Declares the async `ImportStoreAdapter` interface and ships the `MemoryImportStore` implementation used by the canonica…
|
|
- [transform](./src/transform/pipeline.md) — Implements a three-stage row-transform pipeline that any consumer can drive without depending on the canonical `ImportS…
|
|
- [pipeline](./src/transform/pipeline.md) — Implements a three-stage row-transform pipeline that any consumer can drive without depending on the canonical `ImportS…
|
|
- [types](./src/types/types.md) — Defines the cross-module types and constants consumed by every other feature module in `@suaveplan/api-plugin-import`.…
|
|
- [types](./src/types/types.md) — Defines the cross-module types and constants consumed by every other feature module in `@suaveplan/api-plugin-import`.…
|
|
<!-- end:auto-docs -->
|
|
|
|
## Install
|
|
|
|
```bash
|
|
bun add @suaveplan/api-plugin-import
|
|
```
|
|
|
|
## Quick start
|
|
|
|
```ts
|
|
import { createImportPlugin } from "@suaveplan/api-plugin-import";
|
|
|
|
// Register with @suaveplan/api's plugin registry.
|
|
const importPlugin = createImportPlugin({ routePrefix: "/imports" });
|
|
|
|
// Or use the service directly:
|
|
import { ImportService } from "@suaveplan/api-plugin-import";
|
|
|
|
const service = new ImportService();
|
|
const progress = await service.startImport(
|
|
{
|
|
format: "csv",
|
|
data: "name\nAda\nBob",
|
|
mappings: [{ sourceColumn: "name", targetField: "displayName" }],
|
|
},
|
|
{ ownerId: "alice" }
|
|
);
|
|
console.log(progress.successCount); // 2
|
|
```
|
|
|
|
## Modules
|
|
|
|
- [error-codes](./src/error-codes/error-codes.md) — `IMPORT_CODES` /
|
|
`ImportError` registry, 9 codes.
|
|
- [types](./src/types/types.md) — cross-module shared shapes.
|
|
- [parsers/csv](./src/parsers/csv.md) — CSV + streaming CSV parsers.
|
|
- [parsers/json](./src/parsers/json.md) — JSON parser.
|
|
- [parsers/xlsx](./src/parsers/xlsx.md) — TSV-style XLSX parser.
|
|
- [transform/pipeline](./src/transform/pipeline.md) — coerce / validate
|
|
/ computed-field pipeline.
|
|
- [progress/tracker](./src/progress/tracker.md) — `ImportProgressTracker`.
|
|
- [store/memory](./src/store/memory.md) — `ImportStoreAdapter` +
|
|
`MemoryImportStore`.
|
|
- [store/surreal/surreal-import-store](./src/store/surreal/surreal-import-store.md)
|
|
— durable `SurrealImportStore` (SurrealDB-backed). Migration shipped at
|
|
[`surql/tables/import_job.surql`](./surql/tables/import_job.surql).
|
|
- [service/service](./src/service/service.md) — canonical `ImportService`.
|
|
- [routes/problem-json](./src/routes/problem-json.md) — RFC 7807
|
|
renderer.
|
|
- [routes/build-routes](./src/routes/build-routes.md) — 5 auth-gated
|
|
OpenAPI routes.
|
|
- [otel/instruments](./src/otel/instruments.md) — OTel bundle.
|
|
- [schemas/options](./src/schemas/options.md) — plugin options schema.
|
|
- [schemas/schemas](./src/schemas/schemas.md) — public request /
|
|
response Zod surface.
|
|
- [services/registry-augmentation](./src/services/registry-augmentation.md)
|
|
— `ServiceRegistry["importService"]` slot.
|
|
- [plugin/plugin](./src/plugin/plugin.md) — `createImportPlugin`
|
|
factory.
|
|
|
|
## Error codes
|
|
|
|
Every failure path raises an `ImportError` carrying one of nine
|
|
`IMPORT.*` codes — `NOT_FOUND`, `FORBIDDEN`, `VALIDATION`,
|
|
`FILE_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `PROCESSING_FAILED`,
|
|
`STORAGE_ERROR`, `TIMEOUT`, `INVALID_OPTIONS`. Anti-enumeration:
|
|
foreign-owner lookups return `NOT_FOUND`, never `FORBIDDEN`.
|
|
|
|
## SurrealDB store
|
|
|
|
`SurrealImportStore` is the durable alternative to `MemoryImportStore`,
|
|
implementing the same `ImportStoreAdapter` interface against a connected
|
|
`@suaveplan/surrealdb` `SurrealAdapter`. The matching schema migration
|
|
ships co-located at [`surql/tables/import_job.surql`](./surql/tables/import_job.surql);
|
|
hosts apply it through `@suaveplan/api-plugin-database`'s migration
|
|
runner or via `adapter.query(readFileSync(...))`. The store uses
|
|
positional `$p0,$p1,…` SurrealQL placeholders — the canonical pattern
|
|
used by every SurrealDB repository.
|
|
|
|
## Observability
|
|
|
|
Six OpenTelemetry instruments are emitted on the production hot paths
|
|
— call `getImportInstruments()` to reuse the cached bundle, or
|
|
`resetImportInstruments()` in tests after installing a fresh
|
|
`MeterProvider`. Four cover the service (`jobs_total`, `rows_total`,
|
|
`row_errors_total`, `duration_ms`); two cover the store
|
|
(`import.store.writes_total{outcome}` and
|
|
`import.store.reads_total{outcome}` — emitted by `SurrealImportStore`
|
|
on every adapter call so host dashboards can split backend health from
|
|
upstream import success/failure).
|
|
|
|
## License
|
|
|
|
MIT
|
|
|