first commit
This commit is contained in:
141
docs/api-stack/api/api-plugin-import/README.md
Normal file
141
docs/api-stack/api/api-plugin-import/README.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# @suaveplan/api-plugin-import - Tabular-data ingest plugin for @suaveplan/api. CSV / JSON / TSV-XLSX parsers, declarative transform pipeline (coerce + validate + computed), real-time progress tracker, async ImportStoreAdapter (in-memory + SurrealDB backends ship), canonical EventEmitter-based ImportService with owner-scoped jobs, 5 auth-gated OpenAPI Hono routes, RFC 7807 problem+json renderer, 9 IMPORT.* error codes, 6 OTel instruments. Server-tier.
|
||||
|
||||
Tabular-data ingest plugin for `@suaveplan/api`. Parses CSV / JSON /
|
||||
TSV-XLSX payloads, applies declarative column mappings + per-field
|
||||
validation + computed fields, persists owner-scoped import jobs through
|
||||
a pluggable async store, emits real-time progress via an event bus, and
|
||||
exposes five auth-gated OpenAPI routes with full Zod request/response
|
||||
schemas. Ships nine `IMPORT.*` error codes (extending
|
||||
`SuaveplanError`), four OpenTelemetry instruments (`import.jobs_total`,
|
||||
`import.rows_total`, `import.row_errors_total`, `import.duration_ms`),
|
||||
and a hand-rolled RFC 7807 problem+json renderer for the route layer.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [error-codes](./src/error-codes/error-codes.md) — Declares the nine stable `IMPORT.*` error codes used throughout `@suaveplan/api-plugin-import` and exports the `ImportE…
|
||||
- [error-codes](./src/error-codes/error-codes.md) — Declares the nine stable `IMPORT.*` error codes used throughout `@suaveplan/api-plugin-import` and exports the `ImportE…
|
||||
- [otel](./src/otel/instruments.md) — Builds the OpenTelemetry instrument bundle emitted by the canonical `ImportService` so consumers can wire production ob…
|
||||
- [instruments](./src/otel/instruments.md) — Builds the OpenTelemetry instrument bundle emitted by the canonical `ImportService` so consumers can wire production ob…
|
||||
- [csv](./src/parsers/csv.md) — Provides the CSV-family parsers consumed by the canonical `ImportService.preview` / `startImport` pipeline and by any c…
|
||||
- [json](./src/parsers/json.md) — Parses JSON-array input documents into the same `Record<string, unknown>[]` row shape produced by the CSV / XLSX parser…
|
||||
- [xlsx](./src/parsers/xlsx.md) — Provides a lightweight tab-separated-value parser exposed under the `xlsx` format so consumers can ingest spreadsheet-s…
|
||||
- [plugin](./src/plugin/plugin.md) — Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the `@suaveplan/api-plugin` lifecycle. Valida…
|
||||
- [plugin](./src/plugin/plugin.md) — Returns a typed `PluginDescriptor<ImportPluginConfig>` wired onto the `@suaveplan/api-plugin` lifecycle. Valida…
|
||||
- [progress](./src/progress/tracker.md) — Provides the `ImportProgressTracker` class consumed by the canonical `ImportService` (and exported for direct use by CL…
|
||||
- [tracker](./src/progress/tracker.md) — Provides the `ImportProgressTracker` class consumed by the canonical `ImportService` (and exported for direct use by CL…
|
||||
- [progress-sse](./src/routes/progress-sse/progress-sse.md) — Exposes `GET /imports/:id/progress` returning a long-lived `text/event-stream` so clients can subscribe to a job's real…
|
||||
- [progress-sse](./src/routes/progress-sse/progress-sse.md) — Exposes `GET /imports/:id/progress` returning a long-lived `text/event-stream` so clients can subscribe to a job's real…
|
||||
- [build-routes](./src/routes/build-routes.md) — Constructs the `OpenAPIHono` router mounted by `createImportPlugin` at the configured `routePrefix`. Five routes are re…
|
||||
- [problem-json](./src/routes/problem-json.md) — Bridges thrown errors from the route handlers into RFC 7807 `application/problem+json` responses. Reads the HTTP status…
|
||||
- [schemas](./src/schemas/schemas.md) — Carries the full public Zod surface used by the route handlers and any external client (CLI tool, integration test, bro…
|
||||
- [options](./src/schemas/options.md) — Declares the Zod schema for the options accepted by `createImportPlugin`. Parsing options at the factory boundary guara…
|
||||
- [schemas](./src/schemas/schemas.md) — Carries the full public Zod surface used by the route handlers and any external client (CLI tool, integration test, bro…
|
||||
- [service](./src/service/service.md) — Implements the canonical `ImportService` used by `createImportPlugin` and exported for direct programmatic use. Owns th…
|
||||
- [service](./src/service/service.md) — Implements the canonical `ImportService` used by `createImportPlugin` and exported for direct programmatic use. Owns th…
|
||||
- [services](./src/services/registry-augmentation.md) — Augments the `ServiceRegistry` interface from `@suaveplan/api-plugin/service-registry` with the `importService` slot so…
|
||||
- [registry-augmentation](./src/services/registry-augmentation.md) — Augments the `ServiceRegistry` interface from `@suaveplan/api-plugin/service-registry` with the `importService` slot so…
|
||||
- [store](./src/store/memory.md) — Declares the async `ImportStoreAdapter` interface and ships the `MemoryImportStore` implementation used by the canonica…
|
||||
- [surreal](./src/store/surreal/surreal-import-store.md) — Implements the canonical `ImportStoreAdapter` interface against a connected `SurrealAdapter` from `@suaveplan/surrealdb…
|
||||
- [surreal-import-store](./src/store/surreal/surreal-import-store.md) — Implements the canonical `ImportStoreAdapter` interface against a connected `SurrealAdapter` from `@suaveplan/surrealdb…
|
||||
- [memory](./src/store/memory.md) — Declares the async `ImportStoreAdapter` interface and ships the `MemoryImportStore` implementation used by the canonica…
|
||||
- [transform](./src/transform/pipeline.md) — Implements a three-stage row-transform pipeline that any consumer can drive without depending on the canonical `ImportS…
|
||||
- [pipeline](./src/transform/pipeline.md) — Implements a three-stage row-transform pipeline that any consumer can drive without depending on the canonical `ImportS…
|
||||
- [types](./src/types/types.md) — Defines the cross-module types and constants consumed by every other feature module in `@suaveplan/api-plugin-import`.…
|
||||
- [types](./src/types/types.md) — Defines the cross-module types and constants consumed by every other feature module in `@suaveplan/api-plugin-import`.…
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
bun add @suaveplan/api-plugin-import
|
||||
```
|
||||
|
||||
## Quick start
|
||||
|
||||
```ts
|
||||
import { createImportPlugin } from "@suaveplan/api-plugin-import";
|
||||
|
||||
// Register with @suaveplan/api's plugin registry.
|
||||
const importPlugin = createImportPlugin({ routePrefix: "/imports" });
|
||||
|
||||
// Or use the service directly:
|
||||
import { ImportService } from "@suaveplan/api-plugin-import";
|
||||
|
||||
const service = new ImportService();
|
||||
const progress = await service.startImport(
|
||||
{
|
||||
format: "csv",
|
||||
data: "name\nAda\nBob",
|
||||
mappings: [{ sourceColumn: "name", targetField: "displayName" }],
|
||||
},
|
||||
{ ownerId: "alice" }
|
||||
);
|
||||
console.log(progress.successCount); // 2
|
||||
```
|
||||
|
||||
## Modules
|
||||
|
||||
- [error-codes](./src/error-codes/error-codes.md) — `IMPORT_CODES` /
|
||||
`ImportError` registry, 9 codes.
|
||||
- [types](./src/types/types.md) — cross-module shared shapes.
|
||||
- [parsers/csv](./src/parsers/csv.md) — CSV + streaming CSV parsers.
|
||||
- [parsers/json](./src/parsers/json.md) — JSON parser.
|
||||
- [parsers/xlsx](./src/parsers/xlsx.md) — TSV-style XLSX parser.
|
||||
- [transform/pipeline](./src/transform/pipeline.md) — coerce / validate
|
||||
/ computed-field pipeline.
|
||||
- [progress/tracker](./src/progress/tracker.md) — `ImportProgressTracker`.
|
||||
- [store/memory](./src/store/memory.md) — `ImportStoreAdapter` +
|
||||
`MemoryImportStore`.
|
||||
- [store/surreal/surreal-import-store](./src/store/surreal/surreal-import-store.md)
|
||||
— durable `SurrealImportStore` (SurrealDB-backed). Migration shipped at
|
||||
[`surql/tables/import_job.surql`](./surql/tables/import_job.surql).
|
||||
- [service/service](./src/service/service.md) — canonical `ImportService`.
|
||||
- [routes/problem-json](./src/routes/problem-json.md) — RFC 7807
|
||||
renderer.
|
||||
- [routes/build-routes](./src/routes/build-routes.md) — 5 auth-gated
|
||||
OpenAPI routes.
|
||||
- [otel/instruments](./src/otel/instruments.md) — OTel bundle.
|
||||
- [schemas/options](./src/schemas/options.md) — plugin options schema.
|
||||
- [schemas/schemas](./src/schemas/schemas.md) — public request /
|
||||
response Zod surface.
|
||||
- [services/registry-augmentation](./src/services/registry-augmentation.md)
|
||||
— `ServiceRegistry["importService"]` slot.
|
||||
- [plugin/plugin](./src/plugin/plugin.md) — `createImportPlugin`
|
||||
factory.
|
||||
|
||||
## Error codes
|
||||
|
||||
Every failure path raises an `ImportError` carrying one of nine
|
||||
`IMPORT.*` codes — `NOT_FOUND`, `FORBIDDEN`, `VALIDATION`,
|
||||
`FILE_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `PROCESSING_FAILED`,
|
||||
`STORAGE_ERROR`, `TIMEOUT`, `INVALID_OPTIONS`. Anti-enumeration:
|
||||
foreign-owner lookups return `NOT_FOUND`, never `FORBIDDEN`.
|
||||
|
||||
## SurrealDB store
|
||||
|
||||
`SurrealImportStore` is the durable alternative to `MemoryImportStore`,
|
||||
implementing the same `ImportStoreAdapter` interface against a connected
|
||||
`@suaveplan/surrealdb` `SurrealAdapter`. The matching schema migration
|
||||
ships co-located at [`surql/tables/import_job.surql`](./surql/tables/import_job.surql);
|
||||
hosts apply it through `@suaveplan/api-plugin-database`'s migration
|
||||
runner or via `adapter.query(readFileSync(...))`. The store uses
|
||||
positional `$p0,$p1,…` SurrealQL placeholders — the canonical pattern
|
||||
used by every SurrealDB repository.
|
||||
|
||||
## Observability
|
||||
|
||||
Six OpenTelemetry instruments are emitted on the production hot paths
|
||||
— call `getImportInstruments()` to reuse the cached bundle, or
|
||||
`resetImportInstruments()` in tests after installing a fresh
|
||||
`MeterProvider`. Four cover the service (`jobs_total`, `rows_total`,
|
||||
`row_errors_total`, `duration_ms`); two cover the store
|
||||
(`import.store.writes_total{outcome}` and
|
||||
`import.store.reads_total{outcome}` — emitted by `SurrealImportStore`
|
||||
on every adapter call so host dashboards can split backend health from
|
||||
upstream import success/failure).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
Reference in New Issue
Block a user