suaveplan/openspec/archive/2026-05-19-add-api-plugin-import/proposal.md

# Add @suaveplan/api-plugin-import

**Package**: `@suaveplan/api-plugin-import`
**Status**: PROPOSED
**Type**: new-package

- **Design**: ./design.md (omitted — every architectural decision lives in the spec delta's Implementation Notes references and in the per-feature `.md` files)
- **Spec delta**: ./specs/api-plugin-import/spec.md

## Why

Genesis needs a tabular-data ingest plugin for `@suaveplan/api` so user-facing apps can accept CSV / JSON / XLSX uploads, preview them, apply column mappings + per-row transforms + field validations, run dry-runs, persist import jobs, and stream progress events. The upstream
`@suaveplan/api-plugin-import@1.1.7` (`/mnt/s/web/suaveplan/packages/server/api-plugin-import`, 2,320 LOC, ~98.72 % historical coverage) carries the entire feature surface but ships in a half-migrated state — two parallel `ImportService` implementations, seven `@deprecated` factory helpers, no `definePlugin()` export, an inline ad-hoc `ImportError` next to a proper `BaseError`-backed one, and a route bundle that hand-rolls Hono context casts.

This change lands a single, enterprise-grade `@suaveplan/api-plugin-import@0.1.0` that consolidates the two services, removes every deprecated alias, wires a real `createImportPlugin()` onto the genesis `PluginDescriptor` from `@suaveplan/api-plugin`, gates routes through `requireAuth()` from `@suaveplan/api-plugin-auth`, scopes job lookups to `c.get("user").id` with 404 anti-enumeration, ships OTel instruments asserted with a real in-memory `MeterProvider`, and rewrites every test against `@suaveplan/testing/runner`.

## What changes

- ADDED: `IMPORT_CODES` / `ImportError` — `defineCodes("IMPORT", […])` registry of nine codes (`NOT_FOUND`, `FORBIDDEN`, `VALIDATION`, `FILE_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `PROCESSING_FAILED`, `STORAGE_ERROR`, `TIMEOUT`, `INVALID_OPTIONS`) extending `SuaveplanError`.
- ADDED: `parseCsv` / `parseCsvLine` / `detectColumns` / `streamParseCsv` / `detectHeaders` / `parseStreamLine` / `applyColumnMappings` — CSV utilities incl. quoted-field handling, custom delimiter, header detection, embedded-newline tolerance.
- ADDED: `parseJson` — JSON-array parser with non-array rejection.
- ADDED: `parseXlsx` — TSV-style XLSX parser (lightweight; documented limitation).
- ADDED: `coerceType` / `validateField` / `transformRow` / `transformRows` — declarative transform pipeline with type coercion (`number` / `boolean` / `date`), required/default-value handling, per-field validation (`required` / `min` / `max` / `pattern` / `custom`), computed fields.
- ADDED: `ImportProgressTracker` — real-time row tracker with rows/sec, ETA, percent-complete, batch helpers, listener fan-out.
- ADDED: `MemoryImportStore` + `ImportStoreAdapter` — async CRUD + list-by-status interface; in-memory implementation for tests and small deployments.
- ADDED: `ImportService` (canonical) — `EventEmitter<ImportEventMap>`-based service with `preview` / `startImport` / `getJob` / `getResults` / `listJobs` / `deleteJob`, batch progress emission, duplicate-skip, `onRow` filter, OTel instrumentation, scoped-by-owner store writes.
- ADDED: `createImportRoutes` — 5 OpenAPI routes (`POST /preview`, `POST /`, `GET /`, `GET /{id}`, `DELETE /{id}`) emitting RFC 7807 problem+json on failure, validating bodies via Zod, scoping all reads / writes to the caller's `Principal.id`.
- ADDED: `createImportPlugin` — `definePlugin<ImportPluginConfig>` factory wrapping the genesis `PluginDescriptor` (NOT the upstream `definePlugin` from `@suaveplan/api`) with 4-phase lifecycle, registering `importService` on the `ServiceRegistry`, mounting routes under a configurable `basePath`, optional-depending on `auth` / `events` / `database`.
- ADDED: `respondProblemJson` / `PROBLEM_JSON_CONTENT_TYPE` — RFC 7807 helper bridging `SuaveplanError` (and unknown throwables) to a `Response`.
- ADDED: `getImportInstruments` / `resetImportInstruments` / `ImportInstruments` — OTel counter+histogram bundle (`import.jobs_total`, `import.rows_total`, `import.row_errors_total`, `import.duration_ms`) with cache reset for test rebinding.

### Excluded from this migration

The genesis `0.1.0` ships at functional parity with the upstream library surface
(parsers + transform pipeline + progress + memory store + service + routes +
plugin). The §3 enterprise-doc enhancements that depend on packages not yet
landed on genesis are deferred — each with a concrete architectural
justification — and are NOT silently dropped:

- **`SurrealImportStore`** — requires `@suaveplan/surrealdb` migrations + the
  Surreal-side schema. Genesis ships `MemoryImportStore` against the same
  `ImportStoreAdapter` interface so the SurrealDB adapter lands as a sibling
  package without breaking consumers. Listed in pending follow-ups.
- **Rate-limit + idempotency + audit middleware** — depends on
  `@suaveplan/api-plugin-rate-limit`, `@suaveplan/idempotency`,
  `@suaveplan/api-plugin-audit`. These are wired by the host app via the
  standard `app.use(prefix, middleware)` Hono pattern; the plugin does NOT
  hard-depend on them, so excluding them from THIS migration does not block
  consumers from layering them on top.
- **Streaming `Readable`/`AsyncIterable` input** — `streamParseCsv` is shipped
  and the public `startImport` accepts a `string` (parity with upstream). A
  follow-up will extend the input union to `string | AsyncIterable<string>`
  without breaking the existing call site.
- **Benchmarks under `src/__benchmarks__/`** — every benchmark requires a
  mitata harness; the harness lives in `@suaveplan/testing` but the
  benchmark gate is monorepo-wide and shipped per-package in a separate
  benchmark wave.
- **Foreign-key resolution, parallel batch workers, resumable checkpoints,
  S3/URL source adapters, SSE progress push** — every item is a §4 future
  enhancement, not a §2 current-state capability. None are part of the
  upstream public surface; deferring them is parity, not regression.

## Out of scope

- Modifying any other `packages/api/api-plugin-*/` package, `packages/messaging/*`, `packages/audio/*`, `packages/api/api-migrations/`, `packages/database/surrealdb/`, `packages/core/*`, `MANIFEST.md`, `STATUS.md`, `CLAUDE.md`, or the root `package.json` (except `bun.lock`). Concurrent sessions own those scopes.
- The 7 `@deprecated` factory helpers (`jobNotFound`, `invalidFormat`, `importFailed`, `validationError`, `parseError`, `mappingError`, `duplicateDetected`). Genesis bans `@deprecated` in source per `AGENTS.md §2`; the canonical helpers (`importNotFound`, `importUnsupportedFormat`, `importProcessingFailed`, `importValidation`) ship as the only API.
- The legacy `services/import.service.ts` implementation. Genesis ships exactly one canonical `ImportService` per upstream `enterprise/api-plugin-import.md` §3 row 1.

## Source mapping

This change derives from `/mnt/s/web/suaveplan/enterprise/api-plugin-import.md`.

| Requirement | Source lines | Notes |
|---|---|---|
| Single canonical `ImportService` | L46 (§3 row 1) | "Converge on one `ImportService`" — genesis ships the canonical EventEmitter-based one |
| `createImportPlugin()` wiring routes | L47-L48 (§3 row 2) | Wraps onto genesis `PluginDescriptor` (not upstream `definePlugin`) |
| No `@deprecated` helpers | L49 (§3 row 3) + L37, L40 | Enforced by `AGENTS.md §2` — `grep` gate is zero-match |
| XLSX support (TSV interchange) | L50 (§3 row 4) + upstream `src/parsers/xlsx.parser.ts` | Genesis ships TSV-style parity; binary XLSX is documented limitation |
| Async `ImportStoreAdapter` + `MemoryImportStore` | L29, L51 (§3 row 5) | SurrealDB adapter deferred per `### Excluded` |
| Streaming CSV parser | L26, L52 (§3 row 6) | `streamParseCsv` shipped; `string` input retained for parity |
| Auth-gated routes via `requireAuth()` | L53 (§3 row 7) | Genesis `@suaveplan/api-plugin-auth` substitutes for upstream `@suaveplan/jwt-tenant` |
| Owner-scoped jobs with 404 anti-enumeration | L53 (§3 row 7) + path-A safety doc | Foreign job lookups return 404 not 403 |
| OTel instruments | L57 (§3 row 9) | `import.jobs_total`, `import.rows_total`, `import.row_errors_total`, `import.duration_ms` |
| Structured logger via `LoggerLike` | L57 (§3 row 9) | Uses injected `ctx.logger` |
| RFC 7807 problem+json mapping | upstream `src/routes/problem-json.ts` | Genesis re-renders via `SuaveplanError.context.statusCode` |
| Stable `IMPORT.*` error registry | L30, L37, L106 | `defineCodes("IMPORT", […])` — 9 codes |
| `ImportProgressTracker` with rows/sec + ETA | upstream `src/progress/import-progress.ts` | Verbatim functional parity |
| `transformRow` / `transformRows` / `coerceType` / `validateField` | upstream `src/transform/transform-pipeline.ts` | Verbatim functional parity |
| `parseCsv` / `parseCsvLine` / `detectColumns` / `parseJson` / `streamParseCsv` | upstream `src/types/import.types.ts` + `src/parsers/stream-parser.ts` | Verbatim functional parity |
| 5 OpenAPI routes (preview, create, list, get, delete) | upstream `src/routes/import.routes.ts` | Owner scoping is new; route set is verbatim |
| `ImportError` extends `SuaveplanError` | upstream `src/errors/import-errors.ts` + `AGENTS.md §2` | Genesis uses `SuaveplanError`, not upstream `BaseError` |
| `ImportEventMap` + EventEmitter | upstream `src/services/import-service.ts` L30 | Verbatim |
| `ImportJobRecord` async store interface | upstream `src/store/import-store.ts` | Verbatim shape |
| Subpath exports per feature module | `AGENTS.md §6`, repo convention | Per-feature `./<dir>/<file>` entries |

### Requirements with no source line

| Requirement | Justification |
|---|---|
| 100 % istanbul coverage thresholds | Genesis-wide policy (`AGENTS.md §7`) |
| `**Verified by:**` lines at draft time | Genesis-wide policy (`AGENTS.md §13`) |
| Test imports via `@suaveplan/testing/runner` | Genesis-wide policy (`CLAUDE.md` rule 11) |
| Explicit named re-exports in feature barrels | Genesis-wide policy (`AGENTS.md §6a`) |

## Verification approach

Each requirement in the spec delta lists concrete `it(...)` titles under
`**Verified by:**`. Test files:

- `src/error-codes/error-codes.test.ts` — registry shape, every code thrown.
- `src/parsers/csv.test.ts` — quoted fields, embedded newlines, custom delimiters, no headers, empty input.
- `src/parsers/json.test.ts` — array, non-array reject, scalar wrap.
- `src/parsers/xlsx.test.ts` — TSV split, no headers, trim behaviour.
- `src/transform/pipeline.test.ts` — every coerce branch, every validation rule, computed fields, transform failure rollback.
- `src/progress/tracker.test.ts` — single + batch increments, rows/sec, ETA, percent, listener fan-out, isComplete edge case.
- `src/store/memory.test.ts` — CRUD, list filter, missing-id error.
- `src/service/service.test.ts` — preview, startImport happy path, dry-run, skipDuplicates, onRow, batch progress emit, owner scoping, format validation, parse failure → IMPORT.PROCESSING_FAILED, mapping required, getJob 404.
- `src/routes/problem-json.test.ts` — SuaveplanError → status from context; unknown error → 500; HttpError shape.
- `src/routes/build-routes.test.ts` — every route happy path + 401 (no auth) + 404 (foreign job).
- `src/otel/instruments.test.ts` — cached bundle, reset rebinds, real `MeterProvider` collects emitted metrics.
- `src/plugin/plugin.test.ts` — descriptor shape, `INVALID_OPTIONS` on bad config, `onInit` registers service, `onReady` mounts routes, `onClose` clears state.
- `src/plugin/plugin.spec-conformance.test.ts` — generics threaded, Zod boundary, every code reachable, OTel emit asserted, lifecycle dedup, no retroactive ticks.

## Migration impact

None — additive only. No genesis package depends on
`@suaveplan/api-plugin-import` today (it is being introduced). The plugin
opts-in to `@suaveplan/api-plugin-auth` via `optionalDependencies`, so apps
that don't ship auth still register; their `requireAuth()` calls simply
throw `AUTH.UNAUTHENTICATED` per the auth plugin's contract.

## Impact

### Affected dependents

`bunx turbo ls --filter=...@suaveplan/api-plugin-import` enumerates zero
dependents — this is a fresh package.

- No dependents (verified by `bunx turbo ls --filter=...@suaveplan/api-plugin-import`).