Initial Commit
This commit is contained in:
139
openspec/archive/2026-05-19-add-api-plugin-import/proposal.md
Normal file
139
openspec/archive/2026-05-19-add-api-plugin-import/proposal.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Add @suaveplan/api-plugin-import
|
||||
|
||||
**Package**: `@suaveplan/api-plugin-import`
|
||||
**Status**: PROPOSED
|
||||
**Type**: new-package
|
||||
|
||||
- **Design**: ./design.md (omitted — every architectural decision lives in the spec delta's Implementation Notes references and in the per-feature `.md` files)
|
||||
- **Spec delta**: ./specs/api-plugin-import/spec.md
|
||||
|
||||
## Why
|
||||
|
||||
Genesis needs a tabular-data ingest plugin for `@suaveplan/api` so user-facing apps can accept CSV / JSON / XLSX uploads, preview them, apply column mappings + per-row transforms + field validations, run dry-runs, persist import jobs, and stream progress events. The upstream
|
||||
`@suaveplan/api-plugin-import@1.1.7` (`/mnt/s/web/suaveplan/packages/server/api-plugin-import`, 2,320 LOC, ~98.72 % historical coverage) carries the entire feature surface but ships in a half-migrated state — two parallel `ImportService` implementations, seven `@deprecated` factory helpers, no `definePlugin()` export, an inline ad-hoc `ImportError` next to a proper `BaseError`-backed one, and a route bundle that hand-rolls Hono context casts.
|
||||
|
||||
This change lands a single, enterprise-grade `@suaveplan/api-plugin-import@0.1.0` that consolidates the two services, removes every deprecated alias, wires a real `createImportPlugin()` onto the genesis `PluginDescriptor` from `@suaveplan/api-plugin`, gates routes through `requireAuth()` from `@suaveplan/api-plugin-auth`, scopes job lookups to `c.get("user").id` with 404 anti-enumeration, ships OTel instruments asserted with a real in-memory `MeterProvider`, and rewrites every test against `@suaveplan/testing/runner`.
|
||||
|
||||
## What changes
|
||||
|
||||
- ADDED: `IMPORT_CODES` / `ImportError` — `defineCodes("IMPORT", […])` registry of nine codes (`NOT_FOUND`, `FORBIDDEN`, `VALIDATION`, `FILE_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `PROCESSING_FAILED`, `STORAGE_ERROR`, `TIMEOUT`, `INVALID_OPTIONS`) extending `SuaveplanError`.
|
||||
- ADDED: `parseCsv` / `parseCsvLine` / `detectColumns` / `streamParseCsv` / `detectHeaders` / `parseStreamLine` / `applyColumnMappings` — CSV utilities incl. quoted-field handling, custom delimiter, header detection, embedded-newline tolerance.
|
||||
- ADDED: `parseJson` — JSON-array parser with non-array rejection.
|
||||
- ADDED: `parseXlsx` — TSV-style XLSX parser (lightweight; documented limitation).
|
||||
- ADDED: `coerceType` / `validateField` / `transformRow` / `transformRows` — declarative transform pipeline with type coercion (`number` / `boolean` / `date`), required/default-value handling, per-field validation (`required` / `min` / `max` / `pattern` / `custom`), computed fields.
|
||||
- ADDED: `ImportProgressTracker` — real-time row tracker with rows/sec, ETA, percent-complete, batch helpers, listener fan-out.
|
||||
- ADDED: `MemoryImportStore` + `ImportStoreAdapter` — async CRUD + list-by-status interface; in-memory implementation for tests and small deployments.
|
||||
- ADDED: `ImportService` (canonical) — `EventEmitter<ImportEventMap>`-based service with `preview` / `startImport` / `getJob` / `getResults` / `listJobs` / `deleteJob`, batch progress emission, duplicate-skip, `onRow` filter, OTel instrumentation, scoped-by-owner store writes.
|
||||
- ADDED: `createImportRoutes` — 5 OpenAPI routes (`POST /preview`, `POST /`, `GET /`, `GET /{id}`, `DELETE /{id}`) emitting RFC 7807 problem+json on failure, validating bodies via Zod, scoping all reads / writes to the caller's `Principal.id`.
|
||||
- ADDED: `createImportPlugin` — `definePlugin<ImportPluginConfig>` factory wrapping the genesis `PluginDescriptor` (NOT the upstream `definePlugin` from `@suaveplan/api`) with 4-phase lifecycle, registering `importService` on the `ServiceRegistry`, mounting routes under a configurable `basePath`, optional-depending on `auth` / `events` / `database`.
|
||||
- ADDED: `respondProblemJson` / `PROBLEM_JSON_CONTENT_TYPE` — RFC 7807 helper bridging `SuaveplanError` (and unknown throwables) to a `Response`.
|
||||
- ADDED: `getImportInstruments` / `resetImportInstruments` / `ImportInstruments` — OTel counter+histogram bundle (`import.jobs_total`, `import.rows_total`, `import.row_errors_total`, `import.duration_ms`) with cache reset for test rebinding.
|
||||
|
||||
### Excluded from this migration
|
||||
|
||||
The genesis `0.1.0` ships at functional parity with the upstream library surface
|
||||
(parsers + transform pipeline + progress + memory store + service + routes +
|
||||
plugin). The §3 enterprise-doc enhancements that depend on packages not yet
|
||||
landed on genesis are deferred — each with a concrete architectural
|
||||
justification — and are NOT silently dropped:
|
||||
|
||||
- **`SurrealImportStore`** — requires `@suaveplan/surrealdb` migrations + the
|
||||
Surreal-side schema. Genesis ships `MemoryImportStore` against the same
|
||||
`ImportStoreAdapter` interface so the SurrealDB adapter lands as a sibling
|
||||
package without breaking consumers. Listed in pending follow-ups.
|
||||
- **Rate-limit + idempotency + audit middleware** — depends on
|
||||
`@suaveplan/api-plugin-rate-limit`, `@suaveplan/idempotency`,
|
||||
`@suaveplan/api-plugin-audit`. These are wired by the host app via the
|
||||
standard `app.use(prefix, middleware)` Hono pattern; the plugin does NOT
|
||||
hard-depend on them, so excluding them from THIS migration does not block
|
||||
consumers from layering them on top.
|
||||
- **Streaming `Readable`/`AsyncIterable` input** — `streamParseCsv` is shipped
|
||||
and the public `startImport` accepts a `string` (parity with upstream). A
|
||||
follow-up will extend the input union to `string | AsyncIterable<string>`
|
||||
without breaking the existing call site.
|
||||
- **Benchmarks under `src/__benchmarks__/`** — every benchmark requires a
|
||||
mitata harness; the harness lives in `@suaveplan/testing` but the
|
||||
benchmark gate is monorepo-wide and shipped per-package in a separate
|
||||
benchmark wave.
|
||||
- **Foreign-key resolution, parallel batch workers, resumable checkpoints,
|
||||
S3/URL source adapters, SSE progress push** — every item is a §4 future
|
||||
enhancement, not a §2 current-state capability. None are part of the
|
||||
upstream public surface; deferring them is parity, not regression.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Modifying any other `packages/api/api-plugin-*/` package, `packages/messaging/*`, `packages/audio/*`, `packages/api/api-migrations/`, `packages/database/surrealdb/`, `packages/core/*`, `MANIFEST.md`, `STATUS.md`, `CLAUDE.md`, or the root `package.json` (except `bun.lock`). Concurrent sessions own those scopes.
|
||||
- The 7 `@deprecated` factory helpers (`jobNotFound`, `invalidFormat`, `importFailed`, `validationError`, `parseError`, `mappingError`, `duplicateDetected`). Genesis bans `@deprecated` in source per `AGENTS.md §2`; the canonical helpers (`importNotFound`, `importUnsupportedFormat`, `importProcessingFailed`, `importValidation`) ship as the only API.
|
||||
- The legacy `services/import.service.ts` implementation. Genesis ships exactly one canonical `ImportService` per upstream `enterprise/api-plugin-import.md` §3 row 1.
|
||||
|
||||
## Source mapping
|
||||
|
||||
This change derives from `/mnt/s/web/suaveplan/enterprise/api-plugin-import.md`.
|
||||
|
||||
| Requirement | Source lines | Notes |
|
||||
|---|---|---|
|
||||
| Single canonical `ImportService` | L46 (§3 row 1) | "Converge on one `ImportService`" — genesis ships the canonical EventEmitter-based one |
|
||||
| `createImportPlugin()` wiring routes | L47-L48 (§3 row 2) | Wraps onto genesis `PluginDescriptor` (not upstream `definePlugin`) |
|
||||
| No `@deprecated` helpers | L49 (§3 row 3) + L37, L40 | Enforced by `AGENTS.md §2` — `grep` gate is zero-match |
|
||||
| XLSX support (TSV interchange) | L50 (§3 row 4) + upstream `src/parsers/xlsx.parser.ts` | Genesis ships TSV-style parity; binary XLSX is documented limitation |
|
||||
| Async `ImportStoreAdapter` + `MemoryImportStore` | L29, L51 (§3 row 5) | SurrealDB adapter deferred per `### Excluded` |
|
||||
| Streaming CSV parser | L26, L52 (§3 row 6) | `streamParseCsv` shipped; `string` input retained for parity |
|
||||
| Auth-gated routes via `requireAuth()` | L53 (§3 row 7) | Genesis `@suaveplan/api-plugin-auth` substitutes for upstream `@suaveplan/jwt-tenant` |
|
||||
| Owner-scoped jobs with 404 anti-enumeration | L53 (§3 row 7) + path-A safety doc | Foreign job lookups return 404 not 403 |
|
||||
| OTel instruments | L57 (§3 row 9) | `import.jobs_total`, `import.rows_total`, `import.row_errors_total`, `import.duration_ms` |
|
||||
| Structured logger via `LoggerLike` | L57 (§3 row 9) | Uses injected `ctx.logger` |
|
||||
| RFC 7807 problem+json mapping | upstream `src/routes/problem-json.ts` | Genesis re-renders via `SuaveplanError.context.statusCode` |
|
||||
| Stable `IMPORT.*` error registry | L30, L37, L106 | `defineCodes("IMPORT", […])` — 9 codes |
|
||||
| `ImportProgressTracker` with rows/sec + ETA | upstream `src/progress/import-progress.ts` | Verbatim functional parity |
|
||||
| `transformRow` / `transformRows` / `coerceType` / `validateField` | upstream `src/transform/transform-pipeline.ts` | Verbatim functional parity |
|
||||
| `parseCsv` / `parseCsvLine` / `detectColumns` / `parseJson` / `streamParseCsv` | upstream `src/types/import.types.ts` + `src/parsers/stream-parser.ts` | Verbatim functional parity |
|
||||
| 5 OpenAPI routes (preview, create, list, get, delete) | upstream `src/routes/import.routes.ts` | Owner scoping is new; route set is verbatim |
|
||||
| `ImportError` extends `SuaveplanError` | upstream `src/errors/import-errors.ts` + `AGENTS.md §2` | Genesis uses `SuaveplanError`, not upstream `BaseError` |
|
||||
| `ImportEventMap` + EventEmitter | upstream `src/services/import-service.ts` L30 | Verbatim |
|
||||
| `ImportJobRecord` async store interface | upstream `src/store/import-store.ts` | Verbatim shape |
|
||||
| Subpath exports per feature module | `AGENTS.md §6`, repo convention | Per-feature `./<dir>/<file>` entries |
|
||||
|
||||
### Requirements with no source line
|
||||
|
||||
| Requirement | Justification |
|
||||
|---|---|
|
||||
| 100 % istanbul coverage thresholds | Genesis-wide policy (`AGENTS.md §7`) |
|
||||
| `**Verified by:**` lines at draft time | Genesis-wide policy (`AGENTS.md §13`) |
|
||||
| Test imports via `@suaveplan/testing/runner` | Genesis-wide policy (`CLAUDE.md` rule 11) |
|
||||
| Explicit named re-exports in feature barrels | Genesis-wide policy (`AGENTS.md §6a`) |
|
||||
|
||||
## Verification approach
|
||||
|
||||
Each requirement in the spec delta lists concrete `it(...)` titles under
|
||||
`**Verified by:**`. Test files:
|
||||
|
||||
- `src/error-codes/error-codes.test.ts` — registry shape, every code thrown.
|
||||
- `src/parsers/csv.test.ts` — quoted fields, embedded newlines, custom delimiters, no headers, empty input.
|
||||
- `src/parsers/json.test.ts` — array, non-array reject, scalar wrap.
|
||||
- `src/parsers/xlsx.test.ts` — TSV split, no headers, trim behaviour.
|
||||
- `src/transform/pipeline.test.ts` — every coerce branch, every validation rule, computed fields, transform failure rollback.
|
||||
- `src/progress/tracker.test.ts` — single + batch increments, rows/sec, ETA, percent, listener fan-out, isComplete edge case.
|
||||
- `src/store/memory.test.ts` — CRUD, list filter, missing-id error.
|
||||
- `src/service/service.test.ts` — preview, startImport happy path, dry-run, skipDuplicates, onRow, batch progress emit, owner scoping, format validation, parse failure → IMPORT.PROCESSING_FAILED, mapping required, getJob 404.
|
||||
- `src/routes/problem-json.test.ts` — SuaveplanError → status from context; unknown error → 500; HttpError shape.
|
||||
- `src/routes/build-routes.test.ts` — every route happy path + 401 (no auth) + 404 (foreign job).
|
||||
- `src/otel/instruments.test.ts` — cached bundle, reset rebinds, real `MeterProvider` collects emitted metrics.
|
||||
- `src/plugin/plugin.test.ts` — descriptor shape, `INVALID_OPTIONS` on bad config, `onInit` registers service, `onReady` mounts routes, `onClose` clears state.
|
||||
- `src/plugin/plugin.spec-conformance.test.ts` — generics threaded, Zod boundary, every code reachable, OTel emit asserted, lifecycle dedup, no retroactive ticks.
|
||||
|
||||
## Migration impact
|
||||
|
||||
None — additive only. No genesis package depends on
|
||||
`@suaveplan/api-plugin-import` today (it is being introduced). The plugin
|
||||
opts-in to `@suaveplan/api-plugin-auth` via `optionalDependencies`, so apps
|
||||
that don't ship auth still register; their `requireAuth()` calls simply
|
||||
throw `AUTH.UNAUTHENTICATED` per the auth plugin's contract.
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
`bunx turbo ls --filter=...@suaveplan/api-plugin-import` enumerates zero
|
||||
dependents — this is a fresh package.
|
||||
|
||||
- No dependents (verified by `bunx turbo ls --filter=...@suaveplan/api-plugin-import`).
|
||||
@@ -0,0 +1,259 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: api-plugin-import (add-api-plugin-import)
|
||||
|
||||
**Package**: `@suaveplan/api-plugin-import`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Stable IMPORT.* error registry
|
||||
|
||||
The package SHALL register exactly nine stable error codes under the `"IMPORT"` namespace via `defineCodes("IMPORT", […])`, exporting `IMPORT_CODES` and `ImportError extends SuaveplanError<ImportErrorCode>`. Every code SHALL be reachable from a real production code path; deletion to satisfy a coverage gate is forbidden.
|
||||
|
||||
#### Scenario: registry declares the nine canonical codes
|
||||
|
||||
- **WHEN** a caller imports `IMPORT_CODES`
|
||||
- **THEN** the literal keys equal `{NOT_FOUND, FORBIDDEN, VALIDATION, FILE_TOO_LARGE, UNSUPPORTED_FORMAT, PROCESSING_FAILED, STORAGE_ERROR, TIMEOUT, INVALID_OPTIONS}` and every value is namespaced `"IMPORT.<KEY>"`
|
||||
|
||||
#### Scenario: every code is thrown from a production path
|
||||
|
||||
- **WHEN** the test suite drives the full feature surface
|
||||
- **THEN** every key in `IMPORT_CODES` appears as `error.code` on at least one captured throw
|
||||
|
||||
**Verified by:** `src/error-codes/error-codes.test.ts::"declares all nine IMPORT codes"` AND `src/error-codes/error-codes.test.ts::"every IMPORT code is reachable from a production path"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: CSV parsing — quoted, custom-delimiter, header-aware
|
||||
|
||||
The package SHALL parse CSV input into `Record<string, string>[]` rows respecting `"`-quoted fields (including escaped `""` quotes and embedded newlines), custom delimiters, optional headers (defaulting to `column_N` placeholders when absent), and SHALL expose a streaming variant (`streamParseCsv`) that surfaces per-row validation errors without aborting the run.
|
||||
|
||||
#### Scenario: header-first CSV with quoted comma
|
||||
|
||||
- **WHEN** input is `name,city\n"Doe, Jane",NYC`
|
||||
- **THEN** the result is `[{ name: "Doe, Jane", city: "NYC" }]`
|
||||
|
||||
#### Scenario: streaming parser collects validation errors
|
||||
|
||||
- **WHEN** `streamParseCsv` is invoked with `validate` returning `"bad"` on row 1
|
||||
- **THEN** the result reports `errors.length === 1` with `row: 1, message: "bad"` and `validRows` excludes that row
|
||||
|
||||
**Verified by:** `src/parsers/csv.test.ts::"parses quoted fields with embedded commas"` AND `src/parsers/csv.test.ts::"streamParseCsv surfaces per-row validate errors"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: JSON parser rejects non-arrays
|
||||
|
||||
The package SHALL accept a JSON-array input string and return its elements as rows; non-array input SHALL throw with a message starting `JSON data must be an array`.
|
||||
|
||||
#### Scenario: valid JSON array
|
||||
|
||||
- **WHEN** input is `[{"a":1}]`
|
||||
- **THEN** `parseJson(input)` returns `[{ a: 1 }]`
|
||||
|
||||
#### Scenario: invalid non-array input
|
||||
|
||||
- **WHEN** input is `{"a":1}`
|
||||
- **THEN** `parseJson(input)` throws an `Error` whose message contains `JSON data must be an array`
|
||||
|
||||
**Verified by:** `src/parsers/json.test.ts::"returns array elements as rows"` AND `src/parsers/json.test.ts::"throws when input is not a JSON array"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: XLSX (TSV-style) parser
|
||||
|
||||
The package SHALL provide `parseXlsx` that treats `\t`-separated lines as rows with optional header inference; binary `.xlsx` decoding is explicitly out of scope and SHALL be documented as a limitation.
|
||||
|
||||
#### Scenario: TSV input with header row
|
||||
|
||||
- **WHEN** input is `a\tb\n1\t2`
|
||||
- **THEN** `parseXlsx(input)` returns `{ headers: ["a","b"], rows: [{a:"1",b:"2"}], totalRows: 1 }`
|
||||
|
||||
**Verified by:** `src/parsers/xlsx.test.ts::"parses TSV with headers"` AND `src/parsers/xlsx.test.ts::"synthesises column_N headers when hasHeaders is false"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Transform pipeline — coerce, validate, compute
|
||||
|
||||
The package SHALL provide a declarative `transformRow` / `transformRows` pipeline that applies `TransformMapping[]` (source→target with optional type coercion, default value, required flag, transform fn), `ComputedField[]` (target derived from named source columns), and `FieldValidation[]` (`required` / `min` / `max` / `pattern` / `custom`). `coerceType` SHALL succeed/fail explicitly for `number` / `boolean` / `date` / pass-through for `string`. Failed transforms SHALL push a `TransformError` and skip the field, never throw.
|
||||
|
||||
#### Scenario: required field missing produces TransformError
|
||||
|
||||
- **WHEN** `transformRow({}, 0, { mappings: [], validations: [{ field: "x", rule: "required" }] })` runs against an empty `result`
|
||||
- **THEN** the result is `{ success: false, errors: [{ row: 0, field: "x", message: '"x" is required', value: undefined }] }`
|
||||
|
||||
#### Scenario: type coercion failure marks row failed
|
||||
|
||||
- **WHEN** `transformRow({ x: "abc" }, 0, { mappings: [{ sourceColumn: "x", targetField: "x", type: "number" }] })`
|
||||
- **THEN** `errors` contains one entry with `message: "Cannot coerce to number"` and `result.x` is absent
|
||||
|
||||
#### Scenario: computed field aggregates source columns
|
||||
|
||||
- **WHEN** `transformRow({ a: 1, b: 2 }, 0, { mappings: [], computedFields: [{ targetField: "sum", sourceColumns: ["a","b"], compute: v => Number(v.a) + Number(v.b) }] })`
|
||||
- **THEN** `row.sum === 3`
|
||||
|
||||
**Verified by:** `src/transform/pipeline.test.ts::"validateField required produces error for missing value"` AND `src/transform/pipeline.test.ts::"coerceType number returns ok:false for non-numeric input"` AND `src/transform/pipeline.test.ts::"computed field is evaluated from named source columns"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: ImportProgressTracker — rows/sec, ETA, percent
|
||||
|
||||
The package SHALL ship an `ImportProgressTracker` that records single + batch successes/failures, computes `percentComplete` (rounded int), `rowsPerSecond` (2dp), and `estimatedRemainingMs`, fans snapshots to registered listeners, and SHALL expose an `isComplete()` predicate that returns `true` iff `processedRows >= totalRows`.
|
||||
|
||||
#### Scenario: batch success updates totals and notifies listeners
|
||||
|
||||
- **WHEN** `tracker.recordBatchSuccess(5)` runs after a listener is registered
|
||||
- **THEN** the listener receives a snapshot with `successCount: 5` and `processedRows: 5`
|
||||
|
||||
#### Scenario: zero totalRows yields zero percent
|
||||
|
||||
- **WHEN** `new ImportProgressTracker("j", 0).snapshot()` runs
|
||||
- **THEN** `percentComplete === 0` and `rowsPerSecond` is finite (`0` or a small positive number)
|
||||
|
||||
**Verified by:** `src/progress/tracker.test.ts::"recordBatchSuccess fans a snapshot with the cumulative counts"` AND `src/progress/tracker.test.ts::"snapshot guards against zero totalRows"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Async ImportStoreAdapter + MemoryImportStore
|
||||
|
||||
The package SHALL declare an async `ImportStoreAdapter` interface with `create` / `get` / `update` / `delete` / `list` and SHALL ship a `MemoryImportStore` implementing it. `update` on a missing id SHALL throw with `Job "<id>" not found`. `list` SHALL filter by `status` and respect `limit` / `offset`.
|
||||
|
||||
#### Scenario: round-trip create + get
|
||||
|
||||
- **WHEN** a record is created and then fetched by id
|
||||
- **THEN** the fetched value equals the created one
|
||||
|
||||
#### Scenario: update on missing id throws
|
||||
|
||||
- **WHEN** `update("missing", {...})` runs on an empty store
|
||||
- **THEN** it rejects with an error whose message matches `Job "missing" not found`
|
||||
|
||||
**Verified by:** `src/store/memory.test.ts::"round-trips a record via create and get"` AND `src/store/memory.test.ts::"throws when updating a missing id"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: ImportService — preview, startImport, owner-scoped jobs
|
||||
|
||||
The package SHALL ship a canonical `ImportService extends EventEmitter<ImportEventMap>` with `preview(format, data)`, `startImport(config, { ownerId })`, `getJob(id, ownerId)`, `getResults(id, ownerId)`, `listJobs(ownerId)`, `deleteJob(id, ownerId)`. Foreign-owner reads SHALL throw `IMPORT.NOT_FOUND` (anti-enumeration; never `IMPORT.FORBIDDEN` on lookup misses). Format validation SHALL throw `IMPORT.UNSUPPORTED_FORMAT`; parse failures SHALL throw `IMPORT.PROCESSING_FAILED`; missing mappings SHALL throw `IMPORT.VALIDATION`. The service SHALL emit `import.started`, `import.progress`, `import.completed`, `import.failed`, `import.row_error`, `import:row:processed` per the documented `ImportEventMap`. Successful runs SHALL increment `import.jobs_total{outcome:"success"}` and `import.rows_total` OTel counters.
|
||||
|
||||
#### Scenario: preview returns column auto-mapping
|
||||
|
||||
- **WHEN** `preview("csv", "a,b\n1,2")` runs
|
||||
- **THEN** the returned `mappings` has `sourceColumn === targetField` for every detected column
|
||||
|
||||
#### Scenario: dry-run does not persist results
|
||||
|
||||
- **WHEN** `startImport({ format, data, mappings, dryRun: true }, { ownerId })` succeeds
|
||||
- **THEN** `getResults(jobId, ownerId)` returns `[]`
|
||||
|
||||
#### Scenario: foreign-owner lookup returns 404 not 403
|
||||
|
||||
- **WHEN** `getJob(jobId, "other-user")` runs on a job owned by `"alice"`
|
||||
- **THEN** it throws `ImportError` with `code === IMPORT_CODES.NOT_FOUND`
|
||||
|
||||
#### Scenario: parse failure surfaces as IMPORT.PROCESSING_FAILED
|
||||
|
||||
- **WHEN** `startImport({ format: "json", data: "not-json", mappings: [{ sourceColumn: "x", targetField: "x" }] })` runs
|
||||
- **THEN** it throws `ImportError` with `code === IMPORT_CODES.PROCESSING_FAILED`
|
||||
|
||||
**Verified by:** `src/service/service.test.ts::"preview returns identity column mappings"` AND `src/service/service.test.ts::"dryRun does not persist imported rows"` AND `src/service/service.test.ts::"foreign owner lookup returns NOT_FOUND"` AND `src/service/service.test.ts::"parse failure throws PROCESSING_FAILED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 5 Hono routes, auth-gated, owner-scoped
|
||||
|
||||
The package SHALL ship `createImportRoutes({ service, instruments })` returning an `OpenAPIHono` with five routes — `POST /preview`, `POST /`, `GET /`, `GET /{id}`, `DELETE /{id}`. Every route SHALL be wrapped in `requireAuth()` from `@suaveplan/api-plugin-auth`. Job-scoped routes SHALL read `c.get("user").id` as `ownerId`. Lookup misses on jobs owned by a different user SHALL return HTTP 404 with RFC 7807 problem+json (anti-enumeration).
|
||||
|
||||
#### Scenario: anonymous request to POST / returns 401
|
||||
|
||||
- **WHEN** a request reaches `POST /` with `c.get("user") === null`
|
||||
- **THEN** the response is HTTP 401 with `content-type: application/problem+json`
|
||||
|
||||
#### Scenario: foreign-owner GET /{id} returns 404
|
||||
|
||||
- **WHEN** alice owns job `j1` and bob calls `GET /j1`
|
||||
- **THEN** the response is HTTP 404 with problem+json
|
||||
|
||||
**Verified by:** `src/routes/build-routes.test.ts::"POST / returns 401 when caller is anonymous"` AND `src/routes/build-routes.test.ts::"GET /{id} returns 404 when the job is owned by a different user"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: RFC 7807 problem+json mapping
|
||||
|
||||
The package SHALL provide `respondProblemJson(c, err)` that renders any `SuaveplanError` as RFC 7807 `application/problem+json`, reading `error.context.statusCode` when present and defaulting to 500. Unknown throwables SHALL render as a generic 500.
|
||||
|
||||
#### Scenario: SuaveplanError with statusCode 404
|
||||
|
||||
- **WHEN** the error carries `context.statusCode = 404`
|
||||
- **THEN** the response status is 404 and the body is JSON with the `code` and `message` fields
|
||||
|
||||
#### Scenario: unknown throwable yields 500
|
||||
|
||||
- **WHEN** a plain `string` is thrown
|
||||
- **THEN** the response status is 500 with `content-type: application/problem+json`
|
||||
|
||||
**Verified by:** `src/routes/problem-json.test.ts::"renders SuaveplanError with the context statusCode"` AND `src/routes/problem-json.test.ts::"renders unknown throwables as a generic 500"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: OTel instruments bundle
|
||||
|
||||
The package SHALL export `getImportInstruments()` returning a cached bundle of four OpenTelemetry instruments — `import.jobs_total` (Counter, labelled by `outcome`), `import.rows_total` (Counter, labelled by `outcome`), `import.row_errors_total` (Counter), `import.duration_ms` (Histogram). `resetImportInstruments()` SHALL drop the cache so subsequent calls rebind against the current global `MeterProvider`. Production code paths SHALL emit on every `startImport`.
|
||||
|
||||
#### Scenario: cached bundle returns same reference
|
||||
|
||||
- **WHEN** `getImportInstruments()` is called twice without reset
|
||||
- **THEN** the second call returns the same object
|
||||
|
||||
#### Scenario: emit after install collects metrics
|
||||
|
||||
- **WHEN** a fresh `MeterProvider` is installed, `resetImportInstruments()` runs, and a counter is incremented
|
||||
- **THEN** the in-memory exporter contains the metric
|
||||
|
||||
**Verified by:** `src/otel/instruments.test.ts::"getImportInstruments returns a cached bundle"` AND `src/otel/instruments.test.ts::"resetImportInstruments rebinds against a fresh MeterProvider"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: createImportPlugin descriptor
|
||||
|
||||
The package SHALL export `createImportPlugin(options?)` returning a typed `PluginDescriptor<ImportPluginConfig>` whose `name: "import"`, `version: "0.1.0"`, `optionalDependencies: ["auth", "events", "database"]`, and `tags: ["import", "csv", "json", "xlsx"]`. Invalid options SHALL throw `ImportError(IMPORT.INVALID_OPTIONS)` with the `ZodError` as `cause`. `onInit` SHALL register the `ImportService` instance on `ServiceRegistry["importService"]`; `onReady` SHALL mount routes under the configured `routePrefix`; `onClose` SHALL clear all listeners on the service.
|
||||
|
||||
#### Scenario: bad options throws INVALID_OPTIONS
|
||||
|
||||
- **WHEN** `createImportPlugin({ routePrefix: "" })` runs
|
||||
- **THEN** it throws `ImportError` with `code === IMPORT_CODES.INVALID_OPTIONS` and `cause instanceof ZodError`
|
||||
|
||||
#### Scenario: onInit registers the service slot
|
||||
|
||||
- **WHEN** `descriptor.onInit(ctx)` runs
|
||||
- **THEN** `ctx.getService("importService")` is defined and is an `ImportService` instance
|
||||
|
||||
#### Scenario: onReady mounts routes under routePrefix
|
||||
|
||||
- **WHEN** `descriptor.onReady(ctx)` runs with `routePrefix: "/imports"`
|
||||
- **THEN** the host `app.routes` contains an entry whose path starts with `/imports/`
|
||||
|
||||
**Verified by:** `src/plugin/plugin.test.ts::"throws INVALID_OPTIONS on bad config"` AND `src/plugin/plugin.test.ts::"onInit registers the importService slot"` AND `src/plugin/plugin.test.ts::"onReady mounts routes under the configured routePrefix"` AND `src/plugin/plugin.spec-conformance.test.ts::"descriptor carries the canonical metadata"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Subpath exports per feature module
|
||||
|
||||
The package SHALL declare a subpath export in `package.json#exports` for every feature module so consumers can tree-shake. The package root SHALL re-export public symbols via explicit named re-exports; feature-module barrels SHALL NOT use `export *`.
|
||||
|
||||
#### Scenario: subpath export count matches feature count
|
||||
|
||||
- **WHEN** the `exports` map is enumerated
|
||||
- **THEN** it contains entries for `./error-codes/error-codes`, `./parsers/csv`, `./parsers/json`, `./parsers/xlsx`, `./transform/pipeline`, `./progress/tracker`, `./store/memory`, `./service/service`, `./routes/problem-json`, `./routes/build-routes`, `./otel/instruments`, `./schemas/options`, `./schemas/schemas`, `./plugin/plugin`, `./types/types`
|
||||
|
||||
#### Scenario: feature barrels use explicit named re-exports
|
||||
|
||||
- **WHEN** the file tree is scanned for `export *` in `src/<feature>/index.ts`
|
||||
- **THEN** no matches exist (the conformance script enforces this)
|
||||
|
||||
**Verified by:** `src/plugin/plugin.spec-conformance.test.ts::"package exposes a subpath export for every feature module"` AND `src/plugin/plugin.spec-conformance.test.ts::"every feature barrel uses explicit named re-exports"`
|
||||
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
77
openspec/archive/2026-05-19-add-api-plugin-import/tasks.md
Normal file
77
openspec/archive/2026-05-19-add-api-plugin-import/tasks.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Tasks — Add @suaveplan/api-plugin-import
|
||||
|
||||
> Every checkbox MUST reflect work actually done in this change. No retroactive ticking.
|
||||
|
||||
## 1. Spec authoring
|
||||
|
||||
- [x] 1.1 Read `/mnt/s/web/suaveplan/enterprise/api-plugin-import.md` end-to-end
|
||||
- [x] 1.2 Confirm no prior `openspec/specs/api-plugin-import/spec.md` exists
|
||||
- [x] 1.3 Read `openspec/AGENTS.md` §§ 2a, 6a, 7, 12, 13, 14
|
||||
- [x] 1.4 Draft `proposal.md` with `## Source mapping` table + `### Excluded from this migration`
|
||||
- [x] 1.5 Source-mapping table cites every requirement to upstream `enterprise/<pkg>.md` lines OR the "no source line" justification table
|
||||
- [x] 1.6 Draft `specs/api-plugin-import/spec.md`
|
||||
- [x] 1.7 Every requirement carries a `**Verified by:**` line naming concrete `it(...)` titles
|
||||
|
||||
## 2. Implementation
|
||||
|
||||
- [ ] 2.1 `src/error-codes/error-codes.ts` — `defineCodes("IMPORT", […])` (9 codes) + `ImportError extends SuaveplanError`
|
||||
- [ ] 2.2 `src/types/types.ts` — shared types (ImportFormat, ImportStatus, ColumnMapping, RowError, ImportJobRecord, ImportEventMap, …)
|
||||
- [ ] 2.3 `src/parsers/csv.ts` — `parseCsv`, `parseCsvLine`, `detectColumns`, `streamParseCsv`, `detectHeaders`, `parseStreamLine`, `applyColumnMappings`
|
||||
- [ ] 2.4 `src/parsers/json.ts` — `parseJson`
|
||||
- [ ] 2.5 `src/parsers/xlsx.ts` — `parseXlsx` (TSV-style)
|
||||
- [ ] 2.6 `src/transform/pipeline.ts` — `coerceType`, `validateField`, `transformRow`, `transformRows`, `validateRow`, `validateUnique`
|
||||
- [ ] 2.7 `src/progress/tracker.ts` — `ImportProgressTracker`
|
||||
- [ ] 2.8 `src/store/memory.ts` — `ImportStoreAdapter` interface + `MemoryImportStore`
|
||||
- [ ] 2.9 `src/otel/instruments.ts` — `getImportInstruments`, `resetImportInstruments`, `ImportInstruments`
|
||||
- [ ] 2.10 `src/schemas/options.ts` — `ImportPluginOptionsSchema`, `ImportPluginConfig`
|
||||
- [ ] 2.11 `src/schemas/schemas.ts` — public Zod surface (CreateImportJob, Preview, ImportJobResponse, …)
|
||||
- [ ] 2.12 `src/services/registry-augmentation.ts` — `ServiceRegistry` augmentation for `importService`
|
||||
- [ ] 2.13 `src/service/service.ts` — canonical `ImportService` (EventEmitter, OTel-emit, owner-scoped)
|
||||
- [ ] 2.14 `src/routes/problem-json.ts` — RFC 7807 helper
|
||||
- [ ] 2.15 `src/routes/build-routes.ts` — 5 routes with `requireAuth()` + owner scoping
|
||||
- [ ] 2.16 `src/plugin/plugin.ts` — `createImportPlugin` via `definePlugin<ImportPluginConfig>`
|
||||
- [ ] 2.17 `src/index.ts` — barrel with explicit named re-exports
|
||||
|
||||
## 3. Tests
|
||||
|
||||
- [ ] 3.1 `src/error-codes/error-codes.test.ts`
|
||||
- [ ] 3.2 `src/parsers/csv.test.ts`, `json.test.ts`, `xlsx.test.ts`
|
||||
- [ ] 3.3 `src/transform/pipeline.test.ts`
|
||||
- [ ] 3.4 `src/progress/tracker.test.ts`
|
||||
- [ ] 3.5 `src/store/memory.test.ts`
|
||||
- [ ] 3.6 `src/otel/instruments.test.ts`
|
||||
- [ ] 3.7 `src/service/service.test.ts`
|
||||
- [ ] 3.8 `src/routes/problem-json.test.ts`, `build-routes.test.ts`
|
||||
- [ ] 3.9 `src/plugin/plugin.test.ts`, `plugin.spec-conformance.test.ts`
|
||||
- [ ] 3.10 Every `it(...)` carries `{ timeout: 5000 }`
|
||||
- [ ] 3.11 Every test imports from `@suaveplan/testing/runner`
|
||||
|
||||
## 4. Documentation
|
||||
|
||||
- [ ] 4.1 Sibling `.md` for every implementation `.ts` (≥200 words, basic + advanced examples, full API reference)
|
||||
- [ ] 4.2 `README.md` at package root linking every `.md` under `## Modules`
|
||||
|
||||
## 5. Pre-archive 5-gate triple
|
||||
|
||||
- [ ] 5.1 `bun run check`
|
||||
- [ ] 5.2 `bunx turbo lint typecheck test --filter=@suaveplan/api-plugin-import --force`
|
||||
- [ ] 5.3 `bunx turbo test:coverage --filter=@suaveplan/api-plugin-import` → 100/100/100/100 istanbul
|
||||
- [ ] 5.4 `bun scripts/openspec-conformance.ts @suaveplan/api-plugin-import` → 0 errors
|
||||
- [ ] 5.5 `bunx turbo build --filter=@suaveplan/api-plugin-import --force`
|
||||
|
||||
## 6. Pre-archive 9-row checklist (skill §7)
|
||||
|
||||
- [ ] 6.1 Generics threaded; `expectTypeOf` claims pinned
|
||||
- [ ] 6.2 Zod boundary on every public option bundle
|
||||
- [ ] 6.3 Every IMPORT.* code thrown by ≥1 production path
|
||||
- [ ] 6.4 Strict-mode behaviours tested both on and off where applicable (dry-run vs real, skipDuplicates on/off, onRow filter on/off)
|
||||
- [ ] 6.5 OTel emitted from production paths, asserted via real in-memory `MeterProvider`
|
||||
- [ ] 6.6 Lifecycle dedup tested (route prefix collision, double init)
|
||||
- [ ] 6.7 No retroactive tick
|
||||
- [ ] 6.8 No `export *` in feature barrels
|
||||
- [ ] 6.9 No direct `vitest` / `bun:test` imports in test files
|
||||
|
||||
## 7. Archive
|
||||
|
||||
- [ ] 7.1 Fold spec delta into `openspec/specs/api-plugin-import/spec.md`
|
||||
- [ ] 7.2 Move `openspec/changes/add-api-plugin-import/` → `openspec/archive/2026-05-19-add-api-plugin-import/`
|
||||
Reference in New Issue
Block a user