SuavePlan/suaveplan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9b28db724f7c99ad304963ee7d8a876190dd4167
suaveplan/openspec/archive/2026-05-19-add-api-plugin-import/proposal.md
admin fbc70f96e6 Initial Commit
2026-05-27 23:29:53 +01:00

12 KiB
Raw Blame History

Add @suaveplan/api-plugin-import

Package: @suaveplan/api-plugin-import Status: PROPOSED Type: new-package

  • Design: ./design.md (omitted — every architectural decision lives in the spec delta's Implementation Notes references and in the per-feature .md files)
  • Spec delta: ./specs/api-plugin-import/spec.md

Why

Genesis needs a tabular-data ingest plugin for @suaveplan/api so user-facing apps can accept CSV / JSON / XLSX uploads, preview them, apply column mappings + per-row transforms + field validations, run dry-runs, persist import jobs, and stream progress events. The upstream @suaveplan/api-plugin-import@1.1.7 (/mnt/s/web/suaveplan/packages/server/api-plugin-import, 2,320 LOC, ~98.72 % historical coverage) carries the entire feature surface but ships in a half-migrated state — two parallel ImportService implementations, seven @deprecated factory helpers, no definePlugin() export, an inline ad-hoc ImportError next to a proper BaseError-backed one, and a route bundle that hand-rolls Hono context casts.

This change lands a single, enterprise-grade @suaveplan/api-plugin-import@0.1.0 that consolidates the two services, removes every deprecated alias, wires a real createImportPlugin() onto the genesis PluginDescriptor from @suaveplan/api-plugin, gates routes through requireAuth() from @suaveplan/api-plugin-auth, scopes job lookups to c.get("user").id with 404 anti-enumeration, ships OTel instruments asserted with a real in-memory MeterProvider, and rewrites every test against @suaveplan/testing/runner.

What changes

  • ADDED: IMPORT_CODES / ImportError — defineCodes("IMPORT", […]) registry of nine codes (NOT_FOUND, FORBIDDEN, VALIDATION, FILE_TOO_LARGE, UNSUPPORTED_FORMAT, PROCESSING_FAILED, STORAGE_ERROR, TIMEOUT, INVALID_OPTIONS) extending SuaveplanError.
  • ADDED: parseCsv / parseCsvLine / detectColumns / streamParseCsv / detectHeaders / parseStreamLine / applyColumnMappings — CSV utilities incl. quoted-field handling, custom delimiter, header detection, embedded-newline tolerance.
  • ADDED: parseJson — JSON-array parser with non-array rejection.
  • ADDED: parseXlsx — TSV-style XLSX parser (lightweight; documented limitation).
  • ADDED: coerceType / validateField / transformRow / transformRows — declarative transform pipeline with type coercion (number / boolean / date), required/default-value handling, per-field validation (required / min / max / pattern / custom), computed fields.
  • ADDED: ImportProgressTracker — real-time row tracker with rows/sec, ETA, percent-complete, batch helpers, listener fan-out.
  • ADDED: MemoryImportStore + ImportStoreAdapter — async CRUD + list-by-status interface; in-memory implementation for tests and small deployments.
  • ADDED: ImportService (canonical) — EventEmitter<ImportEventMap>-based service with preview / startImport / getJob / getResults / listJobs / deleteJob, batch progress emission, duplicate-skip, onRow filter, OTel instrumentation, scoped-by-owner store writes.
  • ADDED: createImportRoutes — 5 OpenAPI routes (POST /preview, POST /, GET /, GET /{id}, DELETE /{id}) emitting RFC 7807 problem+json on failure, validating bodies via Zod, scoping all reads / writes to the caller's Principal.id.
  • ADDED: createImportPlugin — definePlugin<ImportPluginConfig> factory wrapping the genesis PluginDescriptor (NOT the upstream definePlugin from @suaveplan/api) with 4-phase lifecycle, registering importService on the ServiceRegistry, mounting routes under a configurable basePath, optional-depending on auth / events / database.
  • ADDED: respondProblemJson / PROBLEM_JSON_CONTENT_TYPE — RFC 7807 helper bridging SuaveplanError (and unknown throwables) to a Response.
  • ADDED: getImportInstruments / resetImportInstruments / ImportInstruments — OTel counter+histogram bundle (import.jobs_total, import.rows_total, import.row_errors_total, import.duration_ms) with cache reset for test rebinding.

Excluded from this migration

The genesis 0.1.0 ships at functional parity with the upstream library surface (parsers + transform pipeline + progress + memory store + service + routes + plugin). The §3 enterprise-doc enhancements that depend on packages not yet landed on genesis are deferred — each with a concrete architectural justification — and are NOT silently dropped:

  • SurrealImportStore — requires @suaveplan/surrealdb migrations + the Surreal-side schema. Genesis ships MemoryImportStore against the same ImportStoreAdapter interface so the SurrealDB adapter lands as a sibling package without breaking consumers. Listed in pending follow-ups.
  • Rate-limit + idempotency + audit middleware — depends on @suaveplan/api-plugin-rate-limit, @suaveplan/idempotency, @suaveplan/api-plugin-audit. These are wired by the host app via the standard app.use(prefix, middleware) Hono pattern; the plugin does NOT hard-depend on them, so excluding them from THIS migration does not block consumers from layering them on top.
  • Streaming Readable/AsyncIterable input — streamParseCsv is shipped and the public startImport accepts a string (parity with upstream). A follow-up will extend the input union to string | AsyncIterable<string> without breaking the existing call site.
  • Benchmarks under src/__benchmarks__/ — every benchmark requires a mitata harness; the harness lives in @suaveplan/testing but the benchmark gate is monorepo-wide and shipped per-package in a separate benchmark wave.
  • Foreign-key resolution, parallel batch workers, resumable checkpoints, S3/URL source adapters, SSE progress push — every item is a §4 future enhancement, not a §2 current-state capability. None are part of the upstream public surface; deferring them is parity, not regression.

Out of scope

  • Modifying any other packages/api/api-plugin-*/ package, packages/messaging/*, packages/audio/*, packages/api/api-migrations/, packages/database/surrealdb/, packages/core/*, MANIFEST.md, STATUS.md, CLAUDE.md, or the root package.json (except bun.lock). Concurrent sessions own those scopes.
  • The 7 @deprecated factory helpers (jobNotFound, invalidFormat, importFailed, validationError, parseError, mappingError, duplicateDetected). Genesis bans @deprecated in source per AGENTS.md §2; the canonical helpers (importNotFound, importUnsupportedFormat, importProcessingFailed, importValidation) ship as the only API.
  • The legacy services/import.service.ts implementation. Genesis ships exactly one canonical ImportService per upstream enterprise/api-plugin-import.md §3 row 1.

Source mapping

This change derives from /mnt/s/web/suaveplan/enterprise/api-plugin-import.md.

Requirement Source lines Notes
Single canonical ImportService L46 (§3 row 1) "Converge on one ImportService" — genesis ships the canonical EventEmitter-based one
createImportPlugin() wiring routes L47-L48 (§3 row 2) Wraps onto genesis PluginDescriptor (not upstream definePlugin)
No @deprecated helpers L49 (§3 row 3) + L37, L40 Enforced by AGENTS.md §2 — grep gate is zero-match
XLSX support (TSV interchange) L50 (§3 row 4) + upstream src/parsers/xlsx.parser.ts Genesis ships TSV-style parity; binary XLSX is documented limitation
Async ImportStoreAdapter + MemoryImportStore L29, L51 (§3 row 5) SurrealDB adapter deferred per ### Excluded
Streaming CSV parser L26, L52 (§3 row 6) streamParseCsv shipped; string input retained for parity
Auth-gated routes via requireAuth() L53 (§3 row 7) Genesis @suaveplan/api-plugin-auth substitutes for upstream @suaveplan/jwt-tenant
Owner-scoped jobs with 404 anti-enumeration L53 (§3 row 7) + path-A safety doc Foreign job lookups return 404 not 403
OTel instruments L57 (§3 row 9) import.jobs_total, import.rows_total, import.row_errors_total, import.duration_ms
Structured logger via LoggerLike L57 (§3 row 9) Uses injected ctx.logger
RFC 7807 problem+json mapping upstream src/routes/problem-json.ts Genesis re-renders via SuaveplanError.context.statusCode
Stable IMPORT.* error registry L30, L37, L106 defineCodes("IMPORT", […]) — 9 codes
ImportProgressTracker with rows/sec + ETA upstream src/progress/import-progress.ts Verbatim functional parity
transformRow / transformRows / coerceType / validateField upstream src/transform/transform-pipeline.ts Verbatim functional parity
parseCsv / parseCsvLine / detectColumns / parseJson / streamParseCsv upstream src/types/import.types.ts + src/parsers/stream-parser.ts Verbatim functional parity
5 OpenAPI routes (preview, create, list, get, delete) upstream src/routes/import.routes.ts Owner scoping is new; route set is verbatim
ImportError extends SuaveplanError upstream src/errors/import-errors.ts + AGENTS.md §2 Genesis uses SuaveplanError, not upstream BaseError
ImportEventMap + EventEmitter upstream src/services/import-service.ts L30 Verbatim
ImportJobRecord async store interface upstream src/store/import-store.ts Verbatim shape
Subpath exports per feature module AGENTS.md §6, repo convention Per-feature ./<dir>/<file> entries

Requirements with no source line

Requirement Justification
100 % istanbul coverage thresholds Genesis-wide policy (AGENTS.md §7)
**Verified by:** lines at draft time Genesis-wide policy (AGENTS.md §13)
Test imports via @suaveplan/testing/runner Genesis-wide policy (CLAUDE.md rule 11)
Explicit named re-exports in feature barrels Genesis-wide policy (AGENTS.md §6a)

Verification approach

Each requirement in the spec delta lists concrete it(...) titles under **Verified by:**. Test files:

  • src/error-codes/error-codes.test.ts — registry shape, every code thrown.
  • src/parsers/csv.test.ts — quoted fields, embedded newlines, custom delimiters, no headers, empty input.
  • src/parsers/json.test.ts — array, non-array reject, scalar wrap.
  • src/parsers/xlsx.test.ts — TSV split, no headers, trim behaviour.
  • src/transform/pipeline.test.ts — every coerce branch, every validation rule, computed fields, transform failure rollback.
  • src/progress/tracker.test.ts — single + batch increments, rows/sec, ETA, percent, listener fan-out, isComplete edge case.
  • src/store/memory.test.ts — CRUD, list filter, missing-id error.
  • src/service/service.test.ts — preview, startImport happy path, dry-run, skipDuplicates, onRow, batch progress emit, owner scoping, format validation, parse failure → IMPORT.PROCESSING_FAILED, mapping required, getJob 404.
  • src/routes/problem-json.test.ts — SuaveplanError → status from context; unknown error → 500; HttpError shape.
  • src/routes/build-routes.test.ts — every route happy path + 401 (no auth) + 404 (foreign job).
  • src/otel/instruments.test.ts — cached bundle, reset rebinds, real MeterProvider collects emitted metrics.
  • src/plugin/plugin.test.ts — descriptor shape, INVALID_OPTIONS on bad config, onInit registers service, onReady mounts routes, onClose clears state.
  • src/plugin/plugin.spec-conformance.test.ts — generics threaded, Zod boundary, every code reachable, OTel emit asserted, lifecycle dedup, no retroactive ticks.

Migration impact

None — additive only. No genesis package depends on @suaveplan/api-plugin-import today (it is being introduced). The plugin opts-in to @suaveplan/api-plugin-auth via optionalDependencies, so apps that don't ship auth still register; their requireAuth() calls simply throw AUTH.UNAUTHENTICATED per the auth plugin's contract.

Impact

Affected dependents

bunx turbo ls --filter=...@suaveplan/api-plugin-import enumerates zero dependents — this is a fresh package.

  • No dependents (verified by bunx turbo ls --filter=...@suaveplan/api-plugin-import).
Reference in New Issue View Git Blame Copy Permalink