Initial Commit

openspec/changes/add-import-sse-progress/proposal.md

@@ -0,0 +1,60 @@
+# Change: add-import-sse-progress
+
+## Why
+
+`@suaveplan/api-plugin-import` shipped (archive `2026-05-19-add-api-plugin-import`) with the
+**`## Excluded from this migration`** clause explicitly deferring SSE progress push:
+
+> Foreign-key resolution, parallel batch workers, resumable checkpoints, S3/URL source adapters, **SSE progress push**
+
+This change lifts that single deferral. Consumers can now subscribe to a job's progress
+via `GET /imports/:id/progress` and receive `text/event-stream` frames driven directly off
+the existing `ImportService.bus` events — no polling. The other deferred items
+(foreign-key resolution, parallel workers, checkpoints, S3 sources) remain deferred.
+
+The endpoint reuses:
+
+- the canonical `ImportProgress` payload type,
+- the existing `import.progress` / `import.completed` / `import.failed` events,
+- the existing `IMPORT.*` error registry,
+- the owner-scoping invariants of `ImportService.getJob(jobId, ownerId)`.
+
+The endpoint introduces three new OTel counters scoped to the new feature module:
+`import.sse.connected`, `import.sse.events_sent{kind}`, `import.sse.disconnected{reason}`.
+
+## Affected requirements
+
+ADDED requirement: **SSE progress push** (one new requirement; no MODIFIED / REMOVED entries).
+
+## Source mapping
+
+| Spec scenario | Implementation | Test |
+| --- | --- | --- |
+| GET `/imports/:id/progress` opens an event-stream | `src/routes/progress-sse/progress-sse.ts` | `src/routes/progress-sse/progress-sse.test.ts::"opens a text/event-stream and emits an initial progress frame"` |
+| Each `import.progress` bus emit produces an `event: progress` frame | `src/routes/progress-sse/progress-sse.ts` | `src/routes/progress-sse/progress-sse.test.ts::"emits a progress frame for every bus progress event"` |
+| Terminal `import.completed` emits `event: complete` then closes | `src/routes/progress-sse/progress-sse.ts` | `src/routes/progress-sse/progress-sse.test.ts::"emits a complete frame and closes on import.completed"` |
+| Terminal `import.failed` emits `event: failed` then closes | `src/routes/progress-sse/progress-sse.ts` | `src/routes/progress-sse/progress-sse.test.ts::"emits a failed frame and closes on import.failed"` |
+| Foreign-owner request returns 404 problem+json | `src/routes/progress-sse/progress-sse.ts` | `src/routes/progress-sse/progress-sse.test.ts::"returns 404 problem+json for a foreign-owner job"` |
+| Anonymous request returns 401 problem+json | route is mounted under `requireAuth()` in `build-routes.ts` | `src/routes/progress-sse/progress-sse.test.ts::"returns 401 problem+json when the caller is anonymous"` |
+| Client disconnect cleans up the bus listener | `AbortSignal` wired into the `ReadableStream.cancel` path | `src/routes/progress-sse/progress-sse.test.ts::"detaches the listener and increments disconnect counter on client abort"` |
+| OTel counters increment on connect / event / disconnect | `src/routes/progress-sse/progress-sse.ts` (new instruments wired through `getImportInstruments()`) | `src/routes/progress-sse/progress-sse.test.ts::"increments import.sse counters on connect, send, and disconnect"` |
+
+## Impact
+
+### Affected dependents
+
+Run `bunx turbo ls --filter=...@suaveplan/api-plugin-import` to enumerate.
+
+- `@suaveplan/api` — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason:
+  additive route under an existing prefix; no signature changes to the consumed surface)
+
+Foundation package? No. This is a feature-additive change on a server-tier plugin.
+
+## Out of scope (still deferred from the original `Excluded` list)
+
+- Foreign-key resolution
+- Parallel batch workers
+- Resumable checkpoints
+- S3 / URL source adapters
+- Streaming input parsing
+- Benchmarks

openspec/changes/add-import-sse-progress/specs/api-plugin-import/spec.md

@@ -0,0 +1,92 @@
+<!-- markdownlint-disable MD024 -->
+
+# Spec Delta: api-plugin-import — add SSE progress push
+
+## ADDED Requirements
+
+### Requirement: SSE progress push — `GET /imports/:id/progress`
+
+The package SHALL expose `GET /imports/:id/progress` returning
+`Content-Type: text/event-stream; charset=utf-8`, mounted under the existing
+`requireAuth()` middleware. Frames SHALL be driven directly off the existing
+`ImportService.bus` events (`import.progress`, `import.completed`, `import.failed`);
+NO polling loop over the store is permitted. Every frame SHALL carry the canonical
+`ImportProgress` JSON payload. Three SSE event kinds SHALL be emitted: `progress`
+(every `import.progress` tick), `complete` (terminal, on `import.completed`),
+and `failed` (terminal, on `import.failed`). The stream SHALL close cleanly after
+a terminal event. The handler SHALL respect the request's `AbortSignal` (client
+disconnect) by detaching the bus listener and incrementing
+`import.sse.disconnected{reason: "client"}`. Foreign-owner or missing-job lookups
+SHALL return HTTP 404 RFC 7807 problem+json with `code === IMPORT.NOT_FOUND`
+(anti-enumeration parity with `GET /:id`). Anonymous requests SHALL return HTTP
+401 problem+json via the existing auth middleware.
+
+The package SHALL extend `getImportInstruments()` with three additional counters:
+`import.sse.connected` (Counter), `import.sse.events_sent` (Counter, labelled by
+`kind ∈ "progress" | "complete" | "failed"`), `import.sse.disconnected`
+(Counter, labelled by `reason ∈ "complete" | "failed" | "client"`). The counters
+SHALL be cached on the same bundle as the existing six instruments and SHALL
+rebind through `resetImportInstruments()`.
+
+#### Scenario: opens a text/event-stream and emits an initial progress frame
+
+- **WHEN** an authenticated caller `GET /imports/:id/progress` for a job they own
+- **THEN** the response is HTTP 200 with `content-type: text/event-stream; charset=utf-8`
+  and the first frame is `event: progress\ndata: <ImportProgress JSON>\n\n`
+
+#### Scenario: emits a progress frame for every bus progress event
+
+- **WHEN** the service emits `import.progress` while the client is connected
+- **THEN** the stream writes a frame whose first non-id line is `event: progress`
+  and whose `data:` line decodes to an `ImportProgress` object with the matching
+  `processedRows`
+
+#### Scenario: emits a complete frame and closes on import.completed
+
+- **WHEN** the service emits `import.completed`
+- **THEN** the stream writes a frame whose first non-id line is `event: complete`
+  and then closes; subsequent reads from the stream return `done: true`
+
+#### Scenario: emits a failed frame and closes on import.failed
+
+- **WHEN** the service emits `import.failed`
+- **THEN** the stream writes a frame whose first non-id line is `event: failed`
+  and then closes
+
+#### Scenario: returns 404 problem+json for a foreign-owner job
+
+- **WHEN** alice owns job `j1` and bob requests `GET /imports/j1/progress`
+- **THEN** the response is HTTP 404 with `content-type: application/problem+json`
+  and a body whose `code === "IMPORT.NOT_FOUND"`
+
+#### Scenario: returns 401 problem+json when the caller is anonymous
+
+- **WHEN** an anonymous caller requests `GET /imports/:id/progress`
+- **THEN** the response is HTTP 401 with `content-type: application/problem+json`
+
+#### Scenario: detaches the listener and increments disconnect counter on client abort
+
+- **WHEN** the client aborts the request before a terminal event
+- **THEN** the handler unsubscribes from the bus and increments
+  `import.sse.disconnected{reason:"client"}`
+
+#### Scenario: increments import.sse counters on connect, send, and disconnect
+
+- **WHEN** a connection opens, two progress events fire, and a terminal `complete`
+  event closes the stream
+- **THEN** `import.sse.connected` increments by 1, `import.sse.events_sent{kind}`
+  increments by 3 (2× `progress` + 1× `complete`), and
+  `import.sse.disconnected{reason:"complete"}` increments by 1
+
+**Verified by:**
+`src/routes/progress-sse/progress-sse.test.ts::"opens a text/event-stream and emits an initial progress frame"`
+AND `src/routes/progress-sse/progress-sse.test.ts::"emits a progress frame for every bus progress event"`
+AND `src/routes/progress-sse/progress-sse.test.ts::"emits a complete frame and closes on import.completed"`
+AND `src/routes/progress-sse/progress-sse.test.ts::"emits a failed frame and closes on import.failed"`
+AND `src/routes/progress-sse/progress-sse.test.ts::"returns 404 problem+json for a foreign-owner job"`
+AND `src/routes/progress-sse/progress-sse.test.ts::"returns 401 problem+json when the caller is anonymous"`
+AND `src/routes/progress-sse/progress-sse.test.ts::"detaches the listener and increments disconnect counter on client abort"`
+AND `src/routes/progress-sse/progress-sse.test.ts::"increments import.sse counters on connect, send, and disconnect"`
+AND `src/otel/instruments.test.ts::"bundle exposes the SSE counters"`
+
+<!-- markdownlint-enable MD024 -->

openspec/changes/add-import-sse-progress/tasks.md

@@ -0,0 +1,59 @@
+# Tasks: add-import-sse-progress
+
+## 0. Pre-flight (non-negotiable)
+
+- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
+- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
+- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
+
+## 1. OTel — extend instruments bundle
+
+- [ ] Add `sseConnected: Counter` (`import.sse.connected`) to `ImportInstruments`.
+- [ ] Add `sseEventsSent: Counter` (`import.sse.events_sent`) labelled by `kind`.
+- [ ] Add `sseDisconnected: Counter` (`import.sse.disconnected`) labelled by `reason`.
+- [ ] Update `src/otel/instruments.md` to describe the three new counters.
+- [ ] Extend `src/otel/instruments.test.ts` to assert the new counters round-trip through a fresh `MeterProvider`.
+
+## 2. Route — `GET /:id/progress`
+
+- [ ] Create `src/routes/progress-sse/progress-sse.ts` exporting `registerProgressSseRoute(app, deps)`.
+- [ ] Wire the handler into `createImportRoutes` in `src/routes/build-routes.ts`.
+- [ ] Subscribe to `service.bus` (`import.progress`, `import.completed`, `import.failed`) — NO polling.
+- [ ] Forward the request `AbortSignal` to detach the listener on client disconnect.
+- [ ] Frame format: `id: <jobId>-<seq>`, `event: progress|complete|failed`, JSON-stringified `ImportProgress` payload, double-newline terminator.
+- [ ] Initial frame: snapshot from `service.getJob(id, ownerId)` so the client sees current state before the first bus tick.
+- [ ] Terminal events (`complete` / `failed`) close the stream cleanly.
+
+## 3. Errors
+
+- [ ] Reuse `IMPORT.NOT_FOUND` for foreign-owner / missing jobs (404 problem+json).
+- [ ] Reuse `IMPORT.UNAUTHENTICATED` (via `requireAuth()` middleware already in `build-routes.ts`) for the 401 path.
+- [ ] No new error codes added.
+
+## 4. Tests
+
+- [ ] `src/routes/progress-sse/progress-sse.test.ts` covering eight scenarios listed in `proposal.md#source-mapping`.
+- [ ] Tests use Hono's `app.request()` returning a `Response` with a streamed body; assertions read via `body.getReader()` and decode UTF-8.
+- [ ] No mocks. Real `ImportService`, real Hono, real in-memory `MeterProvider`.
+
+## 5. Docs
+
+- [ ] `src/routes/progress-sse/progress-sse.md` (≥200 words, Purpose / Features / Usage Basic / Usage Advanced / API Reference / Implementation Notes).
+- [ ] Update package `README.md` Features list and routes table to include the new SSE endpoint.
+
+## 6. Subpath export
+
+- [ ] Add `./routes/progress-sse/progress-sse` to `package.json#exports`.
+
+## 7. Gates (in order)
+
+- [ ] `bunx biome check src/`
+- [ ] `bunx turbo lint typecheck test --filter=@suaveplan/api-plugin-import`
+- [ ] `bunx turbo test:coverage --filter=@suaveplan/api-plugin-import` (100/100/100/100)
+- [ ] `bun scripts/openspec-conformance.ts @suaveplan/api-plugin-import` (0 errors)
+- [ ] `bunx turbo build --filter=@suaveplan/api-plugin-import`
+
+## 8. Archive
+
+- [ ] Fold spec delta into `openspec/specs/api-plugin-import/spec.md`.
+- [ ] `mv openspec/changes/add-import-sse-progress openspec/archive/2026-05-20-add-import-sse-progress`.