suaveplan/openspec/changes/schema-registry-completion/proposal.md

# Schema-registry completion

**Package**: `@suaveplan/schema-registry`
**Status**: PROPOSED
**Type**: feature

## Why

Genesis ships only a thin in-memory `SchemaRegistry` (`createRegistry`, `register` / `getLatest` / `getVersion` / `getAll` / `listSubjects` / `deleteSubject` / `setCompatibility` / `checkCompatibility` / `clear`) plus the `checkCompatibility` predicate and a four-code error registry. The legacy `packages/universal/schema-registry/src` shipped a full enterprise schema-registry surface with seven additional capabilities that genesis silently dropped:

1. **Transitive compatibility** — `BACKWARD_TRANSITIVE` / `FORWARD_TRANSITIVE` / `FULL_TRANSITIVE` modes that check a candidate against ALL prior versions, not just the latest. Legacy `src/compatibility/transitive-checker.ts` (L1-L100) ships `checkTransitiveAgainstAll`, `buildTransitiveReport`, `TransitiveReport`, `TransitiveVersionResult`, `isTransitiveMode`, `stripTransitive`. Without these, schema-evolution policies that mandate "every old consumer must keep working" cannot be enforced — a real concern for event-sourced systems and message buses.
2. **Schema diff** — `diffSchemas(a, b)` → `SchemaDiffResult` and `formatDiffReport(result)` → human string, plus the `SchemaDiffChange` and `DiffChangeType` types. Legacy `src/diff/schema-differ.ts` (L1-L175). Without diff, callers cannot surface "what specifically broke" to ops users.
3. **`SchemaStoreAdapter` interface** — the contract that lets the registry persist beyond process memory. Legacy `src/types/registry.types.ts` (L82-L105) declares 8 methods; genesis has no analogous abstraction.
4. **`MemorySchemaStore`** — in-memory implementation of `SchemaStoreAdapter`. Legacy ships this implicitly inside the registry; genesis needs it as a standalone module so consumers can swap stores.
5. **`DatabaseSchemaStore`** — `DatabaseAdapter`-backed implementation. Legacy `src/store/database-store.ts` (L1-L147). Without it, the genesis registry is process-local.
6. **Canonicalisation utilities** — `canonicalizeSchema`, `computeFingerprint`, `djb2Hash`, `flattenFields`, `isOptionalOrHasDefault`. Legacy `src/utils/utils.ts` (L1-L70). The registry needs deterministic fingerprints to deduplicate identical re-registrations and to support hash-based subject indexing.
7. **`SubjectConfig`** — per-subject configuration object including compatibility mode plus metadata. Legacy `src/types/registry.types.ts` (L65-L68) ships a minimal version; the genesis registry hardcodes the compat level on `SubjectState` with no metadata or persisted version cap.

The genesis-original behaviour the registry already ships (auto-incrementing version, `INVALID_SCHEMA` validation, OTel `register.total` counter and `check.duration_ms` histogram) is preserved. This change is purely additive on the existing `SchemaRegistry` surface — no current export is removed or renamed.

## What changes

- ADDED: `CompatibilityLevel` widened to `"BACKWARD" | "FORWARD" | "FULL" | "NONE" | "BACKWARD_TRANSITIVE" | "FORWARD_TRANSITIVE" | "FULL_TRANSITIVE"`. Existing four levels keep their meaning; the three new modes activate per-subject transitive checking.
- ADDED: `checkTransitiveCompatibility(history: readonly SchemaDefinition[], candidate: unknown, mode: CompatibilityLevel)` → `CompatibilityResult` — convenience entry point.
- ADDED: `buildTransitiveReport(history, candidate, mode)` → `TransitiveReport` — verbose per-version compatibility breakdown.
- ADDED: `TransitiveReport` interface — `{ compatible: boolean; mode: CompatibilityLevel; versionResults: TransitiveVersionResult[]; totalVersions: number; failedVersions: number }`.
- ADDED: `TransitiveVersionResult` interface — `{ version: number; compatible: boolean; errors: readonly string[] }`.
- ADDED: `isTransitiveMode(mode)` and `stripTransitive(mode)` helper functions.
- ADDED: `diffSchemas(oldSchema, newSchema)` → `SchemaDiffResult` — structural diff identifying added / removed / modified fields and flagging breaking changes.
- ADDED: `formatDiffReport(result)` → human-readable string — counts plus per-change `+ / - / ~` rendering with `[BREAKING]` annotation.
- ADDED: `SchemaDiffChange` interface — `{ changeType: DiffChangeType; path: string; breaking: boolean; description: string; oldField?: SchemaField; newField?: SchemaField }`.
- ADDED: `SchemaDiffResult` interface — `{ changes: readonly SchemaDiffChange[]; hasBreakingChanges: boolean; summary: string }`.
- ADDED: `DiffChangeType` union — `"added" | "removed" | "modified"`.
- ADDED: `SchemaField` interface — `{ name: string; type: string; required?: boolean; defaultValue?: unknown; fields?: readonly SchemaField[] }`. Used by the diff and canonicalisation utilities. (Legacy registry already had this; genesis dropped it.)
- ADDED: `SchemaStoreAdapter` interface — `getVersions(subject)`, `getVersion(subject, version)`, `getLatest(subject)`, `putVersion(version)`, `getSubjects()`, `hasSubject(subject)`, `deleteSubject(subject)`, `getConfig(subject)`, `setConfig(subject, config)`. All async.
- ADDED: `MemorySchemaStore` class implementing `SchemaStoreAdapter` against an in-memory `Map`.
- ADDED: `DatabaseSchemaStore<TAdapter extends DatabaseAdapter>` class implementing `SchemaStoreAdapter` over `@suaveplan/db`'s `DatabaseAdapter`. Constructor takes `(db, tableName?, configTableName?)` defaulting to `"schema_versions"` / `"schema_configs"`.
- ADDED: `canonicalizeSchema(schema)` → stable JSON string with fields sorted at every level.
- ADDED: `computeFingerprint(schema)` → 8-hex-char djb2 hash of the canonical form.
- ADDED: `djb2Hash(input)` → 8-hex-char hash. Exported as a building block.
- ADDED: `flattenFields(fields, prefix?)` → `Map<string, SchemaField>` keyed by dot-notation paths.
- ADDED: `isOptionalOrHasDefault(field)` → boolean.
- ADDED: `SubjectConfig` interface — `{ compatibility: CompatibilityLevel; maxVersions?: number; metadata?: Record<string, string> }`. Replaces the inline-on-`SubjectState` representation.
- MODIFIED: `SchemaRegistry` constructor — accepts an optional `store?: SchemaStoreAdapter`. When supplied, persistence routes through the store; otherwise the existing in-process `Map` path is used. The synchronous public API is preserved when no store is supplied; when a store IS supplied, an alternative async surface (`registerAsync`, `getLatestAsync`, `getVersionAsync`, `getAllAsync`, `listSubjectsAsync`, `deleteSubjectAsync`, `checkCompatibilityAsync`) is exposed alongside the existing sync methods (which throw if a store is configured but the call is sync).
- MODIFIED: `SchemaDefinition` — `metadata?: Record<string, string>` already supported; ADDED a non-breaking `fingerprint?: string` field that, when present, equals `computeFingerprint(schema as SchemaDefinition)`.
- ADDED: error code `SCHEMA_REGISTRY.STORE_FAILURE` — thrown when the underlying `SchemaStoreAdapter` fails (network, DB).
- ADDED: error code `SCHEMA_REGISTRY.UNKNOWN_COMPATIBILITY_MODE` — thrown when an invalid mode string is supplied.

## Out of scope

- **Confluent Schema Registry HTTP client** — the legacy package never shipped one; the integration with Confluent's REST API is a separate pending change tracked as `schema-registry-confluent-adapter`. Reason for splitting: the Confluent contract is large (subjects, configs, deletes, soft/hard deletes, modes, mode reset, schema-by-id lookup, version-by-schema lookup), authentication varies (basic/bearer/mTLS), and pulling it in here would dilute review of the much smaller transitive + diff + store surface.
- **Avro / Protobuf / JSON-Schema specific format adapters** — `SchemaFormat` already includes `"json-schema" | "zod" | "avro" | "custom"` in genesis. This change ships only the internal Zod-described `SchemaDefinition` shape that drives the diff and compatibility engines; semantic adapters per format land in follow-up changes (`schema-registry-avro-adapter`, `schema-registry-protobuf-adapter`). Reason: each format has bespoke compatibility semantics — Avro reader/writer schema rules differ from JSON Schema's `$ref` resolution.
- **Webhook notifications on schema change** — the legacy package did not ship them; any `register` → fan-out integration is delegated to `@suaveplan/webhook-client` by composition.
- **Schema validation (running data through a schema)** — `@suaveplan/validation` is the canonical home; the registry stores schemas, the validation package validates payloads against them.
- **`zod`-as-`SchemaDefinition` autodiff** — the diff engine works on `SchemaDefinition` (`{ type, fields }`) per the legacy contract. Translating a Zod schema into `SchemaDefinition` is a follow-up; for now consumers either store JSON-Schema-shaped definitions or supply a custom translator.

## Source mapping

This change derives from the legacy `packages/universal/schema-registry/src` tree at `../suaveplan/`. No `enterprise/schema-registry.md` per-feature audit doc exists; source mapping cites the upstream files directly.

| Requirement | Source file | Source lines | Notes |
|---|---|---|---|
| `checkTransitiveCompatibility` (= `checkTransitiveAgainstAll`) | `packages/universal/schema-registry/src/compatibility/transitive-checker.ts` | L18-L35 | Renamed for symmetry with `checkCompatibility`; behaviour verbatim |
| `buildTransitiveReport` | `packages/universal/schema-registry/src/compatibility/transitive-checker.ts` | L58-L84 | Verbatim |
| `TransitiveReport` interface | `packages/universal/schema-registry/src/compatibility/transitive-checker.ts` | L94-L100 | Verbatim shape; widen `mode` to genesis `CompatibilityLevel` |
| `TransitiveVersionResult` interface | `packages/universal/schema-registry/src/compatibility/transitive-checker.ts` | L87-L91 | Verbatim |
| `isTransitiveMode` / `stripTransitive` helpers | `packages/universal/schema-registry/src/compatibility/transitive-checker.ts` | L40-L52 | Verbatim |
| `diffSchemas` | `packages/universal/schema-registry/src/diff/schema-differ.ts` | L16-L40 | Verbatim |
| `formatDiffReport` | `packages/universal/schema-registry/src/diff/schema-differ.ts` | L45-L64 | Verbatim |
| `SchemaDiffChange` interface | `packages/universal/schema-registry/src/types/registry.types.ts` | L118-L132 | Verbatim |
| `SchemaDiffResult` interface | `packages/universal/schema-registry/src/types/registry.types.ts` | L134-L142 | Verbatim |
| `DiffChangeType` union | `packages/universal/schema-registry/src/types/registry.types.ts` | L116 | Verbatim |
| `SchemaField` interface | `packages/universal/schema-registry/src/types/registry.types.ts` | L15-L26 | Verbatim |
| `SchemaStoreAdapter` interface | `packages/universal/schema-registry/src/types/registry.types.ts` | L82-L105 | Verbatim shape; widen mode types to genesis `CompatibilityLevel` |
| `DatabaseSchemaStore` class | `packages/universal/schema-registry/src/store/database-store.ts` | L1-L147 | Re-target `DatabaseAdapter` import to `@suaveplan/db`'s `DatabaseAdapter` (legacy used an inline interface) |
| `canonicalizeSchema` / `computeFingerprint` / `djb2Hash` / `flattenFields` / `isOptionalOrHasDefault` | `packages/universal/schema-registry/src/utils/utils.ts` | L1-L70 | Verbatim |
| `SubjectConfig` (`compatibility`, `maxVersions?`, `metadata?`) | `packages/universal/schema-registry/src/types/registry.types.ts` | L65-L68 | Extended: legacy ships only `compatibility`; genesis adds `maxVersions` and `metadata` (already partially modelled as constructor option). |

### Requirements with no source line

| Requirement | Justification |
|---|---|
| `MemorySchemaStore` standalone class | Legacy ships the in-memory case implicitly inside the registry; genesis exposes it as a standalone, importable module so consumers can swap stores without re-wiring. Symmetric with `DatabaseSchemaStore`. |
| `SchemaRegistry.{registerAsync, getLatestAsync, ...}` async surface | Legacy registry is sync; legacy `SchemaStoreAdapter` is async. Bridging requires either making the whole API async (BREAKING) or shipping a parallel async surface. Genesis chooses the additive route to preserve the existing sync API for in-memory users. |
| `SCHEMA_REGISTRY.STORE_FAILURE` and `SCHEMA_REGISTRY.UNKNOWN_COMPATIBILITY_MODE` codes | Genesis-wide policy — every thrown error routes through `@suaveplan/error` (CLAUDE.md §Workflow rule 2). |
| `SchemaDefinition.fingerprint?` field | Required for de-duplication when an identical schema is re-registered; without it, every retry creates a new version. The fingerprint is computed by the registry at register-time using `computeFingerprint`; consumers do not supply it. |

## Verification approach

Each spec delta requirement carries a `**Verified by:**` line pointing at concrete `it(...)` titles. The test files that will exist after this change ships:

- `packages/core/schema-registry/src/core/transitive/transitive.test.ts` — covers `checkTransitiveCompatibility` against multi-version histories
- `packages/core/schema-registry/src/core/transitive/transitive.spec-conformance.test.ts` — pins `isTransitiveMode` / `stripTransitive` exhaustively over the seven `CompatibilityLevel` values
- `packages/core/schema-registry/src/core/diff/diff.test.ts` — covers `diffSchemas` for added / removed / modified / type-changed cases; `formatDiffReport` rendering
- `packages/core/schema-registry/src/core/diff/diff.spec-conformance.test.ts` — property tests `diffSchemas(a, a) === { changes: [], hasBreakingChanges: false }` and `formatDiffReport(diff(a, a)) === "No changes detected."`
- `packages/core/schema-registry/src/store/memory/memory.test.ts` — covers `MemorySchemaStore` round-trips, subject delete, config get/set
- `packages/core/schema-registry/src/store/memory/memory.spec-conformance.test.ts` — pins the `SchemaStoreAdapter` contract
- `packages/core/schema-registry/src/store/database/database.test.ts` — covers `DatabaseSchemaStore` against an in-memory `DatabaseAdapter` stub
- `packages/core/schema-registry/src/store/database/database.spec-conformance.test.ts` — pins the same contract; verifies `STORE_FAILURE` is raised on adapter exception
- `packages/core/schema-registry/src/utils/canonical/canonical.test.ts` — covers `canonicalizeSchema`, `computeFingerprint`, `djb2Hash`, `flattenFields`, `isOptionalOrHasDefault`
- `packages/core/schema-registry/src/utils/canonical/canonical.spec-conformance.test.ts` — `fast-check` property `canonicalizeSchema(a) === canonicalizeSchema(shuffleFields(a))` (canonicalisation is order-independent)
- `packages/core/schema-registry/src/core/registry/schema-registry.test.ts` — extended to cover the async surface and store wiring
- `packages/core/schema-registry/src/core/registry/schema-registry.spec-conformance.test.ts` — pins the `UNKNOWN_COMPATIBILITY_MODE` and `STORE_FAILURE` error contracts; pins the `SchemaDefinition.fingerprint` invariant (re-register identical schema produces same fingerprint and triggers de-dup)

## Migration

This is an additive change to the `SchemaRegistry` constructor and `CompatibilityLevel` union. No public API is removed; existing in-memory consumers continue to work without modification.

For consumers that want to switch to a persisted store:

- **Old behavior:** `new SchemaRegistry({ defaultCompatibility: "BACKWARD" })` was process-local.
- **New behavior:** `new SchemaRegistry({ defaultCompatibility: "BACKWARD", store: new DatabaseSchemaStore(dbAdapter) })` persists to the supplied store. The async surface (`registerAsync`, `getLatestAsync`, ...) replaces the sync one; calling sync methods on a store-backed registry throws `SCHEMA_REGISTRY.STORE_FAILURE` with `context.reason === "sync_call_on_async_store"`.
- **Search command consumers should run:** `grep -rln "new SchemaRegistry\|createRegistry" packages/ src/`.
- **Recommended replacement:** for in-memory users, no change. For persisted users, switch construction to pass `store` and migrate call sites to the `*Async` variants.

For consumers using `CompatibilityLevel`:

- **Old behavior:** the union was `"BACKWARD" | "FORWARD" | "FULL" | "NONE"`.
- **New behavior:** the union also includes `"BACKWARD_TRANSITIVE" | "FORWARD_TRANSITIVE" | "FULL_TRANSITIVE"`. Exhaustive `switch` statements over `CompatibilityLevel` will surface a TypeScript narrowing miss until consumers add the three new arms (or fall through them to the corresponding non-transitive case).
- **Search command:** `grep -rln "CompatibilityLevel\|defaultCompatibility" packages/ src/`.
- **Recommended replacement:** in any exhaustive `switch`, fall through `BACKWARD_TRANSITIVE` to the `BACKWARD` arm; same for `FORWARD_TRANSITIVE` / `FULL_TRANSITIVE`. The transitive variants only differ at register-time (check against all versions vs. latest).

## Impact

### Affected dependents

Run `bunx turbo ls --filter=...@suaveplan/schema-registry` to enumerate. From `grep -rl "@suaveplan/schema-registry" /home/ubuntu/web/genesis/packages/ --include="package.json"`:

- `@suaveplan/events-schema` — [x] tests run / [x] docs reviewed / [ ] no action needed. The events-schema package depends on the registry to track event-schema versions; it currently uses the in-memory path. After this change it can opt into transitive compatibility (event consumers commonly span multiple schema versions in flight) and persisted storage (event schemas should outlive process restarts). Tests must verify the package keeps building and passing with the new union shape.

For all other monorepo packages: no action — they do not depend on `@suaveplan/schema-registry`. This change is additive only.