suaveplan/openspec/archive/2026-05-02-bitpacker-modular-restructure/proposal.md

## Why

The genesis `bitpacker` at `packages/primitives/bitpacker` is a cut-down greenfield build (~20 files) that lacks the schema-driven packing system, advanced strategies, streaming, delta encoding, game-state schemas, and enterprise hardening present in suaveplan's 400-file implementation. Rather than dumping all 400 files into a single monolithic package, we need a modular architecture where the core is lightweight and every advanced feature is an independently versioned, opt-in plugin.

## What Changes

- **Create `packages/bitpacker/` parent directory** as the home for all bitpacker-related workspace packages.
- **Move core bitpacker** from `packages/primitives/bitpacker/` to `packages/bitpacker/bitpacker/`.
- **Split suaveplan's 400 files into 10 workspace packages:**
  - `@suaveplan/bitpacker` — Core: `BitWriter`, `BitReader`, `BitPacker`, `PackingStrategyManager`, base types.
  - `@suaveplan/bitpacker-strategies` — All built-in packing strategies (string, number, object, array, enum, datetime, varint, map, set, bitarray, blob, bitset, etc.).
  - `@suaveplan/bitpacker-schema` — Schema parsing, validation, token registry, matching, versioning, visualization.
  - `@suaveplan/bitpacker-streaming` — Chunked streaming encode/decode, transform streams.
  - `@suaveplan/bitpacker-delta` — Delta manager, XOR encoder, string diff, number XOR, field-level delta.
  - `@suaveplan/bitpacker-game` — Game state schemas, quantization, prediction, interpolation, reconciliation, state manager.
  - `@suaveplan/bitpacker-network` — Network simulation helpers (jitter, latency, packet loss).
  - `@suaveplan/bitpacker-crypto` — Checksum helpers (CRC32, Adler32, xxHash32), crypto adapters.
  - `@suaveplan/bitpacker-compressed` — Compression pipeline.
  - `@suaveplan/bitpacker-validation` — Safety limits, schema validation, validation reports.
- **Preserve genesis additions** in core: `zigzagEncode`/`zigzagDecode`, `packBits`/`unpackBits`, `peek`/`seek` on `BitReader`, `align()` on both reader and writer.
- **Preserve backward compatibility** where possible: existing `defineBitSchema` call sites should continue to compile.
- **BREAKING**: Package path changes from `@suaveplan/bitpacker` (implicit in primitives) to explicit `@suaveplan/bitpacker` under `packages/bitpacker/bitpacker/`.
- **BREAKING**: `BitpackerError` will eventually extend `SuaveplanError` (migrated from plain `Error`).

## Parity Checklist

Legend: ✅ ported · ⚠️ partial / stub-thin / misplaced (real gap remaining) · 🚫 not ported · ⛔ intentionally not ported (decision recorded in a follow-up change).

Last verified: 2026-05-02 by side-by-side comparison of `/mnt/s/web/suaveplan/packages/universal/bitpacker/src/` against `packages/bitpacker/*/src/`. Line counts cited are implementation files only (excluding tests).

| Capability | Upstream | Migrated | Notes |
|---|---|---|---|
| `BitWriter` / `BitReader` | ✅ | ✅ | Genesis additions preserved (peek, seek, align) |
| `BitPacker` class | ✅ | ⚠️ | Core simplified by design (1203L → 178L): no EventEmitter, no built-in delta/checksum. Documented in `design.md`. |
| `PackingStrategyManager` | ✅ | ✅ | Chain-of-responsibility pattern preserved |
| `PackingContext` | ✅ | ✅ | Uses `writer`/`reader` instead of `bitOps` |
| Integer packing (signed/unsigned) | ✅ | ✅ | NumberPackingStrategy |
| Fixed-point packing | ✅ | ✅ | NumberPackingStrategy |
| String packing (fixed/variable) | ✅ | ✅ | StringPackingStrategy |
| UUID packing | ✅ | ✅ | StringPackingStrategy |
| Float packing (f32/f64) | ✅ | ✅ | StringPackingStrategy |
| Boolean packing | ✅ | ✅ | BooleanPackingStrategy |
| Null packing | ✅ | ✅ | NullPackingStrategy |
| BigInt packing (i64/u64/arrays) | ✅ | ✅ | BigIntPackingStrategy |
| Object packing | ✅ | ✅ | ObjectPackingStrategy |
| Array packing (tuple + dynamic) | ✅ | ✅ | ArrayPackingStrategy |
| Enum packing | ✅ | ✅ | EnumPackingStrategy |
| Optional packing | ✅ | ✅ | OptionalPackingStrategy |
| OneOf (union) packing | ✅ | ✅ | OneOfPackingStrategy |
| Conditional packing | ✅ | ✅ | ConditionalPackingStrategy |
| Version packing | ✅ | ✅ | VersionPackingStrategy |
| DateTime packing | ✅ | ✅ | DateTimePackingStrategy (simplified) |
| VarInt packing | ✅ | ✅ | VarIntPackingStrategy |
| Map packing | ✅ | ✅ | MapPackingStrategy |
| Set packing | ✅ | ✅ | SetPackingStrategy |
| BitArray packing | ✅ | ✅ | BitArrayPackingStrategy |
| Blob packing | ✅ | ✅ | BlobPackingStrategy |
| Bitset packing | ✅ | ✅ | BitsetPackingStrategy |
| Defs / $ref packing | ✅ | ✅ | DefsPackingStrategy |
| CustomType packing | ✅ | ✅ | CustomTypePackingStrategy |
| Schema registry | ✅ | ⚠️ | `bitpacker-schema/registry/registry.ts` is a 50L `Map` wrapper; old 576L surface (InMemory + LocalStorage storage adapters, factory helpers) NOT ported |
| Schema validation | ✅ | ✅ | validateData / assertValid in bitpacker-schema; full surface in bitpacker-validation (~4249L vs old ~4250L impl) |
| Schema matching | ✅ | ✅ | Ported at `bitpacker-schema/matching/matching.ts` (165L → 173L) |
| Schema migration | ✅ | ✅ | Ported at `bitpacker-schema/migration/migration.ts` (521L → 479L, ~92%) |
| Schema versioning | ✅ | ✅ | Ported at `bitpacker-schema/versioning/versioning.ts` (435L → 365L, ~84%) |
| Schema visualization | ✅ | ✅ | Ported at `bitpacker-schema/visualization/visualization.ts` (893L → 847L, ~95%) |
| Delta manager | ✅ | ❌ | Old 269L `delta-manager.ts` + 149L `delta-state.ts` NOT ported. Genesis has only a 35L inline `DeltaManager` class in `bitpacker-delta/index.ts` using `JSON.stringify` + `Date.now()` (deterministic-clock rule violation). |
| XOR delta encoder | ✅ | ⚠️ | Ported at `bitpacker-delta/xor-encoder.ts` (177L → 60L, ~34%) |
| String diff delta | ✅ | ⚠️ | Ported at `bitpacker-delta/string-diff.ts` (226L → 113L, ~50%) |
| Number XOR delta | ✅ | ⚠️ | Old 232L `number-xor.ts` folded into `xor-encoder.ts`'s `numberXor` export; richer surface lost |
| Game quantization | ✅ | ✅ | quantizeVector / dequantizeVector in bitpacker-game |
| Game prediction | ✅ | ✅ | All 5 files ported in `bitpacker-game/prediction/` (kalman-filter, motion-math, position-predictor, prediction-algorithms, rotation-predictor + entry; 827L → 700L, ~85%) |
| Game interpolation | ✅ | ⚠️ | Ported at `bitpacker-game/interpolation/interpolation.ts` (745L → 447L, ~60% — significant surface trim) |
| Game reconciliation | ✅ | ✅ | Ported at `bitpacker-game/reconciliation/reconciliation.ts` (615L → 478L, ~78%) |
| Game state manager | ✅ | ✅ | GameStateManager in bitpacker-game |
| Game `bitpacker.ts` enhancements | ✅ | ✅ | Ported at `bitpacker-game/bitpacker/bitpacker.ts` (388L → 384L) |
| Game schema-factory | ✅ | ✅ | Ported at `bitpacker-game/schema-factory/schema-factory.ts` (50L → 50L) |
| Network simulation | ✅ | ⚠️ | **Misplaced**: full 622L implementation lives at `bitpacker-game/src/network/network-simulation.ts`; `bitpacker-network/src/index.ts` is a 48L stub using module-scope `Math.random()` (rule violation). Needs to move to `bitpacker-network` and the stub deleted. |
| CRC32 checksum | ✅ | ⚠️ | Functional, but inlined into 80L `bitpacker-crypto/index.ts`. Old 176L `crc32.ts` had separate `appendCrc32`/`verifyAndStripCrc32` helpers — only `appendCrc32`/`verifyAndStripCrc32` survive; helper API surface is flat. |
| Adler32 checksum | ✅ | ⚠️ | Functional, but inlined into 80L `bitpacker-crypto/index.ts`. Old 121L `adler32.ts` had `appendAdler32`/`verifyAdler32` helpers — currently no append/verify wrappers in genesis. |
| xxHash32 checksum | ✅ | ✅ | Ported at `bitpacker-crypto/xxhash32.ts` (re-exported from `index.ts`) |
| Crypto provider | ✅ | ⛔ | Replaced by `Packer` / `Authority` in `restore-crypto-keychain-packer`'s new `@suaveplan/crypto`; auxiliary `SchemaFingerprint` + `HashAlgorithmRegistry` in `bitpacker-crypto-rewrite` (slim). |
| Crypto registry | ✅ | ⛔ | Replaced by `Authority` (in-memory + future HSM/TPM/threshold) in `restore-crypto-keychain-packer`'s `@suaveplan/crypto`. |
| Crypto adapter — noop | ✅ | ⛔ | Replaced by `@suaveplan/testing` fakes against `Authority`; see `restore-crypto-keychain-packer`. |
| Crypto adapter — libsodium | ✅ | ⛔ | Underlying primitives covered by `@suaveplan/crypto-sodium`; opinionated layer is `Packer` + `InMemoryAuthority` in `restore-crypto-keychain-packer`. |
| Crypto adapter — elliptic | ✅ | ⛔ | Underlying primitives covered by `@suaveplan/crypto-curve`; opinionated layer is `Packer` + `InMemoryAuthority` in `restore-crypto-keychain-packer`. |
| Streaming `StreamingPacker` class | ✅ | ❌ | 406L `streaming/streaming.ts` NOT ported. Genesis `bitpacker-streaming/index.ts` (87L) has only chunked helpers (`packToChunks`/`unpackFromChunks{Async}`); the top-level streaming encoder class is missing. |
| Streaming transform | ✅ | ⚠️ | Ported at `bitpacker-streaming/transform/transform.ts` (544L → 365L, ~67%) |
| Compression pipeline | ✅ | ❌ | `bitpacker-compressed/index.ts` is 61L of standalone RLE labelled "placeholder for a more sophisticated compression algorithm". Old 663L (`compressed/bitpacker.ts` 327 + `compressed/compression.ts` 336) `CompressedBitPacker` class NOT ported. |
| Safety limits | ✅ | ✅ | SafetyChecker in bitpacker-validation |
| `defineBitSchema` (legacy) | ✅ | ✅ | Preserved in core for backward compatibility |
| `zigzagEncode`/`zigzagDecode` | ✅ | ✅ | Preserved in core |
| Iterators (`pack-iterator`) | ✅ | ✅ | Ported at `bitpacker/src/iterators/pack-iterator/` |
| Custom-type registry | ✅ | ✅ | Ported at `bitpacker/src/handlers/custom-type-registry.ts` |
| Performance utilities (buffer-pool / lru-cache / monitor) | ✅ | ✅ | Ported at `bitpacker/src/performance/` |
| Serialization | ✅ | ✅ | Ported at `bitpacker/src/serialization/` |

## Outstanding gaps (real, not parity-table-stale)

These are the items above marked ❌ or 🚫 — the work that genuinely remains:

1. ~~**Crypto subsystem** — provider, registry, and noop / libsodium / elliptic adapters (~1906L total). No genesis equivalent exists. Tracked separately by the `bitpacker-full-port` change; should land in a dedicated `bitpacker-crypto-port` follow-up.~~ **Resolved 2026-05-02** in two parallel changes: (a) **`restore-crypto-keychain-packer`** — new opinionated `@suaveplan/crypto` layer with `Packer` / `Unpacker` / `Authority` / `AuditLog` / `RevocationList` / `Keychain` / `Vault` consolidating the 8+ inline sign-then-pack / encrypt-pack / Merkle-log patterns. (b) **`bitpacker-crypto-rewrite` (slim)** — auxiliary bit-packer-side primitives (`SchemaFingerprint`, `HashAlgorithmRegistry`) that don't overlap (a). Decision recorded in `packages/bitpacker/bitpacker-crypto/SUPERSEDED.md`. Predecessor proposals `bitpacker-crypto-port` and the `bitpacker-crypto-rewrite` first draft (with seven primitives) are preserved as decision-history entries.
2. **`StreamingPacker` class** in `bitpacker-streaming` — 406L upstream, currently absent.
3. **`CompressedBitPacker` class** in `bitpacker-compressed` — 663L upstream replaced by 61L placeholder RLE.
4. **`DeltaManager` + `delta-state`** in `bitpacker-delta` — 418L upstream replaced by a 35L inline class that violates the deterministic-clock rule.
5. **Network-simulation misplacement** — the full implementation lives in `bitpacker-game/network/`; `bitpacker-network` itself is a stub.
6. **`SchemaRegistry` storage adapters** — InMemory and LocalStorage adapter classes + factory helpers (the bulk of the 576L upstream file) not ported.

Items marked ⚠️ (XOR encoder, string diff, interpolation, transform, CRC32/Adler32 helper API) have shipped most of the upstream surface but lost some helper functions / refinements; tracked as "polish" follow-ups, not blockers.

## Capabilities

### New Capabilities
- `bitpacker`: Low-level bit I/O and schema-driven packing.
- `bitpacker-strategies`: Modular packing strategies for every schema token.
- `bitpacker-schema`: Schema registry, validation, versioning, and visualization.
- `bitpacker-streaming`: Chunked and transform-stream packing/unpacking.
- `bitpacker-delta`: Delta encoding for efficient repeated state transmission.
- `bitpacker-game`: Game-specific schemas, quantization, prediction, reconciliation.
- `bitpacker-network`: Network simulation and latency/jitter helpers.
- `bitpacker-crypto`: Checksum and crypto adapter integrations.
- `bitpacker-compressed`: Compression pipeline integration.
- `bitpacker-validation`: Safety limits and runtime schema validation.

### Modified Capabilities
- `packer`: The packer package at `packages/data/packer/` imports bitpacker; its imports and types may need updating to point to the new core package location.

## Impact

### Affected dependents

- `@suaveplan/crypto` — imports `defineBitSchema`; compiles green with new core
- `@suaveplan/network-plugin-protocol` — imports `defineBitSchema`; compiles green
- `@suaveplan/network-plugin-registry` — imports `defineBitSchema`; compiles green
- `@suaveplan/packer` — imports bitpacker for envelope encoding; must be verified
- `@suaveplan/network-*` — imports bitpacker for protocol encoding; must be verified
- `@suaveplan/storage-*` — imports bitpacker for payload encoding; must be verified

### Breaking changes

The genesis `BitWriter`/`BitReader`/`defineBitSchema` API surface is preserved where possible, but:
- `defineBitSchema` return type will change to match suaveplan's richer `BitSchema` interface.
- Error types change from plain `Error` to `BitpackerError extends SuaveplanError`.
- New runtime deps: `@suaveplan/schema-registry`, `@suaveplan/opentelemetry`, `@suaveplan/feature-flags`.

## Migration

For consumers of the genesis cut-down API:
- `BitWriter` / `BitReader` — unchanged signatures, but now live under subpath imports.
- `defineBitSchema` — return type gains `pack`/`unpack` methods and metadata; existing call sites compile without change.
- Error handling — catch `BitpackerError` instead of `Error` for programmatic handling.