suaveplan/openspec/archive/2026-05-02-bitpacker-modular-restructure/proposal.md

Why

The genesis bitpacker at packages/primitives/bitpacker is a cut-down greenfield build (~20 files) that lacks the schema-driven packing system, advanced strategies, streaming, delta encoding, game-state schemas, and enterprise hardening present in suaveplan's 400-file implementation. Rather than dumping all 400 files into a single monolithic package, we need a modular architecture where the core is lightweight and every advanced feature is an independently versioned, opt-in plugin.

What Changes

• Create packages/bitpacker/ parent directory as the home for all bitpacker-related workspace packages.
• Move core bitpacker from packages/primitives/bitpacker/ to packages/bitpacker/bitpacker/.
• Split suaveplan's 400 files into 10 workspace packages:
  • @suaveplan/bitpacker — Core: BitWriter, BitReader, BitPacker, PackingStrategyManager, base types.
  • @suaveplan/bitpacker-strategies — All built-in packing strategies (string, number, object, array, enum, datetime, varint, map, set, bitarray, blob, bitset, etc.).
  • @suaveplan/bitpacker-schema — Schema parsing, validation, token registry, matching, versioning, visualization.
  • @suaveplan/bitpacker-streaming — Chunked streaming encode/decode, transform streams.
  • @suaveplan/bitpacker-delta — Delta manager, XOR encoder, string diff, number XOR, field-level delta.
  • @suaveplan/bitpacker-game — Game state schemas, quantization, prediction, interpolation, reconciliation, state manager.
  • @suaveplan/bitpacker-network — Network simulation helpers (jitter, latency, packet loss).
  • @suaveplan/bitpacker-crypto — Checksum helpers (CRC32, Adler32, xxHash32), crypto adapters.
  • @suaveplan/bitpacker-compressed — Compression pipeline.
  • @suaveplan/bitpacker-validation — Safety limits, schema validation, validation reports.
• Preserve genesis additions in core: zigzagEncode/zigzagDecode, packBits/unpackBits, peek/seek on BitReader, align() on both reader and writer.
• Preserve backward compatibility where possible: existing defineBitSchema call sites should continue to compile.
• BREAKING: Package path changes from @suaveplan/bitpacker (implicit in primitives) to explicit @suaveplan/bitpacker under packages/bitpacker/bitpacker/.
• BREAKING: BitpackerError will eventually extend SuaveplanError (migrated from plain Error).

Parity Checklist

Legend: ✅ ported · ⚠️ partial / stub-thin / misplaced (real gap remaining) · 🚫 not ported · ⛔ intentionally not ported (decision recorded in a follow-up change).

Last verified: 2026-05-02 by side-by-side comparison of /mnt/s/web/suaveplan/packages/universal/bitpacker/src/ against packages/bitpacker/*/src/. Line counts cited are implementation files only (excluding tests).

Capability	Upstream	Migrated	Notes
BitWriter / BitReader	✅	✅	Genesis additions preserved (peek, seek, align)
BitPacker class	✅	⚠️	Core simplified by design (1203L → 178L): no EventEmitter, no built-in delta/checksum. Documented in design.md.
PackingStrategyManager	✅	✅	Chain-of-responsibility pattern preserved
PackingContext	✅	✅	Uses writer/reader instead of bitOps
Integer packing (signed/unsigned)	✅	✅	NumberPackingStrategy
Fixed-point packing	✅	✅	NumberPackingStrategy
String packing (fixed/variable)	✅	✅	StringPackingStrategy
UUID packing	✅	✅	StringPackingStrategy
Float packing (f32/f64)	✅	✅	StringPackingStrategy
Boolean packing	✅	✅	BooleanPackingStrategy
Null packing	✅	✅	NullPackingStrategy
BigInt packing (i64/u64/arrays)	✅	✅	BigIntPackingStrategy
Object packing	✅	✅	ObjectPackingStrategy
Array packing (tuple + dynamic)	✅	✅	ArrayPackingStrategy
Enum packing	✅	✅	EnumPackingStrategy
Optional packing	✅	✅	OptionalPackingStrategy
OneOf (union) packing	✅	✅	OneOfPackingStrategy
Conditional packing	✅	✅	ConditionalPackingStrategy
Version packing	✅	✅	VersionPackingStrategy
DateTime packing	✅	✅	DateTimePackingStrategy (simplified)
VarInt packing	✅	✅	VarIntPackingStrategy
Map packing	✅	✅	MapPackingStrategy
Set packing	✅	✅	SetPackingStrategy
BitArray packing	✅	✅	BitArrayPackingStrategy
Blob packing	✅	✅	BlobPackingStrategy
Bitset packing	✅	✅	BitsetPackingStrategy
Defs / $ref packing	✅	✅	DefsPackingStrategy
CustomType packing	✅	✅	CustomTypePackingStrategy
Schema registry	✅	⚠️	bitpacker-schema/registry/registry.ts is a 50L Map wrapper; old 576L surface (InMemory + LocalStorage storage adapters, factory helpers) NOT ported
Schema validation	✅	✅	validateData / assertValid in bitpacker-schema; full surface in bitpacker-validation (~4249L vs old ~4250L impl)
Schema matching	✅	✅	Ported at bitpacker-schema/matching/matching.ts (165L → 173L)
Schema migration	✅	✅	Ported at bitpacker-schema/migration/migration.ts (521L → 479L, ~92%)
Schema versioning	✅	✅	Ported at bitpacker-schema/versioning/versioning.ts (435L → 365L, ~84%)
Schema visualization	✅	✅	Ported at bitpacker-schema/visualization/visualization.ts (893L → 847L, ~95%)
Delta manager	✅	❌	Old 269L delta-manager.ts + 149L delta-state.ts NOT ported. Genesis has only a 35L inline DeltaManager class in bitpacker-delta/index.ts using JSON.stringify + Date.now() (deterministic-clock rule violation).
XOR delta encoder	✅	⚠️	Ported at bitpacker-delta/xor-encoder.ts (177L → 60L, ~34%)
String diff delta	✅	⚠️	Ported at bitpacker-delta/string-diff.ts (226L → 113L, ~50%)
Number XOR delta	✅	⚠️	Old 232L number-xor.ts folded into xor-encoder.ts's numberXor export; richer surface lost
Game quantization	✅	✅	quantizeVector / dequantizeVector in bitpacker-game
Game prediction	✅	✅	All 5 files ported in bitpacker-game/prediction/ (kalman-filter, motion-math, position-predictor, prediction-algorithms, rotation-predictor + entry; 827L → 700L, ~85%)
Game interpolation	✅	⚠️	Ported at bitpacker-game/interpolation/interpolation.ts (745L → 447L, ~60% — significant surface trim)
Game reconciliation	✅	✅	Ported at bitpacker-game/reconciliation/reconciliation.ts (615L → 478L, ~78%)
Game state manager	✅	✅	GameStateManager in bitpacker-game
Game bitpacker.ts enhancements	✅	✅	Ported at bitpacker-game/bitpacker/bitpacker.ts (388L → 384L)
Game schema-factory	✅	✅	Ported at bitpacker-game/schema-factory/schema-factory.ts (50L → 50L)
Network simulation	✅	⚠️	Misplaced: full 622L implementation lives at bitpacker-game/src/network/network-simulation.ts; bitpacker-network/src/index.ts is a 48L stub using module-scope Math.random() (rule violation). Needs to move to bitpacker-network and the stub deleted.
CRC32 checksum	✅	⚠️	Functional, but inlined into 80L bitpacker-crypto/index.ts. Old 176L crc32.ts had separate appendCrc32/verifyAndStripCrc32 helpers — only appendCrc32/verifyAndStripCrc32 survive; helper API surface is flat.
Adler32 checksum	✅	⚠️	Functional, but inlined into 80L bitpacker-crypto/index.ts. Old 121L adler32.ts had appendAdler32/verifyAdler32 helpers — currently no append/verify wrappers in genesis.
xxHash32 checksum	✅	✅	Ported at bitpacker-crypto/xxhash32.ts (re-exported from index.ts)
Crypto provider	✅	⛔	Replaced by Packer / Authority in restore-crypto-keychain-packer's new @suaveplan/crypto; auxiliary SchemaFingerprint + HashAlgorithmRegistry in bitpacker-crypto-rewrite (slim).
Crypto registry	✅	⛔	Replaced by Authority (in-memory + future HSM/TPM/threshold) in restore-crypto-keychain-packer's @suaveplan/crypto.
Crypto adapter — noop	✅	⛔	Replaced by @suaveplan/testing fakes against Authority; see restore-crypto-keychain-packer.
Crypto adapter — libsodium	✅	⛔	Underlying primitives covered by @suaveplan/crypto-sodium; opinionated layer is Packer + InMemoryAuthority in restore-crypto-keychain-packer.
Crypto adapter — elliptic	✅	⛔	Underlying primitives covered by @suaveplan/crypto-curve; opinionated layer is Packer + InMemoryAuthority in restore-crypto-keychain-packer.
Streaming StreamingPacker class	✅	❌	406L streaming/streaming.ts NOT ported. Genesis bitpacker-streaming/index.ts (87L) has only chunked helpers (packToChunks/unpackFromChunks{Async}); the top-level streaming encoder class is missing.
Streaming transform	✅	⚠️	Ported at bitpacker-streaming/transform/transform.ts (544L → 365L, ~67%)
Compression pipeline	✅	❌	bitpacker-compressed/index.ts is 61L of standalone RLE labelled "placeholder for a more sophisticated compression algorithm". Old 663L (compressed/bitpacker.ts 327 + compressed/compression.ts 336) CompressedBitPacker class NOT ported.
Safety limits	✅	✅	SafetyChecker in bitpacker-validation
defineBitSchema (legacy)	✅	✅	Preserved in core for backward compatibility
zigzagEncode/zigzagDecode	✅	✅	Preserved in core
Iterators (pack-iterator)	✅	✅	Ported at bitpacker/src/iterators/pack-iterator/
Custom-type registry	✅	✅	Ported at bitpacker/src/handlers/custom-type-registry.ts
Performance utilities (buffer-pool / lru-cache / monitor)	✅	✅	Ported at bitpacker/src/performance/
Serialization	✅	✅	Ported at bitpacker/src/serialization/

Outstanding gaps (real, not parity-table-stale)

These are the items above marked ❌ or 🚫 — the work that genuinely remains:

1. ~Crypto subsystem — provider, registry, and noop / libsodium / elliptic adapters (1906L total). No genesis equivalent exists. Tracked separately by the bitpacker-full-port change; should land in a dedicated bitpacker-crypto-port follow-up. Resolved 2026-05-02 in two parallel changes: (a) restore-crypto-keychain-packer — new opinionated @suaveplan/crypto layer with Packer / Unpacker / Authority / AuditLog / RevocationList / Keychain / Vault consolidating the 8+ inline sign-then-pack / encrypt-pack / Merkle-log patterns. (b) bitpacker-crypto-rewrite (slim) — auxiliary bit-packer-side primitives (SchemaFingerprint, HashAlgorithmRegistry) that don't overlap (a). Decision recorded in packages/bitpacker/bitpacker-crypto/SUPERSEDED.md. Predecessor proposals bitpacker-crypto-port and the bitpacker-crypto-rewrite first draft (with seven primitives) are preserved as decision-history entries.
2. StreamingPacker class in bitpacker-streaming — 406L upstream, currently absent.
3. CompressedBitPacker class in bitpacker-compressed — 663L upstream replaced by 61L placeholder RLE.
4. DeltaManager + delta-state in bitpacker-delta — 418L upstream replaced by a 35L inline class that violates the deterministic-clock rule.
5. Network-simulation misplacement — the full implementation lives in bitpacker-game/network/; bitpacker-network itself is a stub.
6. SchemaRegistry storage adapters — InMemory and LocalStorage adapter classes + factory helpers (the bulk of the 576L upstream file) not ported.

Items marked ⚠️ (XOR encoder, string diff, interpolation, transform, CRC32/Adler32 helper API) have shipped most of the upstream surface but lost some helper functions / refinements; tracked as "polish" follow-ups, not blockers.

Capabilities

New Capabilities

• bitpacker: Low-level bit I/O and schema-driven packing.
• bitpacker-strategies: Modular packing strategies for every schema token.
• bitpacker-schema: Schema registry, validation, versioning, and visualization.
• bitpacker-streaming: Chunked and transform-stream packing/unpacking.
• bitpacker-delta: Delta encoding for efficient repeated state transmission.
• bitpacker-game: Game-specific schemas, quantization, prediction, reconciliation.
• bitpacker-network: Network simulation and latency/jitter helpers.
• bitpacker-crypto: Checksum and crypto adapter integrations.
• bitpacker-compressed: Compression pipeline integration.
• bitpacker-validation: Safety limits and runtime schema validation.

Modified Capabilities

• packer: The packer package at packages/data/packer/ imports bitpacker; its imports and types may need updating to point to the new core package location.

Impact

Affected dependents

• @suaveplan/crypto — imports defineBitSchema; compiles green with new core
• @suaveplan/network-plugin-protocol — imports defineBitSchema; compiles green
• @suaveplan/network-plugin-registry — imports defineBitSchema; compiles green
• @suaveplan/packer — imports bitpacker for envelope encoding; must be verified
• @suaveplan/network-* — imports bitpacker for protocol encoding; must be verified
• @suaveplan/storage-* — imports bitpacker for payload encoding; must be verified

Breaking changes

The genesis BitWriter/BitReader/defineBitSchema API surface is preserved where possible, but:

• defineBitSchema return type will change to match suaveplan's richer BitSchema interface.
• Error types change from plain Error to BitpackerError extends SuaveplanError.
• New runtime deps: @suaveplan/schema-registry, @suaveplan/opentelemetry, @suaveplan/feature-flags.

Migration

For consumers of the genesis cut-down API:

• BitWriter / BitReader — unchanged signatures, but now live under subpath imports.
• defineBitSchema — return type gains pack/unpack methods and metadata; existing call sites compile without change.
• Error handling — catch BitpackerError instead of Error for programmatic handling.