# bitpacker Specification ## Purpose `@suaveplan/bitpacker` provides bit-level binary serialization driven by declarative schemas. It supports fixed-width primitives (`u8`, `i16`, etc.), variable-length strings/arrays with max-sized length prefixes, and an adaptive blob type (`"~"`) whose length-prefix width is self-describing via a 2-bit tag so payloads ranging from bytes to ~1 TiB share a schema without wasting bits on small entries. ## Requirements ### Requirement: Adaptive blob schema type `"~"` The package SHALL treat schema value `"~"` as a binary blob whose length prefix is self-describing via a 2-bit width tag selecting 2, 3, 4, or 5 bytes of length. Pack input and unpack output SHALL be `Uint8Array` with no encoding. #### Scenario: tag selected for a small payload - **WHEN** packing 200 bytes with schema field `data: "~"` - **THEN** the written bits are `[tag=0][2-byte length=200][200 raw bytes]` #### Scenario: tag selected for a large payload - **WHEN** packing 17 MiB - **THEN** the tag is `2` and the length prefix is 4 bytes #### Scenario: payload over 1 TiB rejected - **WHEN** packing a buffer whose length exceeds `0xFFFFFFFFFF` - **THEN** the packer throws `BitpackerError(VALUE_OUT_OF_RANGE)` #### Scenario: round-trip preserves bytes - **WHEN** a `Uint8Array` is packed then unpacked - **THEN** the output is byte-identical to the input **Verified by:** `packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"selects 2-byte length tag for payloads under 64 KiB"` AND `packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"selects 4-byte length tag for payloads over 16 MiB"` AND `packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"throws on payloads exceeding 1 TiB"` AND `packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"round-trip preserves bytes byte-identical"` ### Requirement: Blob strategy runs before string strategy The strategy chain SHALL resolve `"~"` via `BlobPackingStrategy` before `StringPackingStrategy` sees it, so the reserved keyword MUST NOT be misinterpreted as a string schema. #### Scenario: registerDefaultStrategies orders blob before string - **WHEN** `registerDefaultStrategies(manager)` runs and `manager.find("~")` is called - **THEN** the returned strategy is `BlobPackingStrategy`, not `StringPackingStrategy` **Verified by:** `packages/bitpacker/bitpacker-strategies/src/register-default-strategies.test.ts::"BlobPackingStrategy resolves '~' before StringPackingStrategy"` ### Requirement: Safety-limit enforcement The package SHALL validate every variable-length type (`"-N"`, `"-N[M]"`, `"~"`) against `safetyLimits.maxBufferSize` via the `SafetyChecker.checkBufferSize(byteLength)` API before allocating a write buffer. #### Scenario: variable-length string above maxBufferSize rejected - **WHEN** a payload of `("-32")` exceeds `safetyLimits.maxBufferSize` - **THEN** `SafetyChecker.checkBufferSize` throws #### Scenario: dynamic array above maxBufferSize rejected - **WHEN** an array `("-32[16]")` whose total bytes exceed `safetyLimits.maxBufferSize` is checked - **THEN** `SafetyChecker.checkBufferSize` throws #### Scenario: adaptive blob above maxBufferSize rejected - **WHEN** a blob `"~"` whose length exceeds `safetyLimits.maxBufferSize` is checked - **THEN** `SafetyChecker.checkBufferSize` throws **Verified by:** `packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects variable-length string above maxBufferSize"` AND `packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects dynamic array above maxBufferSize"` AND `packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects adaptive blob above maxBufferSize"` ### Requirement: Public types are Zod-validated at package boundaries Every package in the bitpacker family SHALL expose a Zod schema for each public input type and SHALL validate untrusted inputs at the package boundary (the entry method that accepts a value from outside the package). Internal recursion MUST NOT re-validate. #### Scenario: invalid input rejected at boundary - **WHEN** a caller passes a value that does not match the boundary's Zod schema - **THEN** the package throws a `BitpackerError` whose `cause` is the Zod error and whose code is `INVALID_SCHEMA` #### Scenario: internal recursion does not re-validate - **WHEN** the strategy chain recurses into a sub-schema that has already been validated at the entry boundary - **THEN** no Zod schema is parsed during the recursion (verified by spying on the schema parser's call count) **Verified by:** Per-package boundary schema tests in `packages/bitpacker//src/**/*.schema.test.ts` ### Requirement: Public schemas registered with schema-registry Every package in the bitpacker family SHALL register its public Zod schemas with `@suaveplan/schema-registry` under namespace `"bitpacker"`, with the package name as the schema name and `version: 1`. The registration is idempotent — re-importing the package's `register-schemas` subpath against the same registry is a no-op. #### Scenario: schemas present after registration - **WHEN** a consumer imports the package's `./register-schemas` subpath and calls the exported `registerBitPackerSchemas(registry)` - **THEN** `registry.getLatest("bitpacker:@suaveplan/")` returns a populated entry at `version: 1` **Verified by:** `packages/bitpacker//src/register-schemas.test.ts` per package ### Requirement: OTel spans emitted at hot-path boundaries Every package in the bitpacker family SHALL emit one OTel span per top-level pack/unpack call, named `bitpacker.` for the core package and `bitpacker_.` for sub-packages. Per-chunk events MUST be nested span events on the parent span, not standalone spans. Tracer/meter names follow the genesis convention `@suaveplan/`. #### Scenario: span emitted on pack - **WHEN** a consumer calls a top-level pack/unpack method on `BitPacker` - **THEN** exactly one span is emitted with the canonical name (`bitpacker.pack` or `bitpacker.unpack`) and a `success` attribute of `"true"` or `"false"` **Verified by:** `packages/bitpacker/bitpacker/src/otel/instruments.test.ts` and per-package `instruments.test.ts` files ### Requirement: fast-check property tests cover round-trip soundness Every package that exposes a pack/unpack pair SHALL ship a `__property__/*.property.test.ts` file using `fast-check` to assert that for every value matching the schema's arbitrary, `unpack(pack(v)) ≡ v`. #### Scenario: round-trip property holds for every covered schema kind - **WHEN** a fast-check arbitrary generates 100 values for a schema kind - **THEN** every generated value round-trips byte-identically **Verified by:** `packages/bitpacker/bitpacker/src/__property__/round-trip.property.test.ts` AND `packages/bitpacker/bitpacker-strategies/src/__property__/round-trip.property.test.ts` ### Requirement: mitata benchmarks have committed baselines Every package that exposes a public encoder/decoder SHALL ship a `bench/*.bench.ts` file using `mitata`, with baseline timings committed to `scripts/bench/bitpacker-baselines.json`. The CI gate that fails on >10% regression is tracked as a follow-on Turbo `bench:check` task; running `bun bench/pack-unpack.bench.ts` in any covered package produces measurements that can be compared against the baseline manually. #### Scenario: baseline file present and parseable - **WHEN** `scripts/bench/bitpacker-baselines.json` is read - **THEN** the file parses as JSON and contains a `baselines["@suaveplan/bitpacker"]` block with median nanosecond timings for u8/u32 pack/unpack **Verified by:** Manual: `bun bench/pack-unpack.bench.ts` in `packages/bitpacker/bitpacker/` ### Requirement: `_version` strategy supports opt-in forward tolerance The `VersionPackingStrategy` in `@suaveplan/bitpacker-strategies` SHALL accept an optional `_tolerance: "strict" | "forward"` field on the version-schema shape. When `_tolerance` is absent or `"strict"`, the strategy SHALL throw `BitpackerError(INVALID_SCHEMA)` on any wire-version mismatch (current behaviour). When `_tolerance: "forward"` is set and the schema also carries an `_allowedVersions: number[]`, the strategy SHALL accept any wire version listed in that array and proceed with the schema's `_schema` field for decoding. When `_tolerance: "forward"` is set without `_allowedVersions`, the strategy SHALL accept any wire version >= 0 and proceed with `_schema`. The on-wire byte format is unchanged: a 16-bit unsigned big-endian version followed by the schema-encoded payload. #### Scenario: strict mode rejects mismatched wire version - **WHEN** a consumer unpacks bytes whose wire version differs from the schema's `_version` and `_tolerance` is absent or `"strict"` - **THEN** the strategy throws `BitpackerError(INVALID_SCHEMA)` with the expected and received versions in the error context #### Scenario: forward-tolerant mode accepts a listed version - **WHEN** a consumer unpacks bytes whose wire version is listed in `_allowedVersions` and `_tolerance: "forward"` is set - **THEN** the strategy decodes the payload via `_schema` without throwing #### Scenario: forward-tolerant mode without allowlist accepts any version - **WHEN** a consumer unpacks bytes with `_tolerance: "forward"` and no `_allowedVersions` field - **THEN** the strategy decodes any wire version >= 0 via `_schema` #### Scenario: forward-tolerant mode rejects a version outside the allowlist - **WHEN** a consumer unpacks bytes whose wire version is NOT in the supplied `_allowedVersions`, with `_tolerance: "forward"` - **THEN** the strategy throws `BitpackerError(INVALID_SCHEMA)` with the expected version, the received version, and the allowed-versions list in the error context **Verified by:** `packages/bitpacker/bitpacker-strategies/src/tests/strategies.test.ts::describe("version")` (four scenarios) AND `packages/bitpacker/bitpacker-strategies/src/__property__/round-trip.property.test.ts::"forward-tolerant unpack equals strict-unpack output for any same-shape version"`