suaveplan/openspec/specs/bitpacker/spec.md

bitpacker Specification

Purpose

@suaveplan/bitpacker provides bit-level binary serialization driven by declarative schemas. It supports fixed-width primitives (u8, i16, etc.), variable-length strings/arrays with max-sized length prefixes, and an adaptive blob type ("~") whose length-prefix width is self-describing via a 2-bit tag so payloads ranging from bytes to ~1 TiB share a schema without wasting bits on small entries.

Requirements

Requirement: Adaptive blob schema type "~"

The package SHALL treat schema value "~" as a binary blob whose length prefix is self-describing via a 2-bit width tag selecting 2, 3, 4, or 5 bytes of length. Pack input and unpack output SHALL be Uint8Array with no encoding.

Scenario: tag selected for a small payload

• WHEN packing 200 bytes with schema field data: "~"
• THEN the written bits are [tag=0][2-byte length=200][200 raw bytes]

Scenario: tag selected for a large payload

• WHEN packing 17 MiB
• THEN the tag is 2 and the length prefix is 4 bytes

Scenario: payload over 1 TiB rejected

• WHEN packing a buffer whose length exceeds 0xFFFFFFFFFF
• THEN the packer throws BitpackerError(VALUE_OUT_OF_RANGE)

Scenario: round-trip preserves bytes

• WHEN a Uint8Array is packed then unpacked
• THEN the output is byte-identical to the input

Verified by: packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"selects 2-byte length tag for payloads under 64 KiB" AND packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"selects 4-byte length tag for payloads over 16 MiB" AND packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"throws on payloads exceeding 1 TiB" AND packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"round-trip preserves bytes byte-identical"

Requirement: Blob strategy runs before string strategy

The strategy chain SHALL resolve "~" via BlobPackingStrategy before StringPackingStrategy sees it, so the reserved keyword MUST NOT be misinterpreted as a string schema.

Scenario: registerDefaultStrategies orders blob before string

• WHEN registerDefaultStrategies(manager) runs and manager.find("~") is called
• THEN the returned strategy is BlobPackingStrategy, not StringPackingStrategy

Verified by: packages/bitpacker/bitpacker-strategies/src/register-default-strategies.test.ts::"BlobPackingStrategy resolves '~' before StringPackingStrategy"

Requirement: Safety-limit enforcement

The package SHALL validate every variable-length type ("-N", "-N[M]", "~") against safetyLimits.maxBufferSize via the SafetyChecker.checkBufferSize(byteLength) API before allocating a write buffer.

Scenario: variable-length string above maxBufferSize rejected

• WHEN a payload of ("-32") exceeds safetyLimits.maxBufferSize
• THEN SafetyChecker.checkBufferSize throws

Scenario: dynamic array above maxBufferSize rejected

• WHEN an array ("-32[16]") whose total bytes exceed safetyLimits.maxBufferSize is checked
• THEN SafetyChecker.checkBufferSize throws

Scenario: adaptive blob above maxBufferSize rejected

• WHEN a blob "~" whose length exceeds safetyLimits.maxBufferSize is checked
• THEN SafetyChecker.checkBufferSize throws

Verified by: packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects variable-length string above maxBufferSize" AND packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects dynamic array above maxBufferSize" AND packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects adaptive blob above maxBufferSize"

Requirement: Public types are Zod-validated at package boundaries

Every package in the bitpacker family SHALL expose a Zod schema for each public input type and SHALL validate untrusted inputs at the package boundary (the entry method that accepts a value from outside the package). Internal recursion MUST NOT re-validate.

Scenario: invalid input rejected at boundary

• WHEN a caller passes a value that does not match the boundary's Zod schema
• THEN the package throws a BitpackerError whose cause is the Zod error and whose code is INVALID_SCHEMA

Scenario: internal recursion does not re-validate

• WHEN the strategy chain recurses into a sub-schema that has already been validated at the entry boundary
• THEN no Zod schema is parsed during the recursion (verified by spying on the schema parser's call count)

Verified by: Per-package boundary schema tests in packages/bitpacker/<each-pkg>/src/**/*.schema.test.ts

Requirement: Public schemas registered with schema-registry

Every package in the bitpacker family SHALL register its public Zod schemas with @suaveplan/schema-registry under namespace "bitpacker", with the package name as the schema name and version: 1. The registration is idempotent — re-importing the package's register-schemas subpath against the same registry is a no-op.

Scenario: schemas present after registration

• WHEN a consumer imports the package's ./register-schemas subpath and calls the exported registerBitPacker<Pkg>Schemas(registry)
• THEN registry.getLatest("bitpacker:@suaveplan/<pkg>") returns a populated entry at version: 1

Verified by: packages/bitpacker/<each-pkg>/src/register-schemas.test.ts per package

Requirement: OTel spans emitted at hot-path boundaries

Every package in the bitpacker family SHALL emit one OTel span per top-level pack/unpack call, named bitpacker.<operation> for the core package and bitpacker_<sub>.<operation> for sub-packages. Per-chunk events MUST be nested span events on the parent span, not standalone spans. Tracer/meter names follow the genesis convention @suaveplan/<pkg>.

Scenario: span emitted on pack

• WHEN a consumer calls a top-level pack/unpack method on BitPacker
• THEN exactly one span is emitted with the canonical name (bitpacker.pack or bitpacker.unpack) and a success attribute of "true" or "false"

Verified by: packages/bitpacker/bitpacker/src/otel/instruments.test.ts and per-package instruments.test.ts files

Requirement: fast-check property tests cover round-trip soundness

Every package that exposes a pack/unpack pair SHALL ship a __property__/*.property.test.ts file using fast-check to assert that for every value matching the schema's arbitrary, unpack(pack(v)) ≡ v.

Scenario: round-trip property holds for every covered schema kind

• WHEN a fast-check arbitrary generates 100 values for a schema kind
• THEN every generated value round-trips byte-identically

Verified by: packages/bitpacker/bitpacker/src/__property__/round-trip.property.test.ts AND packages/bitpacker/bitpacker-strategies/src/__property__/round-trip.property.test.ts

Requirement: mitata benchmarks have committed baselines

Every package that exposes a public encoder/decoder SHALL ship a bench/*.bench.ts file using mitata, with baseline timings committed to scripts/bench/bitpacker-baselines.json. The CI gate that fails on >10% regression is tracked as a follow-on Turbo bench:check task; running bun bench/pack-unpack.bench.ts in any covered package produces measurements that can be compared against the baseline manually.

Scenario: baseline file present and parseable

• WHEN scripts/bench/bitpacker-baselines.json is read
• THEN the file parses as JSON and contains a baselines["@suaveplan/bitpacker"] block with median nanosecond timings for u8/u32 pack/unpack

Verified by: Manual: bun bench/pack-unpack.bench.ts in packages/bitpacker/bitpacker/

Requirement: _version strategy supports opt-in forward tolerance

The VersionPackingStrategy in @suaveplan/bitpacker-strategies SHALL accept an optional _tolerance: "strict" | "forward" field on the version-schema shape. When _tolerance is absent or "strict", the strategy SHALL throw BitpackerError(INVALID_SCHEMA) on any wire-version mismatch (current behaviour). When _tolerance: "forward" is set and the schema also carries an _allowedVersions: number[], the strategy SHALL accept any wire version listed in that array and proceed with the schema's _schema field for decoding. When _tolerance: "forward" is set without _allowedVersions, the strategy SHALL accept any wire version >= 0 and proceed with _schema. The on-wire byte format is unchanged: a 16-bit unsigned big-endian version followed by the schema-encoded payload.

Scenario: strict mode rejects mismatched wire version

• WHEN a consumer unpacks bytes whose wire version differs from the schema's _version and _tolerance is absent or "strict"
• THEN the strategy throws BitpackerError(INVALID_SCHEMA) with the expected and received versions in the error context

Scenario: forward-tolerant mode accepts a listed version

• WHEN a consumer unpacks bytes whose wire version is listed in _allowedVersions and _tolerance: "forward" is set
• THEN the strategy decodes the payload via _schema without throwing

Scenario: forward-tolerant mode without allowlist accepts any version

• WHEN a consumer unpacks bytes with _tolerance: "forward" and no _allowedVersions field
• THEN the strategy decodes any wire version >= 0 via _schema

Scenario: forward-tolerant mode rejects a version outside the allowlist

• WHEN a consumer unpacks bytes whose wire version is NOT in the supplied _allowedVersions, with _tolerance: "forward"
• THEN the strategy throws BitpackerError(INVALID_SCHEMA) with the expected version, the received version, and the allowed-versions list in the error context

Verified by: packages/bitpacker/bitpacker-strategies/src/tests/strategies.test.ts::describe("version") (four scenarios) AND packages/bitpacker/bitpacker-strategies/src/__property__/round-trip.property.test.ts::"forward-tolerant unpack equals strict-unpack output for any same-shape version"