152 lines
10 KiB
Markdown
152 lines
10 KiB
Markdown
# bitpacker Specification
|
|
|
|
## Purpose
|
|
|
|
`@suaveplan/bitpacker` provides bit-level binary serialization driven by declarative schemas. It supports fixed-width primitives (`u8`, `i16`, etc.), variable-length strings/arrays with max-sized length prefixes, and an adaptive blob type (`"~"`) whose length-prefix width is self-describing via a 2-bit tag so payloads ranging from bytes to ~1 TiB share a schema without wasting bits on small entries.
|
|
|
|
## Requirements
|
|
|
|
### Requirement: Adaptive blob schema type `"~"`
|
|
|
|
The package SHALL treat schema value `"~"` as a binary blob whose length prefix is self-describing via a 2-bit width tag selecting 2, 3, 4, or 5 bytes of length. Pack input and unpack output SHALL be `Uint8Array` with no encoding.
|
|
|
|
#### Scenario: tag selected for a small payload
|
|
|
|
- **WHEN** packing 200 bytes with schema field `data: "~"`
|
|
- **THEN** the written bits are `[tag=0][2-byte length=200][200 raw bytes]`
|
|
|
|
#### Scenario: tag selected for a large payload
|
|
|
|
- **WHEN** packing 17 MiB
|
|
- **THEN** the tag is `2` and the length prefix is 4 bytes
|
|
|
|
#### Scenario: payload over 1 TiB rejected
|
|
|
|
- **WHEN** packing a buffer whose length exceeds `0xFFFFFFFFFF`
|
|
- **THEN** the packer throws `BitpackerError(VALUE_OUT_OF_RANGE)`
|
|
|
|
#### Scenario: round-trip preserves bytes
|
|
|
|
- **WHEN** a `Uint8Array` is packed then unpacked
|
|
- **THEN** the output is byte-identical to the input
|
|
|
|
**Verified by:** `packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"selects 2-byte length tag for payloads under 64 KiB"` AND `packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"selects 4-byte length tag for payloads over 16 MiB"` AND `packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"throws on payloads exceeding 1 TiB"` AND `packages/bitpacker/bitpacker-strategies/src/strategies/blob/blob.test.ts::"round-trip preserves bytes byte-identical"`
|
|
|
|
### Requirement: Blob strategy runs before string strategy
|
|
|
|
The strategy chain SHALL resolve `"~"` via `BlobPackingStrategy` before `StringPackingStrategy` sees it, so the reserved keyword MUST NOT be misinterpreted as a string schema.
|
|
|
|
#### Scenario: registerDefaultStrategies orders blob before string
|
|
|
|
- **WHEN** `registerDefaultStrategies(manager)` runs and `manager.find("~")` is called
|
|
- **THEN** the returned strategy is `BlobPackingStrategy`, not `StringPackingStrategy`
|
|
|
|
**Verified by:** `packages/bitpacker/bitpacker-strategies/src/register-default-strategies.test.ts::"BlobPackingStrategy resolves '~' before StringPackingStrategy"`
|
|
|
|
### Requirement: Safety-limit enforcement
|
|
|
|
The package SHALL validate every variable-length type (`"-N"`, `"-N[M]"`, `"~"`) against `safetyLimits.maxBufferSize` via the `SafetyChecker.checkBufferSize(byteLength)` API before allocating a write buffer.
|
|
|
|
#### Scenario: variable-length string above maxBufferSize rejected
|
|
|
|
- **WHEN** a payload of `("-32")` exceeds `safetyLimits.maxBufferSize`
|
|
- **THEN** `SafetyChecker.checkBufferSize` throws
|
|
|
|
#### Scenario: dynamic array above maxBufferSize rejected
|
|
|
|
- **WHEN** an array `("-32[16]")` whose total bytes exceed `safetyLimits.maxBufferSize` is checked
|
|
- **THEN** `SafetyChecker.checkBufferSize` throws
|
|
|
|
#### Scenario: adaptive blob above maxBufferSize rejected
|
|
|
|
- **WHEN** a blob `"~"` whose length exceeds `safetyLimits.maxBufferSize` is checked
|
|
- **THEN** `SafetyChecker.checkBufferSize` throws
|
|
|
|
**Verified by:** `packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects variable-length string above maxBufferSize"` AND `packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects dynamic array above maxBufferSize"` AND `packages/bitpacker/bitpacker-validation/src/safety/safety-checks.test.ts::"rejects adaptive blob above maxBufferSize"`
|
|
|
|
### Requirement: Public types are Zod-validated at package boundaries
|
|
|
|
Every package in the bitpacker family SHALL expose a Zod schema for each public input type and SHALL validate untrusted inputs at the package boundary (the entry method that accepts a value from outside the package). Internal recursion MUST NOT re-validate.
|
|
|
|
#### Scenario: invalid input rejected at boundary
|
|
|
|
- **WHEN** a caller passes a value that does not match the boundary's Zod schema
|
|
- **THEN** the package throws a `BitpackerError` whose `cause` is the Zod error and whose code is `INVALID_SCHEMA`
|
|
|
|
#### Scenario: internal recursion does not re-validate
|
|
|
|
- **WHEN** the strategy chain recurses into a sub-schema that has already been validated at the entry boundary
|
|
- **THEN** no Zod schema is parsed during the recursion (verified by spying on the schema parser's call count)
|
|
|
|
**Verified by:** Per-package boundary schema tests in `packages/bitpacker/<each-pkg>/src/**/*.schema.test.ts`
|
|
|
|
### Requirement: Public schemas registered with schema-registry
|
|
|
|
Every package in the bitpacker family SHALL register its public Zod schemas with `@suaveplan/schema-registry` under namespace `"bitpacker"`, with the package name as the schema name and `version: 1`. The registration is idempotent — re-importing the package's `register-schemas` subpath against the same registry is a no-op.
|
|
|
|
#### Scenario: schemas present after registration
|
|
|
|
- **WHEN** a consumer imports the package's `./register-schemas` subpath and calls the exported `registerBitPacker<Pkg>Schemas(registry)`
|
|
- **THEN** `registry.getLatest("bitpacker:@suaveplan/<pkg>")` returns a populated entry at `version: 1`
|
|
|
|
**Verified by:** `packages/bitpacker/<each-pkg>/src/register-schemas.test.ts` per package
|
|
|
|
### Requirement: OTel spans emitted at hot-path boundaries
|
|
|
|
Every package in the bitpacker family SHALL emit one OTel span per top-level pack/unpack call, named `bitpacker.<operation>` for the core package and `bitpacker_<sub>.<operation>` for sub-packages. Per-chunk events MUST be nested span events on the parent span, not standalone spans. Tracer/meter names follow the genesis convention `@suaveplan/<pkg>`.
|
|
|
|
#### Scenario: span emitted on pack
|
|
|
|
- **WHEN** a consumer calls a top-level pack/unpack method on `BitPacker`
|
|
- **THEN** exactly one span is emitted with the canonical name (`bitpacker.pack` or `bitpacker.unpack`) and a `success` attribute of `"true"` or `"false"`
|
|
|
|
**Verified by:** `packages/bitpacker/bitpacker/src/otel/instruments.test.ts` and per-package `instruments.test.ts` files
|
|
|
|
### Requirement: fast-check property tests cover round-trip soundness
|
|
|
|
Every package that exposes a pack/unpack pair SHALL ship a `__property__/*.property.test.ts` file using `fast-check` to assert that for every value matching the schema's arbitrary, `unpack(pack(v)) ≡ v`.
|
|
|
|
#### Scenario: round-trip property holds for every covered schema kind
|
|
|
|
- **WHEN** a fast-check arbitrary generates 100 values for a schema kind
|
|
- **THEN** every generated value round-trips byte-identically
|
|
|
|
**Verified by:** `packages/bitpacker/bitpacker/src/__property__/round-trip.property.test.ts` AND `packages/bitpacker/bitpacker-strategies/src/__property__/round-trip.property.test.ts`
|
|
|
|
### Requirement: mitata benchmarks have committed baselines
|
|
|
|
Every package that exposes a public encoder/decoder SHALL ship a `bench/*.bench.ts` file using `mitata`, with baseline timings committed to `scripts/bench/bitpacker-baselines.json`. The CI gate that fails on >10% regression is tracked as a follow-on Turbo `bench:check` task; running `bun bench/pack-unpack.bench.ts` in any covered package produces measurements that can be compared against the baseline manually.
|
|
|
|
#### Scenario: baseline file present and parseable
|
|
|
|
- **WHEN** `scripts/bench/bitpacker-baselines.json` is read
|
|
- **THEN** the file parses as JSON and contains a `baselines["@suaveplan/bitpacker"]` block with median nanosecond timings for u8/u32 pack/unpack
|
|
|
|
**Verified by:** Manual: `bun bench/pack-unpack.bench.ts` in `packages/bitpacker/bitpacker/`
|
|
|
|
### Requirement: `_version` strategy supports opt-in forward tolerance
|
|
|
|
The `VersionPackingStrategy` in `@suaveplan/bitpacker-strategies` SHALL accept an optional `_tolerance: "strict" | "forward"` field on the version-schema shape. When `_tolerance` is absent or `"strict"`, the strategy SHALL throw `BitpackerError(INVALID_SCHEMA)` on any wire-version mismatch (current behaviour). When `_tolerance: "forward"` is set and the schema also carries an `_allowedVersions: number[]`, the strategy SHALL accept any wire version listed in that array and proceed with the schema's `_schema` field for decoding. When `_tolerance: "forward"` is set without `_allowedVersions`, the strategy SHALL accept any wire version >= 0 and proceed with `_schema`. The on-wire byte format is unchanged: a 16-bit unsigned big-endian version followed by the schema-encoded payload.
|
|
|
|
#### Scenario: strict mode rejects mismatched wire version
|
|
|
|
- **WHEN** a consumer unpacks bytes whose wire version differs from the schema's `_version` and `_tolerance` is absent or `"strict"`
|
|
- **THEN** the strategy throws `BitpackerError(INVALID_SCHEMA)` with the expected and received versions in the error context
|
|
|
|
#### Scenario: forward-tolerant mode accepts a listed version
|
|
|
|
- **WHEN** a consumer unpacks bytes whose wire version is listed in `_allowedVersions` and `_tolerance: "forward"` is set
|
|
- **THEN** the strategy decodes the payload via `_schema` without throwing
|
|
|
|
#### Scenario: forward-tolerant mode without allowlist accepts any version
|
|
|
|
- **WHEN** a consumer unpacks bytes with `_tolerance: "forward"` and no `_allowedVersions` field
|
|
- **THEN** the strategy decodes any wire version >= 0 via `_schema`
|
|
|
|
#### Scenario: forward-tolerant mode rejects a version outside the allowlist
|
|
|
|
- **WHEN** a consumer unpacks bytes whose wire version is NOT in the supplied `_allowedVersions`, with `_tolerance: "forward"`
|
|
- **THEN** the strategy throws `BitpackerError(INVALID_SCHEMA)` with the expected version, the received version, and the allowed-versions list in the error context
|
|
|
|
**Verified by:** `packages/bitpacker/bitpacker-strategies/src/tests/strategies.test.ts::describe("version")` (four scenarios) AND `packages/bitpacker/bitpacker-strategies/src/__property__/round-trip.property.test.ts::"forward-tolerant unpack equals strict-unpack output for any same-shape version"`
|