first commit
This commit is contained in:
139
docs/foundation/data/packer/README.md
Normal file
139
docs/foundation/data/packer/README.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# @suaveplan/packer - Binary packing/unpacking: whole-file (gzip + AEAD + ed25519) and chunked (locked v1 byte format, streaming pipeline, server-signed manifests, peer-verifiable).
|
||||
|
||||
## Summary
|
||||
|
||||
Binary packing/unpacking utilities. Two modes:
|
||||
|
||||
- **Whole-file** (`@suaveplan/packer`) — gzip + AEAD (XChaCha20-Poly1305) + ed25519 sign in one shot, schema-driven via `@suaveplan/bitpacker`. Self-contained pack-and-go for small assets.
|
||||
- **Chunked** (`@suaveplan/packer/chunked`) — chunk-addressed streaming pack/unpack/verify emitting the locked v1 byte format. Used by `@suaveplan/drive-plugin-chunks` for P2P-distributable file chunks.
|
||||
|
||||
Status: scaffolding. See the design document for the in-flight port from `../suaveplan/packages/universal/packer`.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
bun add @suaveplan/packer
|
||||
```
|
||||
|
||||
## Quick start
|
||||
|
||||
```ts
|
||||
import { Packer, Unpacker } from "@suaveplan/packer";
|
||||
|
||||
const p = new Packer();
|
||||
const u = new Unpacker();
|
||||
const packed = await p.pack(new Uint8Array([1, 2, 3]));
|
||||
const result = await u.unpack(packed);
|
||||
// result.metadata.integrity carries BLAKE3 entryDigest + Merkle root
|
||||
```
|
||||
|
||||
## Modules
|
||||
|
||||
- [integrity/integrity](./src/integrity/integrity.md) — BLAKE3 per-entry digest, bundle-level Merkle root, `computeWholeFileIntegrity`, `bytesEqual`.
|
||||
- [observability/observability](./src/observability/observability.md) — OpenTelemetry counters, histograms, and spans; `createPackerObservability`, `withPackerSpan`, `withPackerSpanSync`.
|
||||
- [i18n/messages](./src/i18n/messages.md) — `PACKER_MESSAGES` English-defaults catalog and `defaultPackerResolver` / `resolvePackerMessage` for `MessageResolver` injection.
|
||||
- [packer/packer](./src/packer/packer.md) — whole-file `Packer.pack` (gzip + AEAD + ed25519 + BLAKE3 integrity).
|
||||
- [unpacker/unpacker](./src/unpacker/unpacker.md) — whole-file `Unpacker.unpack` with mandatory BLAKE3 integrity verification.
|
||||
- [chunked/packer/packer](./src/chunked/packer/packer.md) — streaming v1 chunked pack pipeline.
|
||||
- [chunked/unpacker/unpacker](./src/chunked/unpacker/unpacker.md) — chunked unpack (`unpackChunks`, `unpackChunksWithManifest`).
|
||||
- [chunked/manifest/manifest](./src/chunked/manifest/manifest.md) — server-signed manifest builder/verifier carrying `entryDigest` + `merkleRoot`.
|
||||
- [chunked/codec/chunk](./src/chunked/codec/chunk.md) and [chunked/codec/header](./src/chunked/codec/header.md) — v1 byte-layout encoder/parser.
|
||||
- [chunked/crypto/aead](./src/chunked/crypto/aead.md) — XChaCha20-Poly1305 wrapper + per-chunk nonce derivation.
|
||||
- [chunked/verifier/chunk-verifier](./src/chunked/verifier/chunk-verifier.md) — server-pubkey-only verifier for peer-mesh consumers.
|
||||
- [streaming/normalize-source](./src/streaming/normalize-source.md) — source-shape normaliser.
|
||||
- [error-codes](./src/error-codes.md) — `PACKER_CODES` registry, including `PACKER.INTEGRITY_FAILED`.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [abstract](./src/abstract/abstract.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [abstract](./src/abstract/abstract.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [chunk](./src/chunked/codec/chunk.md) — Encodes and parses the wire-format v1 chunk in its entirety: `metadata(100) || sig(64) || ciphertext(payloadLen)`. Comp…
|
||||
- [header](./src/chunked/codec/header.md) — Encodes and parses the 100-byte metadata portion of a v1 chunk on its own (without sig or ciphertext). Every chunk on t…
|
||||
- [crypto](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [aead](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [manifest](./src/chunked/manifest/manifest.md) — Builds, signs, and verifies the v1 manifest — the server-signed object that maps a `fileId` to its full chunk list, enc…
|
||||
- [manifest](./src/chunked/manifest/manifest.md) — Builds, signs, and verifies the v1 manifest — the server-signed object that maps a `fileId` to its full chunk list, enc…
|
||||
- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [chunk-verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [flags](./src/flags/flags.md) — `flags.ts` wires `@suaveplan/packer` into the `@suaveplan/feature-flags` ecosystem so future wire-format versions can b…
|
||||
- [flags](./src/flags/flags.md) — `flags.ts` wires `@suaveplan/packer` into the `@suaveplan/feature-flags` ecosystem so future wire-format versions can b…
|
||||
- [i18n](./src/i18n/messages.md) — `@suaveplan/packer/i18n` carries the per-package message catalog every error thrown by `@suaveplan/packer` flows throug…
|
||||
- [messages](./src/i18n/messages.md) — `@suaveplan/packer/i18n` carries the per-package message catalog every error thrown by `@suaveplan/packer` flows throug…
|
||||
- [idempotency](./src/idempotency/idempotency.md) — Repeat packs over identical inputs are a guaranteed no-op: the second call returns the cached `PackedData` without re-r…
|
||||
- [idempotency](./src/idempotency/idempotency.md) — Repeat packs over identical inputs are a guaranteed no-op: the second call returns the cached `PackedData` without re-r…
|
||||
- [integrity](./src/integrity/integrity.md) — `@suaveplan/packer/integrity` provides the content-addressable integrity layer the rest of the package uses to detect a…
|
||||
- [integrity](./src/integrity/integrity.md) — `@suaveplan/packer/integrity` provides the content-addressable integrity layer the rest of the package uses to detect a…
|
||||
- [observability](./src/observability/observability.md) — `@suaveplan/packer/observability` centralises every OpenTelemetry counter, histogram, and span the package emits, so a…
|
||||
- [observability](./src/observability/observability.md) — `@suaveplan/packer/observability` centralises every OpenTelemetry counter, histogram, and span the package emits, so a…
|
||||
- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [plugins](./src/plugins/plugins.md) — `plugins.ts` exposes a small, opinionated extension surface that lets downstream packages contribute behaviour to `Pack…
|
||||
- [plugins](./src/plugins/plugins.md) — `plugins.ts` exposes a small, opinionated extension surface that lets downstream packages contribute behaviour to `Pack…
|
||||
- [retry](./src/retry/retryable-read.md) — Wraps `ReadableStream` reads with the retry primitives from `@suaveplan/retry` so transient IO failures (network blip,…
|
||||
- [retryable-read](./src/retry/retryable-read.md) — Wraps `ReadableStream` reads with the retry primitives from `@suaveplan/retry` so transient IO failures (network blip,…
|
||||
- [runtime](./src/runtime/runtime.md) — Every production code path in `@suaveplan/packer` reads wall-clock time and randomness through an injected `PackerRunti…
|
||||
- [runtime](./src/runtime/runtime.md) — Every production code path in `@suaveplan/packer` reads wall-clock time and randomness through an injected `PackerRunti…
|
||||
- [schemas](./src/schemas/registration.md) — `registration.ts` wires every Zod schema exported from `packer.schemas.ts` and a canonical `packer-chunk-format` wire-f…
|
||||
- [registration](./src/schemas/registration.md) — `registration.ts` wires every Zod schema exported from `packer.schemas.ts` and a canonical `packer-chunk-format` wire-f…
|
||||
- [streaming](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [normalize-source](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [error-codes](./src/error-codes.md) — Defines the stable error code registry for `@suaveplan/packer`. Every failure surfaced by the package — invalid inputs,…
|
||||
<!-- end:auto-docs --> - [abstract](./src/abstract/abstract.md) — <!-- begin:auto-docs --> _(no submodules)_- [chunk](./src/chunked/codec/chunk.md) — Encodes and parses the wire-format v1 chunk in its entirety: `metadata(100) || sig(64) || ciphertext(payloadLen)`. Comp…
|
||||
- [header](./src/chunked/codec/header.md) — Encodes and parses the 100-byte metadata portion of a v1 chunk on its own (without sig or ciphertext). Every chunk on t…
|
||||
- [crypto](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_- [aead](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest](./src/chunked/manifest/manifest.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest](./src/chunked/manifest/manifest.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_- [chunk-verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [streaming](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_- [normalize-source](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Defines the stable error code registry for `@suaveplan/packer`. Every failure surfaced by the package — invalid inputs,…
|
||||
- [abstract](./src/abstract/abstract.md) — <!-- begin:auto-docs --> _(no submodules)_- [chunk](./src/chunked/codec/chunk.md) — Encodes and parses the wire-format v1 chunk in its entirety: `metadata(100) || sig(64) || ciphertext(payloadLen)`. Comp…
|
||||
- [header](./src/chunked/codec/header.md) — Encodes and parses the 100-byte metadata portion of a v1 chunk on its own (without sig or ciphertext). Every chunk on t…
|
||||
- [crypto](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_- [aead](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest](./src/chunked/manifest/manifest.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest](./src/chunked/manifest/manifest.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_- [chunk-verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [streaming](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_- [normalize-source](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Defines the stable error code registry for `@suaveplan/packer`. Every failure surfaced by the package — invalid inputs,…
|
||||
- [abstract](./src/abstract/abstract.md) — <!-- begin:auto-docs --> _(no submodules)_- [chunk](./src/chunked/codec/chunk.md) — Encodes and parses the wire-format v1 chunk in its entirety: `metadata(100) || sig(64) || ciphertext(payloadLen)`. Comp…
|
||||
- [header](./src/chunked/codec/header.md) — Encodes and parses the 100-byte metadata portion of a v1 chunk on its own (without sig or ciphertext). Every chunk on t…
|
||||
- [crypto](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_- [aead](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest](./src/chunked/manifest/manifest.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest](./src/chunked/manifest/manifest.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_- [chunk-verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [streaming](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_- [normalize-source](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Defines the stable error code registry for `@suaveplan/packer`. Every failure surfaced by the package — invalid inputs,…
|
||||
- [abstract](./src/abstract/abstract.md) — <!-- begin:auto-docs --> _(no submodules)_- [chunk](./src/chunked/codec/chunk.md) — Encodes and parses the wire-format v1 chunk in its entirety: `metadata(100) || sig(64) || ciphertext(payloadLen)`. Comp…
|
||||
- [header](./src/chunked/codec/header.md) — Encodes and parses the 100-byte metadata portion of a v1 chunk on its own (without sig or ciphertext). Every chunk on t…
|
||||
- [crypto](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_ - [aead](./src/chunked/crypto/aead.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest](./src/chunked/manifest/manifest.md) — <!-- begin:auto-docs --> _(no submodules)_ - [manifest](./src/chunked/manifest/manifest.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_ - [packer](./src/chunked/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_ - [unpacker](./src/chunked/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_ - [chunk-verifier](./src/chunked/verifier/chunk-verifier.md) — <!-- begin:auto-docs --> _(no submodules)_- [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_ - [packer](./src/packer/packer.md) — <!-- begin:auto-docs --> _(no submodules)_- [streaming](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_ - [normalize-source](./src/streaming/normalize-source.md) — <!-- begin:auto-docs --> _(no submodules)_- [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_ - [unpacker](./src/unpacker/unpacker.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Defines the stable error code registry for `@suaveplan/packer`. Every failure surfaced by the package — invalid inputs,…
|
||||
|
||||
## Subpaths
|
||||
|
||||
| Subpath | Surface |
|
||||
| --- | --- |
|
||||
| `@suaveplan/packer` | `Packer`, `Unpacker`, whole-file types |
|
||||
| `@suaveplan/packer/chunked` | `ChunkedPacker`, `ChunkedUnpacker`, `ChunkVerifier`, manifest helpers |
|
||||
| `@suaveplan/packer/streaming` | `normalizeSource(...)` source-normalization helper |
|
||||
| `@suaveplan/packer/errors` | `PACKER_CODES`, `PackerError` |
|
||||
| `@suaveplan/packer/flags` | `defaultPackerFlags`, `resolvePackerFlags`, `PACKER_FLAG_KEYS` |
|
||||
| `@suaveplan/packer/plugins` | `PackerPlugin`, `PluginPipeline` |
|
||||
| `@suaveplan/packer/schemas` | `registerPackerSchemas`, `getPackerSchemaRegistry`, `PACKER_CHUNK_FORMAT_V1` |
|
||||
|
||||
## Benchmarks
|
||||
|
||||
Baseline numbers collected on bun 1.3.14 (x64-linux) — Intel(R) Xeon(R)
|
||||
CPU E5-2696 v4 @ 2.20GHz, WSL2. Run locally with `bun run benchmark`.
|
||||
|
||||
| Case | Throughput (avg) | Notes |
|
||||
| --- | --- | --- |
|
||||
| whole-file pack — 1 KiB | ~254 µs/iter | gzip+sha256+blake3 hot path |
|
||||
| whole-file unpack — 1 KiB | ~47 µs/iter | integrity verify + gunzip |
|
||||
| chunked pack — single chunk | ~654 µs/iter | XChaCha20-Poly1305 + ed25519 |
|
||||
| chunked pack — 1000 chunks | ~562 ms/iter | 1 MiB streamed in 1 KiB chunks |
|
||||
| integrity verify — 1 KiB | ~21 µs/iter | BLAKE3 entry + Merkle root |
|
||||
| plugin overhead — pack + noop plugin | ~147 µs/iter | observed over the 1 KiB pack baseline |
|
||||
|
||||
The benchmark suite is advisory: `bun run benchmark` runs `mitata` and
|
||||
emits the numbers above. The suite is intentionally excluded from the
|
||||
test gate (vitest excludes `src/__benchmarks__/**`) — regression
|
||||
tracking is opt-in for local profiling.
|
||||
|
||||
## Chunk binary format — v1
|
||||
|
||||
```text
|
||||
metadata(100 B) ‖ sig(64 B) ‖ ciphertext(payloadLen B)
|
||||
```
|
||||
|
||||
See the specification for the byte-exact field offsets pinned in `PACKER-CHUNK-01`.
|
||||
|
||||
Reference in New Issue
Block a user