165 lines
8.3 KiB
Markdown
165 lines
8.3 KiB
Markdown
# OpenSpec Change: packer-port
|
||
|
||
- **Change ID**: packer-port
|
||
- **Package**: `@suaveplan/packer` (new)
|
||
- **Status**: PROPOSED
|
||
- **Date**: 2026-04-28
|
||
|
||
## Summary
|
||
|
||
Port `@suaveplan/packer` from `../suaveplan/packages/universal/packer` into genesis. Two modes ship as subpaths:
|
||
|
||
- **`@suaveplan/packer`** — whole-file pack/unpack: gzip + AEAD + ed25519 sign in one shot, schema-driven via bitpacker. Self-contained pack-and-go for small assets. Carried over with minor cleanup.
|
||
- **`@suaveplan/packer/chunked`** — chunk-addressed streaming pack/unpack/verify emitting the locked **v1 byte format** specced in `drive-plugin-chunks-port`. Partial rewrite: keeps source's pipeline (streaming source normalisation, AEAD streaming, per-chunk signing, verifier) but drops bitpacker on the chunked path in favour of a byte-aligned fixed-layout codec.
|
||
|
||
Three structural changes versus source:
|
||
|
||
1. **Chunked wire format = locked v1** (164-byte fixed header + ciphertext). Source's u8/u64/u40 bitpacker schema is replaced. Whole-file mode keeps bitpacker (no fixed-size constraint there).
|
||
2. **No signature recovery.** Source uses `signWithRecovery`/`verifyWithRecovery` (1-bit recovery flag); v1 is plain `ed25519_sign` because every consumer knows the server pubkey out-of-band. Saves a bit, simplifies verify path.
|
||
3. **Streaming-first chunked pipeline.** Encryption runs through a streaming AEAD; the ciphertext stream is chopped at `payload_per_chunk = chunkSize - 164` byte offsets. Last chunk is naturally short (no zero-padding). Header `payloadLen` is authoritative.
|
||
|
||
## Source-side scope
|
||
|
||
`../suaveplan/packages/universal/packer/` v2.0.0 — 37 `.ts` files, ~3 460 src + ~4 997 test LOC.
|
||
|
||
```
|
||
src/
|
||
├── abstract/ — AbstractPacker base (compression / AEAD / signing helpers)
|
||
├── streaming/ — normalizeSource(Uint8Array | ReadableStream | AsyncIterable)
|
||
├── packer/ — Packer (whole-file)
|
||
├── unpacker/ — Unpacker (whole-file)
|
||
├── chunked/
|
||
│ ├── packer/ — ChunkedPacker
|
||
│ ├── unpacker/ — ChunkedUnpacker
|
||
│ └── verifier/ — ChunkVerifier
|
||
├── types/
|
||
└── integration/
|
||
```
|
||
|
||
Source whole-file public surface: `Packer`, `Unpacker`, `signedPacketSchema`. Source chunked surface: `ChunkedPacker`, `ChunkedUnpacker`, `ChunkVerifier`, `chunkedPacketSchema` (replaced by v1 codec).
|
||
|
||
## Dependencies
|
||
|
||
All present in genesis:
|
||
|
||
- `@suaveplan/bitpacker` (`packages/primitives/bitpacker`) — used by whole-file mode
|
||
- `@suaveplan/crypto` (`packages/crypto/crypto`) — `encrypt`/`decrypt`, AEAD streaming, ed25519 sign/verify, hashing
|
||
- `@suaveplan/events` (`packages/events`) — `EventEmitter` base for progress events
|
||
- `@suaveplan/mime` (`packages/primitives/mime`) — MIME detection on pack input
|
||
- `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/streams`, `@opentelemetry/api`, `zod`
|
||
|
||
No new packages are required. No source-side blockers.
|
||
|
||
## What changes (port-time)
|
||
|
||
### 1. Layout
|
||
|
||
```
|
||
packages/data/packer/
|
||
├── package.json # @suaveplan/packer @ 0.0.0
|
||
├── tsconfig.json # extends @suaveplan/typescript-config/library
|
||
├── src/
|
||
│ ├── abstract/ # ported with minor cleanup
|
||
│ ├── streaming/ # ported as-is
|
||
│ ├── packer/ # whole-file Packer (bitpacker schema)
|
||
│ ├── unpacker/ # whole-file Unpacker
|
||
│ ├── chunked/
|
||
│ │ ├── codec/ # NEW: v1 byte-format header encode/decode
|
||
│ │ ├── packer/ # ChunkedPacker — emits v1 chunks via streaming pipeline
|
||
│ │ ├── unpacker/ # ChunkedUnpacker — consumes v1 chunks
|
||
│ │ └── verifier/ # ChunkVerifier — sig + AEAD verify, no decrypt
|
||
│ ├── error-codes.ts # PACKER_CODES via defineCodes
|
||
│ ├── schemas/ # Zod boundary schemas for options
|
||
│ └── index.ts # barrel — whole-file surface
|
||
└── src/chunked/index.ts # barrel — chunked surface (subpath ./chunked)
|
||
```
|
||
|
||
### 2. Subpath exports
|
||
|
||
```jsonc
|
||
"exports": {
|
||
".": { "import": "./dist/index.js", "types": "./dist/index.d.ts" },
|
||
"./chunked": { "import": "./dist/chunked/index.js", "types": "./dist/chunked/index.d.ts" },
|
||
"./streaming":{ "import": "./dist/streaming/index.js", "types": "./dist/streaming/index.d.ts" },
|
||
"./errors": { "import": "./dist/error-codes.js", "types": "./dist/error-codes.d.ts" }
|
||
}
|
||
```
|
||
|
||
### 3. Chunked v1 codec (fixed-layout, byte-aligned, big-endian)
|
||
|
||
Header layout per `drive-plugin-chunks-port` `CHUNKS-FORMAT-V1-01`. Repeated here for completeness:
|
||
|
||
```
|
||
metadata(100 B):
|
||
0..2 version uint16 = 0x0001
|
||
2..34 fileId bytes(32) // content hash of plaintext
|
||
34..38 chunkId uint32
|
||
38..42 totalChunks uint32
|
||
42..50 fileSize uint64
|
||
50..54 chunkSize uint32 // budget for non-final chunks
|
||
54..58 payloadLen uint32
|
||
58..60 flags uint16 // v1 MUST be 0x0000
|
||
60..84 nonce bytes(24) // XChaCha20 nonce
|
||
84..100 aeadTag bytes(16) // Poly1305 tag
|
||
|
||
sig(64 B):
|
||
100..164 ed25519 signature over metadata[0..100] ‖ ciphertext[0..payloadLen]
|
||
|
||
ciphertext(payloadLen B):
|
||
164..(164+payloadLen)
|
||
```
|
||
|
||
Last chunk: `payloadLen = fileSize − chunkId * (chunkSize − 164)`. Non-final chunks: `payloadLen = chunkSize − 164`. No zero padding.
|
||
|
||
### 4. Streaming chunked pipeline (write path)
|
||
|
||
```
|
||
PackSource ─► normalizeSource ─► AsyncIterable<Uint8Array>
|
||
─► hash(plaintext) → fileId (sha256, 32 B)
|
||
─► aead-encrypt-stream (XChaCha20-Poly1305, file key + per-chunk nonce)
|
||
─► chop at (chunkSize − 164) byte boundaries
|
||
─► for each slice:
|
||
derive nonce = baseNonce ⊕ (chunkId as 8 B little-endian, zero-extended to 24 B)
|
||
build metadata (100 B)
|
||
ed25519-sign(metadata ‖ ciphertext) → 64 B sig
|
||
emit metadata ‖ sig ‖ ciphertext
|
||
─► finally: build manifest (server-signed, fileId-addressed)
|
||
```
|
||
|
||
End-to-end streaming. No whole-file buffering.
|
||
|
||
### 5. Streaming chunked pipeline (read/verify path)
|
||
|
||
```
|
||
chunk-bytes ─► parseHeader (structural validation: version, flags, payloadLen invariants)
|
||
─► verifySig (metadata ‖ ciphertext) — reject on fail
|
||
─► aead-decrypt(ciphertext, nonce, aeadTag, fileKey)
|
||
─► emit plaintext slice
|
||
```
|
||
|
||
`ChunkVerifier` performs structural-parse + sig-verify only (no decrypt) — used by mesh peers that forward chunks without holding the file key.
|
||
|
||
### 6. Drops
|
||
|
||
- `signWithRecovery`/`verifyWithRecovery` calls on the chunked path — replaced with plain `ed25519_sign`/`ed25519_verify`. Whole-file mode keeps recovery semantics if the source had them (audit during port).
|
||
- Bitpacker schema for chunks (`chunkedPacketSchema`) — replaced by the v1 byte codec.
|
||
- Source's `EventEmitter` -based progress events — replaced by genesis `@suaveplan/events` `EventBus<TMap>` typed events.
|
||
|
||
### 7. Error codes
|
||
|
||
`PACKER_CODES = defineCodes("PACKER", [INVALID_SOURCE, COMPRESSION_FAILED, ENCRYPTION_FAILED, SIGNATURE_FAILED, VERIFICATION_FAILED, INVALID_FORMAT, VERSION_UNSUPPORTED, FLAGS_RESERVED_NONZERO, PAYLOAD_LEN_INVALID, ABORTED] as const)`. `PackerError extends SuaveplanError<PackerErrorCode>`. Frozen.
|
||
|
||
## Impact
|
||
|
||
- New package `packages/data/packer/` at `0.1.0`.
|
||
- New OpenSpec capability `packer` at `openspec/specs/packer/spec.md`.
|
||
- Unblocks `drive-plugin-chunks-port` (which depends on the chunked subpath).
|
||
- Source-side compatibility: **wire-format break** on the chunked side. Whole-file packed data from source v2.0.0 is unaffected (whole-file format unchanged).
|
||
|
||
## Out of scope
|
||
|
||
- Source's `integration/integration.test.ts` end-to-end suite — re-authored as conformance tests against the new format, not ported byte-for-byte.
|
||
- Compression on chunked path — v1 reserved `flags` bit 0 for `compressed`; not implemented in v1.
|
||
- Algorithm agility — v1 hardwires XChaCha20-Poly1305 + ed25519. Algo selector via `flags` bit 1 reserved for v2.
|
||
- Browser-specific `packer-browser` — separate change after this lands.
|