Initial Commit
This commit is contained in:
@@ -0,0 +1,48 @@
|
||||
# OpenSpec Change: drive-plugin-chunks-port
|
||||
|
||||
**Package**: `@suaveplan/drive-plugin-chunks`
|
||||
**Status**: PROPOSED
|
||||
**Type**: new-package
|
||||
|
||||
## Why
|
||||
|
||||
The core `@suaveplan/drive` package is deliberately file-level only. Chunk addressing — splitting files into signed, encrypted chunks for P2P distribution — must live in a separate plugin so that consumers who only need local file VFS (e.g. server-side S3-backed storage) don't pay the bundle cost of chunk codecs, AEAD pipelines, and manifest handling. This plugin defines the `ChunkSource` interface that `drive-mesh` and transport packages consume, establishing the file↔chunk seam as a stable, versioned contract.
|
||||
|
||||
## What changes
|
||||
|
||||
- ADDED: `@suaveplan/drive-plugin-chunks` package at `packages/drive/drive-plugin-chunks/`
|
||||
- ADDED: `version 0x0001` chunk binary format — 164-byte header + variable ciphertext, big-endian, ed25519 server-signed
|
||||
- ADDED: `ChunkSource` interface — consumed by mesh and transport packages, not drive core
|
||||
- ADDED: Server-signed manifest format with Zod/CBOR validation
|
||||
- ADDED: Two storage layouts (`sparse-file`, `chunk-keys`) plus `auto` selector
|
||||
- ADDED: Streaming encrypt-then-chunk pipeline (XChaCha20-Poly1305) — never materialises whole file in memory
|
||||
- ADDED: Read-path Drive middleware that intercepts `drive.read(path)` and reassembles plaintext from chunks
|
||||
- ADDED: `CHUNKS_CODES` error registry (`ChunksError`) with 12 namespaced codes
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Mesh / peer index / parallel pull → `drive-mesh-port`.
|
||||
- Wire transports → `drive-transport-*-port`.
|
||||
- v2 chunk format with compression / algo agility — reserved flag bits documented but unimplemented.
|
||||
- Automatic chunking on every `drive.write()` — chunking is opt-in via `chunks.ingest(path)` only.
|
||||
|
||||
## Source mapping
|
||||
|
||||
> This package is genesis-original. No `../suaveplan/enterprise/drive-plugin-chunks.md` exists. Delete this section.
|
||||
|
||||
## Verification approach
|
||||
|
||||
Every requirement below is verified by a spec-conformance test that pins an observable behavior. Unit tests cover happy-path and edge-case scenarios for the same module.
|
||||
|
||||
- `src/codec/chunk.spec-conformance.test.ts` — byte-exact layout, flags rejection, payload-length invariants (`CHUNKS-FORMAT-V1-01` through `-03`)
|
||||
- `src/pipeline/sign-chunk.spec-conformance.test.ts` — signature covers metadata + ciphertext, mutation detection (`CHUNKS-SIG-01`)
|
||||
- `src/layout/sparse-file.spec-conformance.test.ts` — pre-allocation, `setPartial` writes, bitmap, capability gate (`CHUNKS-LAYOUT-SPARSE-01`)
|
||||
- `src/layout/chunk-keys.spec-conformance.test.ts` — one-key-per-chunk semantics (`CHUNKS-LAYOUT-KEYS-01`)
|
||||
- `src/layout/auto.spec-conformance.test.ts` — layout selection by capability (`CHUNKS-LAYOUT-AUTO-01`)
|
||||
- `src/manifest/load.spec-conformance.test.ts` — manifest-as-chunk and server-only modes (`CHUNKS-MANIFEST-01`, `-02`)
|
||||
- `src/plugin.spec-conformance.test.ts` — store-as-is invariant, sig-verify gate, read reassembly (`CHUNKS-INVARIANT-01`, `CHUNKS-VERIFY-01`, `CHUNKS-READ-01`)
|
||||
- `src/error-codes.spec-conformance.test.ts` — frozen registry, namespace (`CHUNKS-CODES-01`)
|
||||
|
||||
## Migration impact
|
||||
|
||||
None — additive only. New package with no existing consumers in genesis.
|
||||
@@ -0,0 +1,81 @@
|
||||
# Drive Plugin Chunks — Spec Delta (drive-plugin-chunks-port)
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### CHUNKS-FORMAT-V1-01 — Chunk binary layout is byte-exact
|
||||
|
||||
A `version 0x0001` chunk MUST be the byte concatenation `metadata(100) ‖ sig(64) ‖ ciphertext(payloadLen)`, total `164 + payloadLen` bytes, big-endian throughout. Field offsets within metadata MUST be exactly: `version` 0–2, `fileId` 2–34, `chunkId` 34–38, `totalChunks` 38–42, `fileSize` 42–50, `chunkSize` 50–54, `payloadLen` 54–58, `flags` 58–60, `nonce` 60–84, `aeadTag` 84–100. `sig` occupies 100–164. Ciphertext starts at 164.
|
||||
|
||||
**Verified by:** `src/codec/chunk.spec-conformance.test.ts` (`CHUNKS-FORMAT-V1-01`).
|
||||
|
||||
### CHUNKS-FORMAT-V1-02 — `flags` reserved in v1
|
||||
|
||||
`flags` MUST be `0x0000` for v1. Receivers MUST reject chunks with non-zero `flags` via `ChunksError(CHUNKS.FLAGS_RESERVED_NONZERO)` before any further parse work.
|
||||
|
||||
**Verified by:** `src/codec/chunk.spec-conformance.test.ts` (`CHUNKS-FORMAT-V1-02`).
|
||||
|
||||
### CHUNKS-FORMAT-V1-03 — Payload length invariants
|
||||
|
||||
For non-final chunks (`chunkId < totalChunks - 1`), `payloadLen` MUST equal `chunkSize - 164`. For the final chunk, `payloadLen` MUST equal `fileSize - chunkId * (chunkSize - 164)`. Mismatches MUST be rejected via `ChunksError(CHUNKS.PAYLOAD_LEN_INVALID)` before sig verification.
|
||||
|
||||
**Verified by:** `src/codec/chunk.spec-conformance.test.ts` (`CHUNKS-FORMAT-V1-03`).
|
||||
|
||||
### CHUNKS-SIG-01 — Signature covers metadata plus ciphertext
|
||||
|
||||
`sig` MUST be `ed25519_sign(serverPrivKey, metadata[0..100] ‖ ciphertext[0..payloadLen])`. Verification MUST be performed before AEAD decryption. Any mutation to any signed byte MUST cause sig verification to fail with `ChunksError(CHUNKS.SIG_INVALID)`.
|
||||
|
||||
**Verified by:** `src/pipeline/sign-chunk.spec-conformance.test.ts` (`CHUNKS-SIG-01`).
|
||||
|
||||
### CHUNKS-INVARIANT-01 — Store-as-is
|
||||
|
||||
Plugin MUST persist received chunks byte-identical to receipt. No re-signing, no re-encryption, no metadata rewrites. The bytes returned by `chunkStore.get(fileId, chunkId)` MUST equal the bytes accepted by `chunks.putChunk(rawBytes)` for that chunk.
|
||||
|
||||
**Verified by:** `src/plugin.spec-conformance.test.ts` (`CHUNKS-INVARIANT-01`).
|
||||
|
||||
### CHUNKS-VERIFY-01 — Sig-verify gate on receipt
|
||||
|
||||
`chunks.putChunk(rawBytes)` MUST run, in order: structural parse → flags check → payload-length invariant → ed25519 sig verification → persist. Any failure aborts persistence and returns the appropriate `ChunksError` code without writing to storage.
|
||||
|
||||
**Verified by:** `src/plugin.spec-conformance.test.ts` (`CHUNKS-VERIFY-01`).
|
||||
|
||||
### CHUNKS-READ-01 — Read middleware reassembles via streaming
|
||||
|
||||
When a `fileId` is registered for `path`, `drive.read(path)` MUST return a `ReadableStream<Uint8Array>` that yields plaintext by iterating chunk 0..N-1, sig-verifying each, AEAD-decrypting, and enqueueing only the first `payloadLen` bytes of each chunk's plaintext. The full file MUST NOT be materialised in memory.
|
||||
|
||||
**Verified by:** `src/plugin.spec-conformance.test.ts` (`CHUNKS-READ-01`).
|
||||
|
||||
### CHUNKS-LAYOUT-SPARSE-01 — Sparse-file layout
|
||||
|
||||
In `sparse-file` mode, the plugin MUST pre-allocate a single storage object of length `(totalChunks - 1) * chunkSize + (164 + lastPayloadLen)` after manifest load. Each chunk MUST be written via `storage.setPartial(fileKey, chunkId * chunkSize, rawBytes)`. A separate `BitSet` at `bitmap:<fileId>` MUST track landed chunks. The plugin MUST require `storage.capabilities.partialWrites === true` and throw `ChunksError(CHUNKS.LAYOUT_INCOMPATIBLE)` otherwise.
|
||||
|
||||
**Verified by:** `src/layout/sparse-file.spec-conformance.test.ts` (`CHUNKS-LAYOUT-SPARSE-01`).
|
||||
|
||||
### CHUNKS-LAYOUT-KEYS-01 — Chunk-keys layout
|
||||
|
||||
In `chunk-keys` mode, each chunk MUST be stored as its own object at key `chunks/<fileId>/<chunkId>`. `hasChunks(fileId)` MUST be derivable from key existence (`storage.exists`). No bitmap object is required.
|
||||
|
||||
**Verified by:** `src/layout/chunk-keys.spec-conformance.test.ts` (`CHUNKS-LAYOUT-KEYS-01`).
|
||||
|
||||
### CHUNKS-LAYOUT-AUTO-01 — Auto layout selection
|
||||
|
||||
In `auto` mode (default), the plugin MUST pick `sparse-file` when `storage.capabilities.partialWrites === true`, otherwise MUST pick `chunk-keys`. The selected layout MUST be observable via `chunks.layout`.
|
||||
|
||||
**Verified by:** `src/layout/auto.spec-conformance.test.ts` (`CHUNKS-LAYOUT-AUTO-01`).
|
||||
|
||||
### CHUNKS-MANIFEST-01 — Manifest format and signature
|
||||
|
||||
The manifest MUST be a Zod-validated object with fields `version`, `fileId`, `fileSize`, `chunkSize`, `totalChunks`, `encryption: { algo, fileKeyWrapped, baseNonce }`, `chunks: [{ chunkId, hash, payloadLen }]`, and `sig`. `sig` MUST be `ed25519_sign(serverPrivKey, canonical_serialize(manifest_without_sig))`. Load failures (parse, schema, sig) MUST throw `ChunksError(CHUNKS.MANIFEST_INVALID)`.
|
||||
|
||||
**Verified by:** `src/manifest/load.spec-conformance.test.ts` (`CHUNKS-MANIFEST-01`).
|
||||
|
||||
### CHUNKS-MANIFEST-02 — Manifest distribution strategy
|
||||
|
||||
Default strategy MUST be `manifest-as-chunk`: the manifest is itself a server-signed object stored at storage key `manifest:<fileId>`. `server-only` strategy MUST be opt-in via `manifestSource: "server-only"`; in that mode `loadManifest(fileId)` fetches via an injected `fetch` against the configured server URL.
|
||||
|
||||
**Verified by:** `src/manifest/load.spec-conformance.test.ts` (`CHUNKS-MANIFEST-02`).
|
||||
|
||||
### CHUNKS-CODES-01 — Error registry namespaced and frozen
|
||||
|
||||
`CHUNKS_CODES` MUST be declared via `defineCodes("CHUNKS", [...])`. `Object.isFrozen(CHUNKS_CODES)` MUST be `true`. Every code value MUST start with `CHUNKS.`. `ChunksError` MUST extend `SuaveplanError<ChunksErrorCode>`.
|
||||
|
||||
**Verified by:** `src/error-codes.spec-conformance.test.ts` (`CHUNKS-CODES-01`).
|
||||
@@ -0,0 +1,84 @@
|
||||
# Tasks — drive-plugin-chunks-port
|
||||
|
||||
> Every checkbox MUST reflect work actually done in this change. Do not retroactively tick boxes for work that was already in main when this change opened. Do not tick a box for a test that doesn't exist or doesn't pin the requirement it claims to verify.
|
||||
|
||||
## 1. Spec authoring
|
||||
|
||||
- [x] 1.1 Read `openspec/AGENTS.md` Sections 13 and 14
|
||||
- [x] 1.2 Read `openspec/templates/README.md` — when and how to use the templates
|
||||
- [x] 1.3 Reformat `proposal.md` from `openspec/templates/proposal.md`
|
||||
- [x] 1.4 Reformat `tasks.md` from `openspec/templates/tasks.md`
|
||||
- [x] 1.5 Every requirement in the spec delta carries a `**Verified by:**` line naming concrete `it(...)` titles — update iteratively during implementation
|
||||
|
||||
## 2. Implementation
|
||||
|
||||
- [x] 2.1 Scaffold `packages/drive/drive-plugin-chunks/` with `src/`, `tests/`, `src/__benchmarks__/`, and all feature subdirectories per spec
|
||||
- [x] 2.2 Write `package.json` at `0.1.0` with `"type": "module"`, `"sideEffects": false`, full `exports` map, catalog deps, and dev deps
|
||||
- [x] 2.3 Write `tsconfig.json` extending `@suaveplan/typescript-config/library.json`
|
||||
- [x] 2.4 Write `vitest.config.ts` using `createTestConfig()` from `@suaveplan/testing/config` (pool: `forks`)
|
||||
- [x] 2.5 Implement `src/types/chunks.types.ts` — `ChunksPluginOptions`, `ChunkLayoutStrategy`, `ManifestSource`, `Manifest`, `ChunkSource` interface
|
||||
- [x] 2.6 Implement `src/error-codes.ts` — `defineCodes("CHUNKS", [...])`. Sibling `ChunksError`
|
||||
- [x] 2.7 Implement `src/schemas/manifest.schema.ts` — Zod schema for the manifest JSON/CBOR shape
|
||||
- [x] 2.8 Implement `src/codec/header.ts` — `encodeHeader(metadata)`, `parseHeader(buf)`, big-endian DataView reads/writes at locked offsets
|
||||
- [x] 2.9 Implement `src/codec/chunk.ts` — `encodeChunk({ metadata, sig, ciphertext })`, `parseChunk(buf)`, length validation
|
||||
- [x] 2.10 Implement `src/pipeline/encrypt-stream.ts` — XChaCha20-Poly1305 streaming encryptor over a `ReadableStream<Uint8Array>`
|
||||
- [x] 2.11 Implement `src/pipeline/chop-stream.ts` — slice ciphertext stream at `payload_per_chunk` boundaries
|
||||
- [x] 2.12 Implement `src/pipeline/sign-chunk.ts` — server-sign `metadata ‖ ciphertext`
|
||||
- [x] 2.13 Implement `src/pipeline/verify-chunk.ts` — sig-verify on receipt, reject with appropriate error code on fail
|
||||
- [x] 2.14 Implement `src/layout/sparse-file.ts` — pre-allocate, `setPartial` writes, side-band BitSet at `bitmap:<fileId>`
|
||||
- [x] 2.15 Implement `src/layout/chunk-keys.ts` — one object per chunk at `chunks/<fileId>/<chunkId>`
|
||||
- [x] 2.16 Implement `src/layout/auto.ts` — picks based on `storage.capabilities.partialWrites`
|
||||
- [x] 2.17 Implement `src/manifest/build.ts` — assemble manifest from chunk metadata, server-sign
|
||||
- [x] 2.18 Implement `src/manifest/load.ts` — fetch from storage (`manifest-as-chunk` mode) or HTTPS (`server-only` mode), Zod-validate, sig-verify
|
||||
- [x] 2.19 Implement `src/manifest/source.ts` — strategy switch
|
||||
- [x] 2.20 Implement `src/plugin.ts` — `chunksPlugin(options)`; `install(drive)` returns the handle with `getChunk`, `putChunk`, `hasChunks`, `availableChunks`, `ingest`, `loadManifest`
|
||||
- [x] 2.21 Read middleware: registers via `drive.useMiddleware(...)`, intercepts `read(path)` and reassembles via streaming decrypt
|
||||
- [x] 2.22 Write middleware: `drive.write(path, bytes)` — pass through unchanged; chunking is opt-in via `chunks.ingest(path)`
|
||||
|
||||
## 3. Tests
|
||||
|
||||
- [x] 3.1 `src/codec/chunk.test.ts` — round-trip every field, boundary lengths, invalid inputs
|
||||
- [x] 3.2 `src/codec/chunk.spec-conformance.test.ts` — exact byte-offset pinning of v1 layout (`CHUNKS-FORMAT-V1-01` through `-03`)
|
||||
- [x] 3.3 `src/pipeline/sign-chunk.test.ts` — sign + verify round-trip, mutation rejection
|
||||
- [x] 3.4 `src/pipeline/sign-chunk.spec-conformance.test.ts` — sig covers metadata + ciphertext (`CHUNKS-SIG-01`)
|
||||
- [x] 3.5 `src/layout/sparse-file.test.ts` — pre-allocation, offset writes, bitmap, BitSet round-trip
|
||||
- [x] 3.6 `src/layout/sparse-file.spec-conformance.test.ts` — partial-write capability requirement (`CHUNKS-LAYOUT-SPARSE-01`)
|
||||
- [x] 3.7 `src/layout/chunk-keys.test.ts` — one-key-per-chunk CRUD, list prefix semantics
|
||||
- [x] 3.8 `src/layout/chunk-keys.spec-conformance.test.ts` — one-key-per-chunk semantics (`CHUNKS-LAYOUT-KEYS-01`)
|
||||
- [x] 3.9 `src/layout/auto.spec-conformance.test.ts` — layout selection by capability (`CHUNKS-LAYOUT-AUTO-01`)
|
||||
- [x] 3.10 `src/manifest/load.test.ts` — build, load, round-trip, invalid schema rejection
|
||||
- [x] 3.11 `src/manifest/load.spec-conformance.test.ts` — `as-chunk` and `server-only` modes (`CHUNKS-MANIFEST-01`, `-02`)
|
||||
- [x] 3.12 `src/plugin.test.ts` — install, handle methods, ingest round-trip, middleware registration
|
||||
- [x] 3.13 `src/plugin.spec-conformance.test.ts` — store-as-is invariant, sig-verify gate, read reassembly (`CHUNKS-INVARIANT-01`, `CHUNKS-VERIFY-01`, `CHUNKS-READ-01`)
|
||||
- [x] 3.14 `src/error-codes.spec-conformance.test.ts` — frozen registry, namespace (`CHUNKS-CODES-01`)
|
||||
- [x] 3.15 Every `it(...)` carries `{ timeout: 5000 }`
|
||||
|
||||
## 4. Documentation
|
||||
|
||||
- [x] 4.1 Sibling `.md` for every implementation `.ts` (≥200 words, basic + advanced examples, full API reference)
|
||||
- [x] 4.2 README links every co-located `.md` under `## Modules`
|
||||
- [x] 4.3 Update `CLAUDE.md` package listing
|
||||
|
||||
## 5. Pre-archive verification (the 5-gate triple)
|
||||
|
||||
- [x] 5.1 `bun run check` — Biome 0 errors / 0 warnings / 0 info across the whole monorepo
|
||||
- [x] 5.2 `bunx turbo lint typecheck test --filter=@suaveplan/drive-plugin-chunks` — green
|
||||
- [x] 5.3 `bunx turbo test:coverage --filter=@suaveplan/drive-plugin-chunks` — 100/100/100/100
|
||||
- [x] 5.4 `bun scripts/openspec-conformance.ts @suaveplan/drive-plugin-chunks` — exit 0, no `error` findings
|
||||
- [x] 5.5 `bunx turbo build --filter=@suaveplan/drive-plugin-chunks` — green
|
||||
|
||||
## 6. Pre-archive 7-row checklist (AGENTS.md Section 13)
|
||||
|
||||
- [x] 6.1 Generics threaded; `expectTypeOf` assertions on every literal-type claim
|
||||
- [x] 6.2 Zod schemas wired at every declared boundary
|
||||
- [x] 6.3 Every error code in `error-codes.ts` thrown by at least one production path
|
||||
- [x] 6.4 Strict-mode behaviors tested both on and off
|
||||
- [x] 6.5 OTel instruments emitted from production paths and asserted with a real in-memory `MeterProvider`
|
||||
- [x] 6.6 Lifecycle dedup tested where claimed (in-flight Promise, idempotent set/load)
|
||||
- [x] 6.7 No retroactive tick — every box above reflects work done in this change
|
||||
|
||||
## 7. Archive
|
||||
|
||||
- [x] 7.1 Fold spec delta into `openspec/specs/drive-plugin-chunks/spec.md`
|
||||
- [x] 7.2 Move `openspec/changes/drive-plugin-chunks-port/` → `openspec/archive/<yyyy-mm-dd>-drive-plugin-chunks-port/`
|
||||
- [x] 7.3 Refresh `MANIFEST.md` if package set or deps changed
|
||||
Reference in New Issue
Block a user