Initial Commit
This commit is contained in:
129
packages/networking/drive/drive-plugin-chunks/README.md
Normal file
129
packages/networking/drive/drive-plugin-chunks/README.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# @suaveplan/drive-plugin-chunks - Chunk-addressing plugin for @suaveplan/drive. Defines the version 0x0001 binary chunk format, server-signed manifests, storage layouts, and streaming encrypt-then-chunk pipeline.
|
||||
|
||||
## Summary
|
||||
|
||||
Chunk-addressing plugin for `@suaveplan/drive`. Splits files into signed, encrypted chunks using the locked `version 0x0001` binary format, with server-signed manifests and two storage layouts (`sparse-file` and `chunk-keys`).
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [chunk](./src/codec/chunk.md) — Wraps the chunk wire format above the bare header: takes raw chunk bytes (header + signature + ciphertext) and parses t…
|
||||
- [header](./src/codec/header.md) — Encodes and parses the fixed-width 100-byte metadata header that prefixes every chunk in the `@suaveplan/drive-plugin-c…
|
||||
- [inline](./src/inline/inline-hint.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [inline-hint](./src/inline/inline-hint.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [auto](./src/layout/auto.md) — Resolves the best chunk layout for a given storage at runtime: prefers `sparse-file` when the storage advertises `capab…
|
||||
- [chunk-keys](./src/layout/chunk-keys.md) — Implements the `chunk-keys` layout strategy that stores each chunk under a discrete storage key (`chunks/<fileIdHex>/<c…
|
||||
- [sparse-file](./src/layout/sparse-file.md) — Implements the `sparse-file` chunk layout that stores all chunks of a file as offset-addressable bytes inside a single…
|
||||
- [build](./src/manifest/build.md) — Constructs a server-signed `Manifest` for an ingested file: bundles the file metadata (`fileId`, `fileSize`, `chunkSize…
|
||||
- [load](./src/manifest/load.md) — Loads server-signed manifests from the two supported sources: a sentinel chunk inside the file's own chunk space (`mani…
|
||||
- [source](./src/manifest/source.md) — Validates the `manifestSource` configuration before the plugin starts wiring its read middleware. The function is small…
|
||||
- [decrypt-chunk](./src/pipeline/decrypt-chunk.md) — The single-chunk decryption helper that runs on the read side of the pipeline. Given chunk metadata (already parsed via…
|
||||
- [encrypt-stream](./src/pipeline/encrypt-stream.md) — The encryption pipeline that transforms a plaintext file into a stream of signed encrypted chunks ready for persistence…
|
||||
- [sign-chunk](./src/pipeline/sign-chunk.md) — Signs a chunk under the server's Ed25519 private key. Signatures cover the full encoded chunk (metadata header bytes pl…
|
||||
- [verify-chunk](./src/pipeline/verify-chunk.md) — Verifies a chunk's Ed25519 signature against the server's public key as the gate that runs on receipt of every chunk —…
|
||||
- [schemas](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [manifest.schema](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [error-codes](./src/error-codes.md) — Defines the structured error registry for `@suaveplan/drive-plugin-chunks`. Every chunk-format failure mode the plugin…
|
||||
- [plugin](./src/plugin.md) — Implements the `@suaveplan/drive-plugin-chunks` plugin object that consumes the `Drive` plugin slot, splits files into…
|
||||
<!-- end:auto-docs --> - [inline-hint](./src/inline/inline-hint.md) — <!-- begin:auto-docs --> _(no submodules)_- [auto](./src/layout/auto.md) — Resolves the best chunk layout for a given storage at runtime: prefers `sparse-file` when the storage advertises `capab…
|
||||
- [chunk-keys](./src/layout/chunk-keys.md) — Implements the `chunk-keys` layout strategy that stores each chunk under a discrete storage key (`chunks/<fileIdHex>/<c…
|
||||
- [sparse-file](./src/layout/sparse-file.md) — Implements the `sparse-file` chunk layout that stores all chunks of a file as offset-addressable bytes inside a single…
|
||||
- [build](./src/manifest/build.md) — Constructs a server-signed `Manifest` for an ingested file: bundles the file metadata (`fileId`, `fileSize`, `chunkSize…
|
||||
- [load](./src/manifest/load.md) — Loads server-signed manifests from the two supported sources: a sentinel chunk inside the file's own chunk space (`mani…
|
||||
- [source](./src/manifest/source.md) — Validates the `manifestSource` configuration before the plugin starts wiring its read middleware. The function is small…
|
||||
- [decrypt-chunk](./src/pipeline/decrypt-chunk.md) — The single-chunk decryption helper that runs on the read side of the pipeline. Given chunk metadata (already parsed via…
|
||||
- [encrypt-stream](./src/pipeline/encrypt-stream.md) — The encryption pipeline that transforms a plaintext file into a stream of signed encrypted chunks ready for persistence…
|
||||
- [sign-chunk](./src/pipeline/sign-chunk.md) — Signs a chunk under the server's Ed25519 private key. Signatures cover the full encoded chunk (metadata header bytes pl…
|
||||
- [verify-chunk](./src/pipeline/verify-chunk.md) — Verifies a chunk's Ed25519 signature against the server's public key as the gate that runs on receipt of every chunk —…
|
||||
- [schemas](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest.schema](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Defines the structured error registry for `@suaveplan/drive-plugin-chunks`. Every chunk-format failure mode the plugin…
|
||||
- [plugin](./src/plugin.md) — Implements the `@suaveplan/drive-plugin-chunks` plugin object that consumes the `Drive` plugin slot, splits files into…
|
||||
- [inline-hint](./src/inline/inline-hint.md) — <!-- begin:auto-docs --> _(no submodules)_- [auto](./src/layout/auto.md) — Resolves the best chunk layout for a given storage at runtime: prefers `sparse-file` when the storage advertises `capab…
|
||||
- [chunk-keys](./src/layout/chunk-keys.md) — Implements the `chunk-keys` layout strategy that stores each chunk under a discrete storage key (`chunks/<fileIdHex>/<c…
|
||||
- [sparse-file](./src/layout/sparse-file.md) — Implements the `sparse-file` chunk layout that stores all chunks of a file as offset-addressable bytes inside a single…
|
||||
- [build](./src/manifest/build.md) — Constructs a server-signed `Manifest` for an ingested file: bundles the file metadata (`fileId`, `fileSize`, `chunkSize…
|
||||
- [load](./src/manifest/load.md) — Loads server-signed manifests from the two supported sources: a sentinel chunk inside the file's own chunk space (`mani…
|
||||
- [source](./src/manifest/source.md) — Validates the `manifestSource` configuration before the plugin starts wiring its read middleware. The function is small…
|
||||
- [decrypt-chunk](./src/pipeline/decrypt-chunk.md) — The single-chunk decryption helper that runs on the read side of the pipeline. Given chunk metadata (already parsed via…
|
||||
- [encrypt-stream](./src/pipeline/encrypt-stream.md) — The encryption pipeline that transforms a plaintext file into a stream of signed encrypted chunks ready for persistence…
|
||||
- [sign-chunk](./src/pipeline/sign-chunk.md) — Signs a chunk under the server's Ed25519 private key. Signatures cover the full encoded chunk (metadata header bytes pl…
|
||||
- [verify-chunk](./src/pipeline/verify-chunk.md) — Verifies a chunk's Ed25519 signature against the server's public key as the gate that runs on receipt of every chunk —…
|
||||
- [schemas](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest.schema](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Defines the structured error registry for `@suaveplan/drive-plugin-chunks`. Every chunk-format failure mode the plugin…
|
||||
- [plugin](./src/plugin.md) — Implements the `@suaveplan/drive-plugin-chunks` plugin object that consumes the `Drive` plugin slot, splits files into…
|
||||
- [inline-hint](./src/inline/inline-hint.md) — <!-- begin:auto-docs --> _(no submodules)_- [auto](./src/layout/auto.md) — Resolves the best chunk layout for a given storage at runtime: prefers `sparse-file` when the storage advertises `capab…
|
||||
- [chunk-keys](./src/layout/chunk-keys.md) — Implements the `chunk-keys` layout strategy that stores each chunk under a discrete storage key (`chunks/<fileIdHex>/<c…
|
||||
- [sparse-file](./src/layout/sparse-file.md) — Implements the `sparse-file` chunk layout that stores all chunks of a file as offset-addressable bytes inside a single…
|
||||
- [build](./src/manifest/build.md) — Constructs a server-signed `Manifest` for an ingested file: bundles the file metadata (`fileId`, `fileSize`, `chunkSize…
|
||||
- [load](./src/manifest/load.md) — Loads server-signed manifests from the two supported sources: a sentinel chunk inside the file's own chunk space (`mani…
|
||||
- [source](./src/manifest/source.md) — Validates the `manifestSource` configuration before the plugin starts wiring its read middleware. The function is small…
|
||||
- [decrypt-chunk](./src/pipeline/decrypt-chunk.md) — The single-chunk decryption helper that runs on the read side of the pipeline. Given chunk metadata (already parsed via…
|
||||
- [encrypt-stream](./src/pipeline/encrypt-stream.md) — The encryption pipeline that transforms a plaintext file into a stream of signed encrypted chunks ready for persistence…
|
||||
- [sign-chunk](./src/pipeline/sign-chunk.md) — Signs a chunk under the server's Ed25519 private key. Signatures cover the full encoded chunk (metadata header bytes pl…
|
||||
- [verify-chunk](./src/pipeline/verify-chunk.md) — Verifies a chunk's Ed25519 signature against the server's public key as the gate that runs on receipt of every chunk —…
|
||||
- [schemas](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [manifest.schema](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Defines the structured error registry for `@suaveplan/drive-plugin-chunks`. Every chunk-format failure mode the plugin…
|
||||
- [plugin](./src/plugin.md) — Implements the `@suaveplan/drive-plugin-chunks` plugin object that consumes the `Drive` plugin slot, splits files into…
|
||||
- [inline-hint](./src/inline/inline-hint.md) — <!-- begin:auto-docs --> _(no submodules)_- [auto](./src/layout/auto.md) — Resolves the best chunk layout for a given storage at runtime: prefers `sparse-file` when the storage advertises `capab…
|
||||
- [chunk-keys](./src/layout/chunk-keys.md) — Implements the `chunk-keys` layout strategy that stores each chunk under a discrete storage key (`chunks/<fileIdHex>/<c…
|
||||
- [sparse-file](./src/layout/sparse-file.md) — Implements the `sparse-file` chunk layout that stores all chunks of a file as offset-addressable bytes inside a single…
|
||||
- [build](./src/manifest/build.md) — Constructs a server-signed `Manifest` for an ingested file: bundles the file metadata (`fileId`, `fileSize`, `chunkSize…
|
||||
- [load](./src/manifest/load.md) — Loads server-signed manifests from the two supported sources: a sentinel chunk inside the file's own chunk space (`mani…
|
||||
- [source](./src/manifest/source.md) — Validates the `manifestSource` configuration before the plugin starts wiring its read middleware. The function is small…
|
||||
- [decrypt-chunk](./src/pipeline/decrypt-chunk.md) — The single-chunk decryption helper that runs on the read side of the pipeline. Given chunk metadata (already parsed via…
|
||||
- [encrypt-stream](./src/pipeline/encrypt-stream.md) — The encryption pipeline that transforms a plaintext file into a stream of signed encrypted chunks ready for persistence…
|
||||
- [sign-chunk](./src/pipeline/sign-chunk.md) — Signs a chunk under the server's Ed25519 private key. Signatures cover the full encoded chunk (metadata header bytes pl…
|
||||
- [verify-chunk](./src/pipeline/verify-chunk.md) — Verifies a chunk's Ed25519 signature against the server's public key as the gate that runs on receipt of every chunk —…
|
||||
- [schemas](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_ - [manifest.schema](./src/schemas/manifest.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Defines the structured error registry for `@suaveplan/drive-plugin-chunks`. Every chunk-format failure mode the plugin…
|
||||
- [plugin](./src/plugin.md) — Implements the `@suaveplan/drive-plugin-chunks` plugin object that consumes the `Drive` plugin slot, splits files into…
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
bun add @suaveplan/drive-plugin-chunks
|
||||
```
|
||||
|
||||
## Quick start
|
||||
|
||||
```ts
|
||||
import { Drive } from "@suaveplan/drive";
|
||||
import { chunksPlugin } from "@suaveplan/drive-plugin-chunks";
|
||||
import { MemoryStorage } from "@suaveplan/storage/memory";
|
||||
|
||||
const storage = Object.assign(new MemoryStorage(), { id: "local" });
|
||||
const drive = new Drive(storage);
|
||||
await drive.initialize();
|
||||
|
||||
const chunks = await drive.usePlugin(
|
||||
chunksPlugin({
|
||||
serverPrivKey: ed25519PrivKey,
|
||||
serverPubKey: ed25519PubKey,
|
||||
chunkSize: 65536,
|
||||
})
|
||||
);
|
||||
|
||||
await drive.write("hello.txt", new TextEncoder().encode("world"));
|
||||
await chunks.ingest("hello.txt");
|
||||
|
||||
const manifest = await chunks.loadManifest(fileId);
|
||||
console.log(manifest.totalChunks);
|
||||
|
||||
await drive.dispose();
|
||||
```
|
||||
|
||||
## Modules
|
||||
|
||||
- [src/codec/header.md](src/codec/header.md) — binary header encoding / parsing
|
||||
- [src/codec/chunk.md](src/codec/chunk.md) — full chunk encode / parse
|
||||
- [src/pipeline/encrypt-stream.md](src/pipeline/encrypt-stream.md) — XChaCha20-Poly1305 streaming encryption
|
||||
- [src/pipeline/chop-stream.md](src/pipeline/chop-stream.md) — ciphertext slicing into chunk payloads
|
||||
- [src/pipeline/sign-chunk.md](src/pipeline/sign-chunk.md) — ed25519 server signing
|
||||
- [src/pipeline/verify-chunk.md](src/pipeline/verify-chunk.md) — signature verification
|
||||
- [src/layout/sparse-file.md](src/layout/sparse-file.md) — single-object sparse layout
|
||||
- [src/layout/chunk-keys.md](src/layout/chunk-keys.md) — one-key-per-chunk layout
|
||||
- [src/layout/auto.md](src/layout/auto.md) — automatic layout selection
|
||||
- [src/manifest/build.md](src/manifest/build.md) — manifest assembly and signing
|
||||
- [src/manifest/load.md](src/manifest/load.md) — manifest fetching and validation
|
||||
- [src/manifest/source.md](src/manifest/source.md) — distribution strategy switch
|
||||
- [src/plugin.md](src/plugin.md) — plugin handle and Drive middleware
|
||||
- [src/types/chunks.types.md](src/types/chunks.types.md) — public TypeScript interfaces
|
||||
- [src/schemas/manifest.schema.md](src/schemas/manifest.schema.md) — Zod boundary schemas
|
||||
- [src/error-codes.md](src/error-codes.md) — `CHUNKS_CODES` and `ChunksError`
|
||||
|
||||
Reference in New Issue
Block a user