Initial Commit
This commit is contained in:
365
docs/drive-storage-network.md
Normal file
365
docs/drive-storage-network.md
Normal file
@@ -0,0 +1,365 @@
|
||||
# Drive · Storage · Network — How They Compose
|
||||
|
||||
This is the integration document. It exists to answer one question: *given storage, drive, and network as three separate libraries, what does a complete deployment look like and how does a single byte travel through it?*
|
||||
|
||||
If you want depth on any one layer, read its folder:
|
||||
|
||||
- [`docs/storage/`](./storage/) — backend layer.
|
||||
- [`docs/drive/`](./drive/) — file layer.
|
||||
- [`docs/network/`](./network/) — mesh + failover layer.
|
||||
- [`docs/system-review.md`](./system-review.md) — 16 numbered findings against the design (2 blockers, 9 majors, 5 minors) — read this before implementing.
|
||||
- [`docs/drive/mesh-protocol.md`](./drive/mesh-protocol.md) — the chunk-request wire protocol (three forms, bitpacker bitsets, response piggyback).
|
||||
- [`docs/drive/dehydrated-drives.md`](./drive/dehydrated-drives.md) — server-signed templates that pre-configure drives with policy flags + CDN reversed-ID URL fallback + retroactive key grants.
|
||||
|
||||
This document is the map.
|
||||
|
||||
---
|
||||
|
||||
## The three layers, one paragraph
|
||||
|
||||
**Storage** is `key → bytes`. It knows nothing else.
|
||||
**Drive** is `path → file`, with plugins for chunking, signing, encryption, sharing, and mesh sync. It uses storage as its substrate.
|
||||
**Network** is the resilient tiered mesh that lets every drive talk to every other drive. Drives plug into network via the `drive-mesh` and `drive-transport-*` plugins, communicating only through `NetworkBus` events.
|
||||
|
||||
The three layers compose strictly: drive depends on storage and network; network depends on neither; storage depends on neither. There is exactly one bridging type package — `@suaveplan/types` — that holds the cross-cutting interfaces (`StorageLike`, `ChunkSource`, `ChunkRef`, `DrivePlugin`, `MiddlewareHandler`, `HealthStatus`) so no concrete library has to import another concrete library.
|
||||
|
||||
---
|
||||
|
||||
## The dependency graph (concrete)
|
||||
|
||||
```text
|
||||
@suaveplan/types ◄──────────────── (everything depends on this; types has zero runtime code)
|
||||
▲ ▲
|
||||
│ │
|
||||
@suaveplan/storage @suaveplan/network
|
||||
▲ ▲ ▲ ▲
|
||||
│ │ │ │
|
||||
@suaveplan/storage-s3 @suaveplan/network-plugin-protocol
|
||||
@suaveplan/storage-browser @suaveplan/network-plugin-registry
|
||||
@suaveplan/storage-... @suaveplan/network-transport-webrtc
|
||||
@suaveplan/network-plugin-...
|
||||
▲
|
||||
│
|
||||
┌────────────────────────────────────────┘
|
||||
│
|
||||
@suaveplan/drive ──────► uses storage as substrate
|
||||
▲
|
||||
│
|
||||
@suaveplan/drive-plugin-chunks ──► uses crypto, packer, bitpacker
|
||||
@suaveplan/drive-sharing ──► uses crypto, drive
|
||||
@suaveplan/drive-mesh ──► uses drive, network (via bus only)
|
||||
@suaveplan/drive-transport-https ──► uses drive
|
||||
@suaveplan/drive-transport-webrtc ─► uses drive, network-transport-webrtc (via bus only)
|
||||
@suaveplan/drive-react-sharing ──► uses drive-sharing
|
||||
@suaveplan/drive-webrtc ──► pre-packaged: chunks + transport-webrtc + signalling
|
||||
```
|
||||
|
||||
The "via bus only" qualifier is structural: `drive-mesh` doesn't import `network-plugin-content-delivery`. They communicate through bus topics. This is the genesis discipline that broke the circular-deps mess on main.
|
||||
|
||||
---
|
||||
|
||||
## The file write — end to end
|
||||
|
||||
A user clicks "save" on a 50 MiB photo. The drive is configured with `OpfsStorage` locally, `S3` for cold storage, the chunks plugin, the sharing plugin, the mesh plugin, and a WebRTC transport. Every numbered step is a real boundary in the code.
|
||||
|
||||
```text
|
||||
1. App → drive.write("/photos/wedding/IMG_0001.jpg", bytes)
|
||||
|
||||
2. Drive middleware chain (request side, in registration order):
|
||||
2a. logging → record { op:'write', path, size }
|
||||
2b. chunks → start a TransformStream:
|
||||
bytes ──┐
|
||||
├─ chop into 65,408-byte plaintext chunks
|
||||
│ (chunkSize 65,536 minus 164-byte chunk header+sig)
|
||||
└─ for each:
|
||||
nonce = HKDF(fileKey, fileId ‖ chunkId)
|
||||
cipher = XChaCha20-Poly1305(fileKey, nonce, plaintext, AAD=metadata)
|
||||
sig = Ed25519-sign(driveSecretKey, metadata ‖ aeadTag)
|
||||
emit chunk = metadata(100) ‖ sig(64) ‖ ciphertext
|
||||
2c. sharing → wraps fileKey with the partition KEK; appends wrap blob to manifest
|
||||
2d. mesh → no-op on write, but listens for the post-write hook
|
||||
2e. transport-webrtc → no-op on write
|
||||
|
||||
3. Storage selection per chunk:
|
||||
3a. storage.capabilities.partialWrites === true (OPFS) ? → sparse-file layout
|
||||
3b. storage.setPartial("files/<fileId>", chunkId * chunkSize, chunk)
|
||||
3c. for cold copy in S3: storage.set("files/<fileId>/<chunkId>", chunk)
|
||||
(S3 has partialWrites=false; the chunks plugin auto-picks chunk-keys layout for it)
|
||||
|
||||
4. Manifest assembly:
|
||||
4a. manifest = { fileId, totalChunks=800, version=1, entries: [
|
||||
{ chunkId:0, hash:blake3(chunk0), payloadLen },
|
||||
... 799 more ...
|
||||
]}
|
||||
4b. manifestSig = Ed25519-sign(driveSecretKey, canonicalSerialize(manifest))
|
||||
4c. storage.set("manifests/<fileId>-v1", encodeManifest(manifest, manifestSig))
|
||||
|
||||
5. Drive index update (CAS-versioned):
|
||||
5a. read DirIndex; insert DirEntry { name:"IMG_0001.jpg", type:"file", fileLocId:"L-..." }
|
||||
under /photos/wedding/. Increment version.
|
||||
5b. read FileLocationIndex; insert FileLocation {
|
||||
fileId, version:1, size:52,428,800, checksum:"blake3:...",
|
||||
storageHints: [
|
||||
{ kind:"sparse-file", storageId:"opfs", key:"files/<fileId>", chunkSize:65536 },
|
||||
{ kind:"chunk-keys", storageId:"s3", manifestKey:"manifests/<fileId>-v1" },
|
||||
{ kind:"mesh", manifestKey:"manifests/<fileId>-v1" },
|
||||
]
|
||||
}
|
||||
5c. CAS-write both indexes; on conflict, retry with rebased version.
|
||||
|
||||
6. Drive middleware chain (response side, reverse order):
|
||||
6e. transport-webrtc → no-op
|
||||
6d. mesh → emit `cd:advertise { fileId, chunkBitset }` on NetworkBus
|
||||
tells peers we now have all 800 chunks of this file
|
||||
6c. sharing → strip wrap blob from response (already stored)
|
||||
6b. chunks → return manifest summary
|
||||
6a. logging → record { op:'write', path, size, durationMs }
|
||||
|
||||
7. Network propagation (asynchronous, fire-and-forget):
|
||||
7a. network-plugin-content-delivery hears `cd:advertise` and updates its gossip index
|
||||
7b. network-plugin-content-delivery may push `cd:advertise` to siblings on the relay
|
||||
backbone, depending on QoS class
|
||||
7c. peer drives subscribed to this drive's advertisements update their own
|
||||
per-fileId chunk bitsets
|
||||
|
||||
→ drive.write resolves. The user's UI updates "saving..." → "saved".
|
||||
```
|
||||
|
||||
Every byte that hit storage, every byte that hit the network, was AEAD-encrypted ciphertext. The plaintext lived only in step 2b's TransformStream and never on disk or wire.
|
||||
|
||||
---
|
||||
|
||||
## The file read — end to end (with one chunk missing locally)
|
||||
|
||||
A second user (Bob's drive) wants to read the photo. Bob has the file's location entry but is missing chunks 200–250 locally (e.g., he just synced this drive partially).
|
||||
|
||||
```text
|
||||
1. App → drive.read("/photos/wedding/IMG_0001.jpg")
|
||||
|
||||
2. Drive index resolve:
|
||||
2a. walk DirIndex from root → DirEntry { fileLocId:"L-..." }
|
||||
2b. lookup FileLocation by fileLocId → 800 chunks, manifest in OPFS, mesh-known
|
||||
|
||||
3. Drive middleware (request side):
|
||||
3a. logging → start span { op:'read' }
|
||||
3b. chunks → load manifest, parse 800 entries
|
||||
3c. sharing → unwrap fileKey via partition KEK in our KeyRing
|
||||
3d. mesh → for each chunkId in 0..799, check if available locally
|
||||
result: 0..199 ✓, 200..250 ✗, 251..799 ✓
|
||||
3e. transport-webrtc → ensure peer connections to known content holders are live
|
||||
|
||||
4. Per-chunk fetch loop:
|
||||
For chunkIds 0..199 and 251..799:
|
||||
- storage.getPartial("files/<fileId>", chunkId * chunkSize, +chunkSize)
|
||||
returns the 65,536-byte chunk (header + sig + ciphertext)
|
||||
|
||||
For chunkIds 200..250 (missing locally):
|
||||
- drive-mesh emits `cd:request { fileId, chunkId, requesterId }` per chunk
|
||||
- network-plugin-content-delivery picks top-N candidate peers from gossip index
|
||||
- races them via networkNode.send() → drive-transport-webrtc on each peer
|
||||
- peer receives `cd:fetch`, runs their drive's storage.getPartial(...),
|
||||
sends back `cd:response { fileId, chunkId, bytes }`
|
||||
- first verifying response wins; lagging responses dropped
|
||||
- emits `cd:response` on requester's bus
|
||||
- drive-mesh takes the bytes, hands to chunks plugin
|
||||
|
||||
5. Per-chunk verify+decrypt (chunks plugin):
|
||||
5a. parse 164-byte header → metadata, sig, AAD
|
||||
5b. Ed25519-verify(driveId.pubkey, sig, metadata ‖ aeadTag)
|
||||
→ if fail: drop chunk, blacklist source peer (set score to -Infinity), retry
|
||||
5c. XChaCha20-Poly1305-decrypt(fileKey, nonce, ciphertext, AAD=metadata)
|
||||
→ if AEAD tag fails: drop chunk, retry from a different source
|
||||
|
||||
6. Reassemble:
|
||||
6a. concat all 800 plaintext chunks in chunkId order
|
||||
6b. blake3(concatenated) === manifest.checksum ?
|
||||
→ if not: hard error, manifest is wrong or chunks are corrupted
|
||||
|
||||
7. Drive middleware (response side):
|
||||
7e. transport-webrtc → no-op
|
||||
7d. mesh → emit `cd:advertise` (we now have chunks 200..250)
|
||||
peers learn we are a new source
|
||||
7c. sharing → no-op on read
|
||||
7b. chunks → return Uint8Array(52,428,800)
|
||||
7a. logging → record { op:'read', durationMs, peerFetches:51 }
|
||||
|
||||
→ drive.read resolves with the full photo. Bob's drive is now a holder for chunks
|
||||
200..250; the next time another peer needs them, they may fetch from Bob.
|
||||
```
|
||||
|
||||
The user gets the photo. Three peers cooperated to deliver it. Nobody had the whole file from the source; each peer served the chunks they had cached.
|
||||
|
||||
---
|
||||
|
||||
## What happens when the central server vanishes mid-stream
|
||||
|
||||
A real scenario: 3 minutes into Bob's read, the central server drops off. Result:
|
||||
|
||||
- Storage layer: untouched. Bob's local `OpfsStorage` reads and the WebRTC chunk fetches don't go through the server anyway.
|
||||
- Drive layer: untouched. Every operation continues — every chunk verify uses the writer's per-peer Ed25519 pubkey, which is in Bob's local registry.
|
||||
- Network layer:
|
||||
- `network-plugin-server-bridge` fires `server:disconnected` on the bus.
|
||||
- `network-plugin-failover` keeps running heartbeats among peers; mesh continues.
|
||||
- `network-plugin-governance` won't trigger a server-replacement election in `SERVER_BASED` mode (mesh stops); in `SERVER_TO_CLIENT_FAILOVER` mode it elects an acting Tier 0 from Tier 1 within seconds.
|
||||
- Bob's read completes normally — the chunks are coming from peer drives, not the server.
|
||||
- Any *new* file Bob writes is written locally (signed by Bob's drive identity), advertised to the mesh, and queued in `network-plugin-server-bridge`'s buffer for replay when the server returns.
|
||||
- Token grace periods extend; Bob's `CapabilityToken` doesn't expire mid-outage.
|
||||
- On server reconnect: buffered audit-log entries replay, the acting Tier 0 demotes, the mesh reconciles. Bob's new file becomes server-known.
|
||||
|
||||
This is the "if the server goes down, it's all in place with meshed pockets of users with their nominated hosts" part of the user's brief. It works because:
|
||||
|
||||
1. Every byte is signed by its writer; no server is needed to verify.
|
||||
2. Every chunk is content-addressed; no server is needed to discover them.
|
||||
3. Topology, election, partition handling, and circuit breaking are all peer-side.
|
||||
4. The server is a coordinator and an authority for *new-resource issuance*, not a relay for the data plane.
|
||||
|
||||
---
|
||||
|
||||
## How n drives find each other
|
||||
|
||||
A drive's identity is its Ed25519 public key — its `driveId`. Two drives find each other through one of three routes:
|
||||
|
||||
1. **Direct address** — you give one drive the address of another (hostname/port/peer-id). They handshake and join.
|
||||
2. **Discovery plugin** — `network-discovery-trystero` (or libp2p / GunDB / WebTorrent / hybrid) gives each drive a way to find peers without knowing them in advance. The plugin emits `peer:discovered`; `network-plugin-registry` records them; `drive-mesh` starts gossiping.
|
||||
3. **Server-mediated** — a deployment with a central server can have the server return peer lists on join. This is the cheap default for `SERVER_TO_CLIENT_FAILOVER` mode.
|
||||
|
||||
Once two drives know about each other, they handshake (Ed25519 challenge-response, NET-30), exchange capability bitmaps, register each other in `network-plugin-registry`, and begin announcing chunk inventories on the bus.
|
||||
|
||||
---
|
||||
|
||||
## What lives where — quick reference
|
||||
|
||||
| Question | Layer |
|
||||
| --- | --- |
|
||||
| What bytes are at this key? | Storage |
|
||||
| What backend is this storage? | Storage |
|
||||
| Can this backend do partial writes? | Storage (`capabilities`) |
|
||||
| What's in /photos/2026/? | Drive (DirIndex walk) |
|
||||
| Where are the chunks of file X? | Drive (FileLocation `storageHints`) |
|
||||
| Encrypt/sign these bytes | Drive (chunks plugin) |
|
||||
| Verify a peer's signature on a chunk | Drive (chunks plugin), peer pubkey from network registry |
|
||||
| Who can read this file? | Drive (sharing plugin partition) |
|
||||
| Who's online right now? | Network (registry plugin) |
|
||||
| Where can I get chunk Y? | Drive (mesh plugin requests; network content-delivery responds) |
|
||||
| Is the server up? | Network (server-bridge plugin) |
|
||||
| Who's the host of this mesh? | Network (governance + topology plugins) |
|
||||
| Has there been a partition? | Network (partition plugin) |
|
||||
| Can mesh A talk to mesh B? | Network (auth plugin trust validator + relay plugin) |
|
||||
|
||||
---
|
||||
|
||||
## The migration timeline (executive summary)
|
||||
|
||||
The genesis branch is mid-port from `main`. The shipped/pending split (full detail in each layer's `implementation-status.md`):
|
||||
|
||||
| Layer | Shipped on genesis | Most-pressing pending |
|
||||
| --- | --- | --- |
|
||||
| Storage | `storage`, `storage-s3` | `storage-browser-opfs`, `storage-browser`, `storage-file` (port from main) |
|
||||
| Drive | `drive`, `drive-plugin-chunks` | `drive-mesh`, `drive-transport-https`, `drive-transport-webrtc` (write fresh on genesis); `drive-sharing`, `drive-react-sharing`, `drive-webrtc` (port from main) |
|
||||
| Network | `network` (core), `network-plugin-protocol`, `network-plugin-registry`, `network-transport-webrtc`, `network-failover`, `network-governance`, `network-topology`, `network-transport-websockets`, `network-testing` | 14 of the 21 `network-plugin-*` packages (write fresh on genesis); discovery + libp2p + hyperswarm transports (port from main) |
|
||||
|
||||
The blocking spec change is `enterprise-types-cross-cutting` — it adds the canonical interfaces (`ChunkSource`, `ChunkRef`, `DrivePlugin`, `MiddlewareHandler`) to `@suaveplan/types`. Already implemented; awaits archive. Until it's archived, downstream packages keep redeclaring local copies of these types, which is the cause of the "every AI has its own ideas" feeling. **Land it first.**
|
||||
|
||||
The blocking architectural decision is `enterprise-drive` (monolith) vs `drive-port` (split). The migrated code follows `drive-port`; archive `enterprise-drive` and re-propose its CoW + snapshot ideas as their own plugins.
|
||||
|
||||
Both decisions are zero-code: open the change folders, write archive notes, fold the deltas into the canonical specs. Once that's done, every downstream package gets unblocked.
|
||||
|
||||
---
|
||||
|
||||
## Checklists for common deployments
|
||||
|
||||
### Single-server file storage (e.g., a SaaS backend)
|
||||
|
||||
```ts
|
||||
const drive = new Drive(new S3({ ... }))
|
||||
.usePlugin(chunksPlugin({ chunkSize: 65536, signing: { secretKey } }))
|
||||
.usePlugin(sharingPlugin({ ... }));
|
||||
```
|
||||
|
||||
No network plugin. Storage is one S3 bucket. Encrypted-at-rest, signed by the server's drive identity. Used as the persistent layer for an API server or the receiving end of a `drive-transport-https` upload.
|
||||
|
||||
### Browser drive with offline-first, cloud sync
|
||||
|
||||
```ts
|
||||
const drive = new Drive()
|
||||
.use("local", new OpfsStorage())
|
||||
.use("remote", new S3({ ... }))
|
||||
.usePlugin(chunksPlugin({ ... }))
|
||||
.usePlugin(sharingPlugin({ ... }))
|
||||
.usePlugin(meshPlugin({ networkNode }))
|
||||
.usePlugin(transportWebrtcPlugin({ signaller }))
|
||||
.usePlugin(transportHttpsPlugin({ baseUrl: "https://api.example.com" }));
|
||||
|
||||
const networkNode = new NetworkNode()
|
||||
.use(protocolPlugin())
|
||||
.use(registryPlugin())
|
||||
.use(webrtcPlugin({ ... }))
|
||||
.use(governancePlugin())
|
||||
.use(topologyPlugin())
|
||||
.use(serverBridgePlugin({ url: "wss://api.example.com/mesh", model: "SERVER_TO_CLIENT_FAILOVER" }))
|
||||
.use(authPlugin())
|
||||
.use(cryptoMessagesPlugin());
|
||||
```
|
||||
|
||||
Browser drive that works offline, syncs to peers when online, falls back to S3 via HTTPS, and survives server outages.
|
||||
|
||||
### Pure-peer mesh (no server)
|
||||
|
||||
```ts
|
||||
const networkNode = new NetworkNode()
|
||||
.use(protocolPlugin())
|
||||
.use(registryPlugin())
|
||||
.use(webrtcPlugin({ ... }))
|
||||
.use(discoveryTrysteroPlugin({ ... })) // peer discovery
|
||||
.use(governancePlugin()) // peer-elects host
|
||||
.use(topologyPlugin())
|
||||
.use(authPlugin({ mode: 'PURE_CLIENT' })) // no server tokens
|
||||
.use(cryptoMessagesPlugin());
|
||||
|
||||
const drive = new Drive(new OpfsStorage())
|
||||
.usePlugin(chunksPlugin({ ... }))
|
||||
.usePlugin(meshPlugin({ networkNode }))
|
||||
.usePlugin(transportWebrtcPlugin({ ... }));
|
||||
```
|
||||
|
||||
Two phones in the same room. Trystero-discovery finds each other (BitTorrent / Nostr / MQTT). Governance elects a host. Drives share files end-to-end-encrypted. No server anywhere.
|
||||
|
||||
### Cross-mesh corporate federation
|
||||
|
||||
Add `relayPlugin()` and `crossMeshTrustValidator()` to each mesh's `NetworkNode`, exchange `MeshTrustToken`s out of band, and the `HostBackboneTransport` between Tier 0 hosts of each mesh lets envelopes cross. A user in mesh A can read a file from a drive in mesh B; the chunks travel across the relay backbone, unwrapped only by the authorised peers.
|
||||
|
||||
---
|
||||
|
||||
## What this design buys you
|
||||
|
||||
| Need | Mechanism |
|
||||
| --- | --- |
|
||||
| Storage anywhere (memory / fs / db / browser / S3 / FAT image) | Storage backend plugins; capability matrix routes around limitations. |
|
||||
| Encrypt / sign / decrypt / verify on the fly | Drive chunks plugin: streaming AEAD + Ed25519 + BLAKE3. |
|
||||
| Files navigable by path | Drive directory and file-location indexes (bitpacker-encoded). |
|
||||
| Files shareable between drives, with revocation | Drive sharing plugin: partition KEKs, X25519 sealed-box wrapping. |
|
||||
| n drives that find each other | Discovery plugins + registry plugin. |
|
||||
| Peer-to-peer chunk transfer | drive-mesh + network-plugin-content-delivery + drive-transport-{webrtc,https}. |
|
||||
| Resilient tiered network | Tier 0/1/2 topology + governance + failover + relay. |
|
||||
| Server-down survival | server-bridge offline mode; signed-log buffering; replay on reconnect. |
|
||||
| Cross-mesh interop | Relay plugin + auth plugin trust validators. |
|
||||
| Verifiable audit trail | Audit-mesh signed log chains + replication + compaction. |
|
||||
| Multi-tenant isolation | NamespacedStorage + sharing partitions + per-file fileKey. |
|
||||
| End-to-end encryption with peer relay | Peers serve ciphertext bytes they cannot themselves read. |
|
||||
| Test-driven development | testing-integration fixtures, MemoryTransport, FakeClock, no shared state. |
|
||||
|
||||
This is what "enterprise-quality" looks like in this design: every concern owned by exactly one plugin, every cross-plugin link mediated by a typed event, every byte ciphertext-signed-on-disk-and-on-wire, every operation backed by a canonical OpenSpec capability requirement.
|
||||
|
||||
---
|
||||
|
||||
## How to keep the water clear
|
||||
|
||||
The user's complaint was: *"I feel because of how complex it is, and multiple AI having their own ideas, it's muddied the water."* Three rules that prevent that going forward:
|
||||
|
||||
1. **Specs are canonical, proposals are drafts.** [`openspec/specs/{drive,storage,network}/spec.md`](../openspec/specs/) are the source of truth. Anything in `openspec/changes/` is a proposal in flight; if two proposals conflict, archive one before any code lands. The `enterprise-drive` vs `drive-port` conflict is the cardinal example: archive `enterprise-drive`, keep `drive-port`, re-propose its unique deltas separately.
|
||||
2. **Cross-cutting types live in one place.** `enterprise-types-cross-cutting` was implemented but not archived; until it lands in `openspec/specs/types/spec.md`, every plugin keeps redeclaring local copies. Land it.
|
||||
3. **No plugin imports another plugin.** Cross-plugin communication is `NetworkBus` (network) or `drive:hook` (drive) topics only. `drive-network-integration`'s 5-line responsibility table is the rulebook; treat it as the authoritative interpretation of "who owns what."
|
||||
|
||||
Every time a future AI suggests adding a cross-plugin import, point them at the bus. Every time a future AI proposes a fat package, point them at the split. Every time a future AI invents new types in a leaf package, point them at `@suaveplan/types`.
|
||||
|
||||
That keeps the water clear.
|
||||
Reference in New Issue
Block a user