Initial Commit

openspec/archive/2026-05-12-drive-mesh-port/proposal.md

@@ -0,0 +1,134 @@
+# OpenSpec Change: drive-mesh-port
+
+- **Change ID**: drive-mesh-port
+- **Package**: `@suaveplan/drive-mesh` (new)
+- **Status**: PROPOSED
+- **Date**: 2026-04-28
+
+## Summary
+
+P2P mesh layer over the chunks plugin. `DriveMesh` orchestrates peer discovery, availability gossip (compressed bitsets per file), parallel chunk pull (multi-peer racing window), peer scoring, server fallback, and upload backpressure. Mesh consumes the `ChunkSource` interface returned by `@suaveplan/drive-plugin-chunks` — it does NOT import drive core. Mesh consumes one or more `DriveTransport` implementations (HTTPS, WebRTC, libp2p) for the wire.
+
+## Why mesh is a separate package
+
+A vault running on a server doesn't need mesh — it just wants `drive` + `storage-s3`. A browser session in the CDN scenario needs `drive` + chunks plugin + mesh + `drive-transport-webrtc` + `drive-transport-https` (server fallback). Bundling mesh into drive or chunks would push WebRTC peer code and Bloom-filter logic into every consumer.
+
+## Trust model
+
+- **Server is the root of trust.** Server signs every chunk and every manifest with its ed25519 key.
+- **Peers are caches, never authorities.** Mesh treats every byte from a peer as untrusted until the chunks plugin's sig-verify passes.
+- **Bad-sig peers are punished.** A peer that ever delivers a chunk with an invalid server signature is added to a deny list (configurable: per-session or persistent).
+- Mesh never holds the file decryption key — that's in the manifest, decrypted client-side via the recipient's key.
+
+## API surface
+
+```ts
+const mesh = new DriveMesh({
+    chunks,                  // ChunkSource handle from chunksPlugin
+    transports: [webrtcTransport, httpsTransport],
+    serverPubKey: Uint8Array,
+    serverFallback: { url: string, transport: "https" },
+    scheduler: { concurrency: 8, perChunkTimeoutMs: 4000, racingWindowMs: 200 },
+    peerPolicy: { maxConcurrentUploadsPerPeer: 4, blocklistOnBadSig: true },
+    rng: types.Rng,
+    clock: types.Clock,
+});
+
+await mesh.start();           // open transports, begin advertising
+await mesh.fetchFile(fileId); // resolves manifest, schedules missing chunks
+await mesh.fetchChunks(fileId, [12, 13, 14]);
+mesh.on("chunk:received", ({ fileId, chunkId, fromPeer }) => …);
+await mesh.stop();
+```
+
+## Sub-systems
+
+### 1. Peer index (`PeerIndex`)
+
+Tracks live peers and their advertised availability per file (compressed bitset keyed by `fileId`), latency EMAs, error rates. **This is an index overlay, not a registry.** Peer identity and lifecycle events come from `network-plugin-registry` via the `NetworkBus`. The index only adds file-specific metadata on top.
+
+```ts
+// drive-mesh consumes registry events from the network layer
+bus.on("peer:joined", ({ peerId, tier, latency }) => {
+  driveMesh.peerIndex.add(peerId, { tier, latency });
+});
+bus.on("peer:left", ({ peerId }) => {
+  driveMesh.peerIndex.remove(peerId);
+});
+```
+
+Methods: `addPeer(peerId, transport, metadata)`, `removePeer`, `peersFor(fileId, chunkId)`.
+
+### 2. Availability gossip (`AvailabilityIndex`)
+
+- Each peer maintains a `BitSet` per known fileId of chunks it has.
+- On chunk acquisition, peer pushes `advertise(fileId, BitSet)` to known peers via the transport.
+- On peer-join, peer pulls `advertisement(fileId)` from new neighbours.
+- Bitsets are gzipped on the wire.
+
+### 3. Parallel-pull scheduler (`Scheduler`)
+
+- Takes a list of `(fileId, chunkId)` requests.
+- For each, picks top-K peers by score from `peersFor(fileId, chunkId)`.
+- Issues request to the best peer; after `racingWindowMs`, issues a parallel request to the second-best (cancelled when the first arrives).
+- Per-chunk timeout `perChunkTimeoutMs` triggers retry on next-best peer.
+- Honours `concurrency` global cap.
+- After all peer attempts exhaust, falls back to `serverFallback.url` via the HTTPS transport.
+
+### 4. Peer scoring (`PeerScore`)
+
+- `score = w_latency / latency_ema + w_success * success_rate - w_errors * error_rate`.
+- Updated on every chunk-receive completion or failure.
+- Sig-verify failure → permanent score = -∞ (effectively blocked).
+
+### 5. Upload backpressure (`UploadGovernor`)
+
+- Limits concurrent uploads per peer (`maxConcurrentUploadsPerPeer`).
+- Honours WebRTC datachannel `bufferedAmount` watermark; pauses sends when high.
+- Emits `upload:throttled` events for observability.
+
+### 6. NetworkBus integration
+
+`DriveMesh` listens to `network-plugin-registry` events on the `NetworkBus` to maintain its `PeerIndex` without duplicating peer lifecycle tracking:
+
+```ts
+bus.on("peer:joined", ({ peerId, tier, latency }) => {
+  driveMesh.peerIndex.add(peerId, { tier, latency });
+});
+bus.on("peer:left", ({ peerId }) => {
+  driveMesh.peerIndex.remove(peerId);
+});
+bus.on("peer:updated", ({ peerId, latency }) => {
+  driveMesh.peerIndex.updateLatency(peerId, latency);
+});
+```
+
+This ensures that `drive-mesh` and `network-plugin-registry` share a single source of truth for peer identity. File-specific availability bitsets remain the responsibility of `drive-mesh`'s `AvailabilityIndex`.
+
+### 7. Manifest bootstrap
+
+Defaults to **manifest-as-chunk** (chunks plugin's default). Mesh requests `manifest:<fileId>` from peers like any other chunk; if unavailable, falls back to `serverFallback.url`. `mesh.fetchFile(fileId)` is an end-to-end happy-path that:
+
+1. Fetch manifest (peer or server).
+2. Verify manifest sig.
+3. Diff manifest's chunk list against `chunks.hasChunks(fileId)`.
+4. Schedule missing chunks via Scheduler.
+5. Resolve when bitset is full.
+
+## Error codes
+
+`MESH_CODES = defineCodes("MESH", [PEER_NOT_FOUND, NO_PEERS_AVAILABLE, ALL_PEERS_FAILED, SERVER_FALLBACK_FAILED, MANIFEST_FETCH_FAILED, ABORTED, TRANSPORT_ERROR, PEER_BLOCKED] as const)`. `MeshError extends SuaveplanError<MeshErrorCode>`.
+
+## Dependencies
+
+- **Consumes:** `@suaveplan/drive-plugin-chunks` (interface only — `ChunkSource`), `@suaveplan/drive-transport-https` (server fallback), and any other registered `DriveTransport`. Plus `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/events`, `@suaveplan/streams`, `@opentelemetry/api`, `zod`.
+- **Consumes (NetworkBus events):** `@suaveplan/network-plugin-registry` (`peer:joined`, `peer:left`, `peer:updated`). Drive-mesh does NOT import the registry package directly; it listens to bus events.
+- **Does NOT depend on:** `@suaveplan/drive` core.
+- **Blocked on:** `drive-plugin-chunks-port`, at least one `drive-transport-*-port` (HTTPS minimum for fallback), `network-plugin-registry` (for peer identity events).
+
+## Out of scope
+
+- Wire-protocol implementations → `drive-transport-https-port`, `drive-transport-webrtc-port`.
+- DHT-based peer discovery → future change `drive-mesh-discovery-dht`.
+- BitSwap-style pairwise want-list reconciliation → optimisation in v2.
+- Sharing primitives (envelope encryption, grant tokens) → `@suaveplan/drive-sharing` (separate change).

openspec/archive/2026-05-12-drive-mesh-port/specs/drive-mesh/spec.md

@@ -0,0 +1,75 @@
+# Drive Mesh — Spec Delta (drive-mesh-port)
+
+## ADDED Requirements
+
+### MESH-DEPS-01 — Mesh consumes ChunkSource, not drive core
+
+`@suaveplan/drive-mesh` MUST NOT import `@suaveplan/drive`. It MUST consume only the `ChunkSource` interface exported by `@suaveplan/drive-plugin-chunks` and `DriveTransport` interfaces from transport packages.
+
+**Verified by:** `src/mesh.spec-conformance.test.ts` (`MESH-DEPS-01`); package.json import audit.
+
+### MESH-SCHED-01 — Racing window for parallel pull
+
+`Scheduler` MUST issue a request to the best-scored peer for a chunk; after `racingWindowMs`, MUST issue a parallel request to the next-best peer. The first valid (sig-verified) chunk to return wins; in-flight losers MUST be cancelled.
+
+**Verified by:** `src/scheduler/scheduler.spec-conformance.test.ts` (`MESH-SCHED-01`).
+
+### MESH-SCHED-02 — Per-chunk timeout
+
+If a peer does not return a chunk within `perChunkTimeoutMs`, the request MUST be aborted, the peer's score MUST be penalised, and the next-best peer MUST be tried. Repeated until peer pool is exhausted, then server fallback.
+
+**Verified by:** `src/scheduler/scheduler.spec-conformance.test.ts` (`MESH-SCHED-02`).
+
+### MESH-SCHED-03 — Server fallback
+
+When all candidate peers for a chunk fail, the scheduler MUST attempt `serverFallback.url` via the HTTPS transport. Failure of that fallback MUST throw `MeshError(MESH.ALL_PEERS_FAILED)` with `cause` chain including the most recent peer error.
+
+**Verified by:** `src/scheduler/scheduler.spec-conformance.test.ts` (`MESH-SCHED-03`).
+
+### MESH-SCORE-01 — Bad sig blocks peer
+
+A peer that delivers a chunk failing sig verification MUST have its score set to `-Infinity` (effective ban) and MUST NOT be selected for future chunks. If `peerPolicy.blocklistOnBadSig` is `true`, the peer is added to a persistent deny list.
+
+**Verified by:** `src/scheduler/peer-score.spec-conformance.test.ts` (`MESH-SCORE-01`).
+
+### MESH-SCORE-02 — Score updates on every result
+
+`PeerScore` MUST update latency EMA, success count, and error count on every chunk-receive completion. Score function MUST be deterministic given inputs.
+
+**Verified by:** `src/scheduler/peer-score.spec-conformance.test.ts` (`MESH-SCORE-02`).
+
+### MESH-GOSSIP-01 — Availability bitset gossip
+
+`AvailabilityIndex` MUST encode peer-availability as a `BitSet` per `fileId`, gzip-compressed on the wire. On chunk acquisition, the local peer MUST push an updated advertisement to known peers via the transport. On peer-join, the local peer MUST request advertisements for files it cares about.
+
+**Verified by:** `src/registry/availability-index.spec-conformance.test.ts` (`MESH-GOSSIP-01`).
+
+### MESH-BOOTSTRAP-01 — Manifest bootstrap is peer-first
+
+`fetchManifest(fileId)` MUST first attempt peers via the same transport pipeline (manifest-as-chunk default of the chunks plugin), then fall back to `serverFallback.url` only when no peer can serve it. Sig verification on the manifest MUST run regardless of source.
+
+**Verified by:** `src/bootstrap/fetch-manifest.spec-conformance.test.ts` (`MESH-BOOTSTRAP-01`).
+
+### MESH-FETCH-FILE-01 — End-to-end fetch happy path
+
+`mesh.fetchFile(fileId)` MUST: (1) fetch manifest, (2) sig-verify manifest, (3) diff against `chunks.hasChunks(fileId)`, (4) schedule missing chunks via Scheduler, (5) resolve when the local bitset is full. Each chunk that arrives MUST be sig-verified by the chunks plugin before persistence.
+
+**Verified by:** `src/bootstrap/fetch-file.spec-conformance.test.ts` (`MESH-FETCH-FILE-01`).
+
+### MESH-UPLOAD-01 — Upload backpressure
+
+`UploadGovernor` MUST limit concurrent uploads per peer to `peerPolicy.maxConcurrentUploadsPerPeer`. For WebRTC transports, governor MUST honour datachannel `bufferedAmount` watermark and pause sends when above the high-water mark.
+
+**Verified by:** `src/upload/governor.spec-conformance.test.ts` (`MESH-UPLOAD-01`).
+
+### MESH-CODES-01 — Error registry namespaced and frozen
+
+`MESH_CODES` MUST be declared via `defineCodes("MESH", [...])`. `Object.isFrozen(MESH_CODES)` MUST be `true`. Every code value MUST start with `MESH.`. `MeshError` MUST extend `SuaveplanError<MeshErrorCode>`.
+
+**Verified by:** `src/error-codes.spec-conformance.test.ts` (`MESH-CODES-01`).
+
+### MESH-LIFECYCLE-01 — Start/stop are idempotent
+
+`mesh.start()` and `mesh.stop()` MUST be idempotent. `start` opens all configured transports; `stop` closes them and drains in-flight requests with `MeshError(MESH.ABORTED)`.
+
+**Verified by:** `src/mesh.spec-conformance.test.ts` (`MESH-LIFECYCLE-01`).

openspec/archive/2026-05-12-drive-mesh-port/tasks.md

@@ -0,0 +1,58 @@
+# Tasks: drive-mesh-port
+
+> Blocked on: `drive-plugin-chunks-port`, at least one `drive-transport-*-port`.
+
+## 1. Scaffold
+
+- [ ] 1.1 `packages/drive/drive-mesh/` with `package.json` (`@suaveplan/drive-mesh@0.0.0`), `tsconfig.json`, `vitest.config.ts`, `README.md`.
+- [ ] 1.2 Workspace deps: `@suaveplan/drive-plugin-chunks` (interface only), `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/events`, `@suaveplan/streams`, `@opentelemetry/api`, `zod`.
+
+## 2. Types
+
+- [ ] 2.1 `src/types/mesh.types.ts` — `DriveMeshOptions`, `PeerInfo`, `PeerScore`, `ChunkRequest`, `SchedulerConfig`, `PeerPolicy`.
+- [ ] 2.2 `src/types/transport.types.ts` — `DriveTransport` interface (consumed by mesh, implemented by transport packages): `requestChunk(peerId, fileId, chunkId): AsyncIterable<Uint8Array>`, `advertise(peerId, fileId, bitset)`, `onAdvertisement(handler)`, `peers(): Iterable<PeerInfo>`.
+- [ ] 2.3 `src/error-codes.ts` — `defineCodes("MESH", [...])`.
+
+## 3. Peer registry
+
+- [ ] 3.1 `src/registry/peer-registry.ts` — `addPeer`, `removePeer`, `peersFor(fileId, chunkId)`, score updates.
+- [ ] 3.2 `src/registry/availability-index.ts` — `BitSet`-per-file index, gzip wire encoding.
+
+## 4. Scheduler
+
+- [ ] 4.1 `src/scheduler/scheduler.ts` — concurrency cap, racing window, per-chunk timeout, server fallback.
+- [ ] 4.2 `src/scheduler/peer-score.ts` — EMA latency, success/error rates, sig-fail = -∞.
+
+## 5. Upload governor
+
+- [ ] 5.1 `src/upload/governor.ts` — per-peer concurrency cap, datachannel buffer watermark.
+
+## 6. Manifest bootstrap
+
+- [ ] 6.1 `src/bootstrap/fetch-manifest.ts` — peer-first then server fallback, sig-verify gate.
+- [ ] 6.2 `src/bootstrap/fetch-file.ts` — resolves manifest → diffs chunks → schedules missing.
+
+## 7. DriveMesh class
+
+- [ ] 7.1 `src/mesh.ts` — `start`, `stop`, `fetchFile`, `fetchChunks`, event emitter.
+- [ ] 7.2 Lifecycle: opens transports on `start`, closes on `stop`; idempotent.
+
+## 8. Spec
+
+- [ ] 8.1 `openspec/specs/drive-mesh/spec.md` — `MESH-01` … `MESH-NN` with `**Verified by:**` lines.
+
+## 9. Conformance tests
+
+- [ ] 9.1 `src/scheduler/scheduler.spec-conformance.test.ts` — racing window, parallel pull, server fallback (`MESH-SCHED-01`, `-02`, `-03`).
+- [ ] 9.2 `src/scheduler/peer-score.spec-conformance.test.ts` — bad-sig blocks peer, score updates (`MESH-SCORE-01`, `-02`).
+- [ ] 9.3 `src/registry/availability-index.spec-conformance.test.ts` — bitset advertise/receive, gzip round-trip (`MESH-GOSSIP-01`).
+- [ ] 9.4 `src/bootstrap/fetch-manifest.spec-conformance.test.ts` — peer-first then fallback (`MESH-BOOTSTRAP-01`).
+- [ ] 9.5 `src/bootstrap/fetch-file.spec-conformance.test.ts` — end-to-end happy path with fake transports (`MESH-FETCH-FILE-01`).
+- [ ] 9.6 `src/upload/governor.spec-conformance.test.ts` — per-peer cap, watermark pause (`MESH-UPLOAD-01`).
+- [ ] 9.7 `src/error-codes.spec-conformance.test.ts` — frozen registry (`MESH-CODES-01`).
+
+## 10. Verify
+
+- [ ] 10.1 `bunx turbo lint typecheck test --filter=@suaveplan/drive-mesh` green.
+- [ ] 10.2 `bunx turbo test:coverage --filter=@suaveplan/drive-mesh` 100/100/100/100.
+- [ ] 10.3 `bun scripts/openspec-conformance.ts @suaveplan/drive-mesh` PASS.