suaveplan/docs/drive/mesh-protocol.md

Mesh Protocol — Chunk Requests, Advertisements, Responses

This is the wire-protocol spec for how drives request and exchange chunks across the mesh. It encodes the three-form chunk request, the bitpacker-compressed presence bitset, the response piggyback, and the trust model.

This document is the resolution of system-review.md §1, §2, §10, §11. The spec belongs in openspec/specs/drive-mesh/spec.md once the proposal lands; for now this is the architectural reference.

---

1. Why bitsets

A 50 MiB file at 65,536-byte chunks = 800 chunks. Two clients meeting on the mesh must:

1. Tell each other what they have (advertise).
2. Request what they need.

A naïve protocol uses one envelope per chunk. With 800-chunk files and a 50-peer mesh, every advertisement round = 40,000 envelopes. A bitset is two orders of magnitude smaller. With @suaveplan/bitpacker-compressed (RLE + DEFLATE), a typical 800-bit presence map encodes to <100 bytes when most chunks are present or absent.

The mesh protocol is therefore bitset-first: every advertisement and every dense request uses a bitset. Sparse selections (≤64 chunks) use an array; the cutover threshold matches the bytes-on-wire break-even.

---

2. The three request forms

type ChunkRequest =
  | { kind: 'whole' }
  | { kind: 'specific'; chunkIds: number[] }
  | { kind: 'bitset'; bytes: Uint8Array };

Form	When to use	Wire size (800-chunk file)
whole	Initial fetch; receiver returns everything it has	1 byte (the discriminator)
specific	Sparse selection (≤64 chunks needed)	4 bytes per chunkId × N
bitset	Dense selection (>64 chunks)	typically <100 bytes after bitpacker-compressed

The drive-mesh layer chooses the form per-request based on the chunk-presence delta. Switching is automatic; the receiver supports all three.

ChunkRequestEnvelope

interface ChunkRequestEnvelope extends MessageEnvelope {
  type: 'cd:request';
  payload: Uint8Array;  // bitpacker-encoded ChunkRequestPayload
}

interface ChunkRequestPayload {
  fileId: string;            // 32-byte hex
  request: ChunkRequest;
  priority: 'critical' | 'high' | 'normal' | 'low' | 'bulk';
  maxChunks?: number;        // cap on response payload (default 64)
  signal?: AbortSignal;      // not on wire — local cancellation
}

Encoded via @suaveplan/bitpacker-network for the envelope frame; the bytes of the bitset use @suaveplan/bitpacker-compressed.

Empty-array semantics

The user's brief mentioned "supplying the file id and an empty chunk ids array" to mean "everything." The current spec uses { kind: 'whole' } instead of { kind: 'specific'; chunkIds: [] } to disambiguate — empty array could mean "nothing" or "everything" depending on reader bias. kind: 'whole' is unambiguous; kind: 'specific' with chunkIds: [] is rejected at validation as a no-op request.

---

3. The advertisement envelope

interface ChunkAdvertiseEnvelope extends MessageEnvelope {
  type: 'cd:advertise';
  payload: Uint8Array;  // bitpacker-encoded ChunkAdvertisePayload
}

interface ChunkAdvertisePayload {
  fileId: string;
  presence: Uint8Array;  // bitset, bit i = 1 iff peer has chunk i
  totalChunks: number;
  // Generation counter — increments on every change to this peer's presence
  // map for this file. Lets receivers ignore out-of-order advertisements.
  generation: number;
  timestamp: number;     // injectable Clock
}

Encoding

• presence uses @suaveplan/bitpacker-compressed's RLE encoder. Run-length is ideal for chunk presence because chunks tend to arrive in bursts (sequential downloads) — most bitsets have long 1-runs and long 0-runs.
• presence.length (in bits) MUST equal totalChunks. Padding bits in the final byte MUST be zero.
• A bit set to 1 means: "I have a verified chunk for this position." A bit set to 0 means either "I don't have it" or "I have it but haven't verified the signature yet."

When to advertise

Drives gossip advertisements:

• On startup, for every file in the FileLocationIndex.
• On cd:advertise from another peer (transitive — but only the delta since our last advert; throttled to one outgoing per file per second).
• On every cd:response we receive (we now have new chunks; tell our neighbors).

Generation counter

generation lets receivers handle out-of-order advertisements:

if (advert.generation < lastSeen[peerId][fileId]) drop;
else lastSeen[peerId][fileId] = advert.generation;

Wraps modulo 2^32 — wrap is rare enough that we tolerate a one-cycle anomaly.

---

4. The response envelope

interface ChunkResponseEnvelope extends MessageEnvelope {
  type: 'cd:response';
  payload: Uint8Array;  // bitpacker-encoded ChunkResponsePayload
}

interface ChunkResponsePayload {
  fileId: string;
  chunks: ChunkBytes[];        // ordered by chunkId asc
  presence: Uint8Array;        // piggyback: my full bitset for this file (free advert)
  generation: number;
}

interface ChunkBytes {
  chunkId: number;
  bytes: Uint8Array;            // 164-byte chunk header + ciphertext (signed by writer)
}

Why piggyback presence

A response is the perfect time to update peer-index for (peerId, fileId). The receiver got the chunks they wanted and learned the holder's full inventory in one round-trip. This eliminates a separate advertisement cycle for active partners. Cost: ~100 bytes per response. Benefit: skip ~200 bytes + RTT for a follow-up advertisement.

Per-chunk verification

Each ChunkBytes.bytes is the locked v0x0001 chunk format from encryption.md:

metadata(100) ‖ sig(64) ‖ ciphertext(variable)

The receiver verifies the Ed25519 sig against the writer's pubkey (from the registry), checks the AEAD tag, and only then commits the bytes. A bad sig drops the response and demotes the responder's score (see §6).

---

5. Response sizing and chunking-of-the-response

A 64-chunk response at 65,536 bytes per chunk = 4 MiB. That is too large for a single WebRTC datachannel message (WebRTC SCTP fragmentation breaks down past 16 KiB cleanly). The drive-transport layer fragments the envelope; the protocol layer caps:

• maxChunks defaults to 64; receivers MUST honor it.
• Responses larger than 1 MiB are sent as multiple cd:response envelopes with the same correlation id.
• WebRTC backpressure (bufferedAmount > 16 MiB) blocks further sends — see TRANSPORT-WRTC-04.

For very large transfers, the responder may choose to return a partial response and emit cd:advertise for the rest, letting the requester re-pull on its own pace.

---

6. Trust and verification

Probabilistic trust model

A peer that consistently advertises chunks it doesn't actually have wastes the requester's time even if it never sends a bad signature. The trust model handles this:

For each (peerId, fileId):
  advertised: bitset of "has it according to advertisements"
  delivered:  bitset of "actually delivered when asked"
  missing:    bitset of advertised-but-failed-on-request

  trustScore = 1 - (popcount(missing) / popcount(advertised))

  P(skip request) = max(0, 0.05 - trustScore) * 20  // demotes harshly past 5% miss

Bad signature: immediate trustScore = -Infinity (existing MESH-SCORE-01); we never request from this peer for this file again.

Per-file scope: a peer good for one file doesn't earn blanket trust for others. Avoids reputation laundering.

Sample verification on advertisement

Optionally, the receiver can probe: pick one chunk from presence, request it, verify. If the bit is set but the chunk isn't deliverable, demote. Sampling rate is adaptive: 1-in-100 by default, raised to 1-in-10 if a peer enters "suspect" status.

---

7. Worked example — 800-chunk file, 50% present locally

Alice's drive needs file f-1729... (800 chunks). She has chunks 0–399 locally.

1. Alice constructs a "needed" bitset: 0b0...01...1 (400 zeros, 400 ones).
   bitpacker-compressed RLE encode: ~12 bytes.

2. Alice broadcasts cd:request:
   { fileId: 'f-1729...', request: { kind: 'bitset', bytes: <12 bytes> },
     priority: 'normal', maxChunks: 64 }
   Total envelope: ~50 bytes.

3. Three peers (Bob, Carol, Dave) reply:
   - Bob: has 400-799. Sends 64 chunks (bytes ~4.2 MiB, fragmented), plus
     piggyback presence (RLE: ~10 bytes).
   - Carol: has 400-700. Sends 64 chunks of 400-463.
   - Dave: has 500-700 only. Sends 64 chunks of 500-563.

4. Alice's mesh layer races them; first verifying response wins per chunk.
   - Chunks 400-463: Carol wins (lower latency).
   - Chunks 464-499: Bob wins (Carol doesn't have).
   - Chunks 500-563: race between Bob/Carol/Dave.
   - Chunks 564-700: race between Bob/Carol.
   - Chunks 700-799: Bob wins (sole holder).

5. Each verified chunk is stored, presence bit flipped, gossiped via cd:advertise.

6. Alice's progress UI:
   t=0: ███░░░░░░░ 50%
   t=2s: █████░░░░░ 60%  (Carol's batch verified)
   t=4s: ███████░░░ 80%  (Bob's batch verified)
   t=6s: ██████████ 100%

Total round trips: 3 (one per responder). Total bandwidth (excluding chunk bytes): ~200 bytes of metadata. Without bitset compression, the same exchange would have generated 800 single-chunk requests.

---

8. Error envelopes

interface ChunkErrorEnvelope extends MessageEnvelope {
  type: 'cd:error';
  payload: {
    fileId: string;
    correlationId: string;     // matches the cd:request id
    code: ChunkErrorCode;
    chunkId?: number;          // present for chunk-specific errors
    retryAfterMs?: number;     // backpressure hint
  };
}

type ChunkErrorCode =
  | 'NOT_FOUND'                // requested chunk genuinely absent
  | 'ACCESS_DENIED'            // policy prevents serving (e.g. unsigned trust token)
  | 'BANDWIDTH_EXCEEDED'       // QoS quota hit; retryAfterMs set
  | 'GENERATION_STALE'         // requester's presence info is older than ours
  | 'INTERNAL';

GENERATION_STALE is informational — the responder included its current generation in the error so the requester can update without a separate advertisement.

---

9. Integration with network-plugin-content-delivery

drive-mesh produces and consumes these envelopes via NetworkBus. network-plugin-content-delivery is the transport-level plugin that routes them:

Direction	Bus topic	Producer	Consumer
Outbound	cd:request	drive-mesh	network-plugin-content-delivery (forwards to peers)
Outbound	cd:advertise	drive-mesh	network-plugin-content-delivery (gossips to peers)
Outbound	cd:response	drive-mesh (when serving)	network-plugin-content-delivery (forwards to requester)
Inbound	cd:request	network-plugin-content-delivery	drive-mesh (decides what to serve)
Inbound	cd:response	network-plugin-content-delivery	drive-mesh (verifies + stores)
Inbound	cd:advertise	network-plugin-content-delivery	drive-mesh (updates peer-index)

network-plugin-content-delivery does not parse the bitsets or verify signatures. It is a forwarder + redundancy enforcer. All chunk-level intelligence lives in drive-mesh + drive-plugin-chunks.

---

10. Bitpacker package usage map

Concern	Package
Bitset encode/decode (presence, request)	@suaveplan/bitpacker-compressed
Envelope schema validation	@suaveplan/bitpacker-schema
On-wire framing	@suaveplan/bitpacker-network
Streaming for large responses	@suaveplan/bitpacker-streaming
Future: delta-encoded advertisements	@suaveplan/bitpacker-delta (when generation diffs become large)

The codecs are interoperable — a bitset encoded by bitpacker-compressed is decoded the same way regardless of which sub-package framed the envelope.

---

11. Test contract

The drive-mesh-protocol-v1 change folder will ship these conformance tests:

• tests/three-form-roundtrip.test.ts — every ChunkRequest form roundtrips through encode/decode.
• tests/bitset-rle.test.ts — fast-check property: encode(decode(x)) = x for random bitsets.
• tests/generation-monotonic.test.ts — out-of-order advertisements drop correctly.
• tests/piggyback-presence.test.ts — receiving a cd:response updates peer-index without a separate cd:advertise.
• tests/trust-decay.test.ts — repeatedly-advertising-but-not-delivering peer loses trust score predictably.
• tests/cap-respected.test.ts — maxChunks is honored by responders.

---

Reading further

• mesh-and-sharing.md — the drive-mesh plugin proper.
• encryption.md — chunk format that the responses carry.
• dehydrated-drives.md — how mesh protocol is bypassed when CDN fallback is in play.
• ../network/plugins-catalog.md — network-plugin-content-delivery and network-plugin-relay.
• ../system-review.md — the review this doc resolves.