suaveplan/docs/drive/dehydrated-drives.md

Dehydrated Drives — Templates, Policies, and CDN Fallback

A dehydrated drive is a drive whose folder structure and file metadata are pre-configured by a server-signed template, while the actual file bytes arrive later from peers, the CDN, or local writes. The user sees the full directory tree on day one — files have correct names, sizes, and lock states — and chunks populate transparently as they're requested.

This is the answer to "ship a 2 GB game asset bundle to N players" and "subscribe a user to a curated drive without copying everything up front." It is the resolution of system-review.md §3, §4, §5, §8, §9, §12, §13, §16. The spec belongs in openspec/specs/drive-template/spec.md once the proposal lands.

---

1. The lifecycle

┌──────────────────────────────────────────────────────────────────┐
│ Server                                                           │
│  - Curates the drive: which files, which folders, which chunks   │
│  - Computes hashes, sizes, chunk counts                          │
│  - Signs DriveTemplate with server Ed25519 key                   │
│  - Publishes template (via API, share link, or peer broadcast)   │
└──────────────────────────┬───────────────────────────────────────┘
                           │ signed DriveTemplate (~10-100 KB)
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│ Client                                                           │
│  drive.hydrateFrom(template)                                     │
│   ├─ verify server signature                                     │
│   ├─ instantiate DirIndex from FolderTemplate tree               │
│   ├─ instantiate FileLocationIndex from FileTemplate stubs       │
│   │   (every entry's `storageHints` starts with kind:'mesh' and  │
│   │    optionally kind:'cdn')                                    │
│   ├─ unwrap inline keys → KeyRing (lock icon flips per file)     │
│   ├─ for each file with policy.autoDownload === true:            │
│   │   schedule a fetch via mesh-protocol (see mesh-protocol.md)  │
│   │   on miss, fall through to CDN                               │
│   └─ emit `drive:hydrated` event                                 │
└──────────────────────────────────────────────────────────────────┘

After hydration, the drive looks identical to one that was populated by writes. The user calls read("/photo.jpg") the same way; the read pipeline transparently fetches missing chunks. The dehydrated state is invisible at the API surface — only the presence field on DirEntries hints that bytes haven't arrived.

---

2. The DriveTemplate shape

interface DriveTemplate {
  schemaVersion: 1;                    // pinned; future versions are new packages
  driveId: string;                     // unique drive identity (32-byte hex)
  rootFolder: FolderTemplate;
  cdnFallback?: CdnFallbackConfig;
  policy: GlobalPolicy;
  issuer: string;                      // server pubkey (Ed25519, 32-byte hex)
  issuedAt: number;                    // unix ms
  expiresAt: number;                   // unix ms
  nonce: Uint8Array;                   // 16 bytes; replay protection
  serverSignature: Uint8Array;         // Ed25519 over canonicalSerialize(this without sig)
}

interface FolderTemplate {
  name: string;                        // single path component
  children: (FolderTemplate | FileTemplate)[];
  // depth is capped at 32 (validated by DriveTemplateSchema)
}

interface FileTemplate {
  name: string;
  fileLocId: string;                   // primary key into FileLocationIndex
  fileId: string;                      // 32-byte hex; stable across versions
  size: number;                        // plaintext bytes
  totalChunks: number;
  chunkSize: number;
  checksum: string;                    // 'blake3:<hex>' over plaintext
  flags: PerFilePolicy;
  encryption?: FileEncryption;
  manifestKey: string;                 // where the chunk manifest lives
  manifestSig: Uint8Array;             // detached Ed25519 sig over the manifest's content
}

interface PerFilePolicy {
  autoDownload: boolean;               // eagerly fetch on hydrate?
  meshAllowed: boolean;                // may we cd:request from peers?
  cdnAllowed: boolean;                 // may we hit the CDN?
  retention?: RetentionPolicy;
}

type RetentionPolicy =
  | 'permanent'                        // never evict
  | 'lru'                              // global byte-budget LRU
  | { maxAgeMs: number };              // chunks older than N ms eligible for eviction

interface FileEncryption {
  algorithm: 'xchacha20-poly1305';
  keyDistribution:
    | { kind: 'inline'; wrappedKey: Uint8Array }       // shipped in template, see §7
    | { kind: 'retroactive'; keyAuthority: string };   // peer pubkey will issue grant later
}

interface GlobalPolicy {
  defaultPolicy: PerFilePolicy;        // applied to files whose `flags` is partial
  maxLocalCacheBytes?: number;         // global LRU budget (excludes 'permanent' files)
  maxConcurrentFetches?: number;       // backpressure on hydrate
  trustedKeyAuthorities?: string[];    // pubkeys whose key:grant we accept
}

interface CdnFallbackConfig {
  baseUrl: string;                     // e.g. 'https://chunks.example.com'
  pathScheme: 'reverse-id';            // see §4 for the URL convention
  levels: number;                      // directory depth (typically 3)
  padding: number;                     // chunk ID zero-pad width (typically 9)
  uploadEndpoint?: string;             // for write-back of newly-fetched chunks
}

---

3. Signature, replay protection, and validation

Canonical serialization

The server computes the signature over a deterministic byte representation:

canonicalSerialize(template) =
  schemaVersion(varint) ‖
  driveId(32) ‖
  rootFolder(canonical-folder) ‖
  cdnFallback(canonical-cdn-or-empty) ‖
  policy(canonical-policy) ‖
  issuer(32) ‖
  issuedAt(8) ‖
  expiresAt(8) ‖
  nonce(16)

Encoded via @suaveplan/bitpacker-schema with deterministic field order. JSON is not used — it doesn't have a canonical form across implementations.

Validation order

drive.hydrateFrom(template) validates in this exact order; any failure aborts before instantiating anything:

1. schemaVersion === 1 (else DRIVE.TEMPLATE_VERSION_UNSUPPORTED).
2. Date.now() < expiresAt (else DRIVE.TEMPLATE_EXPIRED).
3. Date.now() - issuedAt < 24*60*60*1000 (24-hour issuance window — DRIVE.TEMPLATE_STALE).
4. nonce not seen in last 1024 templates (LRU cache; DRIVE.TEMPLATE_REPLAY).
5. issuer ∈ globalPolicy.trustedKeyAuthorities (or matches drive-creation key — DRIVE.TEMPLATE_UNTRUSTED_ISSUER).
6. Ed25519 verify serverSignature over canonicalSerialize(template - sig) (DRIVE.TEMPLATE_SIG_INVALID).
7. Recursive folder depth ≤ 32 (DRIVE.TEMPLATE_TOO_DEEP).
8. Per-file: chunkSize > 164 (header overhead), totalChunks * chunkSize >= size, manifestKey non-empty (DRIVE.TEMPLATE_MALFORMED).

Only after all eight pass does hydration begin.

Why nonce + issuedAt

A captured template, even one within its expiresAt, cannot be replayed against the same drive because the nonce is recorded. A client moving between devices must request a fresh template; this is by design — it gives the server a chance to update policy.

---

4. CDN fallback URL convention

When policy.cdnAllowed === true and a chunk is not available from the mesh, the chunks plugin falls back to HTTPS via drive-transport-https. The URL convention is reversed-ID with configurable depth and padding:

<baseUrl>/<reversed[0]>/<reversed[1]>/...<reversed[levels-1]>/<fileId>_<chunkIdPadded>.chunk

Worked example

fileId       = "abcdef1234567890fedcba0987654321abcdef1234567890fedcba0987654321"
reversed     = "1234567890abcdef1234567890abcdef1234567890fedcba0987654321fedcba"
levels       = 3
padding      = 9
chunkId      = 47

URL = https://chunks.example.com/1/2/3/<full-fileId>_000000047.chunk

Why reversed

File IDs in practice cluster by prefix:

• Sequential issuance (f-000001, f-000002, ...) makes prefix-based hashing put millions of files in the same directory.
• Timestamp-prefixed IDs (1729a4f72c3b... from a sortable UUID) cluster by issuance time.
• Hash prefixes are uniform but humans tend to name files in consistent ways (creation tools embed magic bytes, etc.).

Reversing distributes leaves randomly across the directory tree regardless of how IDs were generated. The first three reversed characters give 16³ = 4,096 leaf directories — enough to keep any single directory under 250 entries even at 1 M files.

Configuration trade-offs

levels	Leaf dirs	Chunks/dir at 1M files	When to use
2	256	3,906	<100 K files; minimize HTTP path length
3	4,096	244	default — good balance
4	65,536	15	>10 M files; inode budget allows

Padding (9 digits) supports up to 999,999,999 chunks per file (~64 PB at 64 KiB chunks). Not the bottleneck.

Read protocol

GET https://chunks.example.com/1/2/3/<fileId>_000000047.chunk
Range: bytes=0-                  (optional — supports partial chunk fetch)
Accept: application/octet-stream

Response body is the locked v0x0001 chunk format (metadata(100) ‖ sig(64) ‖ ciphertext). Same bytes the mesh would have returned. The drive's chunks plugin verifies the Ed25519 sig and AEAD tag identically — the CDN never sees plaintext.

CDN as cache vs CDN as authority

The CDN is treated as a cache, not an authority. If a chunk's CDN copy fails signature verification, the drive:

1. Discards the bytes.
2. Falls back to the mesh.
3. Optionally reports the corruption to a configured endpoint.

This means a compromised CDN cannot poison a drive — it can only deny service.

---

5. CDN write-back

CdnFallbackConfig.uploadEndpoint (optional) lets a drive contribute newly-fetched chunks back to the CDN. This is the path that lets a small group of users seed a dehydrated drive without needing every chunk on the CDN at issuance time.

POST <baseUrl>/upload
Content-Type: application/octet-stream
Authorization: Bearer <CdnUploadToken>
Body: <chunk bytes — same locked v0x0001 format>

interface CdnUploadToken {
  fileId: string;
  chunkId: number;
  chunkHash: string;             // BLAKE3 of the chunk bytes (the CDN will verify)
  uploaderPubkey: string;
  expiresAt: number;             // 60-second expiry by default
  serverSignature: Uint8Array;   // Ed25519 by the drive's issuing server
}

The CDN:

1. Validates the token signature against the drive's known issuer key.
2. Hashes the body, compares to chunkHash — reject on mismatch.
3. Writes to the reversed-ID URL.

A CDN that doesn't validate signatures can decline writes (read-only mode); the drive falls back to peer-only seeding for that file.

Why server-issued tokens

Direct PUT (no token) lets anyone poison the CDN. CORS won't help — attackers bypass it. The token couples the upload to a server-attested intent and a specific (fileId, chunkId, chunkHash), and expires fast enough that a stolen token isn't useful.

---

6. Per-file policies in detail

autoDownload

• true → on hydrateFrom(), schedule a fetch for all chunks not present locally. Honors maxConcurrentFetches.
• false → wait until first read(). The read triggers the same fetch path; the user pays the latency on first access.

UX: a drive showing a directory listing renders progress for autoDownload=true files even if the user hasn't opened them yet.

meshAllowed

• true → drive-mesh may emit cd:request for this file's chunks.
• false → mesh layer skipped entirely; only local + CDN paths attempted.

Use meshAllowed=false for confidential files where peers shouldn't even know the file exists. The drive's gossip stops at hydration; a peer scraping the network bus never sees a request for this file.

cdnAllowed

• true → drive-transport-https may fetch from the CDN.
• false → CDN path skipped. Use for files restricted to peer-only distribution (e.g., user-shared content that the CDN doesn't host).

retention

The cache eviction policy. See ../system-review.md §9.

Mode	Behavior
'permanent'	Never evict. The drive's job is to keep these chunks forever.
'lru'	Eligible for global LRU eviction. When maxLocalCacheBytes is exceeded, evict oldest-touched chunks (NOT whole files) until under budget. The file remains visible with reduced chunksPresent; re-fetches on next read.
{ maxAgeMs }	Chunks older than N ms eligible for eviction at the next GC pass. Useful for ephemeral content (tournaments, event streams).

Eviction is at chunk granularity, never file granularity. The DirEntry, FileLocation, and manifest stay; only the bytes go. This means a "dehydrated" file gracefully transitions back to "needs-fetch" without index churn.

Defaults via GlobalPolicy.defaultPolicy

A FileTemplate.flags may omit any field; the missing fields fill from defaultPolicy. This keeps the template compact for drives where every file shares the same policy.

---

7. Encryption keys — inline vs retroactive

Inline (key shipped in template)

encryption: {
  algorithm: 'xchacha20-poly1305',
  keyDistribution: {
    kind: 'inline',
    wrappedKey: Uint8Array,   // wrapped to recipient's X25519 pubkey via crypto_box_seal
  },
}

On hydrateFrom(), the drive unwraps wrappedKey with the recipient's secret key (held in @suaveplan/crypto-vault). On success, the unwrapped fileKey lands in the drive's KeyRing under fileId. The lock icon flips from 🔒 to 🔓 immediately.

This is the simple case: drive operator decides at template-issuance time who gets the key, and the recipient has it from day one.

Retroactive (key arrives later)

encryption: {
  algorithm: 'xchacha20-poly1305',
  keyDistribution: {
    kind: 'retroactive',
    keyAuthority: '<peer-pubkey>',   // the peer that will issue grants
  },
}

On hydrateFrom(), the drive:

1. Records the file as encrypted-locked.
2. Subscribes to bus:key:grant events filtered by fileId.
3. Renders 🔒 in the UI.

When the key authority decides to grant access (or the user requests it via a side channel — e.g., paying for content, joining a group), it issues:

interface KeyGrantEnvelope extends MessageEnvelope {
  type: 'key:grant';
  payload: KeyGrantPayload;
}

interface KeyGrantPayload {
  fileId: string;
  recipient: string;            // recipient pubkey
  wrappedKey: Uint8Array;       // wrapped to recipient via crypto_box_seal
  issuer: string;               // keyAuthority pubkey from the template
  issuerSignature: Uint8Array;  // Ed25519 over (fileId, recipient, wrappedKey, expiresAt)
  expiresAt: number;            // optional — for time-limited grants
}

The drive validates:

1. issuer === template.encryption.keyAuthority for this file.
2. recipient === ourPubkey.
3. Ed25519 sig.
4. Date.now() < expiresAt (if set).

On success, unwrap, store in KeyRing, flip 🔒 → 🔓, emit drive:key:granted event. UI updates.

Revocation

A retroactive key grant can carry an expiresAt. After that timestamp:

• The drive evicts the unwrapped key from the KeyRing.
• The lock icon flips back 🔓 → 🔒.
• A new grant can be issued to extend access.

This is forward revocation only — anything the recipient already decrypted, they have. (See mesh-and-sharing.md §"What revocation actually does".) For destructive revocation, rotate the fileKey server-side and re-issue the template with a new manifestSig.

Trust roots

GlobalPolicy.trustedKeyAuthorities is the set of issuer pubkeys whose key:grant envelopes the drive will accept. Typically just [server-pubkey], but multi-issuer setups (a game with multiple content vendors) can list several. The set is fixed at template issuance — adding new issuers requires a fresh template.

---

8. The Drive.hydrateFrom() API

class Drive {
  hydrateFrom(template: DriveTemplate, options?: HydrateOptions): Promise<HydrateResult>;
}

interface HydrateOptions {
  signal?: AbortSignal;
  onProgress?: (event: HydrateProgressEvent) => void;
  preFetch?: 'none' | 'metadata-only' | 'auto-download-files';
}

interface HydrateResult {
  filesAdded: number;
  bytesScheduled: number;        // sum of sizes of autoDownload files
  unlocked: number;              // count of inline-key files unlocked
  locked: number;                // count of files awaiting retroactive grant
}

type HydrateProgressEvent =
  | { kind: 'verified' }
  | { kind: 'index-built'; entries: number }
  | { kind: 'fetch-scheduled'; fileId: string }
  | { kind: 'key-unlocked'; fileId: string };

preFetch controls hydration aggressiveness:

• 'none' — instantiate indexes only; fetch on first read.
• 'metadata-only' — fetch all manifests but no chunks. Lets the drive verify file integrity claims without the byte cost.
• 'auto-download-files' (default) — honor each file's policy.autoDownload.

---

9. Worked example — game asset bundle

A game wants to ship a 2 GB asset bundle to 10K players. The server:

1. Chunks every asset file at 64 KiB. ~32K chunks across 200 files.
2. Computes Ed25519-signed chunk format for each chunk; uploads to the CDN at /chunks.game.example.com/<reversed-id>/<file>_<chunk>.chunk.
3. Computes a DriveTemplate listing every asset path, fileId, size, manifest, encryption (none for assets, inline for premium DLC).
4. Signs the template; expiry 30 days; nonce per player.
5. Distributes the template via the game launcher.

On the player's machine:

const drive = new Drive(opfsStorage)
  .usePlugin(chunksPlugin({ ... }))
  .usePlugin(meshPlugin({ networkNode }))
  .usePlugin(transportWebrtcPlugin({ ... }))
  .usePlugin(transportHttpsPlugin({ baseUrl: 'https://chunks.game.example.com' }));

await drive.initialize();
const result = await drive.hydrateFrom(template, {
  preFetch: 'auto-download-files',
  onProgress: (e) => updateLoadingScreen(e),
});
// result.filesAdded === 200; result.bytesScheduled === 2_000_000_000;

What happens next:

• For files with autoDownload=true and meshAllowed=true: drive-mesh broadcasts cd:request. Other players already running the game (with chunks cached) respond. Bandwidth comes from peers, not the CDN.
• For files no peer has yet (newest player): drive-transport-https hits the CDN.
• For files with autoDownload=false: deferred until the player navigates to the corresponding game zone; first-access pays a fetch.
• For DLC with inline encryption: keys unwrap immediately; UI shows 🔓.
• For event-bound DLC (retroactive keys, e.g., "this skin unlocks during the seasonal event"): UI shows 🔒. When the season starts, the server broadcasts key:grant; UIs flip to 🔓.

The CDN's bandwidth bill drops dramatically once the player base has any seed. The mesh handles the cold-start of new geographic clusters. The drive UX is identical regardless of where bytes came from.

---

10. Test contract

The drive-template-v1 change folder will ship:

• tests/template-roundtrip.test.ts — canonicalSerialize / sign / verify cycle.
• tests/template-replay.test.ts — second hydrate with same nonce rejected.
• tests/template-expired.test.ts — expired template rejected with correct error code.
• tests/template-tampered.test.ts — bit flip anywhere in the template breaks the sig.
• tests/policy-meshallowed-false.test.ts — meshAllowed: false skips bus emit (verified via test bus interceptor).
• tests/policy-cdnallowed-false.test.ts — same for CDN.
• tests/retention-lru.test.ts — eviction respects maxLocalCacheBytes; permanent files never evicted.
• tests/inline-key-unlock.test.ts — hydrate flips lock icon; read() returns plaintext.
• tests/retroactive-key-grant.test.ts — incoming key:grant flips lock icon; read() works.
• tests/cdn-reverse-id-url.test.ts — URL builder produces correct path for various levels/padding combos.
• tests/cdn-bad-sig.test.ts — corrupted CDN response falls back to mesh.

---

11. Reading further

• mesh-protocol.md — the chunk-request/response/advertise envelopes that hydration uses.
• encryption.md — chunk format that the templates reference.
• mesh-and-sharing.md — partition-based sharing as the alternative to dehydrated drives.
• ../system-review.md — the review this doc resolves.
• ../drive-storage-network.md — the whole composition map; hydration plugs into the read pipeline at the mesh layer.