sparse-file

Purpose

Implements the sparse-file chunk layout that stores all chunks of a file as offset-addressable bytes inside a single sparse storage object. Chunks are written via storage.setPartial(key, offset, bytes) and read back via storage.getPartial. A side-band bitmap key records which chunks have been written so callers can answer hasChunks / availableChunks without paging the full file.

Features

createSparseFileLayout(storage): ChunkLayout — factory that throws CHUNKS.LAYOUT_INCOMPATIBLE if storage.capabilities.partialWrites !== true.
preallocate(storage, fileId, totalChunks, chunkSize, lastPayloadLen) — sets the file size up-front via a single storage.set call so subsequent setPartial writes never extend the underlying object.
Bitmap state tracked in bitmap:<fileIdHex> keys, packed eight bits per byte.
Per-fileId WeakMap caches keep chunkSize and totalChunks reachable inside the layout's read path without forcing every call to consult the manifest.

Usage: Basic Example

import { Drive } from "@suaveplan/drive";
import { chunksPlugin } from "@suaveplan/drive-plugin-chunks";

const drive = new Drive(sparseStorage);
await drive.initialize();
const chunks = await drive.usePlugin(
    chunksPlugin({ serverPrivKey, serverPubKey, layout: "sparse-file" })
);

Usage: Advanced Example

import {
    createSparseFileLayout,
    preallocate,
} from "@suaveplan/drive-plugin-chunks";

const layout = createSparseFileLayout(storage);
await preallocate(storage, fileId, totalChunks, chunkSize, lastPayloadLen);
for (const c of encryptedChunks) {
    await layout.putChunk(fileId, c.metadata.chunkId, c.rawBytes);
}
const present = await layout.hasChunks(fileId);

API Reference

createSparseFileLayout(storage: DriveStorage): ChunkLayout — name "sparse-file".
preallocate(storage, fileId, totalChunks, chunkSize, lastPayloadLen): Promise<void> — populates the WeakMap caches and zero-fills the file/bitmap keys.
Layout methods mirror chunk-keys: putChunk, getChunk, hasChunks, availableChunks.

Implementation Notes

The total file length is computed as (totalChunks - 1) * chunkSize + HEADER_SIZE + lastPayloadLen, matching the encrypt-stream output shape. The 164 in the inline arithmetic is HEADER_SIZE from header.ts. The bitmap is grown lazily inside setBit so the layout can survive a missing or truncated bitmap key (e.g. partial restore from backup) without losing chunks. getChunk returns undefined for missing files (getItemSize === null) and trusts getPartial for present-but-not-yet-populated regions.

2.6 KiB Raw Blame History