suaveplan/docs/storage/backends.md

Storage Backends

This is the catalogue of every concrete AbstractStorage implementation the platform ships or plans to ship. Each entry lists runtime, capabilities, migration status, and the distinct value it adds.

A backend is "shipped" only if it lives under packages/storage/<pkg>/ on the genesis branch. Anything else is referenced from ../suaveplan (the main branch) or planned.

Capability cheat-sheet

Backend	Runtime	partialReads	partialWrites	nativeStreams	atomicWrites	contentAddressable	Status
MemoryStorage	universal	✓	✓	✓ (zero-copy)	✓	✗	shipped
NamespacedStorage	universal	mirrors inner	mirrors inner	mirrors inner	mirrors inner	mirrors inner	shipped
S3 (storage-s3)	server	✓	✗ (RMW)	✓	✓ (single-PUT)	✗	shipped
OpfsStorage (storage-browser-opfs)	browser	✓	✓	✓	partial	✗	pending — main
IndexedDbStorage (storage-browser)	browser	✗ (RMW)	✗ (RMW)	✗	✓ (per-tx)	✗	pending — main
FileStorage (storage-file)	Node	✓	✓ (fs.write)	✓	✓ (rename)	✗	pending — main
SqliteStorage (planned)	Bun + browser	✓ (incrblob)	✓ (incrblob)	partial	✓ (per-tx)	✗	planned — wire to @suaveplan/db-sqlite
FatStorage (planned, FAT16/32)	universal (WASM)	✓	✓	✓ (Node) / emulated	✗	✗	planned — fatfs-wasm or fatfs
ExfatStorage (deferred)	universal	—	—	—	—	—	deferred — node-exfat is parser-only
Ext4Storage (deferred)	server (native)	✓	✓	✗	✓ (journal)	✗	deferred — lwext4 needs a JS binding + GPL audit
ContentAddressedStorage (planned)	wraps any	mirrors inner	✗	mirrors inner	inherited	✓	planned — derive key from BLAKE3(content)
GarageStorage	server (test fixture)	✓	✗ (uses S3)	✓	✓	✗	shipped via testing-integration-garage-s3

Backends in detail

MemoryStorage — @suaveplan/storage

In-process map-of-Uint8Array with TTL, OTel metrics, and zero-copy streaming. Used as the default for tests, ephemeral caches, and the in-browser fallback when no persistent backend is available. Capabilities: partialReads, partialWrites, nativeStreams, atomicWrites. Not content-addressable.

NamespacedStorage — @suaveplan/storage

Wraps another AbstractStorage and prefixes every key. Capability mirror: whatever the inner backend can do, the namespaced view can do. This is the primary tool for tenant/partition/share scoping.

S3 — @suaveplan/storage-s3

AWS-SDK-backed S3-compatible backend (works with AWS S3, Garage, MinIO, etc.). set is a single-PUT atomic write; setPartial falls back to RMW because S3 multipart upload requires the upload to be planned in advance and is not byte-addressable in the way setPartial demands. Adds createPresignedUrl(key, options) for delegated browser uploads/downloads.

OpfsStorage (planned — to be ported from main as storage-browser-opfs)

Origin Private File System inside the browser, accessed through the WASM FileSystemSyncAccessHandle API for true byte-addressable reads/writes. This is the right backend for in-browser drives that need real partial writes without round-tripping through a service worker. Capabilities expected: partialReads, partialWrites, nativeStreams. Not yet on genesis.

IndexedDbStorage (planned — to be ported as storage-browser)

IndexedDB key→Blob store. No native partial reads/writes; both fall back to RMW. Right for older browsers, very large numbers of small keys, and structured indexes. Atomic per-transaction.

FileStorage (planned — to be ported as storage-file)

Node fs.promises backed, one file per key. setPartial maps to fs.write(fd, buf, 0, len, offset). Atomic full writes via rename of a temp file. The straight-line backend for any server environment.

SqliteStorage (planned)

Built on @suaveplan/db-sqlite (already shipped). One blob column per key, with incremental_blob_open for byte-addressable reads/writes. The advantage over FileStorage is that you get a journal, transactions, atomic multi-key updates, and a single file to back up. Useful for embedded apps and the browser via @sqlite.org/sqlite-wasm.

FatStorage — FAT16/FAT32 via WASM

Two reference implementations live in /home/ubuntu/web/tmp/:

• fatfs-wasm (WASM over CHaN's FatFs, MIT). Production-quality. Supports the full surface — read, write, lseek, unlink, rename, mkdir, stat, dir iteration. Synchronous; whole disk is a Uint8Array. Recommended.
• fatfs (pure JS, Apache-2 + BSD-2). Async + native streams, but missing unlink/rename ("Coming Soon™"). Useful only if streaming is critical and you can ship the missing ops yourself.

A FatStorage adapter is a thin mapping:

setPartial(key, offset, bytes) → disk.open(key, WRITE|OPEN_ALWAYS); file.lseek(offset); file.write(bytes); file.close();
read(key) → disk.readFile(key)
delete(key) → disk.unlink(key)
list(prefix) → disk.openDir('/'); iterate; filter

Why FAT? Two real use cases:

1. Removable media simulation. A FAT-formatted Uint8Array can be mounted by a real OS — useful for exporting drive snapshots, generating bootable disk images, or interoperating with embedded devices.
2. Browser-side filesystems with a fixed-size budget. A FAT image inside OPFS gives you POSIX-ish semantics with predictable allocation and trivial backup (it's one blob).

Capabilities: partialReads, partialWrites. nativeStreams=true only for the fatfs (Node) variant; the WASM variant emulates streams via chunked read/write. Not atomic — FAT has no journal.

ExfatStorage — deferred

/home/ubuntu/web/tmp/node-exfat is a parser scaffold, not an FS. It can read VBR/AllocTable/Cluster but exposes no read/write/mkdir/unlink. Until a real exFAT JS binding lands, this backend is deferred. Genesis does not need it — FatStorage covers the FAT family for files <4 GiB; for large files inside an embedded FS, prefer Ext4Storage once it lands.

Ext4Storage — deferred

/home/ubuntu/web/tmp/lwext4 is a C library that supports the full ext2/3/4 surface (extents, sparse files, atomic rename, journal, hardlinks, symlinks, full POSIX permissions, case-sensitive paths). It needs:

1. A Node binding (node-gyp / FFI / N-API). ~2 weeks of work.
2. License clearance — lwext4 is mixed BSD-3 + GPL-2 (the xattr and extents modules are GPL). Either replace the GPL files or accept GPL-2 for the resulting wasm/binding.

Once both are resolved, the Ext4Storage adapter looks like FatStorage: thin map of setPartial → fseek+fwrite. Capabilities will be partialReads, partialWrites, atomicWrites (journaled).

ContentAddressedStorage — planned wrapper

Wraps any AbstractStorage and replaces the caller-supplied key with BLAKE3(content) (or SHA-256(content), configurable). set becomes idempotent for identical content (cheap dedup), delete(key) becomes a refcount, and capabilities.contentAddressable=true. This is the substrate for drive-level deduplication and for content-addressed chunk stores in network-plugin-content-delivery.

GarageStorage — test only

Garage is an S3-compatible distributed object store used as the integration-test target via @suaveplan/testing-integration-garage-s3. From the application's point of view it is just S3; the fixture spins up a Docker container, configures a layout, creates a bucket, and hands you S3Options you can drop into createS3(). This is how every drive integration test that needs a real network-attached object store gets one.

Choosing a backend

The drive layer is backend-agnostic, but you almost always know which backend is right for a given deployment:

Use case	Backend
Tests, ephemeral caches	MemoryStorage
Server-side production, single tenant	FileStorage or SqliteStorage
Server-side production, distributed	S3 (any S3-compatible target)
Browser, persistent, fast random access	OpfsStorage
Browser, structured queries / older browsers	IndexedDbStorage
Removable-media export, fixed-size budget, FAT compat	FatStorage
Content-addressed chunk pool	wrap any backend in ContentAddressedStorage
Multi-tenant scoping	wrap any backend in NamespacedStorage

You can compose: NamespacedStorage(ContentAddressedStorage(S3(...))) gives you per-tenant content-addressed S3 — the same construction used by drive-plugin-chunks for shared chunk pools.

On-disk shape comparison — 100-file drive

When the drive layer above writes 100 files of ~50 MiB each, the actual files-on-disk count varies dramatically by backend:

Backend	On-disk shape for 100 × 50 MiB drive	Layout chosen	Why
MemoryStorage	0 files (in-process map)	sparse-file	partialWrites=true
FileStorage (Node fs)	~202 files (100 chunk-files + 100 manifests + 2 indexes)	sparse-file	partialWrites=true
OpfsStorage (browser)	~202 OPFS handles	sparse-file	partialWrites=true
S3	~80,202 objects (one per chunk)	chunk-keys	partialWrites=false; RMW prohibitive
IndexedDbStorage	~80,202 IDB rows	chunk-keys	partialWrites=false
SqliteStorage	1 file (one .sqlite with ~80k blob rows)	sparse-file via incremental_blob_open	Single-file partition
FatStorage	1 file (FAT disk image, mountable by OS)	sparse-file inside FS-in-a-file	True partition file
ContentAddressedStorage (wrapper)	~80k unique chunks (deduped across files)	chunk-keys	Hashes coalesce identical content

So:

• FileStorage produces ~2 files per logical file (chunk-file + manifest).
• S3 produces ~800 objects per logical file (one per chunk + manifest).
• SqliteStorage and FatStorage produce one file total — pick these for atomic backup, mountable export, or fixed-size budgets.
• The drive's kind: 'inline' storage hint lets sub-1-KiB files skip the chunks pipeline entirely, reducing the count further.

The drive layer never produces "plaintext files mirroring user paths" on any backend, because that would leak structural metadata to whoever holds the bytes (S3 admins, FAT disk inspectors, etc.). Path mirroring is by-design absent. If plaintext path-mirror semantics are needed, skip the drive layer and write to FileStorage directly — the drive's job is to abstract over storage.

Reading further

• architecture.md — the contract every backend implements.
• implementation-status.md — port-from-main status.
• ../drive/plugins.md — how the chunks plugin picks layouts based on backend capabilities.
• ../drive/index-and-paths.md — the same comparison from the drive side, plus how the two indexes work.
• ../drive/dehydrated-drives.md — server-signed templates and CDN fallback that consume the storage layer.