# Storage Backends

This is the catalogue of every concrete `AbstractStorage` implementation the platform ships or plans to ship. Each entry lists runtime, capabilities, migration status, and the distinct value it adds.

A backend is "shipped" only if it lives under `packages/storage/<pkg>/` on the `genesis` branch. Anything else is referenced from `../suaveplan` (the main branch) or planned.

## Capability cheat-sheet

| Backend | Runtime | partialReads | partialWrites | nativeStreams | atomicWrites | contentAddressable | Status |
| --- | --- | :-: | :-: | :-: | :-: | :-: | --- |
| `MemoryStorage` | universal | ✓ | ✓ | ✓ (zero-copy) | ✓ | ✗ | **shipped** |
| `NamespacedStorage` | universal | mirrors inner | mirrors inner | mirrors inner | mirrors inner | mirrors inner | **shipped** |
| `S3` (`storage-s3`) | server | ✓ | ✗ (RMW) | ✓ | ✓ (single-PUT) | ✗ | **shipped** |
| `OpfsStorage` (`storage-browser-opfs`) | browser | ✓ | ✓ | ✓ | partial | ✗ | pending — `main` |
| `IndexedDbStorage` (`storage-browser`) | browser | ✗ (RMW) | ✗ (RMW) | ✗ | ✓ (per-tx) | ✗ | pending — `main` |
| `FileStorage` (`storage-file`) | Node | ✓ | ✓ (`fs.write`) | ✓ | ✓ (rename) | ✗ | pending — `main` |
| `SqliteStorage` (planned) | Bun + browser | ✓ (incrblob) | ✓ (incrblob) | partial | ✓ (per-tx) | ✗ | planned — wire to `@suaveplan/db-sqlite` |
| `FatStorage` (planned, FAT16/32) | universal (WASM) | ✓ | ✓ | ✓ (Node) / emulated | ✗ | ✗ | planned — `fatfs-wasm` or `fatfs` |
| `ExfatStorage` (deferred) | universal | — | — | — | — | — | deferred — `node-exfat` is parser-only |
| `Ext4Storage` (deferred) | server (native) | ✓ | ✓ | ✗ | ✓ (journal) | ✗ | deferred — `lwext4` needs a JS binding + GPL audit |
| `ContentAddressedStorage` (planned) | wraps any | mirrors inner | ✗ | mirrors inner | inherited | ✓ | planned — derive key from `BLAKE3(content)` |
| `GarageStorage` | server (test fixture) | ✓ | ✗ (uses S3) | ✓ | ✓ | ✗ | shipped via `testing-integration-garage-s3` |

## Backends in detail

### `MemoryStorage` — `@suaveplan/storage`

In-process map-of-`Uint8Array` with TTL, OTel metrics, and zero-copy streaming. Used as the default for tests, ephemeral caches, and the in-browser fallback when no persistent backend is available. Capabilities: `partialReads`, `partialWrites`, `nativeStreams`, `atomicWrites`. Not content-addressable.

### `NamespacedStorage` — `@suaveplan/storage`

Wraps another `AbstractStorage` and prefixes every key. Capability mirror: whatever the inner backend can do, the namespaced view can do. This is the primary tool for tenant/partition/share scoping.

### `S3` — `@suaveplan/storage-s3`

AWS-SDK-backed S3-compatible backend (works with AWS S3, Garage, MinIO, etc.). `set` is a single-PUT atomic write; `setPartial` falls back to RMW because S3 multipart upload requires the upload to be planned in advance and is not byte-addressable in the way `setPartial` demands. Adds `createPresignedUrl(key, options)` for delegated browser uploads/downloads.

### `OpfsStorage` (planned — to be ported from `main` as `storage-browser-opfs`)

Origin Private File System inside the browser, accessed through the WASM `FileSystemSyncAccessHandle` API for true byte-addressable reads/writes. This is the right backend for in-browser drives that need real partial writes without round-tripping through a service worker. Capabilities expected: `partialReads`, `partialWrites`, `nativeStreams`. Not yet on genesis.

### `IndexedDbStorage` (planned — to be ported as `storage-browser`)

IndexedDB key→`Blob` store. No native partial reads/writes; both fall back to RMW. Right for older browsers, very large numbers of small keys, and structured indexes. Atomic per-transaction.

### `FileStorage` (planned — to be ported as `storage-file`)

Node `fs.promises` backed, one file per key. `setPartial` maps to `fs.write(fd, buf, 0, len, offset)`. Atomic full writes via `rename` of a temp file. The straight-line backend for any server environment.

### `SqliteStorage` (planned)

Built on `@suaveplan/db-sqlite` (already shipped). One blob column per key, with `incremental_blob_open` for byte-addressable reads/writes. The advantage over `FileStorage` is that you get a journal, transactions, atomic multi-key updates, and a single file to back up. Useful for embedded apps and the browser via `@sqlite.org/sqlite-wasm`.

### `FatStorage` — FAT16/FAT32 via WASM

Two reference implementations live in `/home/ubuntu/web/tmp/`:

- **`fatfs-wasm`** (WASM over CHaN's FatFs, MIT). Production-quality. Supports the full surface — `read`, `write`, `lseek`, `unlink`, `rename`, `mkdir`, `stat`, dir iteration. Synchronous; whole disk is a `Uint8Array`. Recommended.
- **`fatfs`** (pure JS, Apache-2 + BSD-2). Async + native streams, but missing `unlink`/`rename` ("Coming Soon™"). Useful only if streaming is critical and you can ship the missing ops yourself.

A `FatStorage` adapter is a *thin* mapping:

```text
setPartial(key, offset, bytes) → disk.open(key, WRITE|OPEN_ALWAYS); file.lseek(offset); file.write(bytes); file.close();
read(key) → disk.readFile(key)
delete(key) → disk.unlink(key)
list(prefix) → disk.openDir('/'); iterate; filter
```

Why FAT? Two real use cases:

1. **Removable media simulation.** A FAT-formatted `Uint8Array` can be mounted by a real OS — useful for exporting drive snapshots, generating bootable disk images, or interoperating with embedded devices.
2. **Browser-side filesystems with a fixed-size budget.** A FAT image inside OPFS gives you POSIX-ish semantics with predictable allocation and trivial backup (it's one blob).

Capabilities: `partialReads`, `partialWrites`. `nativeStreams=true` only for the `fatfs` (Node) variant; the WASM variant emulates streams via chunked `read`/`write`. **Not** atomic — FAT has no journal.

### `ExfatStorage` — deferred

`/home/ubuntu/web/tmp/node-exfat` is a parser scaffold, not an FS. It can read VBR/AllocTable/Cluster but exposes no `read`/`write`/`mkdir`/`unlink`. Until a real exFAT JS binding lands, this backend is deferred. Genesis does not need it — `FatStorage` covers the FAT family for files <4 GiB; for large files inside an embedded FS, prefer `Ext4Storage` once it lands.

### `Ext4Storage` — deferred

`/home/ubuntu/web/tmp/lwext4` is a C library that supports the full ext2/3/4 surface (extents, sparse files, atomic rename, journal, hardlinks, symlinks, full POSIX permissions, case-sensitive paths). It needs:

1. A Node binding (node-gyp / FFI / N-API). ~2 weeks of work.
2. License clearance — `lwext4` is mixed BSD-3 + GPL-2 (the xattr and extents modules are GPL). Either replace the GPL files or accept GPL-2 for the resulting wasm/binding.

Once both are resolved, the `Ext4Storage` adapter looks like `FatStorage`: thin map of `setPartial → fseek+fwrite`. Capabilities will be `partialReads`, `partialWrites`, `atomicWrites` (journaled).

### `ContentAddressedStorage` — planned wrapper

Wraps any `AbstractStorage` and replaces the caller-supplied key with `BLAKE3(content)` (or `SHA-256(content)`, configurable). `set` becomes idempotent for identical content (cheap dedup), `delete(key)` becomes a refcount, and `capabilities.contentAddressable=true`. This is the substrate for drive-level deduplication and for content-addressed chunk stores in `network-plugin-content-delivery`.

### `GarageStorage` — test only

Garage is an S3-compatible distributed object store used as the integration-test target via `@suaveplan/testing-integration-garage-s3`. From the application's point of view it is just `S3`; the fixture spins up a Docker container, configures a layout, creates a bucket, and hands you `S3Options` you can drop into `createS3()`. This is how every drive integration test that needs a real network-attached object store gets one.

## Choosing a backend

The drive layer is backend-agnostic, but you almost always know which backend is right for a given deployment:

| Use case | Backend |
| --- | --- |
| Tests, ephemeral caches | `MemoryStorage` |
| Server-side production, single tenant | `FileStorage` or `SqliteStorage` |
| Server-side production, distributed | `S3` (any S3-compatible target) |
| Browser, persistent, fast random access | `OpfsStorage` |
| Browser, structured queries / older browsers | `IndexedDbStorage` |
| Removable-media export, fixed-size budget, FAT compat | `FatStorage` |
| Content-addressed chunk pool | wrap any backend in `ContentAddressedStorage` |
| Multi-tenant scoping | wrap any backend in `NamespacedStorage` |

You can compose: `NamespacedStorage(ContentAddressedStorage(S3(...)))` gives you per-tenant content-addressed S3 — the same construction used by `drive-plugin-chunks` for shared chunk pools.

## On-disk shape comparison — 100-file drive

When the drive layer above writes 100 files of ~50 MiB each, the actual files-on-disk count varies dramatically by backend:

| Backend | On-disk shape for 100 × 50 MiB drive | Layout chosen | Why |
|---|---|---|---|
| `MemoryStorage` | 0 files (in-process map) | sparse-file | `partialWrites=true` |
| `FileStorage` (Node fs) | **~202 files** (100 chunk-files + 100 manifests + 2 indexes) | sparse-file | `partialWrites=true` |
| `OpfsStorage` (browser) | ~202 OPFS handles | sparse-file | `partialWrites=true` |
| `S3` | **~80,202 objects** (one per chunk) | chunk-keys | `partialWrites=false`; RMW prohibitive |
| `IndexedDbStorage` | ~80,202 IDB rows | chunk-keys | `partialWrites=false` |
| `SqliteStorage` | **1 file** (one `.sqlite` with ~80k blob rows) | sparse-file via `incremental_blob_open` | Single-file partition |
| `FatStorage` | **1 file** (FAT disk image, mountable by OS) | sparse-file inside FS-in-a-file | True partition file |
| `ContentAddressedStorage` (wrapper) | ~80k unique chunks (deduped across files) | chunk-keys | Hashes coalesce identical content |

So:

- `FileStorage` produces ~2 files per logical file (chunk-file + manifest).
- `S3` produces ~800 objects per logical file (one per chunk + manifest).
- `SqliteStorage` and `FatStorage` produce one file total — pick these for atomic backup, mountable export, or fixed-size budgets.
- The drive's `kind: 'inline'` storage hint lets sub-1-KiB files skip the chunks pipeline entirely, reducing the count further.

The drive layer never produces "plaintext files mirroring user paths" on any backend, because that would leak structural metadata to whoever holds the bytes (S3 admins, FAT disk inspectors, etc.). Path mirroring is by-design absent. If plaintext path-mirror semantics are needed, skip the drive layer and write to `FileStorage` directly — the drive's job is to abstract over storage.

## Reading further

- [`architecture.md`](./architecture.md) — the contract every backend implements.
- [`implementation-status.md`](./implementation-status.md) — port-from-main status.
- [`../drive/plugins.md`](../drive/plugins.md) — how the chunks plugin picks layouts based on backend capabilities.
- [`../drive/index-and-paths.md`](../drive/index-and-paths.md) — the same comparison from the drive side, plus how the two indexes work.
- [`../drive/dehydrated-drives.md`](../drive/dehydrated-drives.md) — server-signed templates and CDN fallback that consume the storage layer.