suaveplan/openspec/changes/add-drive-storage-archive/design.md

Context

@suaveplan/drive consumes any object satisfying the structural DriveStorage view (read/write/delete/list/stat/streams + capabilities flags). Existing storage adapters (storage-file, storage-http, storage-cdn-*, storage-browser-*) all expose flat byte stores. None of them treat their content as a structured archive whose entries should themselves appear as paths in the drive.

Compressed archives have format-specific quirks that prevent a single monolithic adapter:

• Container shape: ZIP and 7z have a trailing central directory (random-access friendly); tar is purely sequential; gzip wraps a single stream with no directory at all.
• Compression coupling: tar separates archiving from compression (.tar.gz, .tar.xz, .tar.zst); zip/7z couple them per-entry; rar couples them per-volume.
• Mutability: zip and 7z support entry-level update via central-directory rewrite; tar requires full rewrite for deletes; gzip is single-stream and must be re-encoded entirely on any change; rar is patent-encumbered for writing.
• Splitting: zip uses .z01...zNN+.zip; rar uses .partN.rar (RAR5) or .r00...r99 (RAR4); 7z uses .7z.NNN. Each has a different sibling-discovery and ordering rule.
• Extension lies: a .zip may be a 7z stream; a .bin may be tgz. The host can only trust magic bytes.

Goals / Non-Goals

Goals:

• One installable package per archive format (zip, tar, gz, tgz, rar, 7z) plus one shared drive-storage-archive-core. Each is a separately versioned library.
• Each adapter implements DriveStorage so new Drive(adapter) works with zero drive-side changes.
• Content-sniff format from magic bytes; extension is advisory only.
• First-class split-archive support with sibling-volume resolution callback so the host storage (file, http, indexeddb) decides how to fetch siblings.
• Three explicit modes: read-only, copy-on-write (in-memory overlay + flush()), rewrite (write-through via host storage).
• Streaming reads (ReadableStream<Uint8Array>) for entries larger than memory.
• Password/encryption hook so credentials never live in the adapter — host supplies via callback.
• Conformance harness in drive-storage-archive-core/testing that every per-format adapter passes.

Non-Goals:

• Writing RAR — patent-encumbered, deliberately read-only.
• New compression formats outside the six listed (no .lz4, .zst standalone, .bz2 standalone in this change — tgz codec slot will accept them later).
• Replacing drive-plugin-chunks chunk addressing; archive entries are not chunked at the drive level.
• Mounting archives inside archives transparently (nested mount is a host concern: openArchiveStorage(openArchiveStorage(...))).
• A unified @suaveplan/storage-archive umbrella package. Each format ships standalone (matches user directive).

Decisions

One package per format, plus a shared core

Choice: Six per-format packages (drive-storage-{zip,tar,gz,tgz,rar,7z}) + drive-storage-archive-core for shared primitives.

Alternative considered: Single drive-storage-archive umbrella package with internal format modules.

Rationale: Per-format isolates large codec dependencies (LZMA SDK is ~150KB, RAR unpacker is server-only) and lets consumers depend only on what they need. Matches existing repo pattern (storage-cdn-bunnycdn, storage-cdn-rackspace, etc.). User explicitly requested separate libraries.

Format detection via magic bytes, extension as hint only

Choice: sniffFormat(bytes: Uint8Array): ArchiveFormat | null in drive-storage-archive-core inspects the first 512 bytes against a magic-byte registry: PK\x03\x04 (zip), Rar!\x1A\x07\x00 / Rar!\x1A\x07\x01\x00 (rar4/5), 7z\xBC\xAF\x27\x1C (7z), \x1F\x8B (gzip), and content-based tar detection (block 0 offset 257 = "ustar" for POSIX, or full-block heuristic for legacy v7).

Rationale: Users routinely mislabel archives. Extension drives default codec selection but is overridden by sniff result with a logged ARCHIVE_FORMAT_MISMATCH warning.

Split-volume resolution via sibling-resolver callback

Choice: openArchiveStorage(source, { resolveSibling: (basename, index) => Promise<Uint8Array | ReadableStream> }).

Alternative considered: Hard-code sibling lookup against the host storage path.

Rationale: Host storage might be HTTP (siblings on different URLs), IndexedDB (siblings in different rows), or a P2P drive (siblings discovered via gossip). The adapter never assumes a filesystem. Each format ships a defaultResolver(hostStorage, primaryPath) helper using its own naming convention (name.z01..zNN, name.partN.rar, name.7z.NNN).

Three explicit mutation modes

Choice: mode: 'read-only' | 'copy-on-write' | 'rewrite'.

Rationale:

• read-only is the safe default. RAR forces this.
• copy-on-write keeps the original archive bytes intact until await adapter.flush(). Mutations land in an in-memory overlay (entry → new bytes or tombstone). Suitable for batch edits.
• rewrite writes through on every mutation by re-encoding the archive into the host storage. Suitable for occasional single edits. Each format documents the cost: O(archive size) per mutation.

Streaming entries via Web Streams

Choice: getEntryStream(path): ReadableStream<Uint8Array> returns a Web Streams ReadableStream. Bulk reads via getEntry(path): Promise<Uint8Array> are a convenience wrapper.

Rationale: Archives often hold multi-GB entries (DB dumps, media). The drive createReadStream/createWriteStream operations already negotiate native vs. polyfill — this slots in cleanly.

Codec abstraction for tgz family

Choice: @suaveplan/drive-storage-tgz accepts a codec: ArchiveCodec parameter. Ships with gzipCodec from @suaveplan/drive-storage-gz. Bzip2/xz/zstd codecs are pluggable (out of scope for this change; slot exists).

Rationale: Avoids forcing every consumer to install xz/zstd deps when they only need .tar.gz.

Error taxonomy

Choice: Each package owns its *_CODES registry via defineCodes (error package convention). drive-storage-archive-core owns ARCHIVE_FORMAT_UNKNOWN, ARCHIVE_FORMAT_MISMATCH, ARCHIVE_SPLIT_MISSING_VOLUME, ARCHIVE_PASSWORD_REQUIRED, ARCHIVE_PASSWORD_INVALID, ARCHIVE_MODE_READ_ONLY, ARCHIVE_FLUSH_REQUIRED, ARCHIVE_CORRUPT, ARCHIVE_UNSUPPORTED_FEATURE, ARCHIVE_CODEC_MISSING. Per-format packages add format-specific codes (e.g., ZIP_ZIP64_REQUIRED, TAR_PAX_HEADER_INVALID, RAR_VERSION_UNSUPPORTED, SEVENZ_BCJ_FILTER_UNSUPPORTED).

Universal vs. server tier

• drive-storage-archive-core, drive-storage-zip, drive-storage-tar, drive-storage-gz, drive-storage-tgz → universal (pure-TS codecs: fflate for deflate, hand-rolled tar block parser, pure-TS gzip via fflate).
• drive-storage-7z → universal via wasm LZMA (preferred) or server via lzma-native binding (decided per perf test in tasks).
• drive-storage-rar → server-only (no royalty-free pure-TS RAR5 unpacker exists with full filter support; uses node-unrar-js wasm or unrar native binding). Browser builds resolve to a stub that throws RAR_UNAVAILABLE_IN_BROWSER.

Public factory shape

import { openArchiveStorage } from "@suaveplan/drive-storage-archive-core";

const archiveStorage = await openArchiveStorage(hostBytes, {
  format: "auto",                 // 'auto' | 'zip' | 'tar' | ...
  mode: "copy-on-write",
  password: async () => promptUser(),
  resolveSibling: async (basename, index) => hostStorage.read(`${basename}.z${String(index).padStart(2, "0")}`),
});

const drive = new Drive(archiveStorage);
await drive.initialize();
await drive.write("inside-archive/notes.txt", new TextEncoder().encode("hello"));
await archiveStorage.flush();    // for copy-on-write

The factory does the sniff, picks the per-format adapter via dynamic import (so unused codecs aren't bundled), and returns the constructed DriveStorage. Each per-format package also exports its constructor directly (ZipArchiveStorage, etc.) for callers who don't need sniffing.

Risks / Trade-offs

• RAR licensing → Mitigation: read-only, vendored via permissive-license wasm unrar; license note in package README. Block writes at type level (mode: 'read-only' only).
• Rewrite cost on tar/gz → Mitigation: document O(archive size) cost in README; recommend copy-on-write + single flush() for batches; expose pendingMutationCount getter.
• Password-encrypted entries leak metadata → Mitigation: document that entry names + sizes are unencrypted in zip (by spec) and 7z header-encryption mode is supported as opt-in. RAR encrypted-headers supported.
• Split-volume ordering ambiguity → Mitigation: each format's defaultResolver enforces canonical ordering (.z01..zNN..zip final; .partN.rar ascending; .7z.001..NNN). Adapter validates volume signatures match before assembly; mismatched volumes → ARCHIVE_SPLIT_MISSING_VOLUME with which-volume detail.
• Browser memory pressure on large archives → Mitigation: lazy entry parsing (central directory only on open); stream-only API for >100MB entries; warn via logger if archive exceeds a configurable threshold.
• Format sniff false positives → Mitigation: magic-byte match plus structural validation (zip → can locate EOCD; tar → header checksum matches; gz → trailing CRC matches). Sniff returns null if structural check fails, falling back to extension.
• Nested archives → Mitigation: not solved here. Host composes adapters explicitly; documented as a recipe in drive-storage-archive-core README.

Migration Plan

• New packages — no existing consumer migration required.
• Each package ships at 0.1.0.
• Drive consumers gain archive support opportunistically by passing an archive storage where they previously passed a flat storage.
• Rollback: each package is independently removable. No drive-side changes mean no rollback risk to existing storage adapters.

Open Questions

• 7z wasm LZMA performance vs. native: settle in tasks via benchmark; if wasm <50% of native throughput on a 100MB archive, ship server-only and provide a wasm browser variant later.
• Whether drive-storage-archive-core should expose a suaveplan-archive CLI bin (list/extract/test) or whether that belongs in a separate archive-cli package. Default: defer CLI to a follow-up change to keep this scope tight.
• ZIP AES encryption (WinZip AE-2) — implement now or follow-up? Default: implement read in v0.1.0, write in v0.2.0.