132 lines
10 KiB
Markdown
132 lines
10 KiB
Markdown
## Context
|
|
|
|
`@suaveplan/drive` consumes any object satisfying the structural `DriveStorage` view (read/write/delete/list/stat/streams + capabilities flags). Existing storage adapters (`storage-file`, `storage-http`, `storage-cdn-*`, `storage-browser-*`) all expose flat byte stores. None of them treat their content as a structured archive whose entries should themselves appear as paths in the drive.
|
|
|
|
Compressed archives have format-specific quirks that prevent a single monolithic adapter:
|
|
|
|
- **Container shape**: ZIP and 7z have a trailing central directory (random-access friendly); tar is purely sequential; gzip wraps a single stream with no directory at all.
|
|
- **Compression coupling**: tar separates archiving from compression (`.tar.gz`, `.tar.xz`, `.tar.zst`); zip/7z couple them per-entry; rar couples them per-volume.
|
|
- **Mutability**: zip and 7z support entry-level update via central-directory rewrite; tar requires full rewrite for deletes; gzip is single-stream and must be re-encoded entirely on any change; rar is patent-encumbered for writing.
|
|
- **Splitting**: zip uses `.z01`..`.zNN`+`.zip`; rar uses `.partN.rar` (RAR5) or `.r00`..`.r99` (RAR4); 7z uses `.7z.NNN`. Each has a different sibling-discovery and ordering rule.
|
|
- **Extension lies**: a `.zip` may be a 7z stream; a `.bin` may be tgz. The host can only trust magic bytes.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
|
|
- One installable package per archive format (zip, tar, gz, tgz, rar, 7z) plus one shared `drive-storage-archive-core`. Each is a separately versioned library.
|
|
- Each adapter implements `DriveStorage` so `new Drive(adapter)` works with zero drive-side changes.
|
|
- Content-sniff format from magic bytes; extension is advisory only.
|
|
- First-class split-archive support with sibling-volume resolution callback so the host storage (file, http, indexeddb) decides how to fetch siblings.
|
|
- Three explicit modes: `read-only`, `copy-on-write` (in-memory overlay + `flush()`), `rewrite` (write-through via host storage).
|
|
- Streaming reads (`ReadableStream<Uint8Array>`) for entries larger than memory.
|
|
- Password/encryption hook so credentials never live in the adapter — host supplies via callback.
|
|
- Conformance harness in `drive-storage-archive-core/testing` that every per-format adapter passes.
|
|
|
|
**Non-Goals:**
|
|
|
|
- Writing RAR — patent-encumbered, deliberately read-only.
|
|
- New compression formats outside the six listed (no `.lz4`, `.zst` standalone, `.bz2` standalone in this change — tgz codec slot will accept them later).
|
|
- Replacing `drive-plugin-chunks` chunk addressing; archive entries are not chunked at the drive level.
|
|
- Mounting archives inside archives transparently (nested mount is a host concern: `openArchiveStorage(openArchiveStorage(...))`).
|
|
- A unified `@suaveplan/storage-archive` umbrella package. Each format ships standalone (matches user directive).
|
|
|
|
## Decisions
|
|
|
|
### One package per format, plus a shared core
|
|
|
|
**Choice:** Six per-format packages (`drive-storage-{zip,tar,gz,tgz,rar,7z}`) + `drive-storage-archive-core` for shared primitives.
|
|
|
|
**Alternative considered:** Single `drive-storage-archive` umbrella package with internal format modules.
|
|
|
|
**Rationale:** Per-format isolates large codec dependencies (LZMA SDK is ~150KB, RAR unpacker is server-only) and lets consumers depend only on what they need. Matches existing repo pattern (`storage-cdn-bunnycdn`, `storage-cdn-rackspace`, etc.). User explicitly requested separate libraries.
|
|
|
|
### Format detection via magic bytes, extension as hint only
|
|
|
|
**Choice:** `sniffFormat(bytes: Uint8Array): ArchiveFormat | null` in `drive-storage-archive-core` inspects the first 512 bytes against a magic-byte registry: `PK\x03\x04` (zip), `Rar!\x1A\x07\x00` / `Rar!\x1A\x07\x01\x00` (rar4/5), `7z\xBC\xAF\x27\x1C` (7z), `\x1F\x8B` (gzip), and content-based tar detection (block 0 offset 257 = `"ustar"` for POSIX, or full-block heuristic for legacy v7).
|
|
|
|
**Rationale:** Users routinely mislabel archives. Extension drives default codec selection but is overridden by sniff result with a logged `ARCHIVE_FORMAT_MISMATCH` warning.
|
|
|
|
### Split-volume resolution via sibling-resolver callback
|
|
|
|
**Choice:** `openArchiveStorage(source, { resolveSibling: (basename, index) => Promise<Uint8Array | ReadableStream> })`.
|
|
|
|
**Alternative considered:** Hard-code sibling lookup against the host storage path.
|
|
|
|
**Rationale:** Host storage might be HTTP (siblings on different URLs), IndexedDB (siblings in different rows), or a P2P drive (siblings discovered via gossip). The adapter never assumes a filesystem. Each format ships a `defaultResolver(hostStorage, primaryPath)` helper using its own naming convention (`name.z01..zNN`, `name.partN.rar`, `name.7z.NNN`).
|
|
|
|
### Three explicit mutation modes
|
|
|
|
**Choice:** `mode: 'read-only' | 'copy-on-write' | 'rewrite'`.
|
|
|
|
**Rationale:**
|
|
|
|
- `read-only` is the safe default. RAR forces this.
|
|
- `copy-on-write` keeps the original archive bytes intact until `await adapter.flush()`. Mutations land in an in-memory overlay (entry → new bytes or tombstone). Suitable for batch edits.
|
|
- `rewrite` writes through on every mutation by re-encoding the archive into the host storage. Suitable for occasional single edits. Each format documents the cost: O(archive size) per mutation.
|
|
|
|
### Streaming entries via Web Streams
|
|
|
|
**Choice:** `getEntryStream(path): ReadableStream<Uint8Array>` returns a Web Streams `ReadableStream`. Bulk reads via `getEntry(path): Promise<Uint8Array>` are a convenience wrapper.
|
|
|
|
**Rationale:** Archives often hold multi-GB entries (DB dumps, media). The `drive` `createReadStream`/`createWriteStream` operations already negotiate native vs. polyfill — this slots in cleanly.
|
|
|
|
### Codec abstraction for tgz family
|
|
|
|
**Choice:** `@suaveplan/drive-storage-tgz` accepts a `codec: ArchiveCodec` parameter. Ships with `gzipCodec` from `@suaveplan/drive-storage-gz`. Bzip2/xz/zstd codecs are pluggable (out of scope for this change; slot exists).
|
|
|
|
**Rationale:** Avoids forcing every consumer to install xz/zstd deps when they only need `.tar.gz`.
|
|
|
|
### Error taxonomy
|
|
|
|
**Choice:** Each package owns its `*_CODES` registry via `defineCodes` (`error` package convention). `drive-storage-archive-core` owns `ARCHIVE_FORMAT_UNKNOWN`, `ARCHIVE_FORMAT_MISMATCH`, `ARCHIVE_SPLIT_MISSING_VOLUME`, `ARCHIVE_PASSWORD_REQUIRED`, `ARCHIVE_PASSWORD_INVALID`, `ARCHIVE_MODE_READ_ONLY`, `ARCHIVE_FLUSH_REQUIRED`, `ARCHIVE_CORRUPT`, `ARCHIVE_UNSUPPORTED_FEATURE`, `ARCHIVE_CODEC_MISSING`. Per-format packages add format-specific codes (e.g., `ZIP_ZIP64_REQUIRED`, `TAR_PAX_HEADER_INVALID`, `RAR_VERSION_UNSUPPORTED`, `SEVENZ_BCJ_FILTER_UNSUPPORTED`).
|
|
|
|
### Universal vs. server tier
|
|
|
|
- `drive-storage-archive-core`, `drive-storage-zip`, `drive-storage-tar`, `drive-storage-gz`, `drive-storage-tgz` → **universal** (pure-TS codecs: `fflate` for deflate, hand-rolled tar block parser, pure-TS gzip via `fflate`).
|
|
- `drive-storage-7z` → **universal** via wasm LZMA (preferred) or **server** via `lzma-native` binding (decided per perf test in tasks).
|
|
- `drive-storage-rar` → **server-only** (no royalty-free pure-TS RAR5 unpacker exists with full filter support; uses `node-unrar-js` wasm or `unrar` native binding). Browser builds resolve to a stub that throws `RAR_UNAVAILABLE_IN_BROWSER`.
|
|
|
|
### Public factory shape
|
|
|
|
```ts
|
|
import { openArchiveStorage } from "@suaveplan/drive-storage-archive-core";
|
|
|
|
const archiveStorage = await openArchiveStorage(hostBytes, {
|
|
format: "auto", // 'auto' | 'zip' | 'tar' | ...
|
|
mode: "copy-on-write",
|
|
password: async () => promptUser(),
|
|
resolveSibling: async (basename, index) => hostStorage.read(`${basename}.z${String(index).padStart(2, "0")}`),
|
|
});
|
|
|
|
const drive = new Drive(archiveStorage);
|
|
await drive.initialize();
|
|
await drive.write("inside-archive/notes.txt", new TextEncoder().encode("hello"));
|
|
await archiveStorage.flush(); // for copy-on-write
|
|
```
|
|
|
|
The factory does the sniff, picks the per-format adapter via dynamic import (so unused codecs aren't bundled), and returns the constructed `DriveStorage`. Each per-format package also exports its constructor directly (`ZipArchiveStorage`, etc.) for callers who don't need sniffing.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **RAR licensing** → Mitigation: read-only, vendored via permissive-license wasm unrar; license note in package README. Block writes at type level (`mode: 'read-only'` only).
|
|
- **Rewrite cost on tar/gz** → Mitigation: document O(archive size) cost in README; recommend `copy-on-write` + single `flush()` for batches; expose `pendingMutationCount` getter.
|
|
- **Password-encrypted entries leak metadata** → Mitigation: document that entry names + sizes are unencrypted in zip (by spec) and 7z header-encryption mode is supported as opt-in. RAR encrypted-headers supported.
|
|
- **Split-volume ordering ambiguity** → Mitigation: each format's `defaultResolver` enforces canonical ordering (`.z01..zNN..zip` final; `.partN.rar` ascending; `.7z.001..NNN`). Adapter validates volume signatures match before assembly; mismatched volumes → `ARCHIVE_SPLIT_MISSING_VOLUME` with which-volume detail.
|
|
- **Browser memory pressure on large archives** → Mitigation: lazy entry parsing (central directory only on open); stream-only API for >100MB entries; warn via logger if archive exceeds a configurable threshold.
|
|
- **Format sniff false positives** → Mitigation: magic-byte match plus structural validation (zip → can locate EOCD; tar → header checksum matches; gz → trailing CRC matches). Sniff returns `null` if structural check fails, falling back to extension.
|
|
- **Nested archives** → Mitigation: not solved here. Host composes adapters explicitly; documented as a recipe in `drive-storage-archive-core` README.
|
|
|
|
## Migration Plan
|
|
|
|
- New packages — no existing consumer migration required.
|
|
- Each package ships at `0.1.0`.
|
|
- Drive consumers gain archive support opportunistically by passing an archive storage where they previously passed a flat storage.
|
|
- Rollback: each package is independently removable. No drive-side changes mean no rollback risk to existing storage adapters.
|
|
|
|
## Open Questions
|
|
|
|
- 7z wasm LZMA performance vs. native: settle in tasks via benchmark; if wasm <50% of native throughput on a 100MB archive, ship server-only and provide a wasm browser variant later.
|
|
- Whether `drive-storage-archive-core` should expose a `suaveplan-archive` CLI bin (list/extract/test) or whether that belongs in a separate `archive-cli` package. Default: defer CLI to a follow-up change to keep this scope tight.
|
|
- ZIP AES encryption (WinZip AE-2) — implement now or follow-up? Default: implement read in v0.1.0, write in v0.2.0.
|