OpenSpec Change: add-storage-s3-multipart

Change ID: add-storage-s3-multipart
Package: @suaveplan/storage-s3 (modify)
Status: PROPOSED
Date: 2026-05-17
Design: ./design.md
Spec delta: ./specs/storage-s3/spec.md

Why

S3.set issues a single PutObjectCommand for every payload, which AWS hard-caps at 5 GiB and which leaves WAN throughput on the table for any object above ~16 MiB; S3.setPartial falls back to a read-modify-write that pulls the entire object into process memory for every write. The storage-set-partial change (archived 2026-04-28) explicitly scoped out the native multipart implementation, leaving storage-s3 as the only adapter without a real large-write path. S3.setMultipart exists but is a separate public method with serial parts and a single coarse error code, so neither set nor setPartial benefit from it. This change closes that gap so large writes and chunk-driven sparse-file workloads (e.g. @suaveplan/drive-plugin-chunks) run on the same fast multipart path as hand-written setMultipart callers, without changing the StorageLike surface.

What Changes

S3.set auto-promotes to multipart when value.byteLength exceeds a configurable threshold (default 16 MiB); below it, behaviour is unchanged.
S3.setPartial delegates to the multipart engine when offset === 0 and the write exceeds the threshold; non-zero-offset patches continue to use the read-modify-write fallback.
New internal engine (multipart/multipart-engine.ts, part-chunker.ts, worker-pool.ts) handles CreateMultipartUpload → bounded-parallel UploadPart (default concurrency 4) → CompleteMultipartUpload, with full abort propagation and guaranteed AbortMultipartUpload cleanup.
S3.setMultipart becomes a thin wrapper over the engine; its public signature is preserved.
S3_CODES gains MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED, MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED; legacy MULTIPART_FAILED remains registered as a back-compat alias.
New subpath exports: ./multipart/multipart-engine, ./multipart/part-chunker, ./multipart/worker-pool.
Integration tests run against the existing @suaveplan/testing-integration-garage-s3 Garage testcontainer fixture — no mocks.

Summary

Wire native S3 multipart upload into the set / setPartial happy path so large writes use CreateMultipartUpload + parallel UploadPart + CompleteMultipartUpload instead of either a single 5 GiB-capped PutObject or the read-modify-write defaultSetPartial fallback. Multipart is an internal optimization: the public StorageLike surface is unchanged; callers may tune behaviour via an opt-in { multipart: { partSize, concurrency, threshold } } option bag.

This is the deferral noted in openspec/archive/2026-04-28-storage-set-partial/proposal.md ("Out of scope — storage-s3 multipart-upload native impl — follow-up") and the storage-s3 row of that change's backend wiring table (setPartial → defaultSetPartial, partialWrites: false).

Current state

S3.set (packages/storage/storage-s3/src/s3/s3.ts:233) issues a single PutObjectCommand regardless of payload size. AWS rejects single PUTs > 5 GiB and recommends multipart for objects > ~100 MiB for throughput.
S3.setPartial (s3/s3.ts:480) delegates straight to AbstractStorage#defaultSetPartial, which reads the whole object, mutates a Uint8Array in process memory, then re-PUTs it. O(N) memory and bandwidth per call, unusable for sparse-chunk drive layouts on real-world object sizes.
S3.setMultipart (s3/s3.ts:602) already implements a correct CreateMPU + serial UploadPart + CompleteMPU loop, but: (a) it is a separate public method callers must opt into by name, so set / setPartial never benefit from it; (b) part uploads are strictly serial, leaving WAN throughput on the table; (c) only a single MULTIPART_FAILED code exists, hiding which stage failed.
S3.capabilities.partialWrites is false. Once multipart is wired through setPartial for whole-object writes, the flag stays false (S3 multipart is not random-access — it cannot patch a byte range of an existing object) but consumers gain a documented large-object path via the same method.

What changes

Auto-promote set to multipart above a threshold. When value.byteLength > options.multipart?.threshold ?? DEFAULT_MULTIPART_THRESHOLD (default 16 MiB), S3.set performs CreateMultipartUpload → parallel UploadPart → CompleteMultipartUpload internally instead of PutObject. Below the threshold, behaviour is unchanged.
Streaming input path. Add an internal helper setStream(key, stream, opts) that chunks a ReadableStream<Uint8Array> into parts on the fly without buffering the whole stream. setMultipart becomes a thin wrapper that delegates to this helper. The byte-array set path uses the same engine via a ReadableStream.from-style adapter so there is one canonical implementation.
Bounded parallel parts. Parts are uploaded with a worker-pool of options.multipart?.concurrency ?? DEFAULT_MULTIPART_CONCURRENCY (default 4) concurrent UploadPart calls. Order is preserved in the final MultipartUpload.Parts list by part number.
Abort + cleanup. Any failure or AbortSignal firing during parts MUST propagate to in-flight part workers (cancelling their underlying S3Client.send via abortSignal) and MUST issue exactly one AbortMultipartUploadCommand for the open UploadId. Cleanup is best-effort (logged, not re-thrown) but is mandatory — leaked MPUs accrue storage cost.
Finer error codes. MULTIPART_FAILED is split via defineCodes into MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED, MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED. MULTIPART_FAILED is retained as a backwards-compatibility alias on the registry (raised only by call sites that pre-date this change and were not migrated; the four new codes cover every multipart call site post-change).
setPartial for whole-object writes. When offset === 0 and the call semantically writes the entire object (caller intent expressed via bytes.byteLength being the whole object), setPartial delegates to the same multipart engine. Random-access patches (non-zero offset) continue to use defaultSetPartial's read-modify-write — multipart cannot patch a byte range and capabilities.partialWrites remains false to advertise this honestly.

Out of scope

Resumable uploads — persisting UploadId + completed-part list across process restarts. Callers needing resume should use a dedicated surface (future drive-plugin-resume or equivalent); this change keeps the StorageLike contract simple.
Random-access patching via multipart. S3 multipart cannot replace an arbitrary byte range of an existing object. capabilities.partialWrites stays false.
Server-side encryption / SSE-KMS option passthrough. Tracked separately.
Copy-part (UploadPartCopy) for large server-side copies. Tracked separately under S3.copy enhancements.

Impact

Affected dependents

Run bunx turbo ls --filter=...@suaveplan/storage-s3 to enumerate. The change is additive on the public surface (set / setPartial signatures unchanged; new options are opt-in; new error codes are siblings of an existing one), so the blast radius is small:

@suaveplan/storage — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: storage-s3 consumes AbstractStorage from storage and adds new behaviour internally; no storage surface changes.)
@suaveplan/drive-plugin-chunks — [x] tests run / [x] docs reviewed / [ ] no action needed (reason: declared consumer of setPartial via DriveStorage; benefits directly from the large-write path. Verify conformance suite still passes.)
@suaveplan/drive-plugin-cache — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: cache layer round-trips byte ranges, never crosses the multipart threshold in practice; surface unchanged.)
@suaveplan/storage-cdn / storage-cdn-* — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: CDN adapters do not consume storage-s3 directly.)

storage-s3 is not a foundation package per §14a, but it sits under drive-* chunk plumbing — the dependent sweep (§13 step 10, bunx turbo test --filter=...@suaveplan/storage-s3) MUST be run before archive.

Migration

No public API changes; existing callers see no behaviour difference below the multipart threshold. Callers who explicitly assert on S3StorageError.code === "S3.MULTIPART_FAILED" should widen the check to the new four-code set:

import { S3_CODES } from "@suaveplan/storage-s3";
const multipartCodes = new Set<string>([
    S3_CODES.MULTIPART_INIT_FAILED,
    S3_CODES.MULTIPART_PART_FAILED,
    S3_CODES.MULTIPART_COMPLETE_FAILED,
    S3_CODES.MULTIPART_ABORT_FAILED,
    S3_CODES.MULTIPART_FAILED, // legacy alias, still emitted by no call site post-change
]);
if (err instanceof S3StorageError && multipartCodes.has(err.code)) { … }

Search command for consumers:

grep -rln "MULTIPART_FAILED" packages/ | grep -v "/dist/\|/coverage/\|/storage-s3/"

9.2 KiB Raw Blame History