# OpenSpec Change: add-storage-s3-multipart

- **Change ID**: add-storage-s3-multipart
- **Package**: `@suaveplan/storage-s3` (modify)
- **Status**: PROPOSED
- **Date**: 2026-05-17
- **Design**: ./design.md
- **Spec delta**: ./specs/storage-s3/spec.md

## Why

`S3.set` issues a single `PutObjectCommand` for every payload, which AWS hard-caps
at 5 GiB and which leaves WAN throughput on the table for any object above
~16 MiB; `S3.setPartial` falls back to a read-modify-write that pulls the entire
object into process memory for every write. The
`storage-set-partial` change (archived 2026-04-28) explicitly scoped out the
native multipart implementation, leaving `storage-s3` as the only adapter
without a real large-write path. `S3.setMultipart` exists but is a separate
public method with serial parts and a single coarse error code, so neither
`set` nor `setPartial` benefit from it. This change closes that gap so
large writes and chunk-driven sparse-file workloads (e.g.
`@suaveplan/drive-plugin-chunks`) run on the same fast multipart path as
hand-written `setMultipart` callers, without changing the `StorageLike` surface.

## What Changes

- `S3.set` auto-promotes to multipart when `value.byteLength` exceeds a
  configurable threshold (default 16 MiB); below it, behaviour is unchanged.
- `S3.setPartial` delegates to the multipart engine when `offset === 0` and
  the write exceeds the threshold; non-zero-offset patches continue to use
  the read-modify-write fallback.
- New internal engine (`multipart/multipart-engine.ts`, `part-chunker.ts`,
  `worker-pool.ts`) handles `CreateMultipartUpload` → bounded-parallel
  `UploadPart` (default concurrency 4) → `CompleteMultipartUpload`, with
  full abort propagation and guaranteed `AbortMultipartUpload` cleanup.
- `S3.setMultipart` becomes a thin wrapper over the engine; its public
  signature is preserved.
- `S3_CODES` gains `MULTIPART_INIT_FAILED`, `MULTIPART_PART_FAILED`,
  `MULTIPART_COMPLETE_FAILED`, `MULTIPART_ABORT_FAILED`; legacy
  `MULTIPART_FAILED` remains registered as a back-compat alias.
- New subpath exports: `./multipart/multipart-engine`, `./multipart/part-chunker`,
  `./multipart/worker-pool`.
- Integration tests run against the existing
  `@suaveplan/testing-integration-garage-s3` Garage testcontainer fixture —
  no mocks.

## Summary

Wire native S3 multipart upload into the `set` / `setPartial` happy path so large
writes use `CreateMultipartUpload` + parallel `UploadPart` + `CompleteMultipartUpload`
instead of either a single 5 GiB-capped `PutObject` or the read-modify-write
`defaultSetPartial` fallback. Multipart is an **internal optimization**: the public
`StorageLike` surface is unchanged; callers may tune behaviour via an opt-in
`{ multipart: { partSize, concurrency, threshold } }` option bag.

This is the deferral noted in
`openspec/archive/2026-04-28-storage-set-partial/proposal.md` ("Out of scope —
`storage-s3` multipart-upload native impl — follow-up") and the
`storage-s3` row of that change's backend wiring table (`setPartial` →
`defaultSetPartial`, `partialWrites: false`).

## Current state

- `S3.set` (`packages/storage/storage-s3/src/s3/s3.ts:233`) issues a single
  `PutObjectCommand` regardless of payload size. AWS rejects single PUTs > 5 GiB
  and recommends multipart for objects > ~100 MiB for throughput.
- `S3.setPartial` (`s3/s3.ts:480`) delegates straight to
  `AbstractStorage#defaultSetPartial`, which reads the whole object, mutates a
  `Uint8Array` in process memory, then re-PUTs it. O(N) memory and bandwidth per
  call, unusable for sparse-chunk drive layouts on real-world object sizes.
- `S3.setMultipart` (`s3/s3.ts:602`) already implements a correct CreateMPU +
  serial UploadPart + CompleteMPU loop, but: (a) it is a separate public method
  callers must opt into by name, so `set` / `setPartial` never benefit from it;
  (b) part uploads are strictly serial, leaving WAN throughput on the table;
  (c) only a single `MULTIPART_FAILED` code exists, hiding which stage failed.
- `S3.capabilities.partialWrites` is `false`. Once multipart is wired through
  `setPartial` for whole-object writes, the flag stays `false` (S3 multipart
  is **not** random-access — it cannot patch a byte range of an existing object)
  but consumers gain a documented large-object path via the same method.

## What changes

1. **Auto-promote `set` to multipart above a threshold.** When
   `value.byteLength > options.multipart?.threshold ?? DEFAULT_MULTIPART_THRESHOLD`
   (default 16 MiB), `S3.set` performs CreateMultipartUpload → parallel
   UploadPart → CompleteMultipartUpload internally instead of `PutObject`. Below
   the threshold, behaviour is unchanged.
2. **Streaming input path.** Add an internal helper `setStream(key, stream, opts)`
   that chunks a `ReadableStream<Uint8Array>` into parts on the fly without
   buffering the whole stream. `setMultipart` becomes a thin wrapper that
   delegates to this helper. The byte-array `set` path uses the same engine via
   a `ReadableStream.from`-style adapter so there is one canonical implementation.
3. **Bounded parallel parts.** Parts are uploaded with a worker-pool of
   `options.multipart?.concurrency ?? DEFAULT_MULTIPART_CONCURRENCY` (default 4)
   concurrent `UploadPart` calls. Order is preserved in the final
   `MultipartUpload.Parts` list by part number.
4. **Abort + cleanup.** Any failure or `AbortSignal` firing during parts MUST
   propagate to in-flight part workers (cancelling their underlying
   `S3Client.send` via `abortSignal`) and MUST issue exactly one
   `AbortMultipartUploadCommand` for the open `UploadId`. Cleanup is best-effort
   (logged, not re-thrown) but is mandatory — leaked MPUs accrue storage cost.
5. **Finer error codes.** `MULTIPART_FAILED` is split via `defineCodes` into
   `MULTIPART_INIT_FAILED`, `MULTIPART_PART_FAILED`, `MULTIPART_COMPLETE_FAILED`,
   `MULTIPART_ABORT_FAILED`. `MULTIPART_FAILED` is retained as a
   backwards-compatibility alias on the registry (raised only by call sites
   that pre-date this change and were not migrated; the four new codes cover
   every multipart call site post-change).
6. **`setPartial` for whole-object writes.** When `offset === 0` and the call
   semantically writes the entire object (caller intent expressed via
   `bytes.byteLength` being the whole object), `setPartial` delegates to the
   same multipart engine. Random-access patches (non-zero offset) continue to
   use `defaultSetPartial`'s read-modify-write — multipart cannot patch a byte
   range and `capabilities.partialWrites` remains `false` to advertise this
   honestly.

## Out of scope

- **Resumable uploads** — persisting `UploadId` + completed-part list across
  process restarts. Callers needing resume should use a dedicated surface
  (future `drive-plugin-resume` or equivalent); this change keeps the
  `StorageLike` contract simple.
- **Random-access patching via multipart.** S3 multipart cannot replace an
  arbitrary byte range of an existing object. `capabilities.partialWrites`
  stays `false`.
- **Server-side encryption / SSE-KMS option passthrough.** Tracked separately.
- **Copy-part (`UploadPartCopy`) for large server-side copies.** Tracked
  separately under `S3.copy` enhancements.

## Impact

### Affected dependents

Run `bunx turbo ls --filter=...@suaveplan/storage-s3` to enumerate. The
change is additive on the public surface (`set` / `setPartial` signatures
unchanged; new options are opt-in; new error codes are siblings of an existing
one), so the blast radius is small:

- `@suaveplan/storage` — [ ] tests run / [ ] docs reviewed / [x] no action needed
  (reason: `storage-s3` consumes `AbstractStorage` from `storage` and adds new
  behaviour internally; no `storage` surface changes.)
- `@suaveplan/drive-plugin-chunks` — [x] tests run / [x] docs reviewed / [ ] no action needed
  (reason: declared consumer of `setPartial` via `DriveStorage`; benefits
  directly from the large-write path. Verify conformance suite still passes.)
- `@suaveplan/drive-plugin-cache` — [ ] tests run / [ ] docs reviewed / [x] no action needed
  (reason: cache layer round-trips byte ranges, never crosses the multipart
  threshold in practice; surface unchanged.)
- `@suaveplan/storage-cdn` / `storage-cdn-*` — [ ] tests run / [ ] docs reviewed / [x] no action needed
  (reason: CDN adapters do not consume `storage-s3` directly.)

`storage-s3` is not a foundation package per §14a, but it sits under `drive-*`
chunk plumbing — the dependent sweep (§13 step 10,
`bunx turbo test --filter=...@suaveplan/storage-s3`) MUST be run before archive.

## Migration

No public API changes; existing callers see no behaviour difference below the
multipart threshold. Callers who explicitly assert on
`S3StorageError.code === "S3.MULTIPART_FAILED"` should widen the check to the
new four-code set:

```ts
import { S3_CODES } from "@suaveplan/storage-s3";
const multipartCodes = new Set<string>([
    S3_CODES.MULTIPART_INIT_FAILED,
    S3_CODES.MULTIPART_PART_FAILED,
    S3_CODES.MULTIPART_COMPLETE_FAILED,
    S3_CODES.MULTIPART_ABORT_FAILED,
    S3_CODES.MULTIPART_FAILED, // legacy alias, still emitted by no call site post-change
]);
if (err instanceof S3StorageError && multipartCodes.has(err.code)) { … }
```

Search command for consumers:

```bash
grep -rln "MULTIPART_FAILED" packages/ | grep -v "/dist/\|/coverage/\|/storage-s3/"
```