suaveplan/openspec/changes/complete-storage-s3/design.md

# Design: complete-storage-s3

Non-normative. The normative contract is `./specs/storage-s3/spec.md`.

## Goals

1. **Full S3 protocol parity** — every observable S3 capability that a
   production consumer needs is reachable from `@suaveplan/storage-s3` via
   either the `AbstractStorage` surface (whole-object IO) or one of the
   adapter's S3-specific extension methods (presign, ACLs, tags, versioning,
   lifecycle, CORS, bucket ops).
2. **Tier-honest capability flags** — `capabilities.partialReads = true`
   (native Range), `partialWrites = false` (S3 has no byte-range PUT),
   `nativeStreams = true`, `atomicWrites = true`, `contentAddressable = false`.
3. **Real-service tests only** — every multi-step path is exercised against
   the Garage testcontainer via
   `@suaveplan/testing-integration-garage-s3`.
4. **No mocks** — failure injection uses a wrapping `S3ClientFactory` that
   intercepts specific commands; the underlying transport stays real.
5. **Composable** — pluggable HTTP transport, pluggable signer, retry via
   `@suaveplan/retry`. Hand-rolled `withRetry` is removed.
6. **100/100/100/100** coverage gate per project rule §7.

## File layout

```text
packages/storage/storage-s3/src/
├── s3/
│   ├── s3.ts                       # top-level S3 class — orchestrates extensions
│   ├── s3.md
│   ├── s3.test.ts
│   ├── s3-branches.test.ts
│   └── index.ts                    # explicit named re-exports
├── core/                           # AbstractStorage methods
│   ├── get/
│   │   ├── get.ts                  # native Range + conditional + SSE-C
│   │   ├── get.md
│   │   ├── get.test.ts
│   │   └── index.ts
│   ├── set/
│   │   ├── set.ts                  # auto-promote, metadata, tagging, SSE, ACL
│   │   ├── set.md
│   │   ├── set.test.ts
│   │   └── index.ts
│   ├── set-stream/
│   │   ├── set-stream.ts
│   │   ├── set-stream.md
│   │   ├── set-stream.test.ts
│   │   └── index.ts
│   ├── set-partial/
│   ├── delete/
│   ├── head/                       # head/stat with versionId + checksum
│   ├── list/                       # listObjectsV2 paginator
│   └── exists/
├── multipart/                      # multipart engine (set + setStream + copy)
│   ├── multipart-engine.ts
│   ├── multipart-engine.md
│   ├── multipart-engine.test.ts
│   ├── part-chunker.ts             # ReadableStream → AsyncIterable<PartInput>
│   ├── part-chunker.md
│   ├── part-chunker.test.ts
│   ├── worker-pool.ts              # bounded-concurrency scheduler + abort
│   ├── worker-pool.md
│   ├── worker-pool.test.ts
│   ├── upload-part-copy.ts         # server-side copy of > 5 GiB objects
│   ├── upload-part-copy.md
│   ├── upload-part-copy.test.ts
│   └── index.ts
├── presign/
│   ├── get-signed-get-url.ts
│   ├── get-signed-put-url.ts
│   ├── get-signed-delete-url.ts
│   ├── presign-options.ts          # TTL + headers + content-type pin
│   └── index.ts
├── sse/
│   ├── sse.ts                      # SSE-S3 / SSE-KMS / SSE-C param builder
│   ├── sse.md
│   ├── sse.test.ts
│   └── index.ts
├── acl/
│   ├── acl.ts                      # canned + grant policies; get/put object ACL
│   ├── acl.md
│   ├── acl.test.ts
│   └── index.ts
├── copy/
│   ├── copy.ts                     # copyObject + cross-bucket
│   ├── copy.md
│   ├── copy.test.ts
│   └── index.ts
├── tagging/
│   ├── tagging.ts                  # get/put/delete + inline-at-set encoding
│   ├── tagging.md
│   ├── tagging.test.ts
│   └── index.ts
├── versioning/
│   ├── versioning.ts               # list versions + versionId on get/head/delete
│   ├── versioning.md
│   ├── versioning.test.ts
│   └── index.ts
├── lifecycle/
│   ├── lifecycle.ts
│   ├── lifecycle.md
│   ├── lifecycle.test.ts
│   └── index.ts
├── cors/
│   ├── cors.ts
│   ├── cors.md
│   ├── cors.test.ts
│   └── index.ts
├── bucket/
│   ├── bucket.ts                   # create/delete/exists/region/listV2 paginator
│   ├── bucket.md
│   ├── bucket.test.ts
│   └── index.ts
├── conditional/
│   ├── conditional.ts              # If-Match / If-None-Match / If-*Modified-Since
│   ├── conditional.md
│   ├── conditional.test.ts
│   └── index.ts
├── checksum/
│   ├── checksum.ts                 # SHA256 + CRC32C integrity guards
│   ├── checksum.md
│   ├── checksum.test.ts
│   └── index.ts
├── transport/
│   ├── transport.ts                # FetchLike adapter
│   ├── transport.md
│   ├── transport.test.ts
│   └── index.ts
├── signer/
│   ├── signer.ts                   # SigV4 plug-point
│   ├── signer.md
│   ├── signer.test.ts
│   └── index.ts
├── retry/
│   ├── retry.ts                    # composes @suaveplan/retry
│   ├── retry.md
│   ├── retry.test.ts
│   └── index.ts
├── constants/
│   └── index.ts                    # all DEFAULT_* + MIN_PART_SIZE + thresholds
├── error-codes.ts                  # ~50 STORAGE_S3.* codes
├── error-codes.md
├── types/
│   ├── s3.types.ts
│   ├── s3.types.md
│   ├── sse.types.ts
│   ├── acl.types.ts
│   ├── tagging.types.ts
│   ├── lifecycle.types.ts
│   ├── cors.types.ts
│   ├── conditional.types.ts
│   ├── checksum.types.ts
│   └── index.ts
├── utils/
│   ├── credential-validator.ts
│   ├── build-key.ts
│   ├── error-mapper.ts             # S3ServiceException → S3StorageError
│   └── index.ts
└── index.ts                        # root barrel — namespace exports

packages/storage/storage-s3/tests/integration/
├── core.test.ts                    # get/set/delete/list/head against Garage
├── multipart.test.ts               # auto-promote + abort + cleanup + UploadPartCopy
├── stream.test.ts                  # setStream
├── presign.test.ts                 # all three presigned variants
├── sse.test.ts                     # SSE-S3 / SSE-KMS / SSE-C
├── acl.test.ts                     # canned + grant + get/put
├── copy.test.ts                    # within-bucket + cross-bucket
├── tagging.test.ts
├── versioning.test.ts
├── lifecycle.test.ts
├── cors.test.ts
├── bucket.test.ts                  # create / delete / exists / region / listV2
├── conditional.test.ts             # 412 / 304 paths
├── range.test.ts
└── checksum.test.ts                # SHA256 + CRC32C verify + mismatch
```

## Capability flags

```ts
public override readonly capabilities: StorageCapabilities = {
    partialReads: true,    // native Range GET via getPartial
    partialWrites: false,  // S3 has no byte-range PUT; multipart is whole-object
    nativeStreams: true,   // setStream + range GET stream
    atomicWrites: true,    // S3 PUT semantics are atomic relative to readers
    contentAddressable: false,
};
```

## Trigger thresholds

| Constant | Default | Min | Max | Rationale |
|---|---|---|---|---|
| `DEFAULT_MULTIPART_THRESHOLD` | 16 MiB | 5 MiB | n/a | Below this a single PUT is faster; at 16 MiB with 5 MiB parts and concurrency 4, every part starts in parallel. |
| `DEFAULT_PART_SIZE` | 8 MiB | 5 MiB (AWS) | 5 GiB (AWS) | Doubles AWS minimum for fewer parts on medium objects. |
| `DEFAULT_MULTIPART_CONCURRENCY` | 4 | 1 | 32 | WAN throughput sweet spot; > 32 stalls on TCP head-of-line per typical link. |
| `DEFAULT_PRESIGN_EXPIRES` | 3600 s | 1 s | 7 days (AWS) | One-hour windows are the AWS console default. |
| `MAX_PARTS` | 10 000 | n/a | n/a | AWS hard limit. |
| `MAX_OBJECT_SIZE` | 5 TiB | n/a | n/a | AWS hard limit. |
| `MAX_TAGS` | 10 | n/a | n/a | AWS hard limit. |

## Multipart engine

Same shape as the superseded `add-storage-s3-multipart` design (incorporated
here verbatim — that design's `multipart-engine.ts`, `part-chunker.ts`,
`worker-pool.ts` are imported wholesale into this change):

```ts
type PartInput = { number: number; body: Uint8Array };

async runMultipart(
    fullKey: string,
    parts: AsyncIterable<PartInput>,
    tuning: Required<MultipartTuning>,
    signal: AbortSignal | undefined,
): Promise<void> {
    const uploadId = await initMultipart(fullKey, signal);
    const completed: { ETag: string; PartNumber: number }[] = [];
    try {
        await runPartWorkers(fullKey, uploadId, parts, tuning.concurrency, completed, signal);
        completed.sort((a, b) => a.PartNumber - b.PartNumber);
        await completeMultipart(fullKey, uploadId, completed, signal);
    } catch (err) {
        await abortMultipart(fullKey, uploadId).catch(/* logged, swallowed */);
        throw err;
    }
}
```

Abort propagation: the engine creates an internal `AbortController`
(`linked`). Its signal is forwarded to every `S3Client.send` call. `linked`
is aborted when either the caller-supplied `signal` fires or any part worker
rejects. The `AbortMultipartUploadCommand` is sent with a **fresh** signal
so cleanup still runs after the caller aborts.

`UploadPartCopy` follows the same engine shape for server-side copy of
objects > 5 GiB. Source range is split across part numbers per AWS
`CopySourceRange` semantics.

## SSE parameter shaping

```ts
export type SseConfig =
    | { kind: "sse-s3" }
    | { kind: "sse-kms"; keyId?: string; bucketKeyEnabled?: boolean }
    | { kind: "sse-c"; key: Uint8Array; keyMd5: Uint8Array };
```

`sse-c` requires the caller to provide the key on every subsequent `get` /
`head` / `copy` source — the adapter does not retain key material. The
`keyMd5` field is validated against `MD5(key)` at the boundary; a mismatch
throws `STORAGE_S3.SSE_KEY_MISMATCH` before the request leaves the process.

## Presign

```ts
export interface PresignOptions {
    readonly expiresIn?: number;            // 1..604800
    readonly requestHeaders?: Record<string, string>;
    readonly responseContentType?: string;
    readonly responseContentDisposition?: string;
    readonly contentType?: string;          // pin on PUT (signed)
    readonly sse?: SseConfig;
    readonly checksumAlgorithm?: "SHA256" | "CRC32C";
    readonly versionId?: string;            // GET only
}
```

`getSignedDeleteUrl` is new and supports `versionId` for versioned-bucket
deletes. TTL validation rejects 0, negative, and > 7 days at the boundary
with `STORAGE_S3.PRESIGN_TTL_INVALID`.

## Conditional ops

`get`, `head`, `delete`, `copy` accept:

```ts
export interface ConditionalOptions {
    readonly ifMatch?: string;             // eTag
    readonly ifNoneMatch?: string;
    readonly ifModifiedSince?: Date;
    readonly ifUnmodifiedSince?: Date;
}
```

The adapter maps HTTP responses to error codes:

| HTTP | Code |
|---|---|
| 412 | `STORAGE_S3.PRECONDITION_FAILED` |
| 304 | `STORAGE_S3.NOT_MODIFIED` (returned as `undefined` from `get`, the code is attached to a returned sentinel for the metadata-aware variant) |

## Retry composition

Replaces the hand-rolled `withRetry` with `@suaveplan/retry`:

```ts
const strategy = createRetryStrategy({
    maxAttempts: options.retry?.maxAttempts ?? 3,
    backoff: "exponential",
    baseDelayMs: options.retry?.baseDelayMs ?? 100,
    maxDelayMs: options.retry?.maxDelayMs ?? 30000,
    classify: (err) => isRetryableS3Error(err) ? "retryable" : "fatal",
});
```

`isRetryableS3Error` keeps its current classification (throttling, 5xx,
`SlowDown`, `RequestTimeout`). Other failure modes are fatal.

## Pluggable transport + signer

```ts
export interface FetchLike {
    (input: URL | string, init?: RequestInit): Promise<Response>;
}

export interface SigV4Signer {
    sign(request: SignableRequest, options: SignOptions): Promise<SignedRequest>;
}
```

Defaults route through the AWS SDK's built-in. Custom transports plug at
`S3ClientFactory.create` via `requestHandler` (Node) or `Fetch` (browser /
edge). Custom signers replace the SDK's `SignatureV4` provider.

## Error-code map (~50 codes)

`defineCodes("STORAGE_S3", [...] as const)` registers:

```
ABORTED, CREDENTIALS_MISSING, CREDENTIALS_INVALID, SIGNATURE_INVALID, TOKEN_EXPIRED,
OBJECT_NOT_FOUND, BUCKET_NOT_FOUND, BUCKET_REGION_MISMATCH, REGION_MISMATCH,
BUCKET_ALREADY_EXISTS, BUCKET_NOT_EMPTY, BUCKET_CREATE_FAILED, BUCKET_DELETE_FAILED,
UPLOAD_FAILED, DELETE_FAILED, LIST_FAILED, HEAD_FAILED, COPY_FAILED, COPY_PART_FAILED,
PRESIGN_FAILED, PRESIGN_TTL_INVALID, PRESIGN_HEADER_INVALID,
MULTIPART_FAILED, MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED,
MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED,
SSE_KEY_MISSING, SSE_KEY_MISMATCH, SSE_KMS_DENIED, SSE_ALGORITHM_UNSUPPORTED,
ACL_INVALID, ACL_PUT_FAILED, ACL_GET_FAILED,
TAG_PUT_FAILED, TAG_DELETE_FAILED, TAG_GET_FAILED, TAG_LIMIT_EXCEEDED, TAG_INVALID,
VERSION_NOT_FOUND, VERSIONING_DISABLED,
LIFECYCLE_INVALID, LIFECYCLE_PUT_FAILED, LIFECYCLE_GET_FAILED, LIFECYCLE_DELETE_FAILED,
CORS_INVALID, CORS_PUT_FAILED, CORS_GET_FAILED, CORS_DELETE_FAILED,
PRECONDITION_FAILED, NOT_MODIFIED,
RANGE_INVALID, RANGE_NOT_SATISFIABLE,
CHECKSUM_MISMATCH, CHECKSUM_ALGORITHM_UNSUPPORTED,
TRANSPORT_FAILED,
```

The legacy 11 are retained at their existing identifiers; the additions
extend the registry. `MULTIPART_FAILED` is documented as a back-compat
alias — no post-change call site raises it.

## Test strategy

Per project rule §2 (no mocks): every integration test uses the Garage
container from `@suaveplan/testing-integration-garage-s3`. Failure injection
uses a wrapping `S3ClientFactory` that intercepts specific `Command`
instances and either delays them or fails them — the underlying transport
remains real.

Garage supports the full S3 surface this change targets, including
multipart, SSE-S3 (and Garage's SSE-C compatibility), tagging, versioning,
lifecycle, CORS, and bucket-level ops. SSE-KMS specifically requires a KMS
plug; Garage's SSE-KMS mode is exercised via its built-in compatibility
endpoint, and where Garage diverges from AWS (e.g. KMS key-policy
enforcement), the test asserts the documented Garage behaviour and the
fixture surface is extended in the same change.

For checksums, Garage validates both `ChecksumSHA256` and `ChecksumCRC32C`
when supplied; tests assert round-trip and assert
`STORAGE_S3.CHECKSUM_MISMATCH` on deliberately corrupted bytes.

## Decisions & trade-offs

| Decision | Alternative | Why |
|---|---|---|
| One change for full parity | Phase 1 (multipart only) + Phase 2 (everything else) | User explicitly required full parity, no cherry-picking. The superseded change folder confirms the prior split was rolled back. |
| Native Range GET in `getPartial` | Keep `defaultGetPartial` fetch-then-slice | Eliminates O(N) bandwidth waste; `partialReads: true` becomes truthful. |
| `setPartial` non-zero offset stays r-m-w | Emulate via multipart UploadPartCopy + new-part | S3 multipart cannot patch a byte range — emulation would download the prefix and suffix anyway, no win. |
| Compose `@suaveplan/retry` | Keep hand-rolled `withRetry` | Single retry policy across all `@suaveplan/*` packages; classification logic still lives in `isRetryableS3Error`. |
| SSE-C key material is not retained | Cache MD5 + plaintext key | Forcing per-call provision keeps the adapter stateless w.r.t. secret material. |
| Garage over MinIO for integration | MinIO / LocalStack | Already shipped fixture; faster CI; full multipart + tagging + lifecycle + CORS support. |
| `capabilities.partialWrites` stays `false` | Flip to `true` because multipart engine handles whole-object writes | Capability flag describes byte-range patching, which S3 cannot do. Flipping would lie to `drive-plugin-chunks` sparse-file layout. |
| Inline `tags` at `set` time via `x-amz-tagging` header | Two-round-trip put-then-tag | Single PUT is faster and atomic. |