# Design: complete-storage-s3 Non-normative. The normative contract is `./specs/storage-s3/spec.md`. ## Goals 1. **Full S3 protocol parity** — every observable S3 capability that a production consumer needs is reachable from `@suaveplan/storage-s3` via either the `AbstractStorage` surface (whole-object IO) or one of the adapter's S3-specific extension methods (presign, ACLs, tags, versioning, lifecycle, CORS, bucket ops). 2. **Tier-honest capability flags** — `capabilities.partialReads = true` (native Range), `partialWrites = false` (S3 has no byte-range PUT), `nativeStreams = true`, `atomicWrites = true`, `contentAddressable = false`. 3. **Real-service tests only** — every multi-step path is exercised against the Garage testcontainer via `@suaveplan/testing-integration-garage-s3`. 4. **No mocks** — failure injection uses a wrapping `S3ClientFactory` that intercepts specific commands; the underlying transport stays real. 5. **Composable** — pluggable HTTP transport, pluggable signer, retry via `@suaveplan/retry`. Hand-rolled `withRetry` is removed. 6. **100/100/100/100** coverage gate per project rule §7. ## File layout ```text packages/storage/storage-s3/src/ ├── s3/ │ ├── s3.ts # top-level S3 class — orchestrates extensions │ ├── s3.md │ ├── s3.test.ts │ ├── s3-branches.test.ts │ └── index.ts # explicit named re-exports ├── core/ # AbstractStorage methods │ ├── get/ │ │ ├── get.ts # native Range + conditional + SSE-C │ │ ├── get.md │ │ ├── get.test.ts │ │ └── index.ts │ ├── set/ │ │ ├── set.ts # auto-promote, metadata, tagging, SSE, ACL │ │ ├── set.md │ │ ├── set.test.ts │ │ └── index.ts │ ├── set-stream/ │ │ ├── set-stream.ts │ │ ├── set-stream.md │ │ ├── set-stream.test.ts │ │ └── index.ts │ ├── set-partial/ │ ├── delete/ │ ├── head/ # head/stat with versionId + checksum │ ├── list/ # listObjectsV2 paginator │ └── exists/ ├── multipart/ # multipart engine (set + setStream + copy) │ ├── multipart-engine.ts │ ├── multipart-engine.md │ ├── multipart-engine.test.ts │ ├── part-chunker.ts # ReadableStream → AsyncIterable │ ├── part-chunker.md │ ├── part-chunker.test.ts │ ├── worker-pool.ts # bounded-concurrency scheduler + abort │ ├── worker-pool.md │ ├── worker-pool.test.ts │ ├── upload-part-copy.ts # server-side copy of > 5 GiB objects │ ├── upload-part-copy.md │ ├── upload-part-copy.test.ts │ └── index.ts ├── presign/ │ ├── get-signed-get-url.ts │ ├── get-signed-put-url.ts │ ├── get-signed-delete-url.ts │ ├── presign-options.ts # TTL + headers + content-type pin │ └── index.ts ├── sse/ │ ├── sse.ts # SSE-S3 / SSE-KMS / SSE-C param builder │ ├── sse.md │ ├── sse.test.ts │ └── index.ts ├── acl/ │ ├── acl.ts # canned + grant policies; get/put object ACL │ ├── acl.md │ ├── acl.test.ts │ └── index.ts ├── copy/ │ ├── copy.ts # copyObject + cross-bucket │ ├── copy.md │ ├── copy.test.ts │ └── index.ts ├── tagging/ │ ├── tagging.ts # get/put/delete + inline-at-set encoding │ ├── tagging.md │ ├── tagging.test.ts │ └── index.ts ├── versioning/ │ ├── versioning.ts # list versions + versionId on get/head/delete │ ├── versioning.md │ ├── versioning.test.ts │ └── index.ts ├── lifecycle/ │ ├── lifecycle.ts │ ├── lifecycle.md │ ├── lifecycle.test.ts │ └── index.ts ├── cors/ │ ├── cors.ts │ ├── cors.md │ ├── cors.test.ts │ └── index.ts ├── bucket/ │ ├── bucket.ts # create/delete/exists/region/listV2 paginator │ ├── bucket.md │ ├── bucket.test.ts │ └── index.ts ├── conditional/ │ ├── conditional.ts # If-Match / If-None-Match / If-*Modified-Since │ ├── conditional.md │ ├── conditional.test.ts │ └── index.ts ├── checksum/ │ ├── checksum.ts # SHA256 + CRC32C integrity guards │ ├── checksum.md │ ├── checksum.test.ts │ └── index.ts ├── transport/ │ ├── transport.ts # FetchLike adapter │ ├── transport.md │ ├── transport.test.ts │ └── index.ts ├── signer/ │ ├── signer.ts # SigV4 plug-point │ ├── signer.md │ ├── signer.test.ts │ └── index.ts ├── retry/ │ ├── retry.ts # composes @suaveplan/retry │ ├── retry.md │ ├── retry.test.ts │ └── index.ts ├── constants/ │ └── index.ts # all DEFAULT_* + MIN_PART_SIZE + thresholds ├── error-codes.ts # ~50 STORAGE_S3.* codes ├── error-codes.md ├── types/ │ ├── s3.types.ts │ ├── s3.types.md │ ├── sse.types.ts │ ├── acl.types.ts │ ├── tagging.types.ts │ ├── lifecycle.types.ts │ ├── cors.types.ts │ ├── conditional.types.ts │ ├── checksum.types.ts │ └── index.ts ├── utils/ │ ├── credential-validator.ts │ ├── build-key.ts │ ├── error-mapper.ts # S3ServiceException → S3StorageError │ └── index.ts └── index.ts # root barrel — namespace exports packages/storage/storage-s3/tests/integration/ ├── core.test.ts # get/set/delete/list/head against Garage ├── multipart.test.ts # auto-promote + abort + cleanup + UploadPartCopy ├── stream.test.ts # setStream ├── presign.test.ts # all three presigned variants ├── sse.test.ts # SSE-S3 / SSE-KMS / SSE-C ├── acl.test.ts # canned + grant + get/put ├── copy.test.ts # within-bucket + cross-bucket ├── tagging.test.ts ├── versioning.test.ts ├── lifecycle.test.ts ├── cors.test.ts ├── bucket.test.ts # create / delete / exists / region / listV2 ├── conditional.test.ts # 412 / 304 paths ├── range.test.ts └── checksum.test.ts # SHA256 + CRC32C verify + mismatch ``` ## Capability flags ```ts public override readonly capabilities: StorageCapabilities = { partialReads: true, // native Range GET via getPartial partialWrites: false, // S3 has no byte-range PUT; multipart is whole-object nativeStreams: true, // setStream + range GET stream atomicWrites: true, // S3 PUT semantics are atomic relative to readers contentAddressable: false, }; ``` ## Trigger thresholds | Constant | Default | Min | Max | Rationale | |---|---|---|---|---| | `DEFAULT_MULTIPART_THRESHOLD` | 16 MiB | 5 MiB | n/a | Below this a single PUT is faster; at 16 MiB with 5 MiB parts and concurrency 4, every part starts in parallel. | | `DEFAULT_PART_SIZE` | 8 MiB | 5 MiB (AWS) | 5 GiB (AWS) | Doubles AWS minimum for fewer parts on medium objects. | | `DEFAULT_MULTIPART_CONCURRENCY` | 4 | 1 | 32 | WAN throughput sweet spot; > 32 stalls on TCP head-of-line per typical link. | | `DEFAULT_PRESIGN_EXPIRES` | 3600 s | 1 s | 7 days (AWS) | One-hour windows are the AWS console default. | | `MAX_PARTS` | 10 000 | n/a | n/a | AWS hard limit. | | `MAX_OBJECT_SIZE` | 5 TiB | n/a | n/a | AWS hard limit. | | `MAX_TAGS` | 10 | n/a | n/a | AWS hard limit. | ## Multipart engine Same shape as the superseded `add-storage-s3-multipart` design (incorporated here verbatim — that design's `multipart-engine.ts`, `part-chunker.ts`, `worker-pool.ts` are imported wholesale into this change): ```ts type PartInput = { number: number; body: Uint8Array }; async runMultipart( fullKey: string, parts: AsyncIterable, tuning: Required, signal: AbortSignal | undefined, ): Promise { const uploadId = await initMultipart(fullKey, signal); const completed: { ETag: string; PartNumber: number }[] = []; try { await runPartWorkers(fullKey, uploadId, parts, tuning.concurrency, completed, signal); completed.sort((a, b) => a.PartNumber - b.PartNumber); await completeMultipart(fullKey, uploadId, completed, signal); } catch (err) { await abortMultipart(fullKey, uploadId).catch(/* logged, swallowed */); throw err; } } ``` Abort propagation: the engine creates an internal `AbortController` (`linked`). Its signal is forwarded to every `S3Client.send` call. `linked` is aborted when either the caller-supplied `signal` fires or any part worker rejects. The `AbortMultipartUploadCommand` is sent with a **fresh** signal so cleanup still runs after the caller aborts. `UploadPartCopy` follows the same engine shape for server-side copy of objects > 5 GiB. Source range is split across part numbers per AWS `CopySourceRange` semantics. ## SSE parameter shaping ```ts export type SseConfig = | { kind: "sse-s3" } | { kind: "sse-kms"; keyId?: string; bucketKeyEnabled?: boolean } | { kind: "sse-c"; key: Uint8Array; keyMd5: Uint8Array }; ``` `sse-c` requires the caller to provide the key on every subsequent `get` / `head` / `copy` source — the adapter does not retain key material. The `keyMd5` field is validated against `MD5(key)` at the boundary; a mismatch throws `STORAGE_S3.SSE_KEY_MISMATCH` before the request leaves the process. ## Presign ```ts export interface PresignOptions { readonly expiresIn?: number; // 1..604800 readonly requestHeaders?: Record; readonly responseContentType?: string; readonly responseContentDisposition?: string; readonly contentType?: string; // pin on PUT (signed) readonly sse?: SseConfig; readonly checksumAlgorithm?: "SHA256" | "CRC32C"; readonly versionId?: string; // GET only } ``` `getSignedDeleteUrl` is new and supports `versionId` for versioned-bucket deletes. TTL validation rejects 0, negative, and > 7 days at the boundary with `STORAGE_S3.PRESIGN_TTL_INVALID`. ## Conditional ops `get`, `head`, `delete`, `copy` accept: ```ts export interface ConditionalOptions { readonly ifMatch?: string; // eTag readonly ifNoneMatch?: string; readonly ifModifiedSince?: Date; readonly ifUnmodifiedSince?: Date; } ``` The adapter maps HTTP responses to error codes: | HTTP | Code | |---|---| | 412 | `STORAGE_S3.PRECONDITION_FAILED` | | 304 | `STORAGE_S3.NOT_MODIFIED` (returned as `undefined` from `get`, the code is attached to a returned sentinel for the metadata-aware variant) | ## Retry composition Replaces the hand-rolled `withRetry` with `@suaveplan/retry`: ```ts const strategy = createRetryStrategy({ maxAttempts: options.retry?.maxAttempts ?? 3, backoff: "exponential", baseDelayMs: options.retry?.baseDelayMs ?? 100, maxDelayMs: options.retry?.maxDelayMs ?? 30000, classify: (err) => isRetryableS3Error(err) ? "retryable" : "fatal", }); ``` `isRetryableS3Error` keeps its current classification (throttling, 5xx, `SlowDown`, `RequestTimeout`). Other failure modes are fatal. ## Pluggable transport + signer ```ts export interface FetchLike { (input: URL | string, init?: RequestInit): Promise; } export interface SigV4Signer { sign(request: SignableRequest, options: SignOptions): Promise; } ``` Defaults route through the AWS SDK's built-in. Custom transports plug at `S3ClientFactory.create` via `requestHandler` (Node) or `Fetch` (browser / edge). Custom signers replace the SDK's `SignatureV4` provider. ## Error-code map (~50 codes) `defineCodes("STORAGE_S3", [...] as const)` registers: ``` ABORTED, CREDENTIALS_MISSING, CREDENTIALS_INVALID, SIGNATURE_INVALID, TOKEN_EXPIRED, OBJECT_NOT_FOUND, BUCKET_NOT_FOUND, BUCKET_REGION_MISMATCH, REGION_MISMATCH, BUCKET_ALREADY_EXISTS, BUCKET_NOT_EMPTY, BUCKET_CREATE_FAILED, BUCKET_DELETE_FAILED, UPLOAD_FAILED, DELETE_FAILED, LIST_FAILED, HEAD_FAILED, COPY_FAILED, COPY_PART_FAILED, PRESIGN_FAILED, PRESIGN_TTL_INVALID, PRESIGN_HEADER_INVALID, MULTIPART_FAILED, MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED, MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED, SSE_KEY_MISSING, SSE_KEY_MISMATCH, SSE_KMS_DENIED, SSE_ALGORITHM_UNSUPPORTED, ACL_INVALID, ACL_PUT_FAILED, ACL_GET_FAILED, TAG_PUT_FAILED, TAG_DELETE_FAILED, TAG_GET_FAILED, TAG_LIMIT_EXCEEDED, TAG_INVALID, VERSION_NOT_FOUND, VERSIONING_DISABLED, LIFECYCLE_INVALID, LIFECYCLE_PUT_FAILED, LIFECYCLE_GET_FAILED, LIFECYCLE_DELETE_FAILED, CORS_INVALID, CORS_PUT_FAILED, CORS_GET_FAILED, CORS_DELETE_FAILED, PRECONDITION_FAILED, NOT_MODIFIED, RANGE_INVALID, RANGE_NOT_SATISFIABLE, CHECKSUM_MISMATCH, CHECKSUM_ALGORITHM_UNSUPPORTED, TRANSPORT_FAILED, ``` The legacy 11 are retained at their existing identifiers; the additions extend the registry. `MULTIPART_FAILED` is documented as a back-compat alias — no post-change call site raises it. ## Test strategy Per project rule §2 (no mocks): every integration test uses the Garage container from `@suaveplan/testing-integration-garage-s3`. Failure injection uses a wrapping `S3ClientFactory` that intercepts specific `Command` instances and either delays them or fails them — the underlying transport remains real. Garage supports the full S3 surface this change targets, including multipart, SSE-S3 (and Garage's SSE-C compatibility), tagging, versioning, lifecycle, CORS, and bucket-level ops. SSE-KMS specifically requires a KMS plug; Garage's SSE-KMS mode is exercised via its built-in compatibility endpoint, and where Garage diverges from AWS (e.g. KMS key-policy enforcement), the test asserts the documented Garage behaviour and the fixture surface is extended in the same change. For checksums, Garage validates both `ChecksumSHA256` and `ChecksumCRC32C` when supplied; tests assert round-trip and assert `STORAGE_S3.CHECKSUM_MISMATCH` on deliberately corrupted bytes. ## Decisions & trade-offs | Decision | Alternative | Why | |---|---|---| | One change for full parity | Phase 1 (multipart only) + Phase 2 (everything else) | User explicitly required full parity, no cherry-picking. The superseded change folder confirms the prior split was rolled back. | | Native Range GET in `getPartial` | Keep `defaultGetPartial` fetch-then-slice | Eliminates O(N) bandwidth waste; `partialReads: true` becomes truthful. | | `setPartial` non-zero offset stays r-m-w | Emulate via multipart UploadPartCopy + new-part | S3 multipart cannot patch a byte range — emulation would download the prefix and suffix anyway, no win. | | Compose `@suaveplan/retry` | Keep hand-rolled `withRetry` | Single retry policy across all `@suaveplan/*` packages; classification logic still lives in `isRetryableS3Error`. | | SSE-C key material is not retained | Cache MD5 + plaintext key | Forcing per-call provision keeps the adapter stateless w.r.t. secret material. | | Garage over MinIO for integration | MinIO / LocalStack | Already shipped fixture; faster CI; full multipart + tagging + lifecycle + CORS support. | | `capabilities.partialWrites` stays `false` | Flip to `true` because multipart engine handles whole-object writes | Capability flag describes byte-range patching, which S3 cannot do. Flipping would lie to `drive-plugin-chunks` sparse-file layout. | | Inline `tags` at `set` time via `x-amz-tagging` header | Two-round-trip put-then-tag | Single PUT is faster and atomic. |