suaveplan/openspec/changes/complete-storage-s3/design.md

Design: complete-storage-s3

Non-normative. The normative contract is ./specs/storage-s3/spec.md.

Goals

1. Full S3 protocol parity — every observable S3 capability that a production consumer needs is reachable from @suaveplan/storage-s3 via either the AbstractStorage surface (whole-object IO) or one of the adapter's S3-specific extension methods (presign, ACLs, tags, versioning, lifecycle, CORS, bucket ops).
2. Tier-honest capability flags — capabilities.partialReads = true (native Range), partialWrites = false (S3 has no byte-range PUT), nativeStreams = true, atomicWrites = true, contentAddressable = false.
3. Real-service tests only — every multi-step path is exercised against the Garage testcontainer via @suaveplan/testing-integration-garage-s3.
4. No mocks — failure injection uses a wrapping S3ClientFactory that intercepts specific commands; the underlying transport stays real.
5. Composable — pluggable HTTP transport, pluggable signer, retry via @suaveplan/retry. Hand-rolled withRetry is removed.
6. 100/100/100/100 coverage gate per project rule §7.

File layout

packages/storage/storage-s3/src/
├── s3/
│   ├── s3.ts                       # top-level S3 class — orchestrates extensions
│   ├── s3.md
│   ├── s3.test.ts
│   ├── s3-branches.test.ts
│   └── index.ts                    # explicit named re-exports
├── core/                           # AbstractStorage methods
│   ├── get/
│   │   ├── get.ts                  # native Range + conditional + SSE-C
│   │   ├── get.md
│   │   ├── get.test.ts
│   │   └── index.ts
│   ├── set/
│   │   ├── set.ts                  # auto-promote, metadata, tagging, SSE, ACL
│   │   ├── set.md
│   │   ├── set.test.ts
│   │   └── index.ts
│   ├── set-stream/
│   │   ├── set-stream.ts
│   │   ├── set-stream.md
│   │   ├── set-stream.test.ts
│   │   └── index.ts
│   ├── set-partial/
│   ├── delete/
│   ├── head/                       # head/stat with versionId + checksum
│   ├── list/                       # listObjectsV2 paginator
│   └── exists/
├── multipart/                      # multipart engine (set + setStream + copy)
│   ├── multipart-engine.ts
│   ├── multipart-engine.md
│   ├── multipart-engine.test.ts
│   ├── part-chunker.ts             # ReadableStream → AsyncIterable<PartInput>
│   ├── part-chunker.md
│   ├── part-chunker.test.ts
│   ├── worker-pool.ts              # bounded-concurrency scheduler + abort
│   ├── worker-pool.md
│   ├── worker-pool.test.ts
│   ├── upload-part-copy.ts         # server-side copy of > 5 GiB objects
│   ├── upload-part-copy.md
│   ├── upload-part-copy.test.ts
│   └── index.ts
├── presign/
│   ├── get-signed-get-url.ts
│   ├── get-signed-put-url.ts
│   ├── get-signed-delete-url.ts
│   ├── presign-options.ts          # TTL + headers + content-type pin
│   └── index.ts
├── sse/
│   ├── sse.ts                      # SSE-S3 / SSE-KMS / SSE-C param builder
│   ├── sse.md
│   ├── sse.test.ts
│   └── index.ts
├── acl/
│   ├── acl.ts                      # canned + grant policies; get/put object ACL
│   ├── acl.md
│   ├── acl.test.ts
│   └── index.ts
├── copy/
│   ├── copy.ts                     # copyObject + cross-bucket
│   ├── copy.md
│   ├── copy.test.ts
│   └── index.ts
├── tagging/
│   ├── tagging.ts                  # get/put/delete + inline-at-set encoding
│   ├── tagging.md
│   ├── tagging.test.ts
│   └── index.ts
├── versioning/
│   ├── versioning.ts               # list versions + versionId on get/head/delete
│   ├── versioning.md
│   ├── versioning.test.ts
│   └── index.ts
├── lifecycle/
│   ├── lifecycle.ts
│   ├── lifecycle.md
│   ├── lifecycle.test.ts
│   └── index.ts
├── cors/
│   ├── cors.ts
│   ├── cors.md
│   ├── cors.test.ts
│   └── index.ts
├── bucket/
│   ├── bucket.ts                   # create/delete/exists/region/listV2 paginator
│   ├── bucket.md
│   ├── bucket.test.ts
│   └── index.ts
├── conditional/
│   ├── conditional.ts              # If-Match / If-None-Match / If-*Modified-Since
│   ├── conditional.md
│   ├── conditional.test.ts
│   └── index.ts
├── checksum/
│   ├── checksum.ts                 # SHA256 + CRC32C integrity guards
│   ├── checksum.md
│   ├── checksum.test.ts
│   └── index.ts
├── transport/
│   ├── transport.ts                # FetchLike adapter
│   ├── transport.md
│   ├── transport.test.ts
│   └── index.ts
├── signer/
│   ├── signer.ts                   # SigV4 plug-point
│   ├── signer.md
│   ├── signer.test.ts
│   └── index.ts
├── retry/
│   ├── retry.ts                    # composes @suaveplan/retry
│   ├── retry.md
│   ├── retry.test.ts
│   └── index.ts
├── constants/
│   └── index.ts                    # all DEFAULT_* + MIN_PART_SIZE + thresholds
├── error-codes.ts                  # ~50 STORAGE_S3.* codes
├── error-codes.md
├── types/
│   ├── s3.types.ts
│   ├── s3.types.md
│   ├── sse.types.ts
│   ├── acl.types.ts
│   ├── tagging.types.ts
│   ├── lifecycle.types.ts
│   ├── cors.types.ts
│   ├── conditional.types.ts
│   ├── checksum.types.ts
│   └── index.ts
├── utils/
│   ├── credential-validator.ts
│   ├── build-key.ts
│   ├── error-mapper.ts             # S3ServiceException → S3StorageError
│   └── index.ts
└── index.ts                        # root barrel — namespace exports

packages/storage/storage-s3/tests/integration/
├── core.test.ts                    # get/set/delete/list/head against Garage
├── multipart.test.ts               # auto-promote + abort + cleanup + UploadPartCopy
├── stream.test.ts                  # setStream
├── presign.test.ts                 # all three presigned variants
├── sse.test.ts                     # SSE-S3 / SSE-KMS / SSE-C
├── acl.test.ts                     # canned + grant + get/put
├── copy.test.ts                    # within-bucket + cross-bucket
├── tagging.test.ts
├── versioning.test.ts
├── lifecycle.test.ts
├── cors.test.ts
├── bucket.test.ts                  # create / delete / exists / region / listV2
├── conditional.test.ts             # 412 / 304 paths
├── range.test.ts
└── checksum.test.ts                # SHA256 + CRC32C verify + mismatch

Capability flags

public override readonly capabilities: StorageCapabilities = {
    partialReads: true,    // native Range GET via getPartial
    partialWrites: false,  // S3 has no byte-range PUT; multipart is whole-object
    nativeStreams: true,   // setStream + range GET stream
    atomicWrites: true,    // S3 PUT semantics are atomic relative to readers
    contentAddressable: false,
};

Trigger thresholds

Constant	Default	Min	Max	Rationale
DEFAULT_MULTIPART_THRESHOLD	16 MiB	5 MiB	n/a	Below this a single PUT is faster; at 16 MiB with 5 MiB parts and concurrency 4, every part starts in parallel.
DEFAULT_PART_SIZE	8 MiB	5 MiB (AWS)	5 GiB (AWS)	Doubles AWS minimum for fewer parts on medium objects.
DEFAULT_MULTIPART_CONCURRENCY	4	1	32	WAN throughput sweet spot; > 32 stalls on TCP head-of-line per typical link.
DEFAULT_PRESIGN_EXPIRES	3600 s	1 s	7 days (AWS)	One-hour windows are the AWS console default.
MAX_PARTS	10 000	n/a	n/a	AWS hard limit.
MAX_OBJECT_SIZE	5 TiB	n/a	n/a	AWS hard limit.
MAX_TAGS	10	n/a	n/a	AWS hard limit.

Multipart engine

Same shape as the superseded add-storage-s3-multipart design (incorporated here verbatim — that design's multipart-engine.ts, part-chunker.ts, worker-pool.ts are imported wholesale into this change):

type PartInput = { number: number; body: Uint8Array };

async runMultipart(
    fullKey: string,
    parts: AsyncIterable<PartInput>,
    tuning: Required<MultipartTuning>,
    signal: AbortSignal | undefined,
): Promise<void> {
    const uploadId = await initMultipart(fullKey, signal);
    const completed: { ETag: string; PartNumber: number }[] = [];
    try {
        await runPartWorkers(fullKey, uploadId, parts, tuning.concurrency, completed, signal);
        completed.sort((a, b) => a.PartNumber - b.PartNumber);
        await completeMultipart(fullKey, uploadId, completed, signal);
    } catch (err) {
        await abortMultipart(fullKey, uploadId).catch(/* logged, swallowed */);
        throw err;
    }
}

Abort propagation: the engine creates an internal AbortController (linked). Its signal is forwarded to every S3Client.send call. linked is aborted when either the caller-supplied signal fires or any part worker rejects. The AbortMultipartUploadCommand is sent with a fresh signal so cleanup still runs after the caller aborts.

UploadPartCopy follows the same engine shape for server-side copy of objects > 5 GiB. Source range is split across part numbers per AWS CopySourceRange semantics.

SSE parameter shaping

export type SseConfig =
    | { kind: "sse-s3" }
    | { kind: "sse-kms"; keyId?: string; bucketKeyEnabled?: boolean }
    | { kind: "sse-c"; key: Uint8Array; keyMd5: Uint8Array };

sse-c requires the caller to provide the key on every subsequent get / head / copy source — the adapter does not retain key material. The keyMd5 field is validated against MD5(key) at the boundary; a mismatch throws STORAGE_S3.SSE_KEY_MISMATCH before the request leaves the process.

Presign

export interface PresignOptions {
    readonly expiresIn?: number;            // 1..604800
    readonly requestHeaders?: Record<string, string>;
    readonly responseContentType?: string;
    readonly responseContentDisposition?: string;
    readonly contentType?: string;          // pin on PUT (signed)
    readonly sse?: SseConfig;
    readonly checksumAlgorithm?: "SHA256" | "CRC32C";
    readonly versionId?: string;            // GET only
}

getSignedDeleteUrl is new and supports versionId for versioned-bucket deletes. TTL validation rejects 0, negative, and > 7 days at the boundary with STORAGE_S3.PRESIGN_TTL_INVALID.

Conditional ops

get, head, delete, copy accept:

export interface ConditionalOptions {
    readonly ifMatch?: string;             // eTag
    readonly ifNoneMatch?: string;
    readonly ifModifiedSince?: Date;
    readonly ifUnmodifiedSince?: Date;
}

The adapter maps HTTP responses to error codes:

HTTP	Code
412	STORAGE_S3.PRECONDITION_FAILED
304	STORAGE_S3.NOT_MODIFIED (returned as undefined from get, the code is attached to a returned sentinel for the metadata-aware variant)

Retry composition

Replaces the hand-rolled withRetry with @suaveplan/retry:

const strategy = createRetryStrategy({
    maxAttempts: options.retry?.maxAttempts ?? 3,
    backoff: "exponential",
    baseDelayMs: options.retry?.baseDelayMs ?? 100,
    maxDelayMs: options.retry?.maxDelayMs ?? 30000,
    classify: (err) => isRetryableS3Error(err) ? "retryable" : "fatal",
});

isRetryableS3Error keeps its current classification (throttling, 5xx, SlowDown, RequestTimeout). Other failure modes are fatal.

Pluggable transport + signer

export interface FetchLike {
    (input: URL | string, init?: RequestInit): Promise<Response>;
}

export interface SigV4Signer {
    sign(request: SignableRequest, options: SignOptions): Promise<SignedRequest>;
}

Defaults route through the AWS SDK's built-in. Custom transports plug at S3ClientFactory.create via requestHandler (Node) or Fetch (browser / edge). Custom signers replace the SDK's SignatureV4 provider.

Error-code map (~50 codes)

defineCodes("STORAGE_S3", [...] as const) registers:

ABORTED, CREDENTIALS_MISSING, CREDENTIALS_INVALID, SIGNATURE_INVALID, TOKEN_EXPIRED,
OBJECT_NOT_FOUND, BUCKET_NOT_FOUND, BUCKET_REGION_MISMATCH, REGION_MISMATCH,
BUCKET_ALREADY_EXISTS, BUCKET_NOT_EMPTY, BUCKET_CREATE_FAILED, BUCKET_DELETE_FAILED,
UPLOAD_FAILED, DELETE_FAILED, LIST_FAILED, HEAD_FAILED, COPY_FAILED, COPY_PART_FAILED,
PRESIGN_FAILED, PRESIGN_TTL_INVALID, PRESIGN_HEADER_INVALID,
MULTIPART_FAILED, MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED,
MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED,
SSE_KEY_MISSING, SSE_KEY_MISMATCH, SSE_KMS_DENIED, SSE_ALGORITHM_UNSUPPORTED,
ACL_INVALID, ACL_PUT_FAILED, ACL_GET_FAILED,
TAG_PUT_FAILED, TAG_DELETE_FAILED, TAG_GET_FAILED, TAG_LIMIT_EXCEEDED, TAG_INVALID,
VERSION_NOT_FOUND, VERSIONING_DISABLED,
LIFECYCLE_INVALID, LIFECYCLE_PUT_FAILED, LIFECYCLE_GET_FAILED, LIFECYCLE_DELETE_FAILED,
CORS_INVALID, CORS_PUT_FAILED, CORS_GET_FAILED, CORS_DELETE_FAILED,
PRECONDITION_FAILED, NOT_MODIFIED,
RANGE_INVALID, RANGE_NOT_SATISFIABLE,
CHECKSUM_MISMATCH, CHECKSUM_ALGORITHM_UNSUPPORTED,
TRANSPORT_FAILED,

The legacy 11 are retained at their existing identifiers; the additions extend the registry. MULTIPART_FAILED is documented as a back-compat alias — no post-change call site raises it.

Test strategy

Per project rule §2 (no mocks): every integration test uses the Garage container from @suaveplan/testing-integration-garage-s3. Failure injection uses a wrapping S3ClientFactory that intercepts specific Command instances and either delays them or fails them — the underlying transport remains real.

Garage supports the full S3 surface this change targets, including multipart, SSE-S3 (and Garage's SSE-C compatibility), tagging, versioning, lifecycle, CORS, and bucket-level ops. SSE-KMS specifically requires a KMS plug; Garage's SSE-KMS mode is exercised via its built-in compatibility endpoint, and where Garage diverges from AWS (e.g. KMS key-policy enforcement), the test asserts the documented Garage behaviour and the fixture surface is extended in the same change.

For checksums, Garage validates both ChecksumSHA256 and ChecksumCRC32C when supplied; tests assert round-trip and assert STORAGE_S3.CHECKSUM_MISMATCH on deliberately corrupted bytes.

Decisions & trade-offs

Decision	Alternative	Why
One change for full parity	Phase 1 (multipart only) + Phase 2 (everything else)	User explicitly required full parity, no cherry-picking. The superseded change folder confirms the prior split was rolled back.
Native Range GET in getPartial	Keep defaultGetPartial fetch-then-slice	Eliminates O(N) bandwidth waste; partialReads: true becomes truthful.
setPartial non-zero offset stays r-m-w	Emulate via multipart UploadPartCopy + new-part	S3 multipart cannot patch a byte range — emulation would download the prefix and suffix anyway, no win.
Compose @suaveplan/retry	Keep hand-rolled withRetry	Single retry policy across all @suaveplan/* packages; classification logic still lives in isRetryableS3Error.
SSE-C key material is not retained	Cache MD5 + plaintext key	Forcing per-call provision keeps the adapter stateless w.r.t. secret material.
Garage over MinIO for integration	MinIO / LocalStack	Already shipped fixture; faster CI; full multipart + tagging + lifecycle + CORS support.
capabilities.partialWrites stays false	Flip to true because multipart engine handles whole-object writes	Capability flag describes byte-range patching, which S3 cannot do. Flipping would lie to drive-plugin-chunks sparse-file layout.
Inline tags at set time via x-amz-tagging header	Two-round-trip put-then-tag	Single PUT is faster and atomic.