388 lines
16 KiB
Markdown
388 lines
16 KiB
Markdown
# Design: complete-storage-s3
|
|
|
|
Non-normative. The normative contract is `./specs/storage-s3/spec.md`.
|
|
|
|
## Goals
|
|
|
|
1. **Full S3 protocol parity** — every observable S3 capability that a
|
|
production consumer needs is reachable from `@suaveplan/storage-s3` via
|
|
either the `AbstractStorage` surface (whole-object IO) or one of the
|
|
adapter's S3-specific extension methods (presign, ACLs, tags, versioning,
|
|
lifecycle, CORS, bucket ops).
|
|
2. **Tier-honest capability flags** — `capabilities.partialReads = true`
|
|
(native Range), `partialWrites = false` (S3 has no byte-range PUT),
|
|
`nativeStreams = true`, `atomicWrites = true`, `contentAddressable = false`.
|
|
3. **Real-service tests only** — every multi-step path is exercised against
|
|
the Garage testcontainer via
|
|
`@suaveplan/testing-integration-garage-s3`.
|
|
4. **No mocks** — failure injection uses a wrapping `S3ClientFactory` that
|
|
intercepts specific commands; the underlying transport stays real.
|
|
5. **Composable** — pluggable HTTP transport, pluggable signer, retry via
|
|
`@suaveplan/retry`. Hand-rolled `withRetry` is removed.
|
|
6. **100/100/100/100** coverage gate per project rule §7.
|
|
|
|
## File layout
|
|
|
|
```text
|
|
packages/storage/storage-s3/src/
|
|
├── s3/
|
|
│ ├── s3.ts # top-level S3 class — orchestrates extensions
|
|
│ ├── s3.md
|
|
│ ├── s3.test.ts
|
|
│ ├── s3-branches.test.ts
|
|
│ └── index.ts # explicit named re-exports
|
|
├── core/ # AbstractStorage methods
|
|
│ ├── get/
|
|
│ │ ├── get.ts # native Range + conditional + SSE-C
|
|
│ │ ├── get.md
|
|
│ │ ├── get.test.ts
|
|
│ │ └── index.ts
|
|
│ ├── set/
|
|
│ │ ├── set.ts # auto-promote, metadata, tagging, SSE, ACL
|
|
│ │ ├── set.md
|
|
│ │ ├── set.test.ts
|
|
│ │ └── index.ts
|
|
│ ├── set-stream/
|
|
│ │ ├── set-stream.ts
|
|
│ │ ├── set-stream.md
|
|
│ │ ├── set-stream.test.ts
|
|
│ │ └── index.ts
|
|
│ ├── set-partial/
|
|
│ ├── delete/
|
|
│ ├── head/ # head/stat with versionId + checksum
|
|
│ ├── list/ # listObjectsV2 paginator
|
|
│ └── exists/
|
|
├── multipart/ # multipart engine (set + setStream + copy)
|
|
│ ├── multipart-engine.ts
|
|
│ ├── multipart-engine.md
|
|
│ ├── multipart-engine.test.ts
|
|
│ ├── part-chunker.ts # ReadableStream → AsyncIterable<PartInput>
|
|
│ ├── part-chunker.md
|
|
│ ├── part-chunker.test.ts
|
|
│ ├── worker-pool.ts # bounded-concurrency scheduler + abort
|
|
│ ├── worker-pool.md
|
|
│ ├── worker-pool.test.ts
|
|
│ ├── upload-part-copy.ts # server-side copy of > 5 GiB objects
|
|
│ ├── upload-part-copy.md
|
|
│ ├── upload-part-copy.test.ts
|
|
│ └── index.ts
|
|
├── presign/
|
|
│ ├── get-signed-get-url.ts
|
|
│ ├── get-signed-put-url.ts
|
|
│ ├── get-signed-delete-url.ts
|
|
│ ├── presign-options.ts # TTL + headers + content-type pin
|
|
│ └── index.ts
|
|
├── sse/
|
|
│ ├── sse.ts # SSE-S3 / SSE-KMS / SSE-C param builder
|
|
│ ├── sse.md
|
|
│ ├── sse.test.ts
|
|
│ └── index.ts
|
|
├── acl/
|
|
│ ├── acl.ts # canned + grant policies; get/put object ACL
|
|
│ ├── acl.md
|
|
│ ├── acl.test.ts
|
|
│ └── index.ts
|
|
├── copy/
|
|
│ ├── copy.ts # copyObject + cross-bucket
|
|
│ ├── copy.md
|
|
│ ├── copy.test.ts
|
|
│ └── index.ts
|
|
├── tagging/
|
|
│ ├── tagging.ts # get/put/delete + inline-at-set encoding
|
|
│ ├── tagging.md
|
|
│ ├── tagging.test.ts
|
|
│ └── index.ts
|
|
├── versioning/
|
|
│ ├── versioning.ts # list versions + versionId on get/head/delete
|
|
│ ├── versioning.md
|
|
│ ├── versioning.test.ts
|
|
│ └── index.ts
|
|
├── lifecycle/
|
|
│ ├── lifecycle.ts
|
|
│ ├── lifecycle.md
|
|
│ ├── lifecycle.test.ts
|
|
│ └── index.ts
|
|
├── cors/
|
|
│ ├── cors.ts
|
|
│ ├── cors.md
|
|
│ ├── cors.test.ts
|
|
│ └── index.ts
|
|
├── bucket/
|
|
│ ├── bucket.ts # create/delete/exists/region/listV2 paginator
|
|
│ ├── bucket.md
|
|
│ ├── bucket.test.ts
|
|
│ └── index.ts
|
|
├── conditional/
|
|
│ ├── conditional.ts # If-Match / If-None-Match / If-*Modified-Since
|
|
│ ├── conditional.md
|
|
│ ├── conditional.test.ts
|
|
│ └── index.ts
|
|
├── checksum/
|
|
│ ├── checksum.ts # SHA256 + CRC32C integrity guards
|
|
│ ├── checksum.md
|
|
│ ├── checksum.test.ts
|
|
│ └── index.ts
|
|
├── transport/
|
|
│ ├── transport.ts # FetchLike adapter
|
|
│ ├── transport.md
|
|
│ ├── transport.test.ts
|
|
│ └── index.ts
|
|
├── signer/
|
|
│ ├── signer.ts # SigV4 plug-point
|
|
│ ├── signer.md
|
|
│ ├── signer.test.ts
|
|
│ └── index.ts
|
|
├── retry/
|
|
│ ├── retry.ts # composes @suaveplan/retry
|
|
│ ├── retry.md
|
|
│ ├── retry.test.ts
|
|
│ └── index.ts
|
|
├── constants/
|
|
│ └── index.ts # all DEFAULT_* + MIN_PART_SIZE + thresholds
|
|
├── error-codes.ts # ~50 STORAGE_S3.* codes
|
|
├── error-codes.md
|
|
├── types/
|
|
│ ├── s3.types.ts
|
|
│ ├── s3.types.md
|
|
│ ├── sse.types.ts
|
|
│ ├── acl.types.ts
|
|
│ ├── tagging.types.ts
|
|
│ ├── lifecycle.types.ts
|
|
│ ├── cors.types.ts
|
|
│ ├── conditional.types.ts
|
|
│ ├── checksum.types.ts
|
|
│ └── index.ts
|
|
├── utils/
|
|
│ ├── credential-validator.ts
|
|
│ ├── build-key.ts
|
|
│ ├── error-mapper.ts # S3ServiceException → S3StorageError
|
|
│ └── index.ts
|
|
└── index.ts # root barrel — namespace exports
|
|
|
|
packages/storage/storage-s3/tests/integration/
|
|
├── core.test.ts # get/set/delete/list/head against Garage
|
|
├── multipart.test.ts # auto-promote + abort + cleanup + UploadPartCopy
|
|
├── stream.test.ts # setStream
|
|
├── presign.test.ts # all three presigned variants
|
|
├── sse.test.ts # SSE-S3 / SSE-KMS / SSE-C
|
|
├── acl.test.ts # canned + grant + get/put
|
|
├── copy.test.ts # within-bucket + cross-bucket
|
|
├── tagging.test.ts
|
|
├── versioning.test.ts
|
|
├── lifecycle.test.ts
|
|
├── cors.test.ts
|
|
├── bucket.test.ts # create / delete / exists / region / listV2
|
|
├── conditional.test.ts # 412 / 304 paths
|
|
├── range.test.ts
|
|
└── checksum.test.ts # SHA256 + CRC32C verify + mismatch
|
|
```
|
|
|
|
## Capability flags
|
|
|
|
```ts
|
|
public override readonly capabilities: StorageCapabilities = {
|
|
partialReads: true, // native Range GET via getPartial
|
|
partialWrites: false, // S3 has no byte-range PUT; multipart is whole-object
|
|
nativeStreams: true, // setStream + range GET stream
|
|
atomicWrites: true, // S3 PUT semantics are atomic relative to readers
|
|
contentAddressable: false,
|
|
};
|
|
```
|
|
|
|
## Trigger thresholds
|
|
|
|
| Constant | Default | Min | Max | Rationale |
|
|
|---|---|---|---|---|
|
|
| `DEFAULT_MULTIPART_THRESHOLD` | 16 MiB | 5 MiB | n/a | Below this a single PUT is faster; at 16 MiB with 5 MiB parts and concurrency 4, every part starts in parallel. |
|
|
| `DEFAULT_PART_SIZE` | 8 MiB | 5 MiB (AWS) | 5 GiB (AWS) | Doubles AWS minimum for fewer parts on medium objects. |
|
|
| `DEFAULT_MULTIPART_CONCURRENCY` | 4 | 1 | 32 | WAN throughput sweet spot; > 32 stalls on TCP head-of-line per typical link. |
|
|
| `DEFAULT_PRESIGN_EXPIRES` | 3600 s | 1 s | 7 days (AWS) | One-hour windows are the AWS console default. |
|
|
| `MAX_PARTS` | 10 000 | n/a | n/a | AWS hard limit. |
|
|
| `MAX_OBJECT_SIZE` | 5 TiB | n/a | n/a | AWS hard limit. |
|
|
| `MAX_TAGS` | 10 | n/a | n/a | AWS hard limit. |
|
|
|
|
## Multipart engine
|
|
|
|
Same shape as the superseded `add-storage-s3-multipart` design (incorporated
|
|
here verbatim — that design's `multipart-engine.ts`, `part-chunker.ts`,
|
|
`worker-pool.ts` are imported wholesale into this change):
|
|
|
|
```ts
|
|
type PartInput = { number: number; body: Uint8Array };
|
|
|
|
async runMultipart(
|
|
fullKey: string,
|
|
parts: AsyncIterable<PartInput>,
|
|
tuning: Required<MultipartTuning>,
|
|
signal: AbortSignal | undefined,
|
|
): Promise<void> {
|
|
const uploadId = await initMultipart(fullKey, signal);
|
|
const completed: { ETag: string; PartNumber: number }[] = [];
|
|
try {
|
|
await runPartWorkers(fullKey, uploadId, parts, tuning.concurrency, completed, signal);
|
|
completed.sort((a, b) => a.PartNumber - b.PartNumber);
|
|
await completeMultipart(fullKey, uploadId, completed, signal);
|
|
} catch (err) {
|
|
await abortMultipart(fullKey, uploadId).catch(/* logged, swallowed */);
|
|
throw err;
|
|
}
|
|
}
|
|
```
|
|
|
|
Abort propagation: the engine creates an internal `AbortController`
|
|
(`linked`). Its signal is forwarded to every `S3Client.send` call. `linked`
|
|
is aborted when either the caller-supplied `signal` fires or any part worker
|
|
rejects. The `AbortMultipartUploadCommand` is sent with a **fresh** signal
|
|
so cleanup still runs after the caller aborts.
|
|
|
|
`UploadPartCopy` follows the same engine shape for server-side copy of
|
|
objects > 5 GiB. Source range is split across part numbers per AWS
|
|
`CopySourceRange` semantics.
|
|
|
|
## SSE parameter shaping
|
|
|
|
```ts
|
|
export type SseConfig =
|
|
| { kind: "sse-s3" }
|
|
| { kind: "sse-kms"; keyId?: string; bucketKeyEnabled?: boolean }
|
|
| { kind: "sse-c"; key: Uint8Array; keyMd5: Uint8Array };
|
|
```
|
|
|
|
`sse-c` requires the caller to provide the key on every subsequent `get` /
|
|
`head` / `copy` source — the adapter does not retain key material. The
|
|
`keyMd5` field is validated against `MD5(key)` at the boundary; a mismatch
|
|
throws `STORAGE_S3.SSE_KEY_MISMATCH` before the request leaves the process.
|
|
|
|
## Presign
|
|
|
|
```ts
|
|
export interface PresignOptions {
|
|
readonly expiresIn?: number; // 1..604800
|
|
readonly requestHeaders?: Record<string, string>;
|
|
readonly responseContentType?: string;
|
|
readonly responseContentDisposition?: string;
|
|
readonly contentType?: string; // pin on PUT (signed)
|
|
readonly sse?: SseConfig;
|
|
readonly checksumAlgorithm?: "SHA256" | "CRC32C";
|
|
readonly versionId?: string; // GET only
|
|
}
|
|
```
|
|
|
|
`getSignedDeleteUrl` is new and supports `versionId` for versioned-bucket
|
|
deletes. TTL validation rejects 0, negative, and > 7 days at the boundary
|
|
with `STORAGE_S3.PRESIGN_TTL_INVALID`.
|
|
|
|
## Conditional ops
|
|
|
|
`get`, `head`, `delete`, `copy` accept:
|
|
|
|
```ts
|
|
export interface ConditionalOptions {
|
|
readonly ifMatch?: string; // eTag
|
|
readonly ifNoneMatch?: string;
|
|
readonly ifModifiedSince?: Date;
|
|
readonly ifUnmodifiedSince?: Date;
|
|
}
|
|
```
|
|
|
|
The adapter maps HTTP responses to error codes:
|
|
|
|
| HTTP | Code |
|
|
|---|---|
|
|
| 412 | `STORAGE_S3.PRECONDITION_FAILED` |
|
|
| 304 | `STORAGE_S3.NOT_MODIFIED` (returned as `undefined` from `get`, the code is attached to a returned sentinel for the metadata-aware variant) |
|
|
|
|
## Retry composition
|
|
|
|
Replaces the hand-rolled `withRetry` with `@suaveplan/retry`:
|
|
|
|
```ts
|
|
const strategy = createRetryStrategy({
|
|
maxAttempts: options.retry?.maxAttempts ?? 3,
|
|
backoff: "exponential",
|
|
baseDelayMs: options.retry?.baseDelayMs ?? 100,
|
|
maxDelayMs: options.retry?.maxDelayMs ?? 30000,
|
|
classify: (err) => isRetryableS3Error(err) ? "retryable" : "fatal",
|
|
});
|
|
```
|
|
|
|
`isRetryableS3Error` keeps its current classification (throttling, 5xx,
|
|
`SlowDown`, `RequestTimeout`). Other failure modes are fatal.
|
|
|
|
## Pluggable transport + signer
|
|
|
|
```ts
|
|
export interface FetchLike {
|
|
(input: URL | string, init?: RequestInit): Promise<Response>;
|
|
}
|
|
|
|
export interface SigV4Signer {
|
|
sign(request: SignableRequest, options: SignOptions): Promise<SignedRequest>;
|
|
}
|
|
```
|
|
|
|
Defaults route through the AWS SDK's built-in. Custom transports plug at
|
|
`S3ClientFactory.create` via `requestHandler` (Node) or `Fetch` (browser /
|
|
edge). Custom signers replace the SDK's `SignatureV4` provider.
|
|
|
|
## Error-code map (~50 codes)
|
|
|
|
`defineCodes("STORAGE_S3", [...] as const)` registers:
|
|
|
|
```
|
|
ABORTED, CREDENTIALS_MISSING, CREDENTIALS_INVALID, SIGNATURE_INVALID, TOKEN_EXPIRED,
|
|
OBJECT_NOT_FOUND, BUCKET_NOT_FOUND, BUCKET_REGION_MISMATCH, REGION_MISMATCH,
|
|
BUCKET_ALREADY_EXISTS, BUCKET_NOT_EMPTY, BUCKET_CREATE_FAILED, BUCKET_DELETE_FAILED,
|
|
UPLOAD_FAILED, DELETE_FAILED, LIST_FAILED, HEAD_FAILED, COPY_FAILED, COPY_PART_FAILED,
|
|
PRESIGN_FAILED, PRESIGN_TTL_INVALID, PRESIGN_HEADER_INVALID,
|
|
MULTIPART_FAILED, MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED,
|
|
MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED,
|
|
SSE_KEY_MISSING, SSE_KEY_MISMATCH, SSE_KMS_DENIED, SSE_ALGORITHM_UNSUPPORTED,
|
|
ACL_INVALID, ACL_PUT_FAILED, ACL_GET_FAILED,
|
|
TAG_PUT_FAILED, TAG_DELETE_FAILED, TAG_GET_FAILED, TAG_LIMIT_EXCEEDED, TAG_INVALID,
|
|
VERSION_NOT_FOUND, VERSIONING_DISABLED,
|
|
LIFECYCLE_INVALID, LIFECYCLE_PUT_FAILED, LIFECYCLE_GET_FAILED, LIFECYCLE_DELETE_FAILED,
|
|
CORS_INVALID, CORS_PUT_FAILED, CORS_GET_FAILED, CORS_DELETE_FAILED,
|
|
PRECONDITION_FAILED, NOT_MODIFIED,
|
|
RANGE_INVALID, RANGE_NOT_SATISFIABLE,
|
|
CHECKSUM_MISMATCH, CHECKSUM_ALGORITHM_UNSUPPORTED,
|
|
TRANSPORT_FAILED,
|
|
```
|
|
|
|
The legacy 11 are retained at their existing identifiers; the additions
|
|
extend the registry. `MULTIPART_FAILED` is documented as a back-compat
|
|
alias — no post-change call site raises it.
|
|
|
|
## Test strategy
|
|
|
|
Per project rule §2 (no mocks): every integration test uses the Garage
|
|
container from `@suaveplan/testing-integration-garage-s3`. Failure injection
|
|
uses a wrapping `S3ClientFactory` that intercepts specific `Command`
|
|
instances and either delays them or fails them — the underlying transport
|
|
remains real.
|
|
|
|
Garage supports the full S3 surface this change targets, including
|
|
multipart, SSE-S3 (and Garage's SSE-C compatibility), tagging, versioning,
|
|
lifecycle, CORS, and bucket-level ops. SSE-KMS specifically requires a KMS
|
|
plug; Garage's SSE-KMS mode is exercised via its built-in compatibility
|
|
endpoint, and where Garage diverges from AWS (e.g. KMS key-policy
|
|
enforcement), the test asserts the documented Garage behaviour and the
|
|
fixture surface is extended in the same change.
|
|
|
|
For checksums, Garage validates both `ChecksumSHA256` and `ChecksumCRC32C`
|
|
when supplied; tests assert round-trip and assert
|
|
`STORAGE_S3.CHECKSUM_MISMATCH` on deliberately corrupted bytes.
|
|
|
|
## Decisions & trade-offs
|
|
|
|
| Decision | Alternative | Why |
|
|
|---|---|---|
|
|
| One change for full parity | Phase 1 (multipart only) + Phase 2 (everything else) | User explicitly required full parity, no cherry-picking. The superseded change folder confirms the prior split was rolled back. |
|
|
| Native Range GET in `getPartial` | Keep `defaultGetPartial` fetch-then-slice | Eliminates O(N) bandwidth waste; `partialReads: true` becomes truthful. |
|
|
| `setPartial` non-zero offset stays r-m-w | Emulate via multipart UploadPartCopy + new-part | S3 multipart cannot patch a byte range — emulation would download the prefix and suffix anyway, no win. |
|
|
| Compose `@suaveplan/retry` | Keep hand-rolled `withRetry` | Single retry policy across all `@suaveplan/*` packages; classification logic still lives in `isRetryableS3Error`. |
|
|
| SSE-C key material is not retained | Cache MD5 + plaintext key | Forcing per-call provision keeps the adapter stateless w.r.t. secret material. |
|
|
| Garage over MinIO for integration | MinIO / LocalStack | Already shipped fixture; faster CI; full multipart + tagging + lifecycle + CORS support. |
|
|
| `capabilities.partialWrites` stays `false` | Flip to `true` because multipart engine handles whole-object writes | Capability flag describes byte-range patching, which S3 cannot do. Flipping would lie to `drive-plugin-chunks` sparse-file layout. |
|
|
| Inline `tags` at `set` time via `x-amz-tagging` header | Two-round-trip put-then-tag | Single PUT is faster and atomic. |
|