Initial Commit
This commit is contained in:
387
openspec/changes/complete-storage-s3/design.md
Normal file
387
openspec/changes/complete-storage-s3/design.md
Normal file
@@ -0,0 +1,387 @@
|
||||
# Design: complete-storage-s3
|
||||
|
||||
Non-normative. The normative contract is `./specs/storage-s3/spec.md`.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Full S3 protocol parity** — every observable S3 capability that a
|
||||
production consumer needs is reachable from `@suaveplan/storage-s3` via
|
||||
either the `AbstractStorage` surface (whole-object IO) or one of the
|
||||
adapter's S3-specific extension methods (presign, ACLs, tags, versioning,
|
||||
lifecycle, CORS, bucket ops).
|
||||
2. **Tier-honest capability flags** — `capabilities.partialReads = true`
|
||||
(native Range), `partialWrites = false` (S3 has no byte-range PUT),
|
||||
`nativeStreams = true`, `atomicWrites = true`, `contentAddressable = false`.
|
||||
3. **Real-service tests only** — every multi-step path is exercised against
|
||||
the Garage testcontainer via
|
||||
`@suaveplan/testing-integration-garage-s3`.
|
||||
4. **No mocks** — failure injection uses a wrapping `S3ClientFactory` that
|
||||
intercepts specific commands; the underlying transport stays real.
|
||||
5. **Composable** — pluggable HTTP transport, pluggable signer, retry via
|
||||
`@suaveplan/retry`. Hand-rolled `withRetry` is removed.
|
||||
6. **100/100/100/100** coverage gate per project rule §7.
|
||||
|
||||
## File layout
|
||||
|
||||
```text
|
||||
packages/storage/storage-s3/src/
|
||||
├── s3/
|
||||
│ ├── s3.ts # top-level S3 class — orchestrates extensions
|
||||
│ ├── s3.md
|
||||
│ ├── s3.test.ts
|
||||
│ ├── s3-branches.test.ts
|
||||
│ └── index.ts # explicit named re-exports
|
||||
├── core/ # AbstractStorage methods
|
||||
│ ├── get/
|
||||
│ │ ├── get.ts # native Range + conditional + SSE-C
|
||||
│ │ ├── get.md
|
||||
│ │ ├── get.test.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── set/
|
||||
│ │ ├── set.ts # auto-promote, metadata, tagging, SSE, ACL
|
||||
│ │ ├── set.md
|
||||
│ │ ├── set.test.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── set-stream/
|
||||
│ │ ├── set-stream.ts
|
||||
│ │ ├── set-stream.md
|
||||
│ │ ├── set-stream.test.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── set-partial/
|
||||
│ ├── delete/
|
||||
│ ├── head/ # head/stat with versionId + checksum
|
||||
│ ├── list/ # listObjectsV2 paginator
|
||||
│ └── exists/
|
||||
├── multipart/ # multipart engine (set + setStream + copy)
|
||||
│ ├── multipart-engine.ts
|
||||
│ ├── multipart-engine.md
|
||||
│ ├── multipart-engine.test.ts
|
||||
│ ├── part-chunker.ts # ReadableStream → AsyncIterable<PartInput>
|
||||
│ ├── part-chunker.md
|
||||
│ ├── part-chunker.test.ts
|
||||
│ ├── worker-pool.ts # bounded-concurrency scheduler + abort
|
||||
│ ├── worker-pool.md
|
||||
│ ├── worker-pool.test.ts
|
||||
│ ├── upload-part-copy.ts # server-side copy of > 5 GiB objects
|
||||
│ ├── upload-part-copy.md
|
||||
│ ├── upload-part-copy.test.ts
|
||||
│ └── index.ts
|
||||
├── presign/
|
||||
│ ├── get-signed-get-url.ts
|
||||
│ ├── get-signed-put-url.ts
|
||||
│ ├── get-signed-delete-url.ts
|
||||
│ ├── presign-options.ts # TTL + headers + content-type pin
|
||||
│ └── index.ts
|
||||
├── sse/
|
||||
│ ├── sse.ts # SSE-S3 / SSE-KMS / SSE-C param builder
|
||||
│ ├── sse.md
|
||||
│ ├── sse.test.ts
|
||||
│ └── index.ts
|
||||
├── acl/
|
||||
│ ├── acl.ts # canned + grant policies; get/put object ACL
|
||||
│ ├── acl.md
|
||||
│ ├── acl.test.ts
|
||||
│ └── index.ts
|
||||
├── copy/
|
||||
│ ├── copy.ts # copyObject + cross-bucket
|
||||
│ ├── copy.md
|
||||
│ ├── copy.test.ts
|
||||
│ └── index.ts
|
||||
├── tagging/
|
||||
│ ├── tagging.ts # get/put/delete + inline-at-set encoding
|
||||
│ ├── tagging.md
|
||||
│ ├── tagging.test.ts
|
||||
│ └── index.ts
|
||||
├── versioning/
|
||||
│ ├── versioning.ts # list versions + versionId on get/head/delete
|
||||
│ ├── versioning.md
|
||||
│ ├── versioning.test.ts
|
||||
│ └── index.ts
|
||||
├── lifecycle/
|
||||
│ ├── lifecycle.ts
|
||||
│ ├── lifecycle.md
|
||||
│ ├── lifecycle.test.ts
|
||||
│ └── index.ts
|
||||
├── cors/
|
||||
│ ├── cors.ts
|
||||
│ ├── cors.md
|
||||
│ ├── cors.test.ts
|
||||
│ └── index.ts
|
||||
├── bucket/
|
||||
│ ├── bucket.ts # create/delete/exists/region/listV2 paginator
|
||||
│ ├── bucket.md
|
||||
│ ├── bucket.test.ts
|
||||
│ └── index.ts
|
||||
├── conditional/
|
||||
│ ├── conditional.ts # If-Match / If-None-Match / If-*Modified-Since
|
||||
│ ├── conditional.md
|
||||
│ ├── conditional.test.ts
|
||||
│ └── index.ts
|
||||
├── checksum/
|
||||
│ ├── checksum.ts # SHA256 + CRC32C integrity guards
|
||||
│ ├── checksum.md
|
||||
│ ├── checksum.test.ts
|
||||
│ └── index.ts
|
||||
├── transport/
|
||||
│ ├── transport.ts # FetchLike adapter
|
||||
│ ├── transport.md
|
||||
│ ├── transport.test.ts
|
||||
│ └── index.ts
|
||||
├── signer/
|
||||
│ ├── signer.ts # SigV4 plug-point
|
||||
│ ├── signer.md
|
||||
│ ├── signer.test.ts
|
||||
│ └── index.ts
|
||||
├── retry/
|
||||
│ ├── retry.ts # composes @suaveplan/retry
|
||||
│ ├── retry.md
|
||||
│ ├── retry.test.ts
|
||||
│ └── index.ts
|
||||
├── constants/
|
||||
│ └── index.ts # all DEFAULT_* + MIN_PART_SIZE + thresholds
|
||||
├── error-codes.ts # ~50 STORAGE_S3.* codes
|
||||
├── error-codes.md
|
||||
├── types/
|
||||
│ ├── s3.types.ts
|
||||
│ ├── s3.types.md
|
||||
│ ├── sse.types.ts
|
||||
│ ├── acl.types.ts
|
||||
│ ├── tagging.types.ts
|
||||
│ ├── lifecycle.types.ts
|
||||
│ ├── cors.types.ts
|
||||
│ ├── conditional.types.ts
|
||||
│ ├── checksum.types.ts
|
||||
│ └── index.ts
|
||||
├── utils/
|
||||
│ ├── credential-validator.ts
|
||||
│ ├── build-key.ts
|
||||
│ ├── error-mapper.ts # S3ServiceException → S3StorageError
|
||||
│ └── index.ts
|
||||
└── index.ts # root barrel — namespace exports
|
||||
|
||||
packages/storage/storage-s3/tests/integration/
|
||||
├── core.test.ts # get/set/delete/list/head against Garage
|
||||
├── multipart.test.ts # auto-promote + abort + cleanup + UploadPartCopy
|
||||
├── stream.test.ts # setStream
|
||||
├── presign.test.ts # all three presigned variants
|
||||
├── sse.test.ts # SSE-S3 / SSE-KMS / SSE-C
|
||||
├── acl.test.ts # canned + grant + get/put
|
||||
├── copy.test.ts # within-bucket + cross-bucket
|
||||
├── tagging.test.ts
|
||||
├── versioning.test.ts
|
||||
├── lifecycle.test.ts
|
||||
├── cors.test.ts
|
||||
├── bucket.test.ts # create / delete / exists / region / listV2
|
||||
├── conditional.test.ts # 412 / 304 paths
|
||||
├── range.test.ts
|
||||
└── checksum.test.ts # SHA256 + CRC32C verify + mismatch
|
||||
```
|
||||
|
||||
## Capability flags
|
||||
|
||||
```ts
|
||||
public override readonly capabilities: StorageCapabilities = {
|
||||
partialReads: true, // native Range GET via getPartial
|
||||
partialWrites: false, // S3 has no byte-range PUT; multipart is whole-object
|
||||
nativeStreams: true, // setStream + range GET stream
|
||||
atomicWrites: true, // S3 PUT semantics are atomic relative to readers
|
||||
contentAddressable: false,
|
||||
};
|
||||
```
|
||||
|
||||
## Trigger thresholds
|
||||
|
||||
| Constant | Default | Min | Max | Rationale |
|
||||
|---|---|---|---|---|
|
||||
| `DEFAULT_MULTIPART_THRESHOLD` | 16 MiB | 5 MiB | n/a | Below this a single PUT is faster; at 16 MiB with 5 MiB parts and concurrency 4, every part starts in parallel. |
|
||||
| `DEFAULT_PART_SIZE` | 8 MiB | 5 MiB (AWS) | 5 GiB (AWS) | Doubles AWS minimum for fewer parts on medium objects. |
|
||||
| `DEFAULT_MULTIPART_CONCURRENCY` | 4 | 1 | 32 | WAN throughput sweet spot; > 32 stalls on TCP head-of-line per typical link. |
|
||||
| `DEFAULT_PRESIGN_EXPIRES` | 3600 s | 1 s | 7 days (AWS) | One-hour windows are the AWS console default. |
|
||||
| `MAX_PARTS` | 10 000 | n/a | n/a | AWS hard limit. |
|
||||
| `MAX_OBJECT_SIZE` | 5 TiB | n/a | n/a | AWS hard limit. |
|
||||
| `MAX_TAGS` | 10 | n/a | n/a | AWS hard limit. |
|
||||
|
||||
## Multipart engine
|
||||
|
||||
Same shape as the superseded `add-storage-s3-multipart` design (incorporated
|
||||
here verbatim — that design's `multipart-engine.ts`, `part-chunker.ts`,
|
||||
`worker-pool.ts` are imported wholesale into this change):
|
||||
|
||||
```ts
|
||||
type PartInput = { number: number; body: Uint8Array };
|
||||
|
||||
async runMultipart(
|
||||
fullKey: string,
|
||||
parts: AsyncIterable<PartInput>,
|
||||
tuning: Required<MultipartTuning>,
|
||||
signal: AbortSignal | undefined,
|
||||
): Promise<void> {
|
||||
const uploadId = await initMultipart(fullKey, signal);
|
||||
const completed: { ETag: string; PartNumber: number }[] = [];
|
||||
try {
|
||||
await runPartWorkers(fullKey, uploadId, parts, tuning.concurrency, completed, signal);
|
||||
completed.sort((a, b) => a.PartNumber - b.PartNumber);
|
||||
await completeMultipart(fullKey, uploadId, completed, signal);
|
||||
} catch (err) {
|
||||
await abortMultipart(fullKey, uploadId).catch(/* logged, swallowed */);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Abort propagation: the engine creates an internal `AbortController`
|
||||
(`linked`). Its signal is forwarded to every `S3Client.send` call. `linked`
|
||||
is aborted when either the caller-supplied `signal` fires or any part worker
|
||||
rejects. The `AbortMultipartUploadCommand` is sent with a **fresh** signal
|
||||
so cleanup still runs after the caller aborts.
|
||||
|
||||
`UploadPartCopy` follows the same engine shape for server-side copy of
|
||||
objects > 5 GiB. Source range is split across part numbers per AWS
|
||||
`CopySourceRange` semantics.
|
||||
|
||||
## SSE parameter shaping
|
||||
|
||||
```ts
|
||||
export type SseConfig =
|
||||
| { kind: "sse-s3" }
|
||||
| { kind: "sse-kms"; keyId?: string; bucketKeyEnabled?: boolean }
|
||||
| { kind: "sse-c"; key: Uint8Array; keyMd5: Uint8Array };
|
||||
```
|
||||
|
||||
`sse-c` requires the caller to provide the key on every subsequent `get` /
|
||||
`head` / `copy` source — the adapter does not retain key material. The
|
||||
`keyMd5` field is validated against `MD5(key)` at the boundary; a mismatch
|
||||
throws `STORAGE_S3.SSE_KEY_MISMATCH` before the request leaves the process.
|
||||
|
||||
## Presign
|
||||
|
||||
```ts
|
||||
export interface PresignOptions {
|
||||
readonly expiresIn?: number; // 1..604800
|
||||
readonly requestHeaders?: Record<string, string>;
|
||||
readonly responseContentType?: string;
|
||||
readonly responseContentDisposition?: string;
|
||||
readonly contentType?: string; // pin on PUT (signed)
|
||||
readonly sse?: SseConfig;
|
||||
readonly checksumAlgorithm?: "SHA256" | "CRC32C";
|
||||
readonly versionId?: string; // GET only
|
||||
}
|
||||
```
|
||||
|
||||
`getSignedDeleteUrl` is new and supports `versionId` for versioned-bucket
|
||||
deletes. TTL validation rejects 0, negative, and > 7 days at the boundary
|
||||
with `STORAGE_S3.PRESIGN_TTL_INVALID`.
|
||||
|
||||
## Conditional ops
|
||||
|
||||
`get`, `head`, `delete`, `copy` accept:
|
||||
|
||||
```ts
|
||||
export interface ConditionalOptions {
|
||||
readonly ifMatch?: string; // eTag
|
||||
readonly ifNoneMatch?: string;
|
||||
readonly ifModifiedSince?: Date;
|
||||
readonly ifUnmodifiedSince?: Date;
|
||||
}
|
||||
```
|
||||
|
||||
The adapter maps HTTP responses to error codes:
|
||||
|
||||
| HTTP | Code |
|
||||
|---|---|
|
||||
| 412 | `STORAGE_S3.PRECONDITION_FAILED` |
|
||||
| 304 | `STORAGE_S3.NOT_MODIFIED` (returned as `undefined` from `get`, the code is attached to a returned sentinel for the metadata-aware variant) |
|
||||
|
||||
## Retry composition
|
||||
|
||||
Replaces the hand-rolled `withRetry` with `@suaveplan/retry`:
|
||||
|
||||
```ts
|
||||
const strategy = createRetryStrategy({
|
||||
maxAttempts: options.retry?.maxAttempts ?? 3,
|
||||
backoff: "exponential",
|
||||
baseDelayMs: options.retry?.baseDelayMs ?? 100,
|
||||
maxDelayMs: options.retry?.maxDelayMs ?? 30000,
|
||||
classify: (err) => isRetryableS3Error(err) ? "retryable" : "fatal",
|
||||
});
|
||||
```
|
||||
|
||||
`isRetryableS3Error` keeps its current classification (throttling, 5xx,
|
||||
`SlowDown`, `RequestTimeout`). Other failure modes are fatal.
|
||||
|
||||
## Pluggable transport + signer
|
||||
|
||||
```ts
|
||||
export interface FetchLike {
|
||||
(input: URL | string, init?: RequestInit): Promise<Response>;
|
||||
}
|
||||
|
||||
export interface SigV4Signer {
|
||||
sign(request: SignableRequest, options: SignOptions): Promise<SignedRequest>;
|
||||
}
|
||||
```
|
||||
|
||||
Defaults route through the AWS SDK's built-in. Custom transports plug at
|
||||
`S3ClientFactory.create` via `requestHandler` (Node) or `Fetch` (browser /
|
||||
edge). Custom signers replace the SDK's `SignatureV4` provider.
|
||||
|
||||
## Error-code map (~50 codes)
|
||||
|
||||
`defineCodes("STORAGE_S3", [...] as const)` registers:
|
||||
|
||||
```
|
||||
ABORTED, CREDENTIALS_MISSING, CREDENTIALS_INVALID, SIGNATURE_INVALID, TOKEN_EXPIRED,
|
||||
OBJECT_NOT_FOUND, BUCKET_NOT_FOUND, BUCKET_REGION_MISMATCH, REGION_MISMATCH,
|
||||
BUCKET_ALREADY_EXISTS, BUCKET_NOT_EMPTY, BUCKET_CREATE_FAILED, BUCKET_DELETE_FAILED,
|
||||
UPLOAD_FAILED, DELETE_FAILED, LIST_FAILED, HEAD_FAILED, COPY_FAILED, COPY_PART_FAILED,
|
||||
PRESIGN_FAILED, PRESIGN_TTL_INVALID, PRESIGN_HEADER_INVALID,
|
||||
MULTIPART_FAILED, MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED,
|
||||
MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED,
|
||||
SSE_KEY_MISSING, SSE_KEY_MISMATCH, SSE_KMS_DENIED, SSE_ALGORITHM_UNSUPPORTED,
|
||||
ACL_INVALID, ACL_PUT_FAILED, ACL_GET_FAILED,
|
||||
TAG_PUT_FAILED, TAG_DELETE_FAILED, TAG_GET_FAILED, TAG_LIMIT_EXCEEDED, TAG_INVALID,
|
||||
VERSION_NOT_FOUND, VERSIONING_DISABLED,
|
||||
LIFECYCLE_INVALID, LIFECYCLE_PUT_FAILED, LIFECYCLE_GET_FAILED, LIFECYCLE_DELETE_FAILED,
|
||||
CORS_INVALID, CORS_PUT_FAILED, CORS_GET_FAILED, CORS_DELETE_FAILED,
|
||||
PRECONDITION_FAILED, NOT_MODIFIED,
|
||||
RANGE_INVALID, RANGE_NOT_SATISFIABLE,
|
||||
CHECKSUM_MISMATCH, CHECKSUM_ALGORITHM_UNSUPPORTED,
|
||||
TRANSPORT_FAILED,
|
||||
```
|
||||
|
||||
The legacy 11 are retained at their existing identifiers; the additions
|
||||
extend the registry. `MULTIPART_FAILED` is documented as a back-compat
|
||||
alias — no post-change call site raises it.
|
||||
|
||||
## Test strategy
|
||||
|
||||
Per project rule §2 (no mocks): every integration test uses the Garage
|
||||
container from `@suaveplan/testing-integration-garage-s3`. Failure injection
|
||||
uses a wrapping `S3ClientFactory` that intercepts specific `Command`
|
||||
instances and either delays them or fails them — the underlying transport
|
||||
remains real.
|
||||
|
||||
Garage supports the full S3 surface this change targets, including
|
||||
multipart, SSE-S3 (and Garage's SSE-C compatibility), tagging, versioning,
|
||||
lifecycle, CORS, and bucket-level ops. SSE-KMS specifically requires a KMS
|
||||
plug; Garage's SSE-KMS mode is exercised via its built-in compatibility
|
||||
endpoint, and where Garage diverges from AWS (e.g. KMS key-policy
|
||||
enforcement), the test asserts the documented Garage behaviour and the
|
||||
fixture surface is extended in the same change.
|
||||
|
||||
For checksums, Garage validates both `ChecksumSHA256` and `ChecksumCRC32C`
|
||||
when supplied; tests assert round-trip and assert
|
||||
`STORAGE_S3.CHECKSUM_MISMATCH` on deliberately corrupted bytes.
|
||||
|
||||
## Decisions & trade-offs
|
||||
|
||||
| Decision | Alternative | Why |
|
||||
|---|---|---|
|
||||
| One change for full parity | Phase 1 (multipart only) + Phase 2 (everything else) | User explicitly required full parity, no cherry-picking. The superseded change folder confirms the prior split was rolled back. |
|
||||
| Native Range GET in `getPartial` | Keep `defaultGetPartial` fetch-then-slice | Eliminates O(N) bandwidth waste; `partialReads: true` becomes truthful. |
|
||||
| `setPartial` non-zero offset stays r-m-w | Emulate via multipart UploadPartCopy + new-part | S3 multipart cannot patch a byte range — emulation would download the prefix and suffix anyway, no win. |
|
||||
| Compose `@suaveplan/retry` | Keep hand-rolled `withRetry` | Single retry policy across all `@suaveplan/*` packages; classification logic still lives in `isRetryableS3Error`. |
|
||||
| SSE-C key material is not retained | Cache MD5 + plaintext key | Forcing per-call provision keeps the adapter stateless w.r.t. secret material. |
|
||||
| Garage over MinIO for integration | MinIO / LocalStack | Already shipped fixture; faster CI; full multipart + tagging + lifecycle + CORS support. |
|
||||
| `capabilities.partialWrites` stays `false` | Flip to `true` because multipart engine handles whole-object writes | Capability flag describes byte-range patching, which S3 cannot do. Flipping would lie to `drive-plugin-chunks` sparse-file layout. |
|
||||
| Inline `tags` at `set` time via `x-amz-tagging` header | Two-round-trip put-then-tag | Single PUT is faster and atomic. |
|
||||
Reference in New Issue
Block a user