Initial Commit
This commit is contained in:
387
openspec/changes/complete-storage-s3/design.md
Normal file
387
openspec/changes/complete-storage-s3/design.md
Normal file
@@ -0,0 +1,387 @@
|
||||
# Design: complete-storage-s3
|
||||
|
||||
Non-normative. The normative contract is `./specs/storage-s3/spec.md`.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Full S3 protocol parity** — every observable S3 capability that a
|
||||
production consumer needs is reachable from `@suaveplan/storage-s3` via
|
||||
either the `AbstractStorage` surface (whole-object IO) or one of the
|
||||
adapter's S3-specific extension methods (presign, ACLs, tags, versioning,
|
||||
lifecycle, CORS, bucket ops).
|
||||
2. **Tier-honest capability flags** — `capabilities.partialReads = true`
|
||||
(native Range), `partialWrites = false` (S3 has no byte-range PUT),
|
||||
`nativeStreams = true`, `atomicWrites = true`, `contentAddressable = false`.
|
||||
3. **Real-service tests only** — every multi-step path is exercised against
|
||||
the Garage testcontainer via
|
||||
`@suaveplan/testing-integration-garage-s3`.
|
||||
4. **No mocks** — failure injection uses a wrapping `S3ClientFactory` that
|
||||
intercepts specific commands; the underlying transport stays real.
|
||||
5. **Composable** — pluggable HTTP transport, pluggable signer, retry via
|
||||
`@suaveplan/retry`. Hand-rolled `withRetry` is removed.
|
||||
6. **100/100/100/100** coverage gate per project rule §7.
|
||||
|
||||
## File layout
|
||||
|
||||
```text
|
||||
packages/storage/storage-s3/src/
|
||||
├── s3/
|
||||
│ ├── s3.ts # top-level S3 class — orchestrates extensions
|
||||
│ ├── s3.md
|
||||
│ ├── s3.test.ts
|
||||
│ ├── s3-branches.test.ts
|
||||
│ └── index.ts # explicit named re-exports
|
||||
├── core/ # AbstractStorage methods
|
||||
│ ├── get/
|
||||
│ │ ├── get.ts # native Range + conditional + SSE-C
|
||||
│ │ ├── get.md
|
||||
│ │ ├── get.test.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── set/
|
||||
│ │ ├── set.ts # auto-promote, metadata, tagging, SSE, ACL
|
||||
│ │ ├── set.md
|
||||
│ │ ├── set.test.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── set-stream/
|
||||
│ │ ├── set-stream.ts
|
||||
│ │ ├── set-stream.md
|
||||
│ │ ├── set-stream.test.ts
|
||||
│ │ └── index.ts
|
||||
│ ├── set-partial/
|
||||
│ ├── delete/
|
||||
│ ├── head/ # head/stat with versionId + checksum
|
||||
│ ├── list/ # listObjectsV2 paginator
|
||||
│ └── exists/
|
||||
├── multipart/ # multipart engine (set + setStream + copy)
|
||||
│ ├── multipart-engine.ts
|
||||
│ ├── multipart-engine.md
|
||||
│ ├── multipart-engine.test.ts
|
||||
│ ├── part-chunker.ts # ReadableStream → AsyncIterable<PartInput>
|
||||
│ ├── part-chunker.md
|
||||
│ ├── part-chunker.test.ts
|
||||
│ ├── worker-pool.ts # bounded-concurrency scheduler + abort
|
||||
│ ├── worker-pool.md
|
||||
│ ├── worker-pool.test.ts
|
||||
│ ├── upload-part-copy.ts # server-side copy of > 5 GiB objects
|
||||
│ ├── upload-part-copy.md
|
||||
│ ├── upload-part-copy.test.ts
|
||||
│ └── index.ts
|
||||
├── presign/
|
||||
│ ├── get-signed-get-url.ts
|
||||
│ ├── get-signed-put-url.ts
|
||||
│ ├── get-signed-delete-url.ts
|
||||
│ ├── presign-options.ts # TTL + headers + content-type pin
|
||||
│ └── index.ts
|
||||
├── sse/
|
||||
│ ├── sse.ts # SSE-S3 / SSE-KMS / SSE-C param builder
|
||||
│ ├── sse.md
|
||||
│ ├── sse.test.ts
|
||||
│ └── index.ts
|
||||
├── acl/
|
||||
│ ├── acl.ts # canned + grant policies; get/put object ACL
|
||||
│ ├── acl.md
|
||||
│ ├── acl.test.ts
|
||||
│ └── index.ts
|
||||
├── copy/
|
||||
│ ├── copy.ts # copyObject + cross-bucket
|
||||
│ ├── copy.md
|
||||
│ ├── copy.test.ts
|
||||
│ └── index.ts
|
||||
├── tagging/
|
||||
│ ├── tagging.ts # get/put/delete + inline-at-set encoding
|
||||
│ ├── tagging.md
|
||||
│ ├── tagging.test.ts
|
||||
│ └── index.ts
|
||||
├── versioning/
|
||||
│ ├── versioning.ts # list versions + versionId on get/head/delete
|
||||
│ ├── versioning.md
|
||||
│ ├── versioning.test.ts
|
||||
│ └── index.ts
|
||||
├── lifecycle/
|
||||
│ ├── lifecycle.ts
|
||||
│ ├── lifecycle.md
|
||||
│ ├── lifecycle.test.ts
|
||||
│ └── index.ts
|
||||
├── cors/
|
||||
│ ├── cors.ts
|
||||
│ ├── cors.md
|
||||
│ ├── cors.test.ts
|
||||
│ └── index.ts
|
||||
├── bucket/
|
||||
│ ├── bucket.ts # create/delete/exists/region/listV2 paginator
|
||||
│ ├── bucket.md
|
||||
│ ├── bucket.test.ts
|
||||
│ └── index.ts
|
||||
├── conditional/
|
||||
│ ├── conditional.ts # If-Match / If-None-Match / If-*Modified-Since
|
||||
│ ├── conditional.md
|
||||
│ ├── conditional.test.ts
|
||||
│ └── index.ts
|
||||
├── checksum/
|
||||
│ ├── checksum.ts # SHA256 + CRC32C integrity guards
|
||||
│ ├── checksum.md
|
||||
│ ├── checksum.test.ts
|
||||
│ └── index.ts
|
||||
├── transport/
|
||||
│ ├── transport.ts # FetchLike adapter
|
||||
│ ├── transport.md
|
||||
│ ├── transport.test.ts
|
||||
│ └── index.ts
|
||||
├── signer/
|
||||
│ ├── signer.ts # SigV4 plug-point
|
||||
│ ├── signer.md
|
||||
│ ├── signer.test.ts
|
||||
│ └── index.ts
|
||||
├── retry/
|
||||
│ ├── retry.ts # composes @suaveplan/retry
|
||||
│ ├── retry.md
|
||||
│ ├── retry.test.ts
|
||||
│ └── index.ts
|
||||
├── constants/
|
||||
│ └── index.ts # all DEFAULT_* + MIN_PART_SIZE + thresholds
|
||||
├── error-codes.ts # ~50 STORAGE_S3.* codes
|
||||
├── error-codes.md
|
||||
├── types/
|
||||
│ ├── s3.types.ts
|
||||
│ ├── s3.types.md
|
||||
│ ├── sse.types.ts
|
||||
│ ├── acl.types.ts
|
||||
│ ├── tagging.types.ts
|
||||
│ ├── lifecycle.types.ts
|
||||
│ ├── cors.types.ts
|
||||
│ ├── conditional.types.ts
|
||||
│ ├── checksum.types.ts
|
||||
│ └── index.ts
|
||||
├── utils/
|
||||
│ ├── credential-validator.ts
|
||||
│ ├── build-key.ts
|
||||
│ ├── error-mapper.ts # S3ServiceException → S3StorageError
|
||||
│ └── index.ts
|
||||
└── index.ts # root barrel — namespace exports
|
||||
|
||||
packages/storage/storage-s3/tests/integration/
|
||||
├── core.test.ts # get/set/delete/list/head against Garage
|
||||
├── multipart.test.ts # auto-promote + abort + cleanup + UploadPartCopy
|
||||
├── stream.test.ts # setStream
|
||||
├── presign.test.ts # all three presigned variants
|
||||
├── sse.test.ts # SSE-S3 / SSE-KMS / SSE-C
|
||||
├── acl.test.ts # canned + grant + get/put
|
||||
├── copy.test.ts # within-bucket + cross-bucket
|
||||
├── tagging.test.ts
|
||||
├── versioning.test.ts
|
||||
├── lifecycle.test.ts
|
||||
├── cors.test.ts
|
||||
├── bucket.test.ts # create / delete / exists / region / listV2
|
||||
├── conditional.test.ts # 412 / 304 paths
|
||||
├── range.test.ts
|
||||
└── checksum.test.ts # SHA256 + CRC32C verify + mismatch
|
||||
```
|
||||
|
||||
## Capability flags
|
||||
|
||||
```ts
|
||||
public override readonly capabilities: StorageCapabilities = {
|
||||
partialReads: true, // native Range GET via getPartial
|
||||
partialWrites: false, // S3 has no byte-range PUT; multipart is whole-object
|
||||
nativeStreams: true, // setStream + range GET stream
|
||||
atomicWrites: true, // S3 PUT semantics are atomic relative to readers
|
||||
contentAddressable: false,
|
||||
};
|
||||
```
|
||||
|
||||
## Trigger thresholds
|
||||
|
||||
| Constant | Default | Min | Max | Rationale |
|
||||
|---|---|---|---|---|
|
||||
| `DEFAULT_MULTIPART_THRESHOLD` | 16 MiB | 5 MiB | n/a | Below this a single PUT is faster; at 16 MiB with 5 MiB parts and concurrency 4, every part starts in parallel. |
|
||||
| `DEFAULT_PART_SIZE` | 8 MiB | 5 MiB (AWS) | 5 GiB (AWS) | Doubles AWS minimum for fewer parts on medium objects. |
|
||||
| `DEFAULT_MULTIPART_CONCURRENCY` | 4 | 1 | 32 | WAN throughput sweet spot; > 32 stalls on TCP head-of-line per typical link. |
|
||||
| `DEFAULT_PRESIGN_EXPIRES` | 3600 s | 1 s | 7 days (AWS) | One-hour windows are the AWS console default. |
|
||||
| `MAX_PARTS` | 10 000 | n/a | n/a | AWS hard limit. |
|
||||
| `MAX_OBJECT_SIZE` | 5 TiB | n/a | n/a | AWS hard limit. |
|
||||
| `MAX_TAGS` | 10 | n/a | n/a | AWS hard limit. |
|
||||
|
||||
## Multipart engine
|
||||
|
||||
Same shape as the superseded `add-storage-s3-multipart` design (incorporated
|
||||
here verbatim — that design's `multipart-engine.ts`, `part-chunker.ts`,
|
||||
`worker-pool.ts` are imported wholesale into this change):
|
||||
|
||||
```ts
|
||||
type PartInput = { number: number; body: Uint8Array };
|
||||
|
||||
async runMultipart(
|
||||
fullKey: string,
|
||||
parts: AsyncIterable<PartInput>,
|
||||
tuning: Required<MultipartTuning>,
|
||||
signal: AbortSignal | undefined,
|
||||
): Promise<void> {
|
||||
const uploadId = await initMultipart(fullKey, signal);
|
||||
const completed: { ETag: string; PartNumber: number }[] = [];
|
||||
try {
|
||||
await runPartWorkers(fullKey, uploadId, parts, tuning.concurrency, completed, signal);
|
||||
completed.sort((a, b) => a.PartNumber - b.PartNumber);
|
||||
await completeMultipart(fullKey, uploadId, completed, signal);
|
||||
} catch (err) {
|
||||
await abortMultipart(fullKey, uploadId).catch(/* logged, swallowed */);
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Abort propagation: the engine creates an internal `AbortController`
|
||||
(`linked`). Its signal is forwarded to every `S3Client.send` call. `linked`
|
||||
is aborted when either the caller-supplied `signal` fires or any part worker
|
||||
rejects. The `AbortMultipartUploadCommand` is sent with a **fresh** signal
|
||||
so cleanup still runs after the caller aborts.
|
||||
|
||||
`UploadPartCopy` follows the same engine shape for server-side copy of
|
||||
objects > 5 GiB. Source range is split across part numbers per AWS
|
||||
`CopySourceRange` semantics.
|
||||
|
||||
## SSE parameter shaping
|
||||
|
||||
```ts
|
||||
export type SseConfig =
|
||||
| { kind: "sse-s3" }
|
||||
| { kind: "sse-kms"; keyId?: string; bucketKeyEnabled?: boolean }
|
||||
| { kind: "sse-c"; key: Uint8Array; keyMd5: Uint8Array };
|
||||
```
|
||||
|
||||
`sse-c` requires the caller to provide the key on every subsequent `get` /
|
||||
`head` / `copy` source — the adapter does not retain key material. The
|
||||
`keyMd5` field is validated against `MD5(key)` at the boundary; a mismatch
|
||||
throws `STORAGE_S3.SSE_KEY_MISMATCH` before the request leaves the process.
|
||||
|
||||
## Presign
|
||||
|
||||
```ts
|
||||
export interface PresignOptions {
|
||||
readonly expiresIn?: number; // 1..604800
|
||||
readonly requestHeaders?: Record<string, string>;
|
||||
readonly responseContentType?: string;
|
||||
readonly responseContentDisposition?: string;
|
||||
readonly contentType?: string; // pin on PUT (signed)
|
||||
readonly sse?: SseConfig;
|
||||
readonly checksumAlgorithm?: "SHA256" | "CRC32C";
|
||||
readonly versionId?: string; // GET only
|
||||
}
|
||||
```
|
||||
|
||||
`getSignedDeleteUrl` is new and supports `versionId` for versioned-bucket
|
||||
deletes. TTL validation rejects 0, negative, and > 7 days at the boundary
|
||||
with `STORAGE_S3.PRESIGN_TTL_INVALID`.
|
||||
|
||||
## Conditional ops
|
||||
|
||||
`get`, `head`, `delete`, `copy` accept:
|
||||
|
||||
```ts
|
||||
export interface ConditionalOptions {
|
||||
readonly ifMatch?: string; // eTag
|
||||
readonly ifNoneMatch?: string;
|
||||
readonly ifModifiedSince?: Date;
|
||||
readonly ifUnmodifiedSince?: Date;
|
||||
}
|
||||
```
|
||||
|
||||
The adapter maps HTTP responses to error codes:
|
||||
|
||||
| HTTP | Code |
|
||||
|---|---|
|
||||
| 412 | `STORAGE_S3.PRECONDITION_FAILED` |
|
||||
| 304 | `STORAGE_S3.NOT_MODIFIED` (returned as `undefined` from `get`, the code is attached to a returned sentinel for the metadata-aware variant) |
|
||||
|
||||
## Retry composition
|
||||
|
||||
Replaces the hand-rolled `withRetry` with `@suaveplan/retry`:
|
||||
|
||||
```ts
|
||||
const strategy = createRetryStrategy({
|
||||
maxAttempts: options.retry?.maxAttempts ?? 3,
|
||||
backoff: "exponential",
|
||||
baseDelayMs: options.retry?.baseDelayMs ?? 100,
|
||||
maxDelayMs: options.retry?.maxDelayMs ?? 30000,
|
||||
classify: (err) => isRetryableS3Error(err) ? "retryable" : "fatal",
|
||||
});
|
||||
```
|
||||
|
||||
`isRetryableS3Error` keeps its current classification (throttling, 5xx,
|
||||
`SlowDown`, `RequestTimeout`). Other failure modes are fatal.
|
||||
|
||||
## Pluggable transport + signer
|
||||
|
||||
```ts
|
||||
export interface FetchLike {
|
||||
(input: URL | string, init?: RequestInit): Promise<Response>;
|
||||
}
|
||||
|
||||
export interface SigV4Signer {
|
||||
sign(request: SignableRequest, options: SignOptions): Promise<SignedRequest>;
|
||||
}
|
||||
```
|
||||
|
||||
Defaults route through the AWS SDK's built-in. Custom transports plug at
|
||||
`S3ClientFactory.create` via `requestHandler` (Node) or `Fetch` (browser /
|
||||
edge). Custom signers replace the SDK's `SignatureV4` provider.
|
||||
|
||||
## Error-code map (~50 codes)
|
||||
|
||||
`defineCodes("STORAGE_S3", [...] as const)` registers:
|
||||
|
||||
```
|
||||
ABORTED, CREDENTIALS_MISSING, CREDENTIALS_INVALID, SIGNATURE_INVALID, TOKEN_EXPIRED,
|
||||
OBJECT_NOT_FOUND, BUCKET_NOT_FOUND, BUCKET_REGION_MISMATCH, REGION_MISMATCH,
|
||||
BUCKET_ALREADY_EXISTS, BUCKET_NOT_EMPTY, BUCKET_CREATE_FAILED, BUCKET_DELETE_FAILED,
|
||||
UPLOAD_FAILED, DELETE_FAILED, LIST_FAILED, HEAD_FAILED, COPY_FAILED, COPY_PART_FAILED,
|
||||
PRESIGN_FAILED, PRESIGN_TTL_INVALID, PRESIGN_HEADER_INVALID,
|
||||
MULTIPART_FAILED, MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED,
|
||||
MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED,
|
||||
SSE_KEY_MISSING, SSE_KEY_MISMATCH, SSE_KMS_DENIED, SSE_ALGORITHM_UNSUPPORTED,
|
||||
ACL_INVALID, ACL_PUT_FAILED, ACL_GET_FAILED,
|
||||
TAG_PUT_FAILED, TAG_DELETE_FAILED, TAG_GET_FAILED, TAG_LIMIT_EXCEEDED, TAG_INVALID,
|
||||
VERSION_NOT_FOUND, VERSIONING_DISABLED,
|
||||
LIFECYCLE_INVALID, LIFECYCLE_PUT_FAILED, LIFECYCLE_GET_FAILED, LIFECYCLE_DELETE_FAILED,
|
||||
CORS_INVALID, CORS_PUT_FAILED, CORS_GET_FAILED, CORS_DELETE_FAILED,
|
||||
PRECONDITION_FAILED, NOT_MODIFIED,
|
||||
RANGE_INVALID, RANGE_NOT_SATISFIABLE,
|
||||
CHECKSUM_MISMATCH, CHECKSUM_ALGORITHM_UNSUPPORTED,
|
||||
TRANSPORT_FAILED,
|
||||
```
|
||||
|
||||
The legacy 11 are retained at their existing identifiers; the additions
|
||||
extend the registry. `MULTIPART_FAILED` is documented as a back-compat
|
||||
alias — no post-change call site raises it.
|
||||
|
||||
## Test strategy
|
||||
|
||||
Per project rule §2 (no mocks): every integration test uses the Garage
|
||||
container from `@suaveplan/testing-integration-garage-s3`. Failure injection
|
||||
uses a wrapping `S3ClientFactory` that intercepts specific `Command`
|
||||
instances and either delays them or fails them — the underlying transport
|
||||
remains real.
|
||||
|
||||
Garage supports the full S3 surface this change targets, including
|
||||
multipart, SSE-S3 (and Garage's SSE-C compatibility), tagging, versioning,
|
||||
lifecycle, CORS, and bucket-level ops. SSE-KMS specifically requires a KMS
|
||||
plug; Garage's SSE-KMS mode is exercised via its built-in compatibility
|
||||
endpoint, and where Garage diverges from AWS (e.g. KMS key-policy
|
||||
enforcement), the test asserts the documented Garage behaviour and the
|
||||
fixture surface is extended in the same change.
|
||||
|
||||
For checksums, Garage validates both `ChecksumSHA256` and `ChecksumCRC32C`
|
||||
when supplied; tests assert round-trip and assert
|
||||
`STORAGE_S3.CHECKSUM_MISMATCH` on deliberately corrupted bytes.
|
||||
|
||||
## Decisions & trade-offs
|
||||
|
||||
| Decision | Alternative | Why |
|
||||
|---|---|---|
|
||||
| One change for full parity | Phase 1 (multipart only) + Phase 2 (everything else) | User explicitly required full parity, no cherry-picking. The superseded change folder confirms the prior split was rolled back. |
|
||||
| Native Range GET in `getPartial` | Keep `defaultGetPartial` fetch-then-slice | Eliminates O(N) bandwidth waste; `partialReads: true` becomes truthful. |
|
||||
| `setPartial` non-zero offset stays r-m-w | Emulate via multipart UploadPartCopy + new-part | S3 multipart cannot patch a byte range — emulation would download the prefix and suffix anyway, no win. |
|
||||
| Compose `@suaveplan/retry` | Keep hand-rolled `withRetry` | Single retry policy across all `@suaveplan/*` packages; classification logic still lives in `isRetryableS3Error`. |
|
||||
| SSE-C key material is not retained | Cache MD5 + plaintext key | Forcing per-call provision keeps the adapter stateless w.r.t. secret material. |
|
||||
| Garage over MinIO for integration | MinIO / LocalStack | Already shipped fixture; faster CI; full multipart + tagging + lifecycle + CORS support. |
|
||||
| `capabilities.partialWrites` stays `false` | Flip to `true` because multipart engine handles whole-object writes | Capability flag describes byte-range patching, which S3 cannot do. Flipping would lie to `drive-plugin-chunks` sparse-file layout. |
|
||||
| Inline `tags` at `set` time via `x-amz-tagging` header | Two-round-trip put-then-tag | Single PUT is faster and atomic. |
|
||||
301
openspec/changes/complete-storage-s3/proposal.md
Normal file
301
openspec/changes/complete-storage-s3/proposal.md
Normal file
@@ -0,0 +1,301 @@
|
||||
# OpenSpec Change: complete-storage-s3
|
||||
|
||||
- **Change ID**: complete-storage-s3
|
||||
- **Package**: `@suaveplan/storage-s3` (modify — complete to full S3 parity)
|
||||
- **Status**: PROPOSED
|
||||
- **Date**: 2026-05-17
|
||||
- **Design**: ./design.md
|
||||
- **Spec delta**: ./specs/storage-s3/spec.md
|
||||
|
||||
## Why
|
||||
|
||||
`@suaveplan/storage-s3` ships a working but thin `AbstractStorage`
|
||||
implementation: it does `get` / `set` / `delete` / `list` / `head`, a serial
|
||||
multipart `setMultipart`, presigned URLs limited to `GET` / `PUT`, and one copy
|
||||
helper. Almost every other surface S3 exposes — and that consumers actually
|
||||
need to ship production object-store workloads — is absent:
|
||||
|
||||
- No auto-promote of `set` to multipart for large payloads; the existing
|
||||
`setMultipart` is serial and not wired into the `AbstractStorage` surface.
|
||||
- No streaming `set` from `ReadableStream<Uint8Array>` without buffering.
|
||||
- No `getSignedDeleteUrl`; presigning is fixed to `GET` / `PUT` with no TTL
|
||||
semantics, no response/request header pinning, no content-type or
|
||||
content-disposition control.
|
||||
- No server-side encryption (SSE-S3, SSE-KMS, SSE-C) on `set` or `get`.
|
||||
- No object metadata controls — content-type, content-encoding,
|
||||
content-disposition, cache-control, user-defined `x-amz-meta-*`.
|
||||
- No ACLs (canned or grant policies), no object tagging, no versioning
|
||||
surface, no bucket lifecycle, no bucket CORS, no bucket-level ops (create
|
||||
/ delete / exists / region), no conditional ops (`If-Match`,
|
||||
`If-None-Match`, `If-Modified-Since`, `If-Unmodified-Since`).
|
||||
- No range reads via `get`; `getPartial` round-trips the whole object via
|
||||
the default fallback.
|
||||
- No checksum integrity (SHA-256 / CRC32C) per AWS-recommended guard.
|
||||
- Pluggable HTTP transport (`FetchLike`), pluggable SigV4 signer, and retry
|
||||
composition via `@suaveplan/retry` are not yet integrated; retries are
|
||||
hand-rolled in `withRetry`.
|
||||
- Error coverage is 11 codes for a surface that needs ~50.
|
||||
- No real-service test harness wiring against the existing
|
||||
`@suaveplan/testing-integration-garage-s3` fixture for the multi-feature
|
||||
paths above.
|
||||
|
||||
The supersession marker
|
||||
`openspec/changes/_superseded-narrow-deferrals-2026-05-17/add-storage-s3-multipart/`
|
||||
confirms that the prior narrow deferral (multipart only) was rolled back in
|
||||
favour of a single change that lands the full S3 surface at parity. This
|
||||
change is that rollup. No cherry-picking, no Phase 2 split.
|
||||
|
||||
## What Changes
|
||||
|
||||
This change brings `@suaveplan/storage-s3` to **full S3 feature parity** in a
|
||||
single shipped change. Every capability below is in scope; none are deferred.
|
||||
|
||||
### AbstractStorage surface (production-grade)
|
||||
|
||||
- `get`, `set`, `setPartial`, `has`/`exists`, `delete`, `list`, `head`/`stat`
|
||||
— each becomes production-grade with full options, conditional headers,
|
||||
SSE, and checksum integrity passthrough.
|
||||
- `getPartial` uses native S3 `Range:` reads — no more fetch-the-whole-object
|
||||
fallback.
|
||||
- `setPartial` for `offset === 0` AND `bytes.byteLength > threshold` routes
|
||||
through the native multipart engine; non-zero offsets retain
|
||||
`defaultSetPartial` r-m-w because S3 has no byte-range PUT.
|
||||
`capabilities.partialWrites` stays `false` to honestly advertise that.
|
||||
|
||||
### Multipart (auto-promote, bounded concurrency, abort+cleanup)
|
||||
|
||||
- `S3.set` auto-promotes to multipart when `value.byteLength` exceeds a
|
||||
configurable threshold (default 16 MiB).
|
||||
- Internal `multipart-engine` owns `CreateMultipartUpload` → bounded-parallel
|
||||
`UploadPart` (default concurrency 4) → `CompleteMultipartUpload`, with
|
||||
full abort propagation to in-flight workers and guaranteed
|
||||
`AbortMultipartUpload` cleanup.
|
||||
- Per-stage error codes: `MULTIPART_INIT_FAILED`, `MULTIPART_PART_FAILED`,
|
||||
`MULTIPART_COMPLETE_FAILED`, `MULTIPART_ABORT_FAILED`. Legacy
|
||||
`MULTIPART_FAILED` retained as alias.
|
||||
|
||||
### Streaming put
|
||||
|
||||
- `S3.setStream(key, stream, opts?)` consumes `ReadableStream<Uint8Array>`
|
||||
without buffering the whole stream — chunked into parts on the fly with a
|
||||
ceiling of 2 part-sized buffers in memory.
|
||||
|
||||
### Presigned URLs
|
||||
|
||||
- `getSignedGetUrl`, `getSignedPutUrl`, `getSignedDeleteUrl` — each with
|
||||
TTL, custom request headers, response-content-type and
|
||||
response-content-disposition overrides, content-type pinning, and SSE
|
||||
parameter forwarding.
|
||||
|
||||
### Server-side encryption
|
||||
|
||||
- SSE-S3 (`AES256`), SSE-KMS (`aws:kms` with optional `SSEKMSKeyId` +
|
||||
`BucketKeyEnabled`), SSE-C (caller-supplied key material) on every
|
||||
`set`, `setStream`, `setMultipart`, presigned PUT, and `copy`.
|
||||
- `get` accepts the matching SSE-C key for decryption.
|
||||
|
||||
### Object metadata
|
||||
|
||||
- `set` accepts `contentType`, `contentEncoding`, `contentDisposition`,
|
||||
`contentLanguage`, `cacheControl`, `expires`, and a free-form
|
||||
`userMetadata: Record<string, string>` mapped to `x-amz-meta-*`.
|
||||
- `head` / `getMetadata` returns the full metadata bag including
|
||||
`eTag`, `versionId`, `lastModified`, `storageClass`, and the
|
||||
caller-set headers above.
|
||||
|
||||
### ACLs
|
||||
|
||||
- Canned ACLs (`private`, `public-read`, `public-read-write`,
|
||||
`aws-exec-read`, `authenticated-read`, `bucket-owner-read`,
|
||||
`bucket-owner-full-control`, `log-delivery-write`) per put.
|
||||
- Explicit grant policies (`grantRead`, `grantReadACP`, `grantWriteACP`,
|
||||
`grantFullControl`) per put.
|
||||
- `getObjectAcl` / `putObjectAcl` for post-write changes.
|
||||
|
||||
### Copy
|
||||
|
||||
- `copyObject(source, destination, opts?)` — server-side copy within bucket
|
||||
AND across buckets via `{ sourceBucket }`. Supports metadata directive
|
||||
(`COPY` / `REPLACE`), tagging directive, conditional headers, SSE
|
||||
parameter overrides.
|
||||
- `UploadPartCopy` engine for objects > 5 GiB (multipart server-side copy).
|
||||
|
||||
### Tagging
|
||||
|
||||
- `getObjectTags`, `putObjectTags`, `deleteObjectTags`.
|
||||
- `set` accepts inline `tags: Record<string, string>` (URL-encoded into the
|
||||
`x-amz-tagging` header — no extra round trip).
|
||||
|
||||
### Versioning
|
||||
|
||||
- `listObjectVersions` paginator.
|
||||
- `get(key, { versionId })`, `head(key, { versionId })`,
|
||||
`delete(key, { versionId })`.
|
||||
|
||||
### Bucket-level operations
|
||||
|
||||
- `createBucket`, `deleteBucket`, `bucketExists`, `getBucketRegion`,
|
||||
`listObjectsV2` (paginator with `prefix`, `delimiter`, `maxKeys`,
|
||||
`continuationToken`, `startAfter`).
|
||||
- Lifecycle config: `getBucketLifecycle`, `putBucketLifecycle`,
|
||||
`deleteBucketLifecycle`.
|
||||
- CORS config: `getBucketCors`, `putBucketCors`, `deleteBucketCors`.
|
||||
|
||||
### Conditional ops
|
||||
|
||||
- `IfMatch`, `IfNoneMatch`, `IfModifiedSince`, `IfUnmodifiedSince` on
|
||||
`get` / `head` / `delete` / `copy` — translated to the precondition error
|
||||
code `PRECONDITION_FAILED` (412) and `NOT_MODIFIED` (304).
|
||||
|
||||
### Range reads
|
||||
|
||||
- `get(key, { range: { offset, length } })` issues a native `Range:` GET
|
||||
rather than a full-object fetch + slice.
|
||||
|
||||
### Checksums
|
||||
|
||||
- Per-request `SHA256` (default) and `CRC32C` integrity guards per the
|
||||
AWS-recommended additional checksum modes. Returned `ChecksumSHA256` /
|
||||
`ChecksumCRC32C` are surfaced on the metadata bag and verified on
|
||||
`get`-with-checksum.
|
||||
|
||||
### Pluggable HTTP transport, signer, retry
|
||||
|
||||
- `S3Options.transport?: FetchLike` to swap the underlying HTTP transport
|
||||
(browser `fetch`, Node `undici`, custom edge fetch). Default uses the
|
||||
AWS SDK's built-in.
|
||||
- `S3Options.signer?: SigV4Signer` to swap the signer (default uses AWS SDK
|
||||
SigV4 implementation).
|
||||
- Retry is composed through `@suaveplan/retry` (`createRetryStrategy`) —
|
||||
the hand-rolled `withRetry` is replaced. Throttling / 5xx classification
|
||||
preserved.
|
||||
|
||||
### Error codes
|
||||
|
||||
`S3_CODES` expands from 11 to ~50 codes covering every failure path:
|
||||
SSE (`SSE_KEY_MISSING`, `SSE_KEY_MISMATCH`, `SSE_KMS_DENIED`),
|
||||
ACLs (`ACL_INVALID`, `ACL_PUT_FAILED`),
|
||||
tagging (`TAG_PUT_FAILED`, `TAG_DELETE_FAILED`, `TAG_LIMIT_EXCEEDED`),
|
||||
versioning (`VERSION_NOT_FOUND`, `VERSIONING_DISABLED`),
|
||||
lifecycle (`LIFECYCLE_INVALID`, `LIFECYCLE_PUT_FAILED`),
|
||||
CORS (`CORS_INVALID`, `CORS_PUT_FAILED`),
|
||||
bucket ops (`BUCKET_ALREADY_EXISTS`, `BUCKET_NOT_EMPTY`, `BUCKET_DELETE_FAILED`,
|
||||
`BUCKET_CREATE_FAILED`, `BUCKET_REGION_MISMATCH`),
|
||||
conditional (`PRECONDITION_FAILED`, `NOT_MODIFIED`),
|
||||
copy (`COPY_FAILED`, `COPY_PART_FAILED`),
|
||||
multipart (4 new codes above; `MULTIPART_FAILED` retained),
|
||||
presign (`PRESIGN_TTL_INVALID`, `PRESIGN_HEADER_INVALID`),
|
||||
checksums (`CHECKSUM_MISMATCH`, `CHECKSUM_ALGORITHM_UNSUPPORTED`),
|
||||
range (`RANGE_INVALID`),
|
||||
auth (`SIGNATURE_INVALID`, `TOKEN_EXPIRED`),
|
||||
transport (`TRANSPORT_FAILED`).
|
||||
|
||||
### Tests via real Garage container
|
||||
|
||||
- `@suaveplan/testing-integration-garage-s3` (already shipped) hosts
|
||||
every integration test. No mocks. The fixture is extended in this
|
||||
change if any capability surface (e.g. `PutBucketLifecycleCommand`)
|
||||
needs additional helper exposure.
|
||||
|
||||
### 100% coverage gate
|
||||
|
||||
- Per project rule: `100/100/100/100` istanbul, gated by
|
||||
`bunx turbo test:coverage --filter=@suaveplan/storage-s3`.
|
||||
|
||||
## Parity Checklist
|
||||
|
||||
This change has no upstream `../suaveplan/packages/storage-s3/` to migrate
|
||||
from (the upstream package was never authored there — confirmed by `ls`
|
||||
returning `No such file or directory`). The parity target is **the full
|
||||
public S3 API surface** as documented by AWS, not an upstream library.
|
||||
Therefore the table below is structured by S3 capability rather than upstream
|
||||
file:
|
||||
|
||||
| Capability | S3 native | Migrated | Notes |
|
||||
|---|---|---|---|
|
||||
| `get` / `set` / `delete` / `list` / `head` | ✅ | ✅ | Production-grade with full options. |
|
||||
| `setPartial` (whole-object via multipart) | ✅ | ✅ | Non-zero offset retains r-m-w; `partialWrites: false`. |
|
||||
| `getPartial` via native `Range:` | ✅ | ✅ | Replaces fetch-whole-then-slice. |
|
||||
| Multipart auto-promote + parallel + abort | ✅ | ✅ | Default threshold 16 MiB, concurrency 4. |
|
||||
| Streaming `setStream` | ✅ | ✅ | `ReadableStream<Uint8Array>` without buffering. |
|
||||
| Presigned GET / PUT / DELETE | ✅ | ✅ | TTL + headers + content-type pin. |
|
||||
| SSE-S3 / SSE-KMS / SSE-C | ✅ | ✅ | Per put / get / copy. |
|
||||
| Object metadata (content-* + user-metadata) | ✅ | ✅ | Full bag. |
|
||||
| Canned ACLs + grant policies + getObjectAcl/putObjectAcl | ✅ | ✅ | |
|
||||
| `copyObject` (single-PUT + UploadPartCopy) | ✅ | ✅ | Cross-bucket supported. |
|
||||
| Object tagging (get/put/delete + inline at set) | ✅ | ✅ | |
|
||||
| Versioning (list versions, versionId on get/head/delete) | ✅ | ✅ | |
|
||||
| Bucket lifecycle (get/put/delete) | ✅ | ✅ | |
|
||||
| Bucket CORS (get/put/delete) | ✅ | ✅ | |
|
||||
| Bucket ops (create / delete / exists / region) | ✅ | ✅ | |
|
||||
| `listObjectsV2` paginator | ✅ | ✅ | Prefix/delimiter/max/continuation/startAfter. |
|
||||
| Conditional ops (IfMatch / IfNoneMatch / If-*Modified-Since) | ✅ | ✅ | |
|
||||
| Checksums (SHA256, CRC32C) | ✅ | ✅ | |
|
||||
| Pluggable transport + signer + `@suaveplan/retry` | n/a | ✅ | Genesis-only requirement. |
|
||||
| Comprehensive `STORAGE_S3.*` codes | n/a | ✅ | ~50 codes. |
|
||||
| Real Garage integration tests | n/a | ✅ | Via `testing-integration-garage-s3`. |
|
||||
|
||||
No row is 🚫. Every capability lands in this change.
|
||||
|
||||
## Out of scope
|
||||
|
||||
Strictly things outside the S3 protocol identity:
|
||||
|
||||
- **CloudFront integration** (signed-cookie / signed-URL generation against
|
||||
a CloudFront key-pair, OAI / OAC origin-access-control). Belongs in a
|
||||
separate `@suaveplan/storage-cdn-cloudfront` package.
|
||||
- **AWS-account-level operations** — IAM policy authoring, STS role
|
||||
assumption beyond the standard credential provider chain, KMS key
|
||||
creation / rotation. Out of `storage-s3`'s remit.
|
||||
- **Resumable uploads with cross-process persistence** of `UploadId` +
|
||||
completed-part list. Multipart resume mid-process is supported via the
|
||||
engine; cross-restart resume requires a separate coordinator.
|
||||
- **S3 Object Lambda / Object Lock retention modes** — distinct surface,
|
||||
not part of the core object-store contract.
|
||||
- **Bucket policies / public-access blocks** beyond the ACL surface above.
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
Run `bunx turbo ls --filter=...@suaveplan/storage-s3` to enumerate. Expected
|
||||
dependents (verified at implementation time):
|
||||
|
||||
- `@suaveplan/storage` — [ ] tests run / [ ] docs reviewed / [x] no action
|
||||
needed (reason: `storage-s3` consumes `AbstractStorage`; no
|
||||
surface-of-`storage` change.)
|
||||
- `@suaveplan/drive-plugin-chunks` — [x] tests run / [x] docs reviewed /
|
||||
[ ] no action needed (reason: declared consumer of `setPartial` and
|
||||
range reads via `DriveStorage`; benefits directly from the native range
|
||||
GET and large-object write path.)
|
||||
- `@suaveplan/drive-plugin-cache` — [ ] tests run / [ ] docs reviewed /
|
||||
[x] no action needed (reason: cache layer round-trips byte ranges; new
|
||||
range path is observable but contract unchanged.)
|
||||
- `@suaveplan/storage-cdn` / `storage-cdn-*` — [ ] tests run / [ ] docs
|
||||
reviewed / [x] no action needed (reason: CDN adapters do not consume
|
||||
`storage-s3` directly.)
|
||||
|
||||
`storage-s3` is not a foundation package per §14a, but the dependent sweep
|
||||
(§13 step 10, `bunx turbo test --filter=...@suaveplan/storage-s3`) MUST run
|
||||
before archive because the multipart / range / SSE wiring is observable to
|
||||
chunked-drive consumers.
|
||||
|
||||
## Migration
|
||||
|
||||
The change is mostly additive on the public surface; the few behaviour
|
||||
changes that consumers can observe are listed below with grep commands:
|
||||
|
||||
1. **`set` for large payloads now issues multipart** — observable via
|
||||
`S3ServiceException` shape; no behaviour change for size ≤ 16 MiB.
|
||||
2. **`getPartial` now uses native Range** — observable via wire traffic;
|
||||
contract identical (same bytes returned).
|
||||
3. **Retry composition now flows through `@suaveplan/retry`** — observable
|
||||
only if callers introspect retry timing. Error codes preserved.
|
||||
4. **`S3_CODES` gains ~40 new entries** — consumers asserting on the legacy
|
||||
11 codes are unaffected; new failure paths surface the new codes.
|
||||
|
||||
Search command for consumers depending on the legacy error-code surface:
|
||||
|
||||
```bash
|
||||
grep -rln "S3_CODES\." packages/ | grep -v "/dist/\|/coverage/\|/storage-s3/"
|
||||
```
|
||||
642
openspec/changes/complete-storage-s3/specs/storage-s3/spec.md
Normal file
642
openspec/changes/complete-storage-s3/specs/storage-s3/spec.md
Normal file
@@ -0,0 +1,642 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: storage-s3 (complete-storage-s3)
|
||||
|
||||
**Package**: `@suaveplan/storage-s3`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line authored at
|
||||
> draft time. Paths name files this change will create; they do not exist
|
||||
> yet.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: AbstractStorage compliance with honest capability flags
|
||||
|
||||
The `S3` class SHALL extend `AbstractStorage` and implement every abstract
|
||||
method (`get`, `set`, `setPartial`, `delete`, `exists`, `list`,
|
||||
`getMetadata`, `getWithMetadata`, `getSize`, `getPartial`, `getItemSize`,
|
||||
`createReadStream`, `createWriteStream`, `clear`). It SHALL expose
|
||||
`capabilities = { partialReads: true, partialWrites: false,
|
||||
nativeStreams: true, atomicWrites: true, contentAddressable: false }`.
|
||||
`partialWrites` SHALL remain `false` because S3 has no byte-range PUT.
|
||||
|
||||
#### Scenario: every abstract method is implemented
|
||||
|
||||
- **WHEN** a consumer instantiates `new S3(opts)` and inspects the
|
||||
prototype chain
|
||||
- **THEN** every abstract method declared on `AbstractStorage` resolves to
|
||||
a non-inherited concrete implementation, and `s3.capabilities` matches
|
||||
the documented flag set verbatim
|
||||
|
||||
**Verified by:** `src/s3/s3.test.ts::"implements every AbstractStorage method"`
|
||||
AND `src/s3/s3.test.ts::"reports the documented capability flags"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Native range reads via `getPartial`
|
||||
|
||||
`S3.getPartial(key, offset, length, signal?)` SHALL issue a
|
||||
`GetObjectCommand` with `Range: bytes=<offset>-<offset+length-1>` rather
|
||||
than fetching the full object. Out-of-range requests SHALL raise
|
||||
`S3StorageError` with code `STORAGE_S3.RANGE_NOT_SATISFIABLE`. Zero-length
|
||||
requests SHALL return an empty `Uint8Array` without issuing a network
|
||||
call.
|
||||
|
||||
#### Scenario: partial GET issues a single ranged request
|
||||
|
||||
- **WHEN** a caller invokes `s3.getPartial("k", 100, 200)` on an existing
|
||||
10 KiB object
|
||||
- **THEN** the underlying client issues exactly one `GetObjectCommand`
|
||||
with `Range: bytes=100-299` and returns exactly 200 bytes
|
||||
|
||||
#### Scenario: out-of-range raises typed error
|
||||
|
||||
- **WHEN** a caller invokes `s3.getPartial("k", 1024, 1024)` on a 512-byte
|
||||
object
|
||||
- **THEN** the call rejects with `S3StorageError` whose `code` is
|
||||
`STORAGE_S3.RANGE_NOT_SATISFIABLE`
|
||||
|
||||
**Verified by:** `tests/integration/range.test.ts::"partial GET issues a single ranged request"`
|
||||
AND `tests/integration/range.test.ts::"out-of-range raises RANGE_NOT_SATISFIABLE"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Auto-promote `set` to multipart above threshold
|
||||
|
||||
`S3.set(key, value, options?, signal?)` SHALL internally upload `value` via
|
||||
S3 multipart (`CreateMultipartUpload` → bounded-parallel `UploadPart` →
|
||||
`CompleteMultipartUpload`) when `value.byteLength` exceeds
|
||||
`S3Options.multipart.threshold` (default 16 MiB). At or below the threshold
|
||||
the call SHALL issue a single `PutObjectCommand`. The public signature and
|
||||
observable success semantics SHALL NOT change.
|
||||
|
||||
#### Scenario: large byte array triggers multipart
|
||||
|
||||
- **WHEN** a caller invokes `s3.set("k", new Uint8Array(20 * 1024 * 1024))`
|
||||
against an `S3` instance with default tuning
|
||||
- **THEN** the underlying client issues exactly one
|
||||
`CreateMultipartUploadCommand`, one or more `UploadPartCommand`, exactly
|
||||
one `CompleteMultipartUploadCommand`, and zero `PutObjectCommand` for
|
||||
that key
|
||||
|
||||
#### Scenario: small byte array uses single PUT
|
||||
|
||||
- **WHEN** a caller invokes `s3.set("k", new Uint8Array(1024))`
|
||||
- **THEN** the underlying client issues exactly one `PutObjectCommand` and
|
||||
zero `CreateMultipartUploadCommand` for that key
|
||||
|
||||
**Verified by:** `tests/integration/multipart.test.ts::"large byte array triggers multipart"`
|
||||
AND `tests/integration/multipart.test.ts::"small byte array uses single PUT"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Streaming put without whole-stream buffering
|
||||
|
||||
`S3.setStream(key, stream, opts?)` SHALL consume a
|
||||
`ReadableStream<Uint8Array>` and upload it via the multipart engine. At no
|
||||
point SHALL the implementation buffer more than two parts' worth of bytes
|
||||
in memory (in-flight read + accumulating chunker buffer). The final part
|
||||
MAY be shorter than `partSize`.
|
||||
|
||||
#### Scenario: 25 MiB stream produces five 5 MiB parts
|
||||
|
||||
- **WHEN** a caller invokes `s3.setStream("k", stream, { partSize: 5 * 1024 * 1024 })`
|
||||
where `stream` yields exactly 25 MiB
|
||||
- **THEN** the upload completes with exactly 5 `UploadPartCommand` calls
|
||||
and the round-tripped GET equals the input byte-for-byte
|
||||
|
||||
**Verified by:** `src/multipart/part-chunker.test.ts::"emits exact part-size chunks then a ragged final part"`
|
||||
AND `tests/integration/stream.test.ts::"25 MiB stream produces five 5 MiB parts"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Bounded multipart concurrency and ordered parts
|
||||
|
||||
The multipart engine SHALL upload parts with at most
|
||||
`S3Options.multipart.concurrency` (default 4) `UploadPartCommand` calls in
|
||||
flight simultaneously. Concurrency SHALL be validated at construction
|
||||
time to lie in the inclusive range `[1, 32]`; out-of-range values SHALL
|
||||
throw `S3StorageError(STORAGE_S3.MULTIPART_INIT_FAILED, ...)` at
|
||||
construction. The final `MultipartUpload.Parts` list SHALL be sorted
|
||||
ascending by `PartNumber` regardless of completion order.
|
||||
|
||||
#### Scenario: concurrency cap is honoured
|
||||
|
||||
- **WHEN** a caller invokes `setStream` with 12 parts and concurrency 4,
|
||||
using an injected `S3ClientFactory` that records pending `UploadPart`
|
||||
promise count at each tick
|
||||
- **THEN** the recorded in-flight count never exceeds 4 and the final
|
||||
`CompleteMultipartUploadCommand.MultipartUpload.Parts` array is sorted
|
||||
by `PartNumber`
|
||||
|
||||
**Verified by:** `src/multipart/worker-pool.test.ts::"never exceeds the configured concurrency"`
|
||||
AND `tests/integration/multipart.test.ts::"concurrency cap is honoured under injected part latency"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Abort propagation and guaranteed multipart cleanup
|
||||
|
||||
The multipart engine SHALL react to a caller-supplied `AbortSignal` firing
|
||||
mid-upload by cancelling every in-flight `UploadPartCommand` via a linked
|
||||
`AbortController` signal passed to `S3Client.send`, SHALL stop drawing
|
||||
further parts from the input iterator, and SHALL issue exactly one
|
||||
`AbortMultipartUploadCommand` for the open `UploadId` before re-throwing.
|
||||
The cleanup `AbortMultipartUploadCommand` SHALL use a fresh signal so it
|
||||
runs to completion even after the caller aborts.
|
||||
|
||||
#### Scenario: caller abort cancels parts and cleans up
|
||||
|
||||
- **WHEN** a caller starts `setStream` on a 50 MiB stream then calls
|
||||
`controller.abort()` after the second part begins
|
||||
- **THEN** the call rejects, the underlying client receives at least one
|
||||
`AbortMultipartUploadCommand` for the key, and a subsequent
|
||||
`ListMultipartUploadsCommand` for the bucket returns no open uploads
|
||||
for that key
|
||||
|
||||
**Verified by:** `tests/integration/multipart.test.ts::"caller abort cancels parts and cleans up"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Per-stage multipart error codes
|
||||
|
||||
Multipart failures SHALL raise `S3StorageError` with stage-specific codes:
|
||||
`STORAGE_S3.MULTIPART_INIT_FAILED` when `CreateMultipartUpload` fails or
|
||||
returns no `UploadId`; `STORAGE_S3.MULTIPART_PART_FAILED` when an
|
||||
`UploadPart` call fails after retries, with `partNumber` populated in
|
||||
`context`; `STORAGE_S3.MULTIPART_COMPLETE_FAILED` when
|
||||
`CompleteMultipartUpload` fails; `STORAGE_S3.MULTIPART_ABORT_FAILED` when
|
||||
the cleanup `AbortMultipartUpload` itself fails (logged at warn level;
|
||||
MUST NOT replace the original thrown error). `STORAGE_S3.MULTIPART_FAILED`
|
||||
SHALL remain registered as a back-compat alias and SHALL NOT be raised by
|
||||
any post-change call site.
|
||||
|
||||
#### Scenario: part failure surfaces stage-specific code
|
||||
|
||||
- **WHEN** a wrapping `S3ClientFactory` makes the third `UploadPartCommand`
|
||||
reject deterministically and the caller invokes `setStream` on a stream
|
||||
with at least three parts
|
||||
- **THEN** the call rejects with `S3StorageError` whose `code` is
|
||||
`STORAGE_S3.MULTIPART_PART_FAILED` and whose `context.partNumber` is 3
|
||||
|
||||
#### Scenario: cleanup failure does not mask original error
|
||||
|
||||
- **WHEN** an `UploadPart` rejection triggers cleanup, and the injected
|
||||
`AbortMultipartUploadCommand` also rejects
|
||||
- **THEN** the call still rejects with the original
|
||||
`STORAGE_S3.MULTIPART_PART_FAILED`, and a logger record at warn level
|
||||
carries the `STORAGE_S3.MULTIPART_ABORT_FAILED` code
|
||||
|
||||
**Verified by:** `src/multipart/multipart-engine.test.ts::"maps stage failures to dedicated codes"`
|
||||
AND `tests/integration/multipart.test.ts::"cleanup failure is logged and does not mask original error"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: `setPartial` whole-object writes use multipart
|
||||
|
||||
`S3.setPartial(key, offset, bytes, signal?)` SHALL delegate to the
|
||||
multipart engine when `offset === 0` and `bytes.byteLength` exceeds the
|
||||
multipart threshold. For all other cases it SHALL continue to use
|
||||
`AbstractStorage#defaultSetPartial`'s read-modify-write fallback.
|
||||
`S3.capabilities.partialWrites` SHALL remain `false`.
|
||||
|
||||
#### Scenario: zero-offset large write takes multipart path
|
||||
|
||||
- **WHEN** a caller invokes `s3.setPartial("k", 0, new Uint8Array(20 * 1024 * 1024))`
|
||||
- **THEN** the underlying client issues
|
||||
`CreateMultipartUploadCommand` + `UploadPartCommand`(s) +
|
||||
`CompleteMultipartUploadCommand`, and zero `GetObjectCommand` or
|
||||
`PutObjectCommand` for the key
|
||||
|
||||
#### Scenario: non-zero offset uses read-modify-write
|
||||
|
||||
- **WHEN** a caller invokes `s3.setPartial("k", 1024, new Uint8Array(512))`
|
||||
on an existing 4 KiB object
|
||||
- **THEN** the underlying client issues `GetObjectCommand` followed by
|
||||
`PutObjectCommand` and zero `CreateMultipartUploadCommand`
|
||||
|
||||
**Verified by:** `tests/integration/multipart.test.ts::"setPartial at offset zero uses multipart"`
|
||||
AND `tests/integration/multipart.test.ts::"setPartial at non-zero offset uses read-modify-write"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Presigned GET / PUT / DELETE URLs
|
||||
|
||||
`S3` SHALL expose `getSignedGetUrl(key, opts?)`, `getSignedPutUrl(key,
|
||||
opts?)`, and `getSignedDeleteUrl(key, opts?)`. Each SHALL accept
|
||||
`expiresIn` (1 ≤ TTL ≤ 7 days), `requestHeaders`,
|
||||
`responseContentType`, `responseContentDisposition`, `contentType` (PUT
|
||||
only), `sse`, `checksumAlgorithm`, and `versionId` (GET / DELETE only) as
|
||||
applicable. Invalid TTL SHALL raise
|
||||
`STORAGE_S3.PRESIGN_TTL_INVALID`; invalid header SHALL raise
|
||||
`STORAGE_S3.PRESIGN_HEADER_INVALID`.
|
||||
|
||||
#### Scenario: presigned GET round-trips an object
|
||||
|
||||
- **WHEN** a caller invokes `getSignedGetUrl("k", { expiresIn: 60 })` and
|
||||
immediately fetches the returned URL with `fetch()`
|
||||
- **THEN** the response status is 200 and the body equals the stored bytes
|
||||
|
||||
#### Scenario: invalid TTL is rejected at boundary
|
||||
|
||||
- **WHEN** a caller invokes `getSignedGetUrl("k", { expiresIn: 0 })`
|
||||
- **THEN** the call rejects with `S3StorageError` whose `code` is
|
||||
`STORAGE_S3.PRESIGN_TTL_INVALID` before any network call is issued
|
||||
|
||||
**Verified by:** `tests/integration/presign.test.ts::"presigned GET round-trips an object"`
|
||||
AND `src/presign/get-signed-get-url.test.ts::"rejects out-of-range expiresIn at boundary"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Server-side encryption SHALL support SSE-S3, SSE-KMS, SSE-C
|
||||
|
||||
The adapter SHALL accept an `sse?: SseConfig` discriminated union on
|
||||
`S3.set`, `S3.setStream`, `S3.setMultipart`, `S3.copy`, and the presigned
|
||||
PUT builder. The union members are `{ kind: "sse-s3" }`, `{ kind: "sse-kms"; keyId?: string;
|
||||
bucketKeyEnabled?: boolean }`, or `{ kind: "sse-c"; key: Uint8Array;
|
||||
keyMd5: Uint8Array }`. For SSE-C, `keyMd5` SHALL be validated against
|
||||
`MD5(key)` at the boundary; a mismatch SHALL raise
|
||||
`STORAGE_S3.SSE_KEY_MISMATCH` before the request leaves the process.
|
||||
`S3.get` / `S3.head` SHALL accept the matching SSE-C key for decryption;
|
||||
omitting it on an SSE-C-encrypted object SHALL surface
|
||||
`STORAGE_S3.SSE_KEY_MISSING`.
|
||||
|
||||
#### Scenario: SSE-C round-trip with correct key
|
||||
|
||||
- **WHEN** a caller `set`s an object with `sse: { kind: "sse-c", key, keyMd5 }`
|
||||
and `get`s it back with the same `sse` block
|
||||
- **THEN** the round-tripped bytes equal the original
|
||||
|
||||
#### Scenario: SSE-C MD5 mismatch is rejected at boundary
|
||||
|
||||
- **WHEN** a caller `set`s with `sse: { kind: "sse-c", key, keyMd5 }`
|
||||
where `keyMd5` does not equal `MD5(key)`
|
||||
- **THEN** the call rejects with `S3StorageError` whose `code` is
|
||||
`STORAGE_S3.SSE_KEY_MISMATCH` before any network call is issued
|
||||
|
||||
**Verified by:** `tests/integration/sse.test.ts::"SSE-C round-trip with correct key"`
|
||||
AND `src/sse/sse.test.ts::"rejects mismatched keyMd5 at the boundary"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Object metadata (content-\*, cache-control, user-metadata)
|
||||
|
||||
`S3.set` and `S3.setStream` SHALL accept an `objectMetadata?:
|
||||
ObjectMetadata` bag containing `contentType`, `contentEncoding`,
|
||||
`contentDisposition`, `contentLanguage`, `cacheControl`, `expires`, and
|
||||
`userMetadata: Record<string, string>` (mapped to `x-amz-meta-*`).
|
||||
`S3.head` and `S3.getMetadata` SHALL return the full metadata bag
|
||||
including `eTag`, `versionId`, `lastModified`, `storageClass`, every
|
||||
content-\* header, every `x-amz-meta-*` entry, and SSE-algorithm
|
||||
indicator.
|
||||
|
||||
#### Scenario: metadata round-trip
|
||||
|
||||
- **WHEN** a caller `set`s an object with `objectMetadata: { contentType:
|
||||
"image/png", cacheControl: "max-age=3600", userMetadata: { author: "x" }}`
|
||||
and then calls `head` on the key
|
||||
- **THEN** the returned metadata exposes `contentType === "image/png"`,
|
||||
`cacheControl === "max-age=3600"`, and `userMetadata.author === "x"`
|
||||
|
||||
**Verified by:** `tests/integration/core.test.ts::"object metadata round-trips through head"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: ACLs (canned + grant policies + get/put)
|
||||
|
||||
`S3.set` SHALL accept `acl?: CannedAcl | GrantPolicy`. Canned values are
|
||||
`private`, `public-read`, `public-read-write`, `aws-exec-read`,
|
||||
`authenticated-read`, `bucket-owner-read`, `bucket-owner-full-control`,
|
||||
`log-delivery-write`. Grant policies accept `grantRead`, `grantReadACP`,
|
||||
`grantWriteACP`, `grantFullControl`. `S3` SHALL expose `getObjectAcl(key,
|
||||
opts?)` and `putObjectAcl(key, acl, opts?)`. Invalid ACL identifiers SHALL
|
||||
raise `STORAGE_S3.ACL_INVALID` at the boundary.
|
||||
|
||||
#### Scenario: canned ACL is applied at put
|
||||
|
||||
- **WHEN** a caller `set`s an object with `acl: "public-read"` and then
|
||||
calls `getObjectAcl(key)`
|
||||
- **THEN** the returned grants include the `AllUsers` group with `READ`
|
||||
permission
|
||||
|
||||
**Verified by:** `tests/integration/acl.test.ts::"canned ACL is applied at put"`
|
||||
AND `tests/integration/acl.test.ts::"putObjectAcl replaces grants on existing object"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Server-side copy (within-bucket + cross-bucket)
|
||||
|
||||
`S3.copy(source, destination, opts?)` SHALL issue a single
|
||||
`CopyObjectCommand` when the source object is ≤ 5 GiB; for sources
|
||||
> 5 GiB it SHALL route to the multipart server-side copy engine
|
||||
(`UploadPartCopy`). The options bag SHALL support `sourceBucket`
|
||||
(cross-bucket), `metadataDirective: "COPY" | "REPLACE"`,
|
||||
`taggingDirective: "COPY" | "REPLACE"`, conditional headers, and an SSE
|
||||
override. Copy failures SHALL surface `STORAGE_S3.COPY_FAILED` for the
|
||||
single-PUT path or `STORAGE_S3.COPY_PART_FAILED` for the multipart path
|
||||
with `partNumber` in `context`.
|
||||
|
||||
#### Scenario: cross-bucket copy
|
||||
|
||||
- **WHEN** a caller invokes `s3.copy("a", "b", { sourceBucket: "other" })`
|
||||
where `other/a` exists
|
||||
- **THEN** an object `b` exists in the configured bucket with byte-for-byte
|
||||
identical content to `other/a`
|
||||
|
||||
**Verified by:** `tests/integration/copy.test.ts::"cross-bucket copy"`
|
||||
AND `tests/integration/copy.test.ts::"UploadPartCopy is used for sources above five GiB"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Object tagging (get/put/delete + inline at set)
|
||||
|
||||
`S3.set` SHALL accept `tags?: Record<string, string>` URL-encoded into the
|
||||
`x-amz-tagging` header at PUT time (single round trip). `S3` SHALL expose
|
||||
`getObjectTags(key, opts?)`, `putObjectTags(key, tags, opts?)`, and
|
||||
`deleteObjectTags(key, opts?)`. More than `MAX_TAGS` (10) tags on any
|
||||
operation SHALL raise `STORAGE_S3.TAG_LIMIT_EXCEEDED` at the boundary.
|
||||
|
||||
#### Scenario: inline tags at set are queryable
|
||||
|
||||
- **WHEN** a caller invokes `s3.set("k", bytes, { tags: { env: "prod" } })`
|
||||
and then calls `s3.getObjectTags("k")`
|
||||
- **THEN** the returned map equals `{ env: "prod" }`
|
||||
|
||||
**Verified by:** `tests/integration/tagging.test.ts::"inline tags at set are queryable"`
|
||||
AND `tests/integration/tagging.test.ts::"more than ten tags raises TAG_LIMIT_EXCEEDED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Versioning support (list, get/head/delete by versionId)
|
||||
|
||||
`S3` SHALL expose `listObjectVersions(opts?)` as a paginator yielding
|
||||
`ObjectVersion` records. `S3.get`, `S3.head`, and `S3.delete` SHALL
|
||||
accept an `opts.versionId` and forward it to the underlying SDK
|
||||
commands. `STORAGE_S3.VERSION_NOT_FOUND` SHALL surface when the requested
|
||||
version does not exist; `STORAGE_S3.VERSIONING_DISABLED` SHALL surface
|
||||
when a versionId is supplied against a bucket without versioning enabled.
|
||||
|
||||
#### Scenario: get-by-versionId returns the historical bytes
|
||||
|
||||
- **WHEN** a caller `set`s "k" twice (v1, v2) on a versioned bucket and
|
||||
then invokes `s3.get("k", { versionId: v1 })`
|
||||
- **THEN** the returned bytes equal the v1 payload
|
||||
|
||||
**Verified by:** `tests/integration/versioning.test.ts::"get by versionId returns historical bytes"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Bucket lifecycle configuration
|
||||
|
||||
`S3` SHALL expose `getBucketLifecycle(opts?)`, `putBucketLifecycle(rules,
|
||||
opts?)`, and `deleteBucketLifecycle(opts?)`. Rule shape SHALL be validated
|
||||
by a Zod schema at the boundary; invalid rules SHALL raise
|
||||
`STORAGE_S3.LIFECYCLE_INVALID` before the request leaves the process.
|
||||
|
||||
#### Scenario: lifecycle rules round-trip
|
||||
|
||||
- **WHEN** a caller invokes `putBucketLifecycle([{ id: "expire-tmp",
|
||||
prefix: "tmp/", expirationDays: 7 }])` and then calls
|
||||
`getBucketLifecycle()`
|
||||
- **THEN** the returned rules contain a single rule with `id ===
|
||||
"expire-tmp"`, `prefix === "tmp/"`, and `expirationDays === 7`
|
||||
|
||||
**Verified by:** `tests/integration/lifecycle.test.ts::"lifecycle rules round-trip"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Bucket CORS configuration
|
||||
|
||||
`S3` SHALL expose `getBucketCors(opts?)`, `putBucketCors(rules, opts?)`,
|
||||
and `deleteBucketCors(opts?)`. Rule shape SHALL be validated by a Zod
|
||||
schema at the boundary; invalid rules SHALL raise
|
||||
`STORAGE_S3.CORS_INVALID` before the request leaves the process.
|
||||
|
||||
#### Scenario: CORS rules round-trip
|
||||
|
||||
- **WHEN** a caller invokes `putBucketCors([{ allowedOrigins: ["*"],
|
||||
allowedMethods: ["GET"], maxAgeSeconds: 60 }])` and then calls
|
||||
`getBucketCors()`
|
||||
- **THEN** the returned rules contain a single rule equal to the input
|
||||
|
||||
**Verified by:** `tests/integration/cors.test.ts::"CORS rules round-trip"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Bucket-level operations
|
||||
|
||||
`S3` SHALL expose `createBucket(opts?)`, `deleteBucket(opts?)`,
|
||||
`bucketExists(opts?)`, and `getBucketRegion(opts?)`. `createBucket` on an
|
||||
existing bucket SHALL surface `STORAGE_S3.BUCKET_ALREADY_EXISTS`.
|
||||
`deleteBucket` on a non-empty bucket SHALL surface
|
||||
`STORAGE_S3.BUCKET_NOT_EMPTY`. `bucketExists` SHALL return `false` on 404
|
||||
and `true` on 200. `getBucketRegion` SHALL return the bucket's home
|
||||
region as a string.
|
||||
|
||||
#### Scenario: deleteBucket on non-empty bucket raises typed error
|
||||
|
||||
- **WHEN** a caller invokes `deleteBucket()` on a bucket with at least
|
||||
one object
|
||||
- **THEN** the call rejects with `S3StorageError` whose `code` is
|
||||
`STORAGE_S3.BUCKET_NOT_EMPTY`
|
||||
|
||||
**Verified by:** `tests/integration/bucket.test.ts::"deleteBucket on non-empty bucket raises BUCKET_NOT_EMPTY"`
|
||||
AND `tests/integration/bucket.test.ts::"createBucket on existing raises BUCKET_ALREADY_EXISTS"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: `listObjectsV2` paginator with full options
|
||||
|
||||
`S3.list(opts?)` SHALL accept `prefix`, `delimiter`, `maxKeys` (1..1000),
|
||||
`continuationToken`, and `startAfter`, and SHALL return a
|
||||
`StorageListResult` including a `continuationToken` for further pages
|
||||
when the underlying `ListObjectsV2` response is truncated.
|
||||
|
||||
#### Scenario: pagination via continuationToken
|
||||
|
||||
- **WHEN** a bucket holds 2500 objects and a caller pages through with
|
||||
`maxKeys: 1000`, threading the returned `continuationToken` into the
|
||||
next call
|
||||
- **THEN** three calls return 1000 + 1000 + 500 keys with no duplicates
|
||||
and no omissions
|
||||
|
||||
**Verified by:** `tests/integration/bucket.test.ts::"listObjectsV2 paginates via continuationToken"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Conditional operations
|
||||
|
||||
`S3.get`, `S3.head`, `S3.delete`, and `S3.copy` SHALL accept
|
||||
`ConditionalOptions = { ifMatch?, ifNoneMatch?, ifModifiedSince?,
|
||||
ifUnmodifiedSince? }`. The adapter SHALL translate HTTP 412 responses to
|
||||
`STORAGE_S3.PRECONDITION_FAILED` and HTTP 304 responses (where applicable
|
||||
to `get` / `head`) to `STORAGE_S3.NOT_MODIFIED`.
|
||||
|
||||
#### Scenario: IfMatch with stale eTag raises PRECONDITION_FAILED
|
||||
|
||||
- **WHEN** a caller invokes `s3.get("k", { ifMatch: "stale-etag" })` on
|
||||
an existing object whose current eTag is different
|
||||
- **THEN** the call rejects with `S3StorageError` whose `code` is
|
||||
`STORAGE_S3.PRECONDITION_FAILED`
|
||||
|
||||
**Verified by:** `tests/integration/conditional.test.ts::"IfMatch with stale eTag raises PRECONDITION_FAILED"`
|
||||
AND `tests/integration/conditional.test.ts::"IfNoneMatch with current eTag raises NOT_MODIFIED on get"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Checksums (SHA-256 + CRC32C)
|
||||
|
||||
`S3.set` and `S3.setStream` SHALL accept `checksumAlgorithm?:
|
||||
"SHA256" | "CRC32C"`. When supplied, the adapter SHALL compute and send
|
||||
the checksum header (`x-amz-checksum-sha256` / `x-amz-checksum-crc32c`)
|
||||
with the request. `S3.get` SHALL accept `checksumMode: "ENABLED"` and
|
||||
SHALL verify the returned checksum against the body; a mismatch SHALL
|
||||
raise `STORAGE_S3.CHECKSUM_MISMATCH`. Unsupported algorithm strings
|
||||
SHALL raise `STORAGE_S3.CHECKSUM_ALGORITHM_UNSUPPORTED` at the boundary.
|
||||
|
||||
#### Scenario: checksum mismatch raises typed error
|
||||
|
||||
- **WHEN** a caller `set`s with `checksumAlgorithm: "SHA256"` and a
|
||||
wrapping `S3ClientFactory` intercepts the GET response to corrupt one
|
||||
byte before delivery, and the caller `get`s with `checksumMode:
|
||||
"ENABLED"`
|
||||
- **THEN** the call rejects with `S3StorageError` whose `code` is
|
||||
`STORAGE_S3.CHECKSUM_MISMATCH`
|
||||
|
||||
**Verified by:** `tests/integration/checksum.test.ts::"checksum mismatch raises typed error"`
|
||||
AND `tests/integration/checksum.test.ts::"SHA256 and CRC32C round-trip cleanly"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Pluggable HTTP transport and SigV4 signer
|
||||
|
||||
`S3Options` SHALL accept `transport?: FetchLike` and `signer?:
|
||||
SigV4Signer`. When `transport` is provided it SHALL be wired into the
|
||||
underlying `S3Client` via the SDK's `requestHandler` / `Fetch` plug
|
||||
point. When `signer` is provided it SHALL replace the SDK's default
|
||||
`SignatureV4` provider. Defaults preserve the AWS SDK's built-in
|
||||
behaviour and SHALL be observable as the no-op identity when no plug is
|
||||
provided.
|
||||
|
||||
#### Scenario: custom transport is invoked for every request
|
||||
|
||||
- **WHEN** a caller constructs `new S3({ ..., transport: customFetch })`
|
||||
and invokes `s3.head("k")`
|
||||
- **THEN** `customFetch` is called at least once and the underlying AWS
|
||||
SDK's default fetch is not called
|
||||
|
||||
**Verified by:** `src/transport/transport.test.ts::"custom transport is invoked for every request"`
|
||||
AND `src/signer/signer.test.ts::"custom signer replaces the default SigV4 provider"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Retry composition via `@suaveplan/retry`
|
||||
|
||||
The adapter SHALL compose its retry behaviour via `createRetryStrategy`
|
||||
from `@suaveplan/retry` rather than a hand-rolled loop. Classification
|
||||
SHALL preserve the existing semantics (5xx, throttling, `SlowDown`,
|
||||
`RequestTimeout` are retryable; all other failures are fatal).
|
||||
`S3Options.retry` SHALL accept `maxAttempts`, `baseDelayMs`, and
|
||||
`maxDelayMs`; defaults are `3`, `100`, `30000`.
|
||||
|
||||
#### Scenario: retryable failure is retried up to maxAttempts
|
||||
|
||||
- **WHEN** a wrapping `S3ClientFactory` makes the first two
|
||||
`GetObjectCommand` calls fail with a synthetic 503 and the third
|
||||
succeed, with `retry: { maxAttempts: 3 }`
|
||||
- **THEN** the call resolves with the expected bytes and the total number
|
||||
of underlying calls is exactly 3
|
||||
|
||||
**Verified by:** `src/retry/retry.test.ts::"retryable failure is retried up to maxAttempts"`
|
||||
AND `src/retry/retry.test.ts::"fatal failure is not retried"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Comprehensive `STORAGE_S3.*` error code registry
|
||||
|
||||
`S3_CODES` SHALL be registered via `defineCodes("STORAGE_S3", [...] as
|
||||
const)` and SHALL include every code listed in
|
||||
`design.md#error-code-map`. Every registered code SHALL be raised by at
|
||||
least one call site under `src/`; the post-archive openspec conformance
|
||||
check SHALL pass. Legacy codes (the original 11) SHALL remain at their
|
||||
existing identifiers; `MULTIPART_FAILED` SHALL remain registered as a
|
||||
back-compat alias with no new call site.
|
||||
|
||||
#### Scenario: every registered code is reachable
|
||||
|
||||
- **WHEN** the openspec conformance script
|
||||
`bun scripts/openspec-conformance.ts @suaveplan/storage-s3` runs
|
||||
- **THEN** it reports zero `unused-error-code` findings for `STORAGE_S3.*`
|
||||
|
||||
**Verified by:** `src/error-codes.test.ts::"registry exposes every documented code"`
|
||||
AND `src/error-codes.test.ts::"MULTIPART_FAILED is registered as a back-compat alias"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Subpath exports for every feature module
|
||||
|
||||
`package.json` SHALL expose a subpath export entry for every feature
|
||||
module documented in `design.md#file-layout`. The root barrel
|
||||
(`src/index.ts`) SHALL re-export each feature module under a namespace
|
||||
(`export * as multipart from "./multipart/index.js"`, etc.) per project
|
||||
rule §6a. Feature barrels (`src/<feature>/index.ts`) SHALL use explicit
|
||||
named re-exports — no `export *`.
|
||||
|
||||
#### Scenario: subpath import resolves and tree-shakes
|
||||
|
||||
- **WHEN** a consumer writes
|
||||
`import { runMultipart } from "@suaveplan/storage-s3/multipart/multipart-engine"`
|
||||
- **THEN** the import resolves to the engine implementation and the
|
||||
consumer's bundle does not include `S3`, presign, ACL, lifecycle, CORS,
|
||||
bucket, or tagging modules unless separately imported
|
||||
|
||||
**Verified by:** `src/multipart/multipart-engine.test.ts::"is reachable via the dedicated subpath export"`
|
||||
AND `src/s3/s3.test.ts::"feature barrels use explicit named re-exports"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: End-to-end verification against a real S3-compatible service
|
||||
|
||||
The package SHALL exercise every requirement above end-to-end against a
|
||||
real S3-compatible service via testcontainers per project rule §2 (no
|
||||
mocks). The genesis monorepo's
|
||||
`@suaveplan/testing-integration-garage-s3` fixture SHALL be the host. No
|
||||
unit test MAY substitute a mocked AWS SDK for these behaviours; failure
|
||||
injection MAY use a wrapping `S3ClientFactory` that intercepts specific
|
||||
commands, but the underlying transport MUST remain real. Where Garage's
|
||||
behaviour diverges from AWS (e.g. SSE-KMS key-policy enforcement), the
|
||||
test SHALL assert the documented Garage behaviour and the fixture
|
||||
surface SHALL be extended in this same change.
|
||||
|
||||
#### Scenario: integration suite runs against the Garage container
|
||||
|
||||
- **WHEN** the integration test config is executed via
|
||||
`bunx turbo test:integration --filter=@suaveplan/storage-s3`
|
||||
- **THEN** every test in `tests/integration/` uses the Garage-backed
|
||||
`S3Client` from `@suaveplan/testing-integration-garage-s3` rather than
|
||||
a mocked client, and the suite exits zero
|
||||
|
||||
**Verified by:** `tests/integration/core.test.ts::"integration suite is wired to the Garage testcontainer fixture"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: 100% coverage gate
|
||||
|
||||
The package SHALL meet the project-wide `100/100/100/100` istanbul
|
||||
coverage gate (statements / branches / functions / lines). The
|
||||
pre-publish gate `bunx turbo test:coverage
|
||||
--filter=@suaveplan/storage-s3` SHALL hard-fail below 100% on any of the
|
||||
four metrics. Coverage SHALL include unit tests under `src/**/*.test.ts`
|
||||
AND integration tests under `tests/integration/**/*.test.ts`; integration
|
||||
tests SHALL be counted via the merged istanbul report.
|
||||
|
||||
#### Scenario: coverage gate is enforced
|
||||
|
||||
- **WHEN** the publish gate
|
||||
`bunx turbo test:coverage --filter=@suaveplan/storage-s3` is run
|
||||
- **THEN** the gate exits zero only when statements / branches / functions
|
||||
/ lines are all at 100%
|
||||
|
||||
**Verified by:** `vitest.config.ts` thresholds block AND
|
||||
`vitest.integration.config.ts` thresholds block AND
|
||||
`tests/integration/core.test.ts::"merged coverage report is generated"`
|
||||
225
openspec/changes/complete-storage-s3/tasks.md
Normal file
225
openspec/changes/complete-storage-s3/tasks.md
Normal file
@@ -0,0 +1,225 @@
|
||||
# Tasks: complete-storage-s3
|
||||
|
||||
## 0. Pre-flight (non-negotiable)
|
||||
|
||||
- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
|
||||
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
|
||||
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
|
||||
|
||||
## 1. Foundation — constants, error codes, types
|
||||
|
||||
- [ ] 1.1 Extend `src/constants/index.ts` with `DEFAULT_MULTIPART_THRESHOLD = 16 MiB`,
|
||||
`DEFAULT_MULTIPART_CONCURRENCY = 4`, `MIN_PART_SIZE = 5 MiB`,
|
||||
`MAX_PART_SIZE = 5 GiB`, `MAX_PARTS = 10_000`, `MAX_OBJECT_SIZE = 5 TiB`,
|
||||
`MAX_TAGS = 10`, `MAX_PRESIGN_EXPIRES = 7 * 24 * 3600`.
|
||||
- [ ] 1.2 Expand `S3_CODES` via `defineCodes("STORAGE_S3", [...] as const)` to
|
||||
the ~50 codes listed in `design.md#error-code-map`. Retain the legacy 11
|
||||
codes verbatim. Document `MULTIPART_FAILED` as a back-compat alias.
|
||||
- [ ] 1.3 Create `src/types/sse.types.ts`, `acl.types.ts`, `tagging.types.ts`,
|
||||
`lifecycle.types.ts`, `cors.types.ts`, `conditional.types.ts`,
|
||||
`checksum.types.ts`. Wire into `src/types/index.ts`.
|
||||
- [ ] 1.4 Extend `S3Options` in `src/types/s3.types.ts` with `multipart?:
|
||||
MultipartTuning`, `transport?: FetchLike`, `signer?: SigV4Signer`,
|
||||
`defaultSse?: SseConfig`, `defaultAcl?: CannedAcl`.
|
||||
- [ ] 1.5 Create `src/utils/error-mapper.ts` — maps `S3ServiceException`
|
||||
(`$metadata.httpStatusCode`, `name`) to the appropriate `STORAGE_S3.*`
|
||||
code (412 → `PRECONDITION_FAILED`, 304 → `NOT_MODIFIED`, 416 →
|
||||
`RANGE_NOT_SATISFIABLE`, etc.). Sibling `.md` + `.test.ts`.
|
||||
|
||||
## 2. AbstractStorage surface — production-grade
|
||||
|
||||
- [ ] 2.1 `src/core/get/` — native Range GET via `Range:` header; conditional
|
||||
headers (`IfMatch`, `IfNoneMatch`, `IfModifiedSince`, `IfUnmodifiedSince`);
|
||||
SSE-C key passthrough; `versionId` passthrough; checksum verification.
|
||||
Sibling `.md` + `.test.ts`.
|
||||
- [ ] 2.2 `src/core/set/` — auto-promote-to-multipart branch over
|
||||
`multipart.threshold`; single PUT below; SSE / ACL / tagging / metadata /
|
||||
checksum applied at the PUT boundary. Sibling `.md` + `.test.ts`.
|
||||
- [ ] 2.3 `src/core/set-stream/` — `setStream(key, stream, opts?)` that
|
||||
consumes `ReadableStream<Uint8Array>` and routes through the multipart
|
||||
engine without buffering the whole stream. Sibling `.md` + `.test.ts`.
|
||||
- [ ] 2.4 `src/core/set-partial/` — branches: `offset === 0 && bytes.byteLength >
|
||||
threshold` → multipart engine; otherwise `defaultSetPartial`.
|
||||
`capabilities.partialWrites` documented as `false`. Sibling `.md` +
|
||||
`.test.ts`.
|
||||
- [ ] 2.5 `src/core/delete/` — versioned-delete via `versionId`; conditional
|
||||
delete via `IfMatch`. Sibling `.md` + `.test.ts`.
|
||||
- [ ] 2.6 `src/core/head/` — full metadata bag (eTag, versionId, lastModified,
|
||||
storageClass, content-*, user-metadata, checksum, SSE algorithm).
|
||||
Sibling `.md` + `.test.ts`.
|
||||
- [ ] 2.7 `src/core/list/` — `listObjectsV2` paginator with `prefix`,
|
||||
`delimiter`, `maxKeys`, `continuationToken`, `startAfter`. Sibling `.md`
|
||||
+ `.test.ts`.
|
||||
- [ ] 2.8 `src/core/exists/` — branch on 404 / 200; documented to use HEAD.
|
||||
Sibling `.md` + `.test.ts`.
|
||||
|
||||
## 3. Multipart engine (whole-object writes + UploadPartCopy)
|
||||
|
||||
- [ ] 3.1 `src/multipart/part-chunker.ts` + `.md` + `.test.ts` — covers exact
|
||||
multiples, ragged final part, empty stream, mid-read abort.
|
||||
- [ ] 3.2 `src/multipart/worker-pool.ts` + `.md` + `.test.ts` —
|
||||
bounded-concurrency scheduler with linked `AbortController`; covers cap
|
||||
enforcement, first-failure abort-of-peers, caller-abort propagation.
|
||||
- [ ] 3.3 `src/multipart/multipart-engine.ts` + `.md` + `.test.ts` — owns
|
||||
init / parts / complete / abort lifecycle; maps each stage failure to
|
||||
its dedicated code.
|
||||
- [ ] 3.4 `src/multipart/upload-part-copy.ts` + `.md` + `.test.ts` —
|
||||
server-side multipart copy for objects > 5 GiB. Splits source by
|
||||
`CopySourceRange`.
|
||||
- [ ] 3.5 `src/multipart/index.ts` — explicit named re-exports only (§6a).
|
||||
|
||||
## 4. S3-specific extensions
|
||||
|
||||
- [ ] 4.1 `src/presign/get-signed-get-url.ts` + sibling docs + tests — TTL +
|
||||
response-content-* + version + SSE forwarding.
|
||||
- [ ] 4.2 `src/presign/get-signed-put-url.ts` + docs + tests — TTL + content
|
||||
pin + SSE + checksum forwarding.
|
||||
- [ ] 4.3 `src/presign/get-signed-delete-url.ts` + docs + tests — TTL +
|
||||
version-id support.
|
||||
- [ ] 4.4 `src/sse/` — SSE-S3 / SSE-KMS / SSE-C parameter shaping; `keyMd5`
|
||||
validation against `MD5(key)` at the boundary.
|
||||
- [ ] 4.5 `src/acl/` — canned ACL + grant-policy parameter shaping;
|
||||
`getObjectAcl` / `putObjectAcl`.
|
||||
- [ ] 4.6 `src/copy/` — `copyObject` within-bucket + cross-bucket (via
|
||||
`sourceBucket`); metadata directive (`COPY` / `REPLACE`); tagging
|
||||
directive; conditional headers; SSE override; auto-route to
|
||||
`upload-part-copy` when source > 5 GiB.
|
||||
- [ ] 4.7 `src/tagging/` — `getObjectTags` / `putObjectTags` /
|
||||
`deleteObjectTags`; inline `tags` at `set` time URL-encoded into
|
||||
`x-amz-tagging`.
|
||||
- [ ] 4.8 `src/versioning/` — `listObjectVersions` paginator; `versionId`
|
||||
passthrough on `get` / `head` / `delete`.
|
||||
- [ ] 4.9 `src/lifecycle/` — `get` / `put` / `delete` bucket lifecycle rules.
|
||||
- [ ] 4.10 `src/cors/` — `get` / `put` / `delete` bucket CORS rules.
|
||||
- [ ] 4.11 `src/bucket/` — `createBucket` / `deleteBucket` / `bucketExists` /
|
||||
`getBucketRegion`.
|
||||
- [ ] 4.12 `src/conditional/` — shared module that translates
|
||||
`ConditionalOptions` into the right AWS-SDK input fields, used by
|
||||
`get` / `head` / `delete` / `copy`.
|
||||
- [ ] 4.13 `src/checksum/` — SHA-256 + CRC32C computation + verification on
|
||||
single PUT and on `get`-with-checksum.
|
||||
|
||||
## 5. Transport / signer / retry plug-points
|
||||
|
||||
- [ ] 5.1 `src/transport/transport.ts` + docs + tests — `FetchLike` adapter
|
||||
threaded through `S3ClientFactory.create` via `requestHandler` /
|
||||
`Fetch`.
|
||||
- [ ] 5.2 `src/signer/signer.ts` + docs + tests — pluggable `SigV4Signer`;
|
||||
defaults to AWS SDK's `SignatureV4`.
|
||||
- [ ] 5.3 `src/retry/retry.ts` + docs + tests — replace `withRetry` in
|
||||
`s3.ts` with `createRetryStrategy` from `@suaveplan/retry`. Preserve
|
||||
`isRetryableS3Error` classification (5xx / throttling / `SlowDown` /
|
||||
`RequestTimeout`). Delete the old `withRetry` helper.
|
||||
|
||||
## 6. Top-level `S3` class wiring
|
||||
|
||||
- [ ] 6.1 Refactor `src/s3/s3.ts` so it owns construction (`validateCredentials`,
|
||||
`S3Client` build with transport/signer plug-points), capability flags
|
||||
(`partialReads: true`, `nativeStreams: true`, `partialWrites: false`),
|
||||
and delegates each surface to its co-located module. Keep `S3` class
|
||||
≤ 300 lines.
|
||||
- [ ] 6.2 Validate `multipart.partSize ∈ [MIN_PART_SIZE, MAX_PART_SIZE]`,
|
||||
`multipart.threshold ≥ MIN_PART_SIZE`, `multipart.concurrency ∈ [1, 32]`
|
||||
at construction time; throw `S3StorageError(MULTIPART_INIT_FAILED, ...)`.
|
||||
- [ ] 6.3 Update `src/index.ts` root barrel to namespace-re-export every new
|
||||
module (`export * as multipart from "./multipart/index.js"`,
|
||||
`export * as presign from "./presign/index.js"`, etc.) per §6a.
|
||||
|
||||
## 7. Subpath exports + package.json
|
||||
|
||||
- [ ] 7.1 Add subpath exports in `package.json` for every feature module
|
||||
under `src/`: `./core/get/get`, `./core/set/set`, `./core/set-stream/set-stream`,
|
||||
`./multipart/multipart-engine`, `./multipart/part-chunker`,
|
||||
`./multipart/worker-pool`, `./multipart/upload-part-copy`,
|
||||
`./presign/get-signed-get-url`, `./presign/get-signed-put-url`,
|
||||
`./presign/get-signed-delete-url`, `./sse/sse`, `./acl/acl`,
|
||||
`./copy/copy`, `./tagging/tagging`, `./versioning/versioning`,
|
||||
`./lifecycle/lifecycle`, `./cors/cors`, `./bucket/bucket`,
|
||||
`./conditional/conditional`, `./checksum/checksum`,
|
||||
`./transport/transport`, `./signer/signer`, `./retry/retry`,
|
||||
`./utils/error-mapper`.
|
||||
- [ ] 7.2 Add `@suaveplan/retry` to `dependencies` if not already
|
||||
present (it is — verify `workspace:^`).
|
||||
|
||||
## 8. Integration tests against Garage (real service, no mocks)
|
||||
|
||||
- [ ] 8.1 `tests/integration/core.test.ts` — get / set / delete / list / head
|
||||
round-trips against the Garage fixture.
|
||||
- [ ] 8.2 `tests/integration/multipart.test.ts` — 10 cases per design table
|
||||
(auto-promote, concurrency cap, abort+cleanup, per-stage error codes,
|
||||
`setPartial` whole-object, part-size validation).
|
||||
- [ ] 8.3 `tests/integration/stream.test.ts` — `setStream` of 25 MiB → 5
|
||||
parts at 5 MiB; round-trip bytes verify.
|
||||
- [ ] 8.4 `tests/integration/presign.test.ts` — all three presigned URLs;
|
||||
TTL boundary; header pin; response-content-disposition override.
|
||||
- [ ] 8.5 `tests/integration/sse.test.ts` — SSE-S3, SSE-KMS (Garage
|
||||
compatibility), SSE-C with key+MD5 validation; SSE-C mismatch raises
|
||||
`SSE_KEY_MISMATCH` before request leaves process.
|
||||
- [ ] 8.6 `tests/integration/acl.test.ts` — canned + grants + getObjectAcl
|
||||
+ putObjectAcl.
|
||||
- [ ] 8.7 `tests/integration/copy.test.ts` — within-bucket, cross-bucket
|
||||
(second bucket created via fixture extension), > 5 GiB routed via
|
||||
UploadPartCopy (simulated via fixture-level threshold override since
|
||||
a real > 5 GiB object is impractical in CI).
|
||||
- [ ] 8.8 `tests/integration/tagging.test.ts` — inline-at-set, post-write
|
||||
put, get, delete; > 10 tags raises `TAG_LIMIT_EXCEEDED`.
|
||||
- [ ] 8.9 `tests/integration/versioning.test.ts` — list versions, get by
|
||||
versionId, delete by versionId.
|
||||
- [ ] 8.10 `tests/integration/lifecycle.test.ts` — round-trip lifecycle
|
||||
rules; delete; get-after-delete raises 404.
|
||||
- [ ] 8.11 `tests/integration/cors.test.ts` — round-trip CORS rules;
|
||||
delete; get-after-delete raises 404.
|
||||
- [ ] 8.12 `tests/integration/bucket.test.ts` — create / exists / region /
|
||||
delete; create-existing raises `BUCKET_ALREADY_EXISTS`; delete-non-
|
||||
empty raises `BUCKET_NOT_EMPTY`.
|
||||
- [ ] 8.13 `tests/integration/conditional.test.ts` — IfMatch hit → 200;
|
||||
IfMatch miss → `PRECONDITION_FAILED`; IfNoneMatch / If-*Modified-
|
||||
Since paths.
|
||||
- [ ] 8.14 `tests/integration/range.test.ts` — partial GET via native
|
||||
Range; OOB raises `RANGE_NOT_SATISFIABLE`.
|
||||
- [ ] 8.15 `tests/integration/checksum.test.ts` — SHA-256 + CRC32C round
|
||||
trip; deliberately corrupted bytes raise `CHECKSUM_MISMATCH`.
|
||||
- [ ] 8.16 Extend `@suaveplan/testing-integration-garage-s3` fixture
|
||||
surface only as needed (e.g. expose a second-bucket helper for
|
||||
cross-bucket copy; expose `ListPartsCommand` for mid-flight assertions).
|
||||
Any extension lands in this change.
|
||||
|
||||
## 9. Docs
|
||||
|
||||
- [ ] 9.1 Rewrite `README.md` Modules + Features sections to cover the
|
||||
full surface (one bullet per module under §3 in this tasks file).
|
||||
- [ ] 9.2 Co-located `.md` for every implementation file per project rule §4
|
||||
(≥ 200 words, basic + advanced example, API reference, implementation
|
||||
notes).
|
||||
- [ ] 9.3 Update `error-codes.md` to enumerate every code with one-sentence
|
||||
semantics + the call site that raises it.
|
||||
|
||||
## 10. Verify (per §13 pre-ship sequence)
|
||||
|
||||
- [ ] 10.1 Spec reconciliation (§13 step 0): walk every `### Requirement`
|
||||
and `#### Scenario` in `specs/storage-s3/spec.md`; confirm matching
|
||||
impl + verbatim `**Verified by:**` test names.
|
||||
- [ ] 10.2 `bunx turbo lint typecheck test --filter=@suaveplan/storage-s3`
|
||||
green.
|
||||
- [ ] 10.3 `bunx turbo test:coverage --filter=@suaveplan/storage-s3`
|
||||
100/100/100/100.
|
||||
- [ ] 10.4 `bunx turbo test:integration --filter=@suaveplan/storage-s3`
|
||||
green against the Garage testcontainer.
|
||||
- [ ] 10.5 Dependent sweep:
|
||||
`bunx turbo test --filter=...@suaveplan/storage-s3` green.
|
||||
- [ ] 10.6 Banned-pattern / barrel-shape / runner-import scans (§13 steps
|
||||
4 / 4a / 4b) — zero output.
|
||||
- [ ] 10.7 `bun scripts/openspec-conformance.ts @suaveplan/storage-s3`
|
||||
PASS.
|
||||
- [ ] 10.8 `openspec validate complete-storage-s3 --strict` PASS.
|
||||
|
||||
## 11. Archive
|
||||
|
||||
- [ ] 11.1 Refresh root docs per §13b — `MANIFEST.md`, `MISSING.yml`,
|
||||
`STATUS.md`, `SUMMARY.md`; run `bunx markdownlint-cli2 --fix`.
|
||||
- [ ] 11.2 Fold `specs/storage-s3/spec.md` into
|
||||
`openspec/specs/storage-s3/spec.md` (this is the first change to
|
||||
establish that canonical capability file — create the folder under
|
||||
`openspec/specs/` only at this archival step).
|
||||
- [ ] 11.3 Move `openspec/changes/complete-storage-s3/` →
|
||||
`openspec/archive/<yyyy-mm-dd>-complete-storage-s3/`.
|
||||
Reference in New Issue
Block a user