226 lines
13 KiB
Markdown
226 lines
13 KiB
Markdown
# Tasks: complete-storage-s3
|
|
|
|
## 0. Pre-flight (non-negotiable)
|
|
|
|
- [ ] 0.1 Confirm clean working tree (`git status` reports nothing) and no in-flight worktrees with uncommitted work (`git worktree list` then `git status` in each)
|
|
- [ ] 0.2 If any worktrees exist under `.claude/worktrees/`, account for each one: still-active agent, work-pending-merge, or stale-and-prune. Stale worktrees MUST be removed (`git worktree remove`) before opening a new one — code that has been lost twice on main is the failure mode this guards against.
|
|
- [ ] 0.3 Confirm the package set in `MANIFEST.md` matches `find packages -maxdepth 3 -name package.json` count. Drift here means a prior change archived without regenerating the manifest.
|
|
|
|
## 1. Foundation — constants, error codes, types
|
|
|
|
- [ ] 1.1 Extend `src/constants/index.ts` with `DEFAULT_MULTIPART_THRESHOLD = 16 MiB`,
|
|
`DEFAULT_MULTIPART_CONCURRENCY = 4`, `MIN_PART_SIZE = 5 MiB`,
|
|
`MAX_PART_SIZE = 5 GiB`, `MAX_PARTS = 10_000`, `MAX_OBJECT_SIZE = 5 TiB`,
|
|
`MAX_TAGS = 10`, `MAX_PRESIGN_EXPIRES = 7 * 24 * 3600`.
|
|
- [ ] 1.2 Expand `S3_CODES` via `defineCodes("STORAGE_S3", [...] as const)` to
|
|
the ~50 codes listed in `design.md#error-code-map`. Retain the legacy 11
|
|
codes verbatim. Document `MULTIPART_FAILED` as a back-compat alias.
|
|
- [ ] 1.3 Create `src/types/sse.types.ts`, `acl.types.ts`, `tagging.types.ts`,
|
|
`lifecycle.types.ts`, `cors.types.ts`, `conditional.types.ts`,
|
|
`checksum.types.ts`. Wire into `src/types/index.ts`.
|
|
- [ ] 1.4 Extend `S3Options` in `src/types/s3.types.ts` with `multipart?:
|
|
MultipartTuning`, `transport?: FetchLike`, `signer?: SigV4Signer`,
|
|
`defaultSse?: SseConfig`, `defaultAcl?: CannedAcl`.
|
|
- [ ] 1.5 Create `src/utils/error-mapper.ts` — maps `S3ServiceException`
|
|
(`$metadata.httpStatusCode`, `name`) to the appropriate `STORAGE_S3.*`
|
|
code (412 → `PRECONDITION_FAILED`, 304 → `NOT_MODIFIED`, 416 →
|
|
`RANGE_NOT_SATISFIABLE`, etc.). Sibling `.md` + `.test.ts`.
|
|
|
|
## 2. AbstractStorage surface — production-grade
|
|
|
|
- [ ] 2.1 `src/core/get/` — native Range GET via `Range:` header; conditional
|
|
headers (`IfMatch`, `IfNoneMatch`, `IfModifiedSince`, `IfUnmodifiedSince`);
|
|
SSE-C key passthrough; `versionId` passthrough; checksum verification.
|
|
Sibling `.md` + `.test.ts`.
|
|
- [ ] 2.2 `src/core/set/` — auto-promote-to-multipart branch over
|
|
`multipart.threshold`; single PUT below; SSE / ACL / tagging / metadata /
|
|
checksum applied at the PUT boundary. Sibling `.md` + `.test.ts`.
|
|
- [ ] 2.3 `src/core/set-stream/` — `setStream(key, stream, opts?)` that
|
|
consumes `ReadableStream<Uint8Array>` and routes through the multipart
|
|
engine without buffering the whole stream. Sibling `.md` + `.test.ts`.
|
|
- [ ] 2.4 `src/core/set-partial/` — branches: `offset === 0 && bytes.byteLength >
|
|
threshold` → multipart engine; otherwise `defaultSetPartial`.
|
|
`capabilities.partialWrites` documented as `false`. Sibling `.md` +
|
|
`.test.ts`.
|
|
- [ ] 2.5 `src/core/delete/` — versioned-delete via `versionId`; conditional
|
|
delete via `IfMatch`. Sibling `.md` + `.test.ts`.
|
|
- [ ] 2.6 `src/core/head/` — full metadata bag (eTag, versionId, lastModified,
|
|
storageClass, content-*, user-metadata, checksum, SSE algorithm).
|
|
Sibling `.md` + `.test.ts`.
|
|
- [ ] 2.7 `src/core/list/` — `listObjectsV2` paginator with `prefix`,
|
|
`delimiter`, `maxKeys`, `continuationToken`, `startAfter`. Sibling `.md`
|
|
+ `.test.ts`.
|
|
- [ ] 2.8 `src/core/exists/` — branch on 404 / 200; documented to use HEAD.
|
|
Sibling `.md` + `.test.ts`.
|
|
|
|
## 3. Multipart engine (whole-object writes + UploadPartCopy)
|
|
|
|
- [ ] 3.1 `src/multipart/part-chunker.ts` + `.md` + `.test.ts` — covers exact
|
|
multiples, ragged final part, empty stream, mid-read abort.
|
|
- [ ] 3.2 `src/multipart/worker-pool.ts` + `.md` + `.test.ts` —
|
|
bounded-concurrency scheduler with linked `AbortController`; covers cap
|
|
enforcement, first-failure abort-of-peers, caller-abort propagation.
|
|
- [ ] 3.3 `src/multipart/multipart-engine.ts` + `.md` + `.test.ts` — owns
|
|
init / parts / complete / abort lifecycle; maps each stage failure to
|
|
its dedicated code.
|
|
- [ ] 3.4 `src/multipart/upload-part-copy.ts` + `.md` + `.test.ts` —
|
|
server-side multipart copy for objects > 5 GiB. Splits source by
|
|
`CopySourceRange`.
|
|
- [ ] 3.5 `src/multipart/index.ts` — explicit named re-exports only (§6a).
|
|
|
|
## 4. S3-specific extensions
|
|
|
|
- [ ] 4.1 `src/presign/get-signed-get-url.ts` + sibling docs + tests — TTL +
|
|
response-content-* + version + SSE forwarding.
|
|
- [ ] 4.2 `src/presign/get-signed-put-url.ts` + docs + tests — TTL + content
|
|
pin + SSE + checksum forwarding.
|
|
- [ ] 4.3 `src/presign/get-signed-delete-url.ts` + docs + tests — TTL +
|
|
version-id support.
|
|
- [ ] 4.4 `src/sse/` — SSE-S3 / SSE-KMS / SSE-C parameter shaping; `keyMd5`
|
|
validation against `MD5(key)` at the boundary.
|
|
- [ ] 4.5 `src/acl/` — canned ACL + grant-policy parameter shaping;
|
|
`getObjectAcl` / `putObjectAcl`.
|
|
- [ ] 4.6 `src/copy/` — `copyObject` within-bucket + cross-bucket (via
|
|
`sourceBucket`); metadata directive (`COPY` / `REPLACE`); tagging
|
|
directive; conditional headers; SSE override; auto-route to
|
|
`upload-part-copy` when source > 5 GiB.
|
|
- [ ] 4.7 `src/tagging/` — `getObjectTags` / `putObjectTags` /
|
|
`deleteObjectTags`; inline `tags` at `set` time URL-encoded into
|
|
`x-amz-tagging`.
|
|
- [ ] 4.8 `src/versioning/` — `listObjectVersions` paginator; `versionId`
|
|
passthrough on `get` / `head` / `delete`.
|
|
- [ ] 4.9 `src/lifecycle/` — `get` / `put` / `delete` bucket lifecycle rules.
|
|
- [ ] 4.10 `src/cors/` — `get` / `put` / `delete` bucket CORS rules.
|
|
- [ ] 4.11 `src/bucket/` — `createBucket` / `deleteBucket` / `bucketExists` /
|
|
`getBucketRegion`.
|
|
- [ ] 4.12 `src/conditional/` — shared module that translates
|
|
`ConditionalOptions` into the right AWS-SDK input fields, used by
|
|
`get` / `head` / `delete` / `copy`.
|
|
- [ ] 4.13 `src/checksum/` — SHA-256 + CRC32C computation + verification on
|
|
single PUT and on `get`-with-checksum.
|
|
|
|
## 5. Transport / signer / retry plug-points
|
|
|
|
- [ ] 5.1 `src/transport/transport.ts` + docs + tests — `FetchLike` adapter
|
|
threaded through `S3ClientFactory.create` via `requestHandler` /
|
|
`Fetch`.
|
|
- [ ] 5.2 `src/signer/signer.ts` + docs + tests — pluggable `SigV4Signer`;
|
|
defaults to AWS SDK's `SignatureV4`.
|
|
- [ ] 5.3 `src/retry/retry.ts` + docs + tests — replace `withRetry` in
|
|
`s3.ts` with `createRetryStrategy` from `@suaveplan/retry`. Preserve
|
|
`isRetryableS3Error` classification (5xx / throttling / `SlowDown` /
|
|
`RequestTimeout`). Delete the old `withRetry` helper.
|
|
|
|
## 6. Top-level `S3` class wiring
|
|
|
|
- [ ] 6.1 Refactor `src/s3/s3.ts` so it owns construction (`validateCredentials`,
|
|
`S3Client` build with transport/signer plug-points), capability flags
|
|
(`partialReads: true`, `nativeStreams: true`, `partialWrites: false`),
|
|
and delegates each surface to its co-located module. Keep `S3` class
|
|
≤ 300 lines.
|
|
- [ ] 6.2 Validate `multipart.partSize ∈ [MIN_PART_SIZE, MAX_PART_SIZE]`,
|
|
`multipart.threshold ≥ MIN_PART_SIZE`, `multipart.concurrency ∈ [1, 32]`
|
|
at construction time; throw `S3StorageError(MULTIPART_INIT_FAILED, ...)`.
|
|
- [ ] 6.3 Update `src/index.ts` root barrel to namespace-re-export every new
|
|
module (`export * as multipart from "./multipart/index.js"`,
|
|
`export * as presign from "./presign/index.js"`, etc.) per §6a.
|
|
|
|
## 7. Subpath exports + package.json
|
|
|
|
- [ ] 7.1 Add subpath exports in `package.json` for every feature module
|
|
under `src/`: `./core/get/get`, `./core/set/set`, `./core/set-stream/set-stream`,
|
|
`./multipart/multipart-engine`, `./multipart/part-chunker`,
|
|
`./multipart/worker-pool`, `./multipart/upload-part-copy`,
|
|
`./presign/get-signed-get-url`, `./presign/get-signed-put-url`,
|
|
`./presign/get-signed-delete-url`, `./sse/sse`, `./acl/acl`,
|
|
`./copy/copy`, `./tagging/tagging`, `./versioning/versioning`,
|
|
`./lifecycle/lifecycle`, `./cors/cors`, `./bucket/bucket`,
|
|
`./conditional/conditional`, `./checksum/checksum`,
|
|
`./transport/transport`, `./signer/signer`, `./retry/retry`,
|
|
`./utils/error-mapper`.
|
|
- [ ] 7.2 Add `@suaveplan/retry` to `dependencies` if not already
|
|
present (it is — verify `workspace:^`).
|
|
|
|
## 8. Integration tests against Garage (real service, no mocks)
|
|
|
|
- [ ] 8.1 `tests/integration/core.test.ts` — get / set / delete / list / head
|
|
round-trips against the Garage fixture.
|
|
- [ ] 8.2 `tests/integration/multipart.test.ts` — 10 cases per design table
|
|
(auto-promote, concurrency cap, abort+cleanup, per-stage error codes,
|
|
`setPartial` whole-object, part-size validation).
|
|
- [ ] 8.3 `tests/integration/stream.test.ts` — `setStream` of 25 MiB → 5
|
|
parts at 5 MiB; round-trip bytes verify.
|
|
- [ ] 8.4 `tests/integration/presign.test.ts` — all three presigned URLs;
|
|
TTL boundary; header pin; response-content-disposition override.
|
|
- [ ] 8.5 `tests/integration/sse.test.ts` — SSE-S3, SSE-KMS (Garage
|
|
compatibility), SSE-C with key+MD5 validation; SSE-C mismatch raises
|
|
`SSE_KEY_MISMATCH` before request leaves process.
|
|
- [ ] 8.6 `tests/integration/acl.test.ts` — canned + grants + getObjectAcl
|
|
+ putObjectAcl.
|
|
- [ ] 8.7 `tests/integration/copy.test.ts` — within-bucket, cross-bucket
|
|
(second bucket created via fixture extension), > 5 GiB routed via
|
|
UploadPartCopy (simulated via fixture-level threshold override since
|
|
a real > 5 GiB object is impractical in CI).
|
|
- [ ] 8.8 `tests/integration/tagging.test.ts` — inline-at-set, post-write
|
|
put, get, delete; > 10 tags raises `TAG_LIMIT_EXCEEDED`.
|
|
- [ ] 8.9 `tests/integration/versioning.test.ts` — list versions, get by
|
|
versionId, delete by versionId.
|
|
- [ ] 8.10 `tests/integration/lifecycle.test.ts` — round-trip lifecycle
|
|
rules; delete; get-after-delete raises 404.
|
|
- [ ] 8.11 `tests/integration/cors.test.ts` — round-trip CORS rules;
|
|
delete; get-after-delete raises 404.
|
|
- [ ] 8.12 `tests/integration/bucket.test.ts` — create / exists / region /
|
|
delete; create-existing raises `BUCKET_ALREADY_EXISTS`; delete-non-
|
|
empty raises `BUCKET_NOT_EMPTY`.
|
|
- [ ] 8.13 `tests/integration/conditional.test.ts` — IfMatch hit → 200;
|
|
IfMatch miss → `PRECONDITION_FAILED`; IfNoneMatch / If-*Modified-
|
|
Since paths.
|
|
- [ ] 8.14 `tests/integration/range.test.ts` — partial GET via native
|
|
Range; OOB raises `RANGE_NOT_SATISFIABLE`.
|
|
- [ ] 8.15 `tests/integration/checksum.test.ts` — SHA-256 + CRC32C round
|
|
trip; deliberately corrupted bytes raise `CHECKSUM_MISMATCH`.
|
|
- [ ] 8.16 Extend `@suaveplan/testing-integration-garage-s3` fixture
|
|
surface only as needed (e.g. expose a second-bucket helper for
|
|
cross-bucket copy; expose `ListPartsCommand` for mid-flight assertions).
|
|
Any extension lands in this change.
|
|
|
|
## 9. Docs
|
|
|
|
- [ ] 9.1 Rewrite `README.md` Modules + Features sections to cover the
|
|
full surface (one bullet per module under §3 in this tasks file).
|
|
- [ ] 9.2 Co-located `.md` for every implementation file per project rule §4
|
|
(≥ 200 words, basic + advanced example, API reference, implementation
|
|
notes).
|
|
- [ ] 9.3 Update `error-codes.md` to enumerate every code with one-sentence
|
|
semantics + the call site that raises it.
|
|
|
|
## 10. Verify (per §13 pre-ship sequence)
|
|
|
|
- [ ] 10.1 Spec reconciliation (§13 step 0): walk every `### Requirement`
|
|
and `#### Scenario` in `specs/storage-s3/spec.md`; confirm matching
|
|
impl + verbatim `**Verified by:**` test names.
|
|
- [ ] 10.2 `bunx turbo lint typecheck test --filter=@suaveplan/storage-s3`
|
|
green.
|
|
- [ ] 10.3 `bunx turbo test:coverage --filter=@suaveplan/storage-s3`
|
|
100/100/100/100.
|
|
- [ ] 10.4 `bunx turbo test:integration --filter=@suaveplan/storage-s3`
|
|
green against the Garage testcontainer.
|
|
- [ ] 10.5 Dependent sweep:
|
|
`bunx turbo test --filter=...@suaveplan/storage-s3` green.
|
|
- [ ] 10.6 Banned-pattern / barrel-shape / runner-import scans (§13 steps
|
|
4 / 4a / 4b) — zero output.
|
|
- [ ] 10.7 `bun scripts/openspec-conformance.ts @suaveplan/storage-s3`
|
|
PASS.
|
|
- [ ] 10.8 `openspec validate complete-storage-s3 --strict` PASS.
|
|
|
|
## 11. Archive
|
|
|
|
- [ ] 11.1 Refresh root docs per §13b — `MANIFEST.md`, `MISSING.yml`,
|
|
`STATUS.md`, `SUMMARY.md`; run `bunx markdownlint-cli2 --fix`.
|
|
- [ ] 11.2 Fold `specs/storage-s3/spec.md` into
|
|
`openspec/specs/storage-s3/spec.md` (this is the first change to
|
|
establish that canonical capability file — create the folder under
|
|
`openspec/specs/` only at this archival step).
|
|
- [ ] 11.3 Move `openspec/changes/complete-storage-s3/` →
|
|
`openspec/archive/<yyyy-mm-dd>-complete-storage-s3/`.
|