Files
2026-05-27 23:29:53 +01:00

5.4 KiB

Tasks: add-storage-s3-multipart

1. Error codes + constants

  • 1.1 Extend S3_CODES via defineCodes with MULTIPART_INIT_FAILED, MULTIPART_PART_FAILED, MULTIPART_COMPLETE_FAILED, MULTIPART_ABORT_FAILED. Retain MULTIPART_FAILED as a documented legacy alias (no new call site raises it).
  • 1.2 Add DEFAULT_MULTIPART_THRESHOLD = 16 * 1024 * 1024 and DEFAULT_MULTIPART_CONCURRENCY = 4 to src/constants/index.ts. Verify DEFAULT_PART_SIZE floor (≥ 5 MiB) and add a constant MIN_PART_SIZE = 5 * 1024 * 1024.
  • 1.3 Add MultipartTuning interface to src/types/s3.types.ts; extend S3Options with multipart?: MultipartTuning.

2. Internal engine

  • 2.1 Create src/multipart/part-chunker.ts + sibling .md + .test.ts — chunkStream(stream, partSize): AsyncIterable<PartInput>; covers exact multiples, ragged final part, empty stream, abort mid-read.
  • 2.2 Create src/multipart/worker-pool.ts + sibling .md + .test.ts — bounded-concurrency scheduler using Promise.race + a linked AbortController; covers cap enforcement, first-failure aborts peers, caller-abort cancels in-flight workers.
  • 2.3 Create src/multipart/multipart-engine.ts + sibling .md + .test.ts — runMultipart(client, bucket, key, parts, tuning, signal) with initMultipart / completePartsList / abortMultipart helpers; maps each stage failure to its dedicated code.
  • 2.4 Create src/multipart/index.ts — explicit named re-exports only (rule 10 / §6a). No export *.

3. Wire into S3.set + S3.setPartial

  • 3.1 In s3/s3.ts, branch S3.set on value.byteLength > (this.options.multipart?.threshold ?? DEFAULT_MULTIPART_THRESHOLD) to call the engine via a synchronous part iterator built from the buffer; below the threshold keep the existing PutObjectCommand.
  • 3.2 In S3.setPartial, when offset === 0 AND bytes.byteLength > threshold, delegate to the engine (whole-object write); otherwise keep defaultSetPartial. Document in s3/s3.md that capabilities.partialWrites stays false and explain why.
  • 3.3 Rewrite S3.setMultipart as a thin wrapper over the new engine using chunkStream for the input ReadableStream. Preserve public signature; preserve setMultipart subpath export.
  • 3.4 Validate partSize >= MIN_PART_SIZE at construction time; throw S3StorageError(MULTIPART_INIT_FAILED, ...) with a clear message.

4. Subpath exports + barrels

  • 4.1 Add ./multipart/multipart-engine, ./multipart/part-chunker, ./multipart/worker-pool entries to package.json exports.
  • 4.2 Update src/index.ts to re-export the new multipart helpers via a namespace export (export * as multipart from "./multipart/index.js") so consumers can reach them without breaking the root barrel rule.

5. Docs

  • 5.1 Update README.md Modules + Features sections to mention auto multipart, tuning, and the four new error codes.
  • 5.2 Update s3/s3.md (Implementation Notes) — threshold rationale, capability honesty, abort/cleanup contract.
  • 5.3 Add multipart/multipart-engine.md, part-chunker.md, worker-pool.md per §4 (≥200 words, basic + advanced examples, API reference, implementation notes).

6. Integration tests (real Garage container)

  • 6.1 Confirm @suaveplan/testing-integration-garage-s3 fixture surface covers MPU operations end-to-end; if a missing capability is found (e.g. mid-flight ListPartsCommand exposure), extend the fixture in the same change.
  • 6.2 Add tests/integration/multipart.test.ts covering the 10 cases in design.md#test-plan. No mocks — use a wrapping S3ClientFactory to inject controlled delays / failures around the real Garage client.
  • 6.3 Wire integration suite into vitest.integration.config.ts (already present in the package).

7. Verify (per §13 pre-ship sequence)

  • 7.1 Spec reconciliation (§13 step 0): every ### Requirement in specs/storage-s3/spec.md has matching impl + tests; every **Verified by:** path resolves verbatim.
  • 7.2 bunx turbo lint typecheck test --filter=@suaveplan/storage-s3 green.
  • 7.3 bunx turbo test:coverage --filter=@suaveplan/storage-s3 100/100/100/100.
  • 7.4 Integration sweep: bunx turbo test:integration --filter=@suaveplan/storage-s3 green against the Garage testcontainer.
  • 7.5 Dependent sweep: bunx turbo test --filter=...@suaveplan/storage-s3 green.
  • 7.6 Banned-pattern / barrel-shape / runner-import scans (§13 steps 4 / 4a / 4b) — zero output.
  • 7.7 bun scripts/openspec-conformance.ts @suaveplan/storage-s3 PASS.
  • 7.8 Run openspec validate add-storage-s3-multipart --strict — must report PASS.

8. Archive

  • 8.1 Refresh root docs per §13b — MANIFEST.md, MISSING.yml, STATUS.md, SUMMARY.md; run bunx markdownlint-cli2 --fix.
  • 8.2 Fold specs/storage-s3/spec.md into openspec/specs/storage-s3/spec.md (this is the first change to establish that canonical capability file — create the folder under openspec/specs/ only at this archival step, never before).
  • 8.3 Move openspec/changes/add-storage-s3-multipart/ → openspec/archive/<yyyy-mm-dd>-add-storage-s3-multipart/.