Initial Commit

This commit is contained in:
2026-05-27 23:29:53 +01:00
commit fbc70f96e6
10000 changed files with 1102152 additions and 0 deletions

View File

@@ -0,0 +1,216 @@
# OpenSpec Change: enterprise-storage-s3
- **Change ID**: enterprise-storage-s3
- **Package**: `@suaveplan/storage-s3`
- **Category**: universal
- **Target path**: `universal/storage/storage-s3`
- **Status**: PROPOSED
- **Date**: 2026-04-24
- **Reference spec**: `/mnt/s/web/suaveplan/enterprise/storage-s3.md`
---
## Summary
Deliver `@suaveplan/storage-s3` on the genesis branch from scratch at enterprise quality. The package provides an S3-compatible object storage implementation that extends `@suaveplan/storage`'s `AbstractStorage` base class. It implements all `StorageAdapter` methods (get, set, delete, list, copy, move, exists, metadata) backed by `@aws-sdk/client-s3`, registers itself with `StorageFactory` as the `s3` provider, and exposes a `createS3` factory function. Genesis adds structured errors via `@suaveplan/error`, Zod option schemas, OTel instrumentation, retry integration via `@suaveplan/retry`, `AbortSignal` support throughout, presigned URL generation, multipart upload for large objects, credential validation, property-based tests (fast-check), and a mitata benchmark suite. The main-branch implementation uses `@aws-sdk/client-s3` directly; genesis adds AWS SDK v3 presigned URLs via `@aws-sdk/s3-request-presigner`.
---
## Current-state audit (genesis)
The package does not exist on genesis yet — this is a fresh build. No prior source, no test results, no coverage data.
---
## Available workspace deps on genesis
| Package | Role |
|---|---|
| `@suaveplan/storage` (`workspace:^`) | `AbstractStorage`, `StorageFactory`, `StorageError`, all shared types |
| `@suaveplan/error` (`workspace:^`) | Base class for `S3StorageError`; `defineCodes` for `S3.*` codes |
| `@suaveplan/types` (`workspace:^`) | `LoggerLike`, `Clock` |
| `@suaveplan/retry` (`workspace:^`) | `withRetry`, `RetryStrategyOptions` — wired on all S3 API calls |
| `@aws-sdk/client-s3` (`catalog:storage`) | S3 commands (Get, Put, Delete, List, Head, Copy) |
| `@aws-sdk/s3-request-presigner` (`catalog:storage`) | `getSignedUrl` for presigned URL generation |
| `zod` (`catalog:forms`) | Zod schemas for `S3Options` and `PresignOptions` |
| `@opentelemetry/api` (`catalog:observability`) | Spans + counters on every operation |
| `tslib` (`catalog:build`) | Runtime helpers |
Dev deps: `vitest`, `@vitest/coverage-istanbul`, `fast-check`, `mitata` (all `catalog:testing`), `@suaveplan/testing workspace:^`.
---
## Preserved public API surface (from main branch)
### Core class
```ts
class S3 extends AbstractStorage {
constructor(options: S3Options)
// StorageAdapter methods (all from AbstractStorage contract):
get(key: string, options?: StorageOptions): Promise<StorageResult>
set(key: string, value: Uint8Array | string, options?: StorageOptions): Promise<void>
delete(key: string): Promise<void>
list(prefix?: string, options?: StorageListOptions): Promise<StorageListResult>
exists(key: string): Promise<boolean>
metadata(key: string): Promise<StorageMetadata>
copy(source: string, destination: string): Promise<void>
move(source: string, destination: string): Promise<void>
}
```
### Options
```ts
interface S3Options {
bucket: string
region: string
accessKeyId?: string
secretAccessKey?: string
endpoint?: string
keyPrefix?: string
}
```
### Factory registration
The package imports `StorageFactory` from `@suaveplan/storage` and calls `StorageFactory.register('s3', S3)` as a side-effect on package load (via `src/registry.ts`).
### Error codes (genesis addition, not in main)
```ts
type S3ErrorCode =
| 'S3_CREDENTIALS_MISSING'
| 'S3_CREDENTIALS_INVALID'
| 'S3_OBJECT_NOT_FOUND'
| 'S3_BUCKET_NOT_FOUND'
| 'S3_UPLOAD_FAILED'
| 'S3_DELETE_FAILED'
| 'S3_LIST_FAILED'
| 'S3_PRESIGN_FAILED'
| 'S3_MULTIPART_FAILED'
| 'S3_REGION_MISMATCH'
```
---
## New additions for genesis
### 1. Structured `S3StorageError`
`S3StorageError` extends `SuaveplanError` from `@suaveplan/error` carrying an `S3ErrorCode` discriminant. All internal throw sites use named factory helpers. The `StorageError` from `@suaveplan/storage` is preserved for backward compatibility — `S3StorageError` is emitted for S3-specific failures while `StorageError` wraps generic store-layer failures.
### 2. Zod option schemas
- `S3OptionsSchema` — validates all fields including credential format guards (conditional on `isAwsS3` detection).
- `PresignOptionsSchema` — for the new `presign(key, options)` method.
- `MultipartUploadOptionsSchema` — for `setMultipart(key, parts, options)`.
### 3. Retry integration
`S3Options` gains an optional `retry?: RetryStrategyOptions`. When provided, every S3 API call is wrapped with `@suaveplan/retry`'s `withRetry`. Default: no retry (caller opt-in). Retryable S3 errors (5xx, `SlowDown`, `RequestTimeout`) are automatically detected from the AWS SDK error code.
### 4. AbortSignal support
All async methods accept an optional `signal?: AbortSignal` passed through to the AWS SDK `AbortController` interop. Aborted requests throw `S3StorageError(S3_UPLOAD_FAILED)` with `retryable: false`.
### 5. Presigned URL generation
```ts
interface PresignOptions { expiresIn?: number; method?: 'GET' | 'PUT' }
presign(key: string, options?: PresignOptions): Promise<string>
```
Uses `@aws-sdk/s3-request-presigner`'s `getSignedUrl`. Throws `S3StorageError(S3_PRESIGN_FAILED)` on failure.
### 6. Multipart upload
```ts
interface MultipartUploadOptions { partSize?: number; concurrency?: number; signal?: AbortSignal }
setMultipart(key: string, stream: ReadableStream<Uint8Array>, options?: MultipartUploadOptions): Promise<void>
```
Uses `CreateMultipartUploadCommand` + `UploadPartCommand` + `CompleteMultipartUploadCommand`. Aborts the multipart session on error or signal. Default part size: 5 MiB (AWS minimum). Throws `S3StorageError(S3_MULTIPART_FAILED)` on failure.
### 7. OTel instrumentation
| Operation | Signal |
|---|---|
| `get` | span `s3.get`, histogram `s3.get_duration_ms`, counter `s3.get_total` |
| `set` | span `s3.set`, histogram `s3.set_duration_ms`, counter `s3.set_total`, attribute `s3.object_size` |
| `delete` | span `s3.delete`, counter `s3.delete_total` |
| `list` | span `s3.list`, counter `s3.list_total`, attribute `s3.prefix` |
| `copy` / `move` | span `s3.copy` / `s3.move` |
| `presign` | span `s3.presign`, attribute `s3.expires_in` |
| `setMultipart` | span `s3.multipart`, attribute `s3.part_count` |
### 8. Logger injection
`S3Options` gains an optional `logger?: LoggerLike`. When provided, each operation emits structured entries at `debug` level with key, bucket, duration.
### 9. Credential format guard
AWS S3 credential validation (AKIA prefix + 20 chars, secret key length ≥ 40) is preserved from main. For S3-compatible endpoints (non-amazonaws.com), validation is skipped. This logic now throws `S3StorageError` with appropriate code instead of the main-branch `StorageError`.
### 10. Property-based tests (fast-check)
- **Key prefix normalisation**: for arbitrary key strings and prefix strings, assert that the final S3 key is always `prefix + key` with no double slashes.
- **Credential detection**: for arbitrary endpoint URLs, assert that `isAwsS3(endpoint)` returns true iff the hostname contains `amazonaws.com`.
- **List pagination**: for N objects spread across M pages, assert that `list()` returns all N keys.
### 11. Benchmark suite
- `get_hot`: 100 `get()` calls on pre-existing small objects (requires `GARAGE_ENDPOINT` env; skip otherwise).
- `set_1mb`: upload a 1 MiB buffer (requires env; skip otherwise).
- `list_1000`: list 1000 objects in a bucket (requires env; skip otherwise).
---
## File layout
```text
universal/storage/storage-s3/
package.json
tsconfig.json
vitest.config.ts
src/
index.ts # Public barrel
error-codes.ts # S3.* codes + registerS3ErrorCodes
constants/
index.ts # DEFAULT_PART_SIZE, DEFAULT_PRESIGN_EXPIRES
types/
s3.types.ts # S3Options, PresignOptions, MultipartUploadOptions
s3.types.md
schemas.ts # Zod schemas
schemas.md
s3/
s3/
s3.ts # S3 class
s3.md
s3.test.ts
index.ts
registry/
registry/
registry.ts # StorageFactory.register('s3', S3) side-effect
registry.md
index.ts
utils/
credential-validator.ts # isAwsS3, validateCredentials
credential-validator.md
credential-validator.test.ts
__benchmarks__/
s3.bench.ts
tests/
integration.test.ts # Real Garage/MinIO (GARAGE_ENDPOINT guard)
property.test.ts # fast-check property tests
```
---
## Impact
**Downstream packages**: `@suaveplan/drive` uses `@suaveplan/storage` (not `storage-s3` directly); any package needing S3 object storage will add `@suaveplan/storage-s3` as an optional dep.
**Breaking changes from main-branch API**: none. The `S3` class constructor signature is unchanged; new optional fields (`retry`, `signal`, `logger`) are additive; `presign` and `setMultipart` are new methods.
**Workspace dep prerequisites**: `@suaveplan/storage`, `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/retry` must be published to Verdaccio before this package can be built or tested.

View File

@@ -0,0 +1,168 @@
# Tasks: enterprise-storage-s3
**Change ID**: enterprise-storage-s3
**Package**: `@suaveplan/storage-s3`
**Target path**: `universal/storage/storage-s3`
**Prerequisite packages on Verdaccio**: `@suaveplan/storage`, `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/retry`
---
## 1. Scaffold
- [x] Create directory `universal/storage/storage-s3/` with `src/`, `tests/`, `src/__benchmarks__/`, and all feature subdirectories per the layout in `proposal.md`.
- [x] Write `package.json` at version `0.1.0` with:
- `"type": "module"`, `"sideEffects": false`
- `exports` map: top-level `"."` barrel plus subpaths `./s3/s3`, `./types/s3.types`, `./types/schemas`
- `dependencies`: `@suaveplan/storage workspace:^`, `@suaveplan/error workspace:^`, `@suaveplan/types workspace:^`, `@suaveplan/retry workspace:^`, `@aws-sdk/client-s3 catalog:storage`, `@aws-sdk/s3-request-presigner catalog:storage`, `zod catalog:forms`, `@opentelemetry/api catalog:observability`, `tslib catalog:build`
- `devDependencies`: `vitest catalog:testing`, `@vitest/coverage-istanbul catalog:testing`, `fast-check catalog:testing`, `mitata catalog:testing`, `@suaveplan/testing workspace:^`
- [x] Write `tsconfig.json` extending `@suaveplan/typescript-config/library.json`.
- [x] Write `vitest.config.ts` using `createTestConfig()` from `@suaveplan/testing/config` (pool: `forks`).
- [x] Add `lint`, `typecheck`, `test`, `test:coverage`, `build`, `docs:check`, `bench` scripts to `package.json`.
---
## 2. Types + errors
### 2a. S3 types (`src/types/s3.types.ts` + `src/types/s3.types.md`)
- [x] Define `S3Options`: `bucket`, `region`, `accessKeyId?`, `secretAccessKey?`, `endpoint?`, `keyPrefix?` — preserved from main.
- [x] Add genesis-only optional fields: `retry?: RetryStrategyOptions`, `logger?: LoggerLike`, `signal?: AbortSignal` (default for all ops).
- [x] Define `PresignOptions`: `expiresIn?: number`, `method?: 'GET' | 'PUT'`.
- [x] Define `MultipartUploadOptions`: `partSize?: number`, `concurrency?: number`, `signal?: AbortSignal`.
- [x] Write `src/types/s3.types.md` (≥200 words, API reference, examples).
### 2b. Error codes (`src/error-codes.ts`)
- [x] Import `CodeMetadata`, `ErrorCodes`, `SuaveplanError` from `@suaveplan/error`.
- [x] Define `S3ErrorCode` union — ten codes: `S3_CREDENTIALS_MISSING`, `S3_CREDENTIALS_INVALID`, `S3_OBJECT_NOT_FOUND`, `S3_BUCKET_NOT_FOUND`, `S3_UPLOAD_FAILED`, `S3_DELETE_FAILED`, `S3_LIST_FAILED`, `S3_PRESIGN_FAILED`, `S3_MULTIPART_FAILED`, `S3_REGION_MISMATCH`.
- [x] Define `S3_ERROR_METADATA` with `httpStatus`, `severity`, `retryable`, `category`, `description` per code.
- [x] Export `registerS3ErrorCodes()`. Idempotent; called eagerly at module scope.
- [x] Export `S3StorageError` extending `SuaveplanError` carrying `S3ErrorCode` discriminant.
### 2c. Zod schemas (`src/types/schemas.ts` + `src/types/schemas.md`)
- [x] Write `S3OptionsSchema`, `PresignOptionsSchema`, `MultipartUploadOptionsSchema`.
- [x] Write `src/types/schemas.md` (≥200 words).
---
## 3. Constants (`src/constants/index.ts`)
- [x] Export `DEFAULT_PRESIGN_EXPIRES_SECONDS = 3600`, `DEFAULT_MULTIPART_PART_SIZE = 5 * 1024 * 1024`, `DEFAULT_MULTIPART_CONCURRENCY = 4`.
---
## 4. Credential validator (`src/utils/credential-validator.ts`)
- [x] Implement `isAwsS3(endpoint?: string): boolean` — returns `true` if endpoint is undefined or hostname contains `amazonaws.com`.
- [x] Implement `validateCredentials(options: S3Options): void`:
- Throws `S3StorageError(S3_CREDENTIALS_MISSING)` if `accessKeyId` or `secretAccessKey` absent (neither in options nor env).
- Throws `S3StorageError(S3_CREDENTIALS_INVALID)` for invalid AWS AKIA key format when `isAwsS3(options.endpoint)` is true.
- No validation for S3-compatible endpoints.
- [x] Write `credential-validator.test.ts`: valid AWS creds, invalid format, missing creds, S3-compatible endpoint skips validation.
- [x] Write `credential-validator.md` (≥200 words).
---
## 5. S3 class (`src/s3/s3/`)
- [x] Implement `S3 extends AbstractStorage` (import from `@suaveplan/storage`):
- Constructor calls `validateCredentials(options)`, builds `S3Client` with `{ region, credentials: { accessKeyId, secretAccessKey }, endpoint }`.
- Internal `_resolveKey(key): string` — applies `options.keyPrefix` stripping double-slashes.
- **`get(key, options?)`**:
- Sends `GetObjectCommand`; reads response body as `Uint8Array`.
- Throws `S3StorageError(S3_OBJECT_NOT_FOUND)` on `NoSuchKey`.
- OTel span + counter. Retry wrapper if configured.
- **`set(key, value, options?)`**:
- Sends `PutObjectCommand`.
- Throws `S3StorageError(S3_UPLOAD_FAILED)` on error.
- OTel span + histogram (object size attribute).
- **`delete(key)`**:
- Sends `DeleteObjectCommand`.
- Throws `S3StorageError(S3_DELETE_FAILED)` on error.
- **`list(prefix?, options?)`**:
- Sends `ListObjectsV2Command` with pagination (handles `NextContinuationToken`).
- Throws `S3StorageError(S3_LIST_FAILED)` on error.
- **`exists(key)`**: sends `HeadObjectCommand`; returns `true` on 200, `false` on `NoSuchKey`.
- **`metadata(key)`**: parses `HeadObjectCommand` response into `StorageMetadata`.
- **`copy(source, destination)`**: sends `CopyObjectCommand`.
- **`move(source, destination)`**: `copy` then `delete`.
- **`presign(key, options?)`**: uses `getSignedUrl` from `@aws-sdk/s3-request-presigner`. Throws `S3StorageError(S3_PRESIGN_FAILED)`.
- **`setMultipart(key, stream, options?)`**:
- `CreateMultipartUploadCommand` → upload parts with concurrency cap → `CompleteMultipartUploadCommand`.
- On any error: `AbortMultipartUploadCommand` then rethrow as `S3StorageError(S3_MULTIPART_FAILED)`.
- Respects `options.signal` (abort mid-upload).
- All methods accept per-call `signal?: AbortSignal` that overrides the instance-level default.
- Log via injected `logger` at `debug` level.
- [x] Write `s3.test.ts` (unit: inject a mock `S3Client` via constructor option; no vi.mock — DI only). Cover: get hit, get miss (404), set, delete, list with pagination, exists true/false, copy, move, presign, credential validation failure.
- [x] Write `s3.md` (≥200 words, both basic + advanced example with retry config).
---
## 6. Registry (`src/registry/registry/`)
- [x] `registry.ts` — imports `S3` and calls `StorageFactory.register('s3', S3)` as a side-effect on import.
- [x] Write `registry.md` explaining the side-effect and tree-shaking implications.
---
## 7. Public barrel (`src/index.ts`)
- [x] Re-export: `S3`, `S3StorageError`, `S3OptionsSchema`, `PresignOptionsSchema`, `MultipartUploadOptionsSchema`, all types, error-codes, constants.
- [x] Import `./registry/registry` to trigger provider registration.
---
## 8. Property-based tests (`tests/property.test.ts`)
Use `fast-check`. All timeouts: `{ timeout: 30000 }`.
- [x] **Key prefix normalisation**: for arbitrary key and prefix strings (alphanum + `/`), assert `resolveKey(prefix, key)` never produces double slashes and always starts with prefix.
- [x] **`isAwsS3` detection**: for arbitrary endpoint strings, assert `isAwsS3` returns true iff hostname contains `amazonaws.com`.
- [x] **List pagination assembly**: given M pages of N/M keys each, assert the assembled list equals the concatenation of all pages in order.
---
## 9. Integration tests (`tests/integration.test.ts`)
- [x] Gate all tests behind `process.env.GARAGE_ENDPOINT` skip guard.
- [x] Test: put → get → exists → metadata → list → copy → move → delete against a real Garage S3-compatible server.
- [x] Test: multipart upload of a 15 MiB buffer (3 × 5 MiB parts).
- [x] Test: presigned GET URL is accessible without credentials.
---
## 10. Benchmarks (`src/__benchmarks__/s3.bench.ts`)
Use `mitata`. Gate IO benchmarks behind `SKIP_S3_BENCHMARKS` env flag.
- [x] `get_hot_100` (conditional): 100 `get()` calls on a pre-existing 1 KiB object.
- [x] `set_1mb` (conditional): upload a 1 MiB buffer.
- [x] `list_1000` (conditional): list 1000 objects.
- [x] `credential_validate_1k` (pure): 1000 credential validation calls.
---
## 11. Docs
- [x] `src/types/s3.types.md`
- [x] `src/types/schemas.md`
- [x] `src/s3/s3/s3.md`
- [x] `src/registry/registry/registry.md`
- [x] `src/utils/credential-validator.md`
- [x] Root `universal/storage/storage-s3/README.md`.
---
## 12. Verify and publish
- [x] Run `bunx turbo lint --filter=@suaveplan/storage-s3` — zero errors.
- [x] Run `bunx turbo typecheck --filter=@suaveplan/storage-s3` — zero errors.
- [x] Run `bunx turbo test --filter=@suaveplan/storage-s3` — all tests pass.
- [x] Run `bunx turbo test:coverage --filter=@suaveplan/storage-s3` — 100% on all touched modules.
- [x] Run `bun run docs:check --filter=@suaveplan/storage-s3` — zero errors.
- [x] Confirm no `TODO`, `FIXME`, `@deprecated`, untyped `any`, `ts-ignore`, or `.skip` in `src/` or `tests/`.
- [x] Confirm `"sideEffects": false` in `package.json`.
- [x] Run `bun scripts/publish.ts --package @suaveplan/storage-s3`.
- [x] Update `CLAUDE.md` package listing to include `@suaveplan/storage-s3`.
- [x] Archive: move `openspec/changes/enterprise-storage-s3/` to `openspec/archive/2026-04-24-enterprise-storage-s3/` and fold spec delta into `openspec/specs/storage/spec.md`.