first commit

This commit is contained in:
2026-05-29 08:20:56 +01:00
commit 949299f58d
6368 changed files with 527176 additions and 0 deletions

View File

@@ -0,0 +1,44 @@
# @suaveplan/storage-s3 - S3-compatible object storage adapter for @suaveplan/storage: implements AbstractStorage backed by @aws-sdk/client-s3 with presigned URLs, multipart upload, retry integration, and OTel instrumentation.
## Summary
S3-compatible object storage adapter for `@suaveplan/storage`. Extends `AbstractStorage` backed by `@aws-sdk/client-s3` with presigned URL generation, multipart upload, retry integration, and OpenTelemetry instrumentation.
<!-- begin:auto-docs -->
## Documentation tree
- [registry](./src/registry/registry/registry.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [registry](./src/registry/registry/registry.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [utils](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [credential-validator](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
<!-- end:auto-docs --> - [registry](./src/registry/registry/registry.md) — <!-- begin:auto-docs --> _(no submodules)_- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_- [utils](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_- [credential-validator](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_- [registry](./src/registry/registry/registry.md) — <!-- begin:auto-docs --> _(no submodules)_- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_- [utils](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_- [credential-validator](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_- [registry](./src/registry/registry/registry.md) — <!-- begin:auto-docs --> _(no submodules)_- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_- [utils](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_- [credential-validator](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_- [registry](./src/registry/registry/registry.md) — <!-- begin:auto-docs --> _(no submodules)_- [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_ - [s3](./src/s3/s3.md) — <!-- begin:auto-docs --> _(no submodules)_- [utils](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_ - [credential-validator](./src/utils/credential-validator.md) — <!-- begin:auto-docs --> _(no submodules)_
## Installation
```bash
bun add @suaveplan/storage-s3
```
## Quick Start
```ts
import { createS3 } from "@suaveplan/storage-s3";
const storage = createS3({
bucket: "my-bucket",
region: "us-east-1",
accessKeyId: "AKIAIOSFODNN7EXAMPLE",
secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
});
await storage.set("hello.txt", new TextEncoder().encode("hello"));
const data = await storage.get("hello.txt");
```
## Modules
- [`src/s3/s3/s3`](src/s3/s3/s3.md) — `S3` class and `createS3` factory
- [`src/types/s3.types`](src/types/s3.types.md) — `S3Options`, `PresignOptions`, `MultipartUploadOptions`, `S3ClientFactory`
- [`src/utils/credential-validator`](src/utils/credential-validator.md) — `isAwsS3`, `validateCredentials`
- [`src/registry/registry/registry`](src/registry/registry/registry.md) — `registerS3Provider`

View File

@@ -0,0 +1,56 @@
# registry
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Registers the `S3` storage provider with `StorageFactory` from `@suaveplan/storage` under the provider key `"s3"`. This side-effect runs automatically when the module is imported, enabling consumers to use `createStorage("s3", opts)` without manually wiring the class. The registration is idempotent — importing the module multiple times does not register duplicate providers.
Exposing the registration via a named `registerS3Provider` function (in addition to the auto-registration side-effect) allows consumers who tree-shake aggressively to call it explicitly in their initialization code.
## Features
- Auto-registers `S3` with `StorageFactory` on module import
- Named `registerS3Provider()` export for explicit initialization in tree-shaken builds
- Compatible with `StorageFactory.register` module-augmentation pattern
## Usage: Basic Example
```ts
// Import to trigger registration (side-effect)
import "@suaveplan/storage-s3/registry/registry";
import { createStorage } from "@suaveplan/storage";
const storage = createStorage("s3", {
bucket: "my-bucket",
region: "us-east-1",
});
```
## Usage: Advanced Example
```ts
import { registerS3Provider } from "@suaveplan/storage-s3/registry/registry";
import { createStorage } from "@suaveplan/storage";
// Explicit registration (useful when tree-shaking might remove the import)
registerS3Provider();
const storage = createStorage("s3", {
bucket: "prod-bucket",
region: "eu-west-1",
endpoint: "https://garage.example.com",
});
await storage.set("config.json", new TextEncoder().encode(JSON.stringify({})));
```
## API Reference
### `registerS3Provider(): void`
Registers the `S3` class with `StorageFactory` under the key `"s3"`. Safe to call multiple times — subsequent calls are idempotent if `StorageFactory.register` deduplicates by key.
## Implementation Notes
The `StorageFactory.register` call uses a type cast because `StorageFactory` expects a constructor signature matching `AbstractStorage`, while `S3` extends `AbstractStorage` but accepts `S3Options` rather than a generic storage config. The cast is safe because `createStorage("s3", opts)` users are expected to pass `S3Options` compatible objects.

View File

@@ -0,0 +1,140 @@
# S3
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Implements the `AbstractStorage` interface from `@suaveplan/storage` backed by the AWS SDK v3 `@aws-sdk/client-s3`. The `S3` class provides full lifecycle management for objects in any S3-compatible bucket, including AWS S3, Cloudflare R2, MinIO, and Garage. It adds enterprise capabilities not present in the main-branch reference: structured error codes, presigned URL generation, multipart upload with automatic part splitting and abort-on-failure, injectable retry strategy, AbortSignal propagation, and OpenTelemetry span/counter/histogram instrumentation.
The class is intentionally thin: it translates between `AbstractStorage`'s key-value contract and S3's object model, adds a key prefix namespace, and delegates all batching, pattern-listing, and hook machinery to the base class. The `S3ClientFactory` interface enables full testing via in-memory fakes without touching real AWS infrastructure.
## Features
- Implements all `AbstractStorage` abstract methods: `get`, `set`, `delete`, `exists`, `list`, `clear`, `getWithMetadata`, `getMetadata`, `getSize`, `getPartial`, `getItemSize`, `createReadStream`, `createWriteStream`
- S3-specific extensions: `presign` (GET/PUT), `copy`, `move`, `setMultipart`
- Key prefix namespacing: `keyPrefix` option prepends a path segment to all keys
- Retry with exponential backoff for retryable S3 errors (5xx, SlowDown, RequestTimeout)
- AbortSignal propagation to all AWS SDK calls
- OpenTelemetry spans, counters, and histograms for every operation
- Structured `S3StorageError` with stable `S3_CODES` for all failure modes
- Injectable `S3ClientFactory` for testability
- AWS credential format validation via `validateCredentials`
## Usage: Basic Example
```ts
import { S3, createS3 } from "@suaveplan/storage-s3";
const storage = createS3({
bucket: "my-bucket",
region: "us-east-1",
accessKeyId: "AKIAIOSFODNN7EXAMPLE",
secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
});
await storage.set("hello.txt", new TextEncoder().encode("hello"));
const data = await storage.get("hello.txt");
console.log(new TextDecoder().decode(data)); // "hello"
```
## Usage: Advanced Example
```ts
import { S3 } from "@suaveplan/storage-s3";
import type { S3ClientFactory, S3Options } from "@suaveplan/storage-s3";
// Inject a fake client for testing
const fakeClient = new FakeS3Client(); // your in-memory implementation
const fakeFactory: S3ClientFactory = {
create: () => fakeClient as unknown as S3Client,
};
const storage = new S3(
{
bucket: "test-bucket",
region: "us-east-1",
endpoint: "http://localhost:9000",
keyPrefix: "uploads/v2",
retry: { maxAttempts: 3, baseDelayMs: 50 },
},
fakeFactory
);
// Multipart upload
const stream = new ReadableStream<Uint8Array>({
start(ctrl) {
ctrl.enqueue(new Uint8Array(10 * 1024 * 1024)); // 10 MiB
ctrl.close();
},
});
await storage.setMultipart("large-file.bin", stream, { partSize: 5 * 1024 * 1024 });
// Presigned URL
const url = await storage.presign("large-file.bin", { expiresIn: 300, method: "GET" });
console.log(url); // https://... signed URL
```
## API Reference
### `new S3(opts: S3Options, clientFactory?: S3ClientFactory)`
Creates an S3 adapter.
| Param | Type | Description |
|---|---|---|
| `opts` | `S3Options` | Bucket, region, credentials, endpoint, prefix, retry, logger |
| `clientFactory` | `S3ClientFactory?` | Injectable factory; defaults to real AWS SDK client |
**Throws**: `S3StorageError(S3.CREDENTIALS_INVALID)` if AWS credential format is invalid.
### `createS3(opts, clientFactory?): S3`
Factory function equivalent to `new S3(opts, clientFactory)`.
### `get(key, signal?): Promise<Uint8Array | undefined>`
Downloads an object. Returns `undefined` if the key does not exist.
### `set(key, value, options?, signal?): Promise<void>`
Uploads a `Uint8Array` as an object. Throws `S3StorageError(S3.UPLOAD_FAILED)` on failure.
### `delete(key, signal?): Promise<boolean>`
Deletes an object. Returns `false` if it did not exist, `true` if deleted.
### `exists(key, signal?): Promise<boolean>`
Checks existence via `HeadObject`.
### `list(options?, signal?): Promise<StorageListResult>`
Lists objects under an optional prefix. Follows continuation tokens internally.
### `clear(signal?): Promise<void>`
Deletes all objects under the configured key prefix.
### `presign(key, opts?): Promise<string>`
Returns a presigned URL (GET by default). Throws `S3StorageError(S3.PRESIGN_FAILED)`.
### `copy(source, destination, signal?): Promise<void>`
Copies an object within the same bucket. Throws `S3StorageError(S3.UPLOAD_FAILED)`.
### `move(source, destination, signal?): Promise<void>`
Moves an object: copy then delete source.
### `setMultipart(key, stream, opts?): Promise<void>`
Uploads a `ReadableStream` in parts. Automatically aborts the multipart session on error or signal. Throws `S3StorageError(S3.MULTIPART_FAILED)`.
## Implementation Notes
The `withRetry` helper is a lightweight internal implementation rather than using `@suaveplan/retry` directly, to avoid the need for the retry package to be a hard runtime dep on the critical storage path. The `RetryOptions` type is a minimal subset, and consumers can opt in by passing `retry: { maxAttempts: 3 }`. Retryable errors are detected by checking `S3ServiceException.$retryable.throttling`, `name` for `SlowDown`/`RequestTimeout`, and the HTTP status code ≥ 500.
For `setMultipart`, the stream is read in a single pass, buffering chunks until a full `partSize` accumulates. The final part may be smaller than `partSize`. AWS requires each part except the last to be at least 5 MiB, so the default `partSize` matches that minimum. If the multipart session fails after `CreateMultipartUpload`, the session is aborted via `AbortMultipartUploadCommand` to avoid incurring storage costs for incomplete uploads.

View File

@@ -0,0 +1,92 @@
# s3.types
## Purpose
Defines the public TypeScript types for the `@suaveplan/storage-s3` package. These types govern configuration, presigned URL generation, multipart upload, and the injectable S3 client factory interface. They are designed to be stable, serialization-friendly, and usable without importing the main `S3` class.
The types decouple configuration concerns from implementation, enabling dependency injection and testability. The `S3ClientFactory` interface in particular allows tests to inject a `FakeS3Client` without touching real AWS infrastructure.
## Features
- `S3Options` — complete configuration for the S3 adapter including credentials, endpoint, key prefix, retry strategy, and logger injection
- `PresignOptions` — controls presigned URL generation: expiry and HTTP method
- `MultipartUploadOptions` — controls multipart upload: part size, concurrency, and AbortSignal
- `S3ClientFactory` — injectable factory interface for the underlying `S3Client`
- `RetryOptions` — minimal subset of `@suaveplan/retry` `RetryStrategyOptions` for forward-compatible configuration
## Usage: Basic Example
```ts
import type { S3Options } from "@suaveplan/storage-s3";
const options: S3Options = {
bucket: "my-bucket",
region: "us-east-1",
accessKeyId: "AKIAIOSFODNN7EXAMPLE",
secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
};
```
## Usage: Advanced Example
```ts
import type { S3Options, S3ClientFactory } from "@suaveplan/storage-s3";
import { S3Client } from "@aws-sdk/client-s3";
import type { LoggerLike } from "@suaveplan/types";
const logger: LoggerLike = console;
const options: S3Options = {
bucket: "my-bucket",
region: "auto",
endpoint: "https://my-garage-instance.example.com",
keyPrefix: "uploads/",
retry: { maxAttempts: 5, baseDelayMs: 200 },
logger,
};
// Custom factory for testing
const fakeFactory: S3ClientFactory = {
create: (_opts) => new S3Client({ region: "us-east-1" }),
};
```
## API Reference
### `S3Options`
| Field | Type | Required | Description |
|---|---|---|---|
| `bucket` | `string` | Yes | S3 bucket name |
| `region` | `string` | Yes | AWS region or "auto" |
| `accessKeyId` | `string` | No | AWS access key ID |
| `secretAccessKey` | `string` | No | AWS secret access key |
| `endpoint` | `string` | No | Custom S3-compatible endpoint |
| `keyPrefix` | `string` | No | Prefix prepended to all keys |
| `retry` | `RetryOptions` | No | Retry strategy configuration |
| `logger` | `LoggerLike` | No | Structured logger |
### `PresignOptions`
| Field | Type | Required | Description |
|---|---|---|---|
| `expiresIn` | `number` | No | Expiry in seconds (default: 3600) |
| `method` | `"GET" \| "PUT"` | No | HTTP method (default: "GET") |
### `MultipartUploadOptions`
| Field | Type | Required | Description |
|---|---|---|---|
| `partSize` | `number` | No | Part size in bytes (default: 5 MiB) |
| `concurrency` | `number` | No | Max concurrent part uploads (default: 4) |
| `signal` | `AbortSignal` | No | Cancellation signal |
### `S3ClientFactory`
| Method | Params | Returns | Description |
|---|---|---|---|
| `create` | `options: S3Options` | `S3Client` | Creates an AWS SDK S3Client from options |
## Implementation Notes
`RetryOptions` is intentionally a minimal subset of `@suaveplan/retry`'s full options to avoid a circular type dependency. The `S3` class maps these to the full retry strategy internally. `S3ClientFactory` returns the AWS SDK `S3Client` type directly; consumers who want to mock S3 should implement this interface and return an object with matching method signatures rather than extending the SDK class.

View File

@@ -0,0 +1,89 @@
# credential-validator
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides AWS S3 credential format validation and endpoint classification for the `@suaveplan/storage-s3` package. It guards against misconfigured credentials before they hit the AWS API, producing structured `S3StorageError` instances rather than opaque SDK errors. For S3-compatible endpoints (MinIO, Garage, Cloudflare R2) it skips AWS-specific format checks since those providers use different credential formats.
The utility is also used in property-based tests to verify that `isAwsS3` correctly classifies arbitrary endpoint URLs.
## Features
- `isAwsS3(endpoint?)` — classifies an endpoint as AWS S3 proper vs. S3-compatible; uses `URL.hostname` for robustness
- `validateCredentials(options)` — validates access key ID (AKIA prefix + 20 chars) and secret key (≥ 40 chars) only for AWS S3 endpoints
- Allows environment/instance credentials (IAM roles, EC2 instance profiles) when neither key is provided
- Throws `S3StorageError` with stable `S3_CODES` for all failure modes
## Usage: Basic Example
```ts
import { isAwsS3, validateCredentials } from "@suaveplan/storage-s3/utils/credential-validator";
// Classify endpoint
isAwsS3(); // true (no endpoint = AWS)
isAwsS3("https://s3.amazonaws.com"); // true
isAwsS3("https://my-garage.example.com"); // false
// Validate credentials
validateCredentials({ bucket: "b", region: "us-east-1" }); // OK (instance creds)
```
## Usage: Advanced Example
```ts
import { validateCredentials } from "@suaveplan/storage-s3/utils/credential-validator";
import { S3StorageError } from "@suaveplan/storage-s3";
try {
validateCredentials({
bucket: "my-bucket",
region: "us-east-1",
accessKeyId: "BADKEY",
secretAccessKey: "short",
});
} catch (err) {
if (err instanceof S3StorageError) {
console.log(err.code); // "S3.CREDENTIALS_INVALID"
}
}
// S3-compatible endpoint: skip validation
validateCredentials({
bucket: "my-bucket",
region: "garage",
endpoint: "https://my-garage.internal",
accessKeyId: "garage-key", // non-AKIA format is fine
secretAccessKey: "s", // short is fine too
}); // does not throw
```
## API Reference
### `isAwsS3(endpoint?: string): boolean`
Returns `true` if `endpoint` is undefined (defaults to AWS) or its hostname ends with `"amazonaws.com"`. Returns `false` for all other URLs, including malformed ones.
| Param | Type | Description |
|---|---|---|
| `endpoint` | `string?` | Custom endpoint URL; omit for AWS S3 proper |
**Returns**: `boolean`
### `validateCredentials(options: S3Options): void`
Validates credential format when targeting AWS S3. No-op for S3-compatible endpoints.
| Param | Type | Description |
|---|---|---|
| `options` | `S3Options` | Full S3 options including credentials and endpoint |
**Throws**:
- `S3StorageError(S3.CREDENTIALS_MISSING)` — when one of the pair is provided but not the other
- `S3StorageError(S3.CREDENTIALS_INVALID)` — when the access key ID or secret key fails format checks
## Implementation Notes
AWS credentials may be sourced from environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), IAM instance profiles, or ECS task roles — these are handled by the AWS SDK transparently. This validator only checks explicitly provided credentials. The `URL` constructor is used for endpoint parsing because string matching is fragile; any malformed URL string returns `false` from `isAwsS3` rather than throwing.