first commit
This commit is contained in:
140
docs/persistence/storage/storage-s3/src/s3/s3.md
Normal file
140
docs/persistence/storage/storage-s3/src/s3/s3.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# S3
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Implements the `AbstractStorage` interface from `@suaveplan/storage` backed by the AWS SDK v3 `@aws-sdk/client-s3`. The `S3` class provides full lifecycle management for objects in any S3-compatible bucket, including AWS S3, Cloudflare R2, MinIO, and Garage. It adds enterprise capabilities not present in the main-branch reference: structured error codes, presigned URL generation, multipart upload with automatic part splitting and abort-on-failure, injectable retry strategy, AbortSignal propagation, and OpenTelemetry span/counter/histogram instrumentation.
|
||||
|
||||
The class is intentionally thin: it translates between `AbstractStorage`'s key-value contract and S3's object model, adds a key prefix namespace, and delegates all batching, pattern-listing, and hook machinery to the base class. The `S3ClientFactory` interface enables full testing via in-memory fakes without touching real AWS infrastructure.
|
||||
|
||||
## Features
|
||||
|
||||
- Implements all `AbstractStorage` abstract methods: `get`, `set`, `delete`, `exists`, `list`, `clear`, `getWithMetadata`, `getMetadata`, `getSize`, `getPartial`, `getItemSize`, `createReadStream`, `createWriteStream`
|
||||
- S3-specific extensions: `presign` (GET/PUT), `copy`, `move`, `setMultipart`
|
||||
- Key prefix namespacing: `keyPrefix` option prepends a path segment to all keys
|
||||
- Retry with exponential backoff for retryable S3 errors (5xx, SlowDown, RequestTimeout)
|
||||
- AbortSignal propagation to all AWS SDK calls
|
||||
- OpenTelemetry spans, counters, and histograms for every operation
|
||||
- Structured `S3StorageError` with stable `S3_CODES` for all failure modes
|
||||
- Injectable `S3ClientFactory` for testability
|
||||
- AWS credential format validation via `validateCredentials`
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { S3, createS3 } from "@suaveplan/storage-s3";
|
||||
|
||||
const storage = createS3({
|
||||
bucket: "my-bucket",
|
||||
region: "us-east-1",
|
||||
accessKeyId: "AKIAIOSFODNN7EXAMPLE",
|
||||
secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
});
|
||||
|
||||
await storage.set("hello.txt", new TextEncoder().encode("hello"));
|
||||
const data = await storage.get("hello.txt");
|
||||
console.log(new TextDecoder().decode(data)); // "hello"
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { S3 } from "@suaveplan/storage-s3";
|
||||
import type { S3ClientFactory, S3Options } from "@suaveplan/storage-s3";
|
||||
|
||||
// Inject a fake client for testing
|
||||
const fakeClient = new FakeS3Client(); // your in-memory implementation
|
||||
|
||||
const fakeFactory: S3ClientFactory = {
|
||||
create: () => fakeClient as unknown as S3Client,
|
||||
};
|
||||
|
||||
const storage = new S3(
|
||||
{
|
||||
bucket: "test-bucket",
|
||||
region: "us-east-1",
|
||||
endpoint: "http://localhost:9000",
|
||||
keyPrefix: "uploads/v2",
|
||||
retry: { maxAttempts: 3, baseDelayMs: 50 },
|
||||
},
|
||||
fakeFactory
|
||||
);
|
||||
|
||||
// Multipart upload
|
||||
const stream = new ReadableStream<Uint8Array>({
|
||||
start(ctrl) {
|
||||
ctrl.enqueue(new Uint8Array(10 * 1024 * 1024)); // 10 MiB
|
||||
ctrl.close();
|
||||
},
|
||||
});
|
||||
await storage.setMultipart("large-file.bin", stream, { partSize: 5 * 1024 * 1024 });
|
||||
|
||||
// Presigned URL
|
||||
const url = await storage.presign("large-file.bin", { expiresIn: 300, method: "GET" });
|
||||
console.log(url); // https://... signed URL
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `new S3(opts: S3Options, clientFactory?: S3ClientFactory)`
|
||||
|
||||
Creates an S3 adapter.
|
||||
|
||||
| Param | Type | Description |
|
||||
|---|---|---|
|
||||
| `opts` | `S3Options` | Bucket, region, credentials, endpoint, prefix, retry, logger |
|
||||
| `clientFactory` | `S3ClientFactory?` | Injectable factory; defaults to real AWS SDK client |
|
||||
|
||||
**Throws**: `S3StorageError(S3.CREDENTIALS_INVALID)` if AWS credential format is invalid.
|
||||
|
||||
### `createS3(opts, clientFactory?): S3`
|
||||
|
||||
Factory function equivalent to `new S3(opts, clientFactory)`.
|
||||
|
||||
### `get(key, signal?): Promise<Uint8Array | undefined>`
|
||||
|
||||
Downloads an object. Returns `undefined` if the key does not exist.
|
||||
|
||||
### `set(key, value, options?, signal?): Promise<void>`
|
||||
|
||||
Uploads a `Uint8Array` as an object. Throws `S3StorageError(S3.UPLOAD_FAILED)` on failure.
|
||||
|
||||
### `delete(key, signal?): Promise<boolean>`
|
||||
|
||||
Deletes an object. Returns `false` if it did not exist, `true` if deleted.
|
||||
|
||||
### `exists(key, signal?): Promise<boolean>`
|
||||
|
||||
Checks existence via `HeadObject`.
|
||||
|
||||
### `list(options?, signal?): Promise<StorageListResult>`
|
||||
|
||||
Lists objects under an optional prefix. Follows continuation tokens internally.
|
||||
|
||||
### `clear(signal?): Promise<void>`
|
||||
|
||||
Deletes all objects under the configured key prefix.
|
||||
|
||||
### `presign(key, opts?): Promise<string>`
|
||||
|
||||
Returns a presigned URL (GET by default). Throws `S3StorageError(S3.PRESIGN_FAILED)`.
|
||||
|
||||
### `copy(source, destination, signal?): Promise<void>`
|
||||
|
||||
Copies an object within the same bucket. Throws `S3StorageError(S3.UPLOAD_FAILED)`.
|
||||
|
||||
### `move(source, destination, signal?): Promise<void>`
|
||||
|
||||
Moves an object: copy then delete source.
|
||||
|
||||
### `setMultipart(key, stream, opts?): Promise<void>`
|
||||
|
||||
Uploads a `ReadableStream` in parts. Automatically aborts the multipart session on error or signal. Throws `S3StorageError(S3.MULTIPART_FAILED)`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The `withRetry` helper is a lightweight internal implementation rather than using `@suaveplan/retry` directly, to avoid the need for the retry package to be a hard runtime dep on the critical storage path. The `RetryOptions` type is a minimal subset, and consumers can opt in by passing `retry: { maxAttempts: 3 }`. Retryable errors are detected by checking `S3ServiceException.$retryable.throttling`, `name` for `SlowDown`/`RequestTimeout`, and the HTTP status code ≥ 500.
|
||||
|
||||
For `setMultipart`, the stream is read in a single pass, buffering chunks until a full `partSize` accumulates. The final part may be smaller than `partSize`. AWS requires each part except the last to be at least 5 MiB, so the default `partSize` matches that minimum. If the multipart session fails after `CreateMultipartUpload`, the session is aborted via `AbortMultipartUploadCommand` to avoid incurring storage costs for incomplete uploads.
|
||||
Reference in New Issue
Block a user