first commit
This commit is contained in:
100
docs/foundation/flow/queue/src/worker/queue-worker.md
Normal file
100
docs/foundation/flow/queue/src/worker/queue-worker.md
Normal file
@@ -0,0 +1,100 @@
|
||||
# queue-worker
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
`QueueWorker<T>` drains a `QueueLike<T>` queue by calling a user-supplied handler for each job. It manages concurrency (up to N simultaneous jobs), implements exponential-backoff retry, forwards exhausted jobs to an optional dead-letter queue, and honours `AbortSignal` for graceful shutdown.
|
||||
|
||||
The worker is intentionally decoupled from any specific queue implementation — it accepts any object that satisfies `QueueLike<T>`, allowing `MemoryQueue`, `PriorityQueue`, `DelayedQueue`, or a custom adapter to be swapped in.
|
||||
|
||||
## Features
|
||||
|
||||
- Configurable concurrency: processes up to N jobs simultaneously
|
||||
- Exponential-backoff retry with configurable `maxAttempts`, `backoffMs`, and `backoffMultiplier`
|
||||
- Dead-letter queue forwarding when all retries are exhausted
|
||||
- `AbortSignal` cancellation: throws `QueueWorkerAbortedError` on abort
|
||||
- `WorkerStats` snapshot (processed, succeeded, failed, dead, retried)
|
||||
- `runWorker()` convenience wrapper for fire-and-forget usage
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createMemoryQueue } from "@suaveplan/queue/queue/memory-queue";
|
||||
import { QueueWorker } from "@suaveplan/queue/worker/queue-worker";
|
||||
|
||||
const queue = createMemoryQueue<string>();
|
||||
await queue.enqueue("hello");
|
||||
await queue.enqueue("world");
|
||||
|
||||
const worker = new QueueWorker(queue, async (job) => {
|
||||
console.log("Processing:", job.payload);
|
||||
});
|
||||
await worker.run();
|
||||
console.log(worker.stats); // { processed: 2, succeeded: 2, failed: 0, dead: 0, retried: 0 }
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { createMemoryQueue } from "@suaveplan/queue/queue/memory-queue";
|
||||
import { runWorker } from "@suaveplan/queue/worker/queue-worker";
|
||||
|
||||
const mainQueue = createMemoryQueue<{ taskId: string }>();
|
||||
const deadLetterQueue = createMemoryQueue<unknown>();
|
||||
|
||||
await mainQueue.enqueue({ taskId: "a" });
|
||||
await mainQueue.enqueue({ taskId: "b" });
|
||||
|
||||
let callCount = 0;
|
||||
const stats = await runWorker(
|
||||
mainQueue,
|
||||
async (job) => {
|
||||
callCount++;
|
||||
if (callCount <= 4) throw new Error("transient failure"); // triggers retries
|
||||
// succeeds on later attempts
|
||||
},
|
||||
{
|
||||
concurrency: 2,
|
||||
retryPolicy: { maxAttempts: 2, backoffMs: 10, backoffMultiplier: 2 },
|
||||
deadLetterQueue,
|
||||
}
|
||||
);
|
||||
|
||||
console.log(stats.dead); // jobs that exhausted all retries
|
||||
console.log(deadLetterQueue.size()); // dead-letter queue depth
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `QueueWorker<T>`
|
||||
|
||||
#### `new QueueWorker(queue, handler, options?)`
|
||||
|
||||
- `queue: QueueLike<T>` — source queue
|
||||
- `handler: (job: Job<T>) => Promise<void>` — async job processor; throw to fail the job
|
||||
- `options.concurrency` — default `1`
|
||||
- `options.retryPolicy` — default `{ maxAttempts: 3, backoffMs: 1000, backoffMultiplier: 2 }`
|
||||
- `options.deadLetterQueue?` — receives exhausted jobs
|
||||
- `options.signal?` — `AbortSignal` for cancellation
|
||||
|
||||
#### `worker.run(): Promise<void>`
|
||||
|
||||
Drains the queue. Resolves when empty. Throws `QueueWorkerAbortedError` on abort.
|
||||
|
||||
#### `worker.stats: WorkerStats`
|
||||
|
||||
Snapshot: `{ processed, succeeded, failed, dead, retried }`.
|
||||
|
||||
#### `worker.running: boolean`
|
||||
|
||||
`true` while `run()` has not resolved.
|
||||
|
||||
### `runWorker<T>(queue, handler, options?): Promise<WorkerStats>`
|
||||
|
||||
Convenience factory: creates a worker, runs it, and returns stats.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The drain loop maintains a `Set<Promise<void>>` of in-flight job promises. When `slots.size` reaches `concurrency`, it awaits `Promise.race(slots)` to free up a slot before dequeuing the next job. This achieves true concurrent processing without spawning unbound promises. Dead-letter forwarding errors wrap the original error as `QueueDeadLetterFailedError` so callers can distinguish delivery failures from job failures.
|
||||
Reference in New Issue
Block a user