68 lines
3.4 KiB
Markdown
68 lines
3.4 KiB
Markdown
# `dispatchWithRetry`
|
|
|
|
## Purpose
|
|
|
|
`dispatchWithRetry` is the core retry loop of `WorkerManager`. It picks a worker via the configured `LoadBalancingStrategy`, executes the task, and retries up to `maxTaskRetries` on failure, sleeping `taskRetryDelayMs` between attempts via `@suaveplan/retry`'s `sleep`. The loop is bounded (no recursive self-call) and honours `AbortSignal` mid-flight — interrupting the sleep, surfacing `TaskCancelledError` thrown from the worker, and short-circuiting cleanly. Implementation is factored out of `manager.ts` so each file stays under the 300-LOC budget.
|
|
|
|
## Features
|
|
|
|
- Re-selects a worker on every iteration so a single bad worker can't capture every retry.
|
|
- Emits OTel `queue_dist.manager.assign` span with `task_id` and `worker_id` attributes; histogram + counter per attempt.
|
|
- Bounded loop — no recursion stack growth on long retry counts.
|
|
- `TaskCancelledError` thrown by the worker propagates immediately without further retries.
|
|
- Dead-letter via `config.onFailedTask(task)` callback when retries exhaust.
|
|
- Per-attempt retry count tracked via `config.trackRetryCount(taskId, attempts)`.
|
|
|
|
## Usage: Basic Example
|
|
|
|
`dispatchWithRetry` is internal to `manager.ts`; consumers use `WorkerManager.assignTask`. Direct usage looks like:
|
|
|
|
```ts
|
|
import { dispatchWithRetry } from "@suaveplan/queue-distribution/manager/manager-dispatch";
|
|
|
|
const result = await dispatchWithRetry(
|
|
task,
|
|
input,
|
|
{
|
|
pool,
|
|
balancer,
|
|
maxTaskRetries: 3,
|
|
taskRetryDelayMs: 250,
|
|
clock: Date.now,
|
|
onFailedTask: (t) => deadLetter.add(t),
|
|
trackRetryCount: (id, n) => retries.set(id, n),
|
|
},
|
|
);
|
|
```
|
|
|
|
## Usage: Advanced Example
|
|
|
|
The dispatcher consults `pickWorker` and `runOnce` (helpers in this file) to apply the strategy and run a single attempt with full OTel + balancer-callback emission. To extend behaviour without forking, swap the `LoadBalancingStrategy` or wrap the `onFailedTask` callback.
|
|
|
|
```ts
|
|
import { dispatchWithRetry } from "@suaveplan/queue-distribution/manager/manager-dispatch";
|
|
|
|
await dispatchWithRetry(task, input, {
|
|
pool,
|
|
balancer: {
|
|
selectWorker: (workers) => workers[0] ?? null,
|
|
notifyTaskCompleted: (worker, t, success, durationMs) => metrics.record(success, durationMs),
|
|
getName: () => "first",
|
|
},
|
|
maxTaskRetries: 5,
|
|
taskRetryDelayMs: 500,
|
|
clock: () => Date.now(),
|
|
onFailedTask: (task) => alertingClient.fire(task.id),
|
|
trackRetryCount: () => undefined,
|
|
}, ctrl.signal);
|
|
```
|
|
|
|
## API Reference
|
|
|
|
- `interface DispatchConfig` — `pool`, `balancer`, `maxTaskRetries`, `taskRetryDelayMs`, `clock`, `onFailedTask`, `trackRetryCount`.
|
|
- `function dispatchWithRetry<I, O>(task, input, config, signal?): Promise<O>` — throws `NoAvailableWorkersError`, `StrategyNoWorkerSelectedError`, `TaskCancelledError`, or `TaskRetriesExhaustedError`.
|
|
|
|
## Implementation Notes
|
|
|
|
The sleep between retries is routed through `@suaveplan/retry`'s `sleep(ms, signal)` so aborting the supplied signal cleanly interrupts the wait and rethrows as `TaskCancelledError`. The reason the abort handler uses `.catch(() => throw new TaskCancelledError(...))` rather than checking the rejection type is that `sleep` only ever rejects via `AbortSignal` — so any rejection is by definition a cancellation. This keeps the catch arm single-branch and 100%-coverable, and avoids a dead-code arm for non-existent non-abort rejections.
|