`dispatchWithRetry`

Purpose

dispatchWithRetry is the core retry loop of WorkerManager. It picks a worker via the configured LoadBalancingStrategy, executes the task, and retries up to maxTaskRetries on failure, sleeping taskRetryDelayMs between attempts via @suaveplan/retry's sleep. The loop is bounded (no recursive self-call) and honours AbortSignal mid-flight — interrupting the sleep, surfacing TaskCancelledError thrown from the worker, and short-circuiting cleanly. Implementation is factored out of manager.ts so each file stays under the 300-LOC budget.

Features

Re-selects a worker on every iteration so a single bad worker can't capture every retry.
Emits OTel queue_dist.manager.assign span with task_id and worker_id attributes; histogram + counter per attempt.
Bounded loop — no recursion stack growth on long retry counts.
TaskCancelledError thrown by the worker propagates immediately without further retries.
Dead-letter via config.onFailedTask(task) callback when retries exhaust.
Per-attempt retry count tracked via config.trackRetryCount(taskId, attempts).

Usage: Basic Example

dispatchWithRetry is internal to manager.ts; consumers use WorkerManager.assignTask. Direct usage looks like:

import { dispatchWithRetry } from "@suaveplan/queue-distribution/manager/manager-dispatch";

const result = await dispatchWithRetry(
    task,
    input,
    {
        pool,
        balancer,
        maxTaskRetries: 3,
        taskRetryDelayMs: 250,
        clock: Date.now,
        onFailedTask: (t) => deadLetter.add(t),
        trackRetryCount: (id, n) => retries.set(id, n),
    },
);

Usage: Advanced Example

The dispatcher consults pickWorker and runOnce (helpers in this file) to apply the strategy and run a single attempt with full OTel + balancer-callback emission. To extend behaviour without forking, swap the LoadBalancingStrategy or wrap the onFailedTask callback.

import { dispatchWithRetry } from "@suaveplan/queue-distribution/manager/manager-dispatch";

await dispatchWithRetry(task, input, {
    pool,
    balancer: {
        selectWorker: (workers) => workers[0] ?? null,
        notifyTaskCompleted: (worker, t, success, durationMs) => metrics.record(success, durationMs),
        getName: () => "first",
    },
    maxTaskRetries: 5,
    taskRetryDelayMs: 500,
    clock: () => Date.now(),
    onFailedTask: (task) => alertingClient.fire(task.id),
    trackRetryCount: () => undefined,
}, ctrl.signal);

API Reference

interface DispatchConfig — pool, balancer, maxTaskRetries, taskRetryDelayMs, clock, onFailedTask, trackRetryCount.
function dispatchWithRetry<I, O>(task, input, config, signal?): Promise<O> — throws NoAvailableWorkersError, StrategyNoWorkerSelectedError, TaskCancelledError, or TaskRetriesExhaustedError.

Implementation Notes

The sleep between retries is routed through @suaveplan/retry's sleep(ms, signal) so aborting the supplied signal cleanly interrupts the wait and rethrows as TaskCancelledError. The reason the abort handler uses .catch(() => throw new TaskCancelledError(...)) rather than checking the rejection type is that sleep only ever rejects via AbortSignal — so any rejection is by definition a cancellation. This keeps the catch arm single-branch and 100%-coverable, and avoids a dead-code arm for non-existent non-abort rejections.

3.4 KiB Raw Blame History

dispatchWithRetry