Move docs/ package mirror into document-navigator plugin
Relocate the ~6k-file docs tree from the repo root into plugins/document-navigator/docs/ so the plugin is self-contained and portable. docs/superpowers/ (specs and plans) stays at the repo root. Moved via cp+rm rather than git mv because directory rename() fails with EPERM on this WSL /mnt DrvFs mount; git rename-detection pairs the identical-content files at commit time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,67 @@
|
||||
# `dispatchWithRetry`
|
||||
|
||||
## Purpose
|
||||
|
||||
`dispatchWithRetry` is the core retry loop of `WorkerManager`. It picks a worker via the configured `LoadBalancingStrategy`, executes the task, and retries up to `maxTaskRetries` on failure, sleeping `taskRetryDelayMs` between attempts via `@suaveplan/retry`'s `sleep`. The loop is bounded (no recursive self-call) and honours `AbortSignal` mid-flight — interrupting the sleep, surfacing `TaskCancelledError` thrown from the worker, and short-circuiting cleanly. Implementation is factored out of `manager.ts` so each file stays under the 300-LOC budget.
|
||||
|
||||
## Features
|
||||
|
||||
- Re-selects a worker on every iteration so a single bad worker can't capture every retry.
|
||||
- Emits OTel `queue_dist.manager.assign` span with `task_id` and `worker_id` attributes; histogram + counter per attempt.
|
||||
- Bounded loop — no recursion stack growth on long retry counts.
|
||||
- `TaskCancelledError` thrown by the worker propagates immediately without further retries.
|
||||
- Dead-letter via `config.onFailedTask(task)` callback when retries exhaust.
|
||||
- Per-attempt retry count tracked via `config.trackRetryCount(taskId, attempts)`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
`dispatchWithRetry` is internal to `manager.ts`; consumers use `WorkerManager.assignTask`. Direct usage looks like:
|
||||
|
||||
```ts
|
||||
import { dispatchWithRetry } from "@suaveplan/queue-distribution/manager/manager-dispatch";
|
||||
|
||||
const result = await dispatchWithRetry(
|
||||
task,
|
||||
input,
|
||||
{
|
||||
pool,
|
||||
balancer,
|
||||
maxTaskRetries: 3,
|
||||
taskRetryDelayMs: 250,
|
||||
clock: Date.now,
|
||||
onFailedTask: (t) => deadLetter.add(t),
|
||||
trackRetryCount: (id, n) => retries.set(id, n),
|
||||
},
|
||||
);
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
The dispatcher consults `pickWorker` and `runOnce` (helpers in this file) to apply the strategy and run a single attempt with full OTel + balancer-callback emission. To extend behaviour without forking, swap the `LoadBalancingStrategy` or wrap the `onFailedTask` callback.
|
||||
|
||||
```ts
|
||||
import { dispatchWithRetry } from "@suaveplan/queue-distribution/manager/manager-dispatch";
|
||||
|
||||
await dispatchWithRetry(task, input, {
|
||||
pool,
|
||||
balancer: {
|
||||
selectWorker: (workers) => workers[0] ?? null,
|
||||
notifyTaskCompleted: (worker, t, success, durationMs) => metrics.record(success, durationMs),
|
||||
getName: () => "first",
|
||||
},
|
||||
maxTaskRetries: 5,
|
||||
taskRetryDelayMs: 500,
|
||||
clock: () => Date.now(),
|
||||
onFailedTask: (task) => alertingClient.fire(task.id),
|
||||
trackRetryCount: () => undefined,
|
||||
}, ctrl.signal);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `interface DispatchConfig` — `pool`, `balancer`, `maxTaskRetries`, `taskRetryDelayMs`, `clock`, `onFailedTask`, `trackRetryCount`.
|
||||
- `function dispatchWithRetry<I, O>(task, input, config, signal?): Promise<O>` — throws `NoAvailableWorkersError`, `StrategyNoWorkerSelectedError`, `TaskCancelledError`, or `TaskRetriesExhaustedError`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The sleep between retries is routed through `@suaveplan/retry`'s `sleep(ms, signal)` so aborting the supplied signal cleanly interrupts the wait and rethrows as `TaskCancelledError`. The reason the abort handler uses `.catch(() => throw new TaskCancelledError(...))` rather than checking the rejection type is that `sleep` only ever rejects via `AbortSignal` — so any rejection is by definition a cancellation. This keeps the catch arm single-branch and 100%-coverable, and avoids a dead-code arm for non-existent non-abort rejections.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Manager supervisor helpers
|
||||
|
||||
## Purpose
|
||||
|
||||
The supervisor module owns the worker-health side of `WorkerManager`: periodic health checks across the pool, restart of `UNHEALTHY` workers up to a configurable limit, and structured failure handling. It is split into pure functions taking a `SupervisorConfig` so each unit is testable in isolation and so `manager.ts` stays under the 300-LOC budget.
|
||||
|
||||
## Features
|
||||
|
||||
- `checkAllWorkersHealth(config, signal?)` — fan-out over every worker, returns map of id → `WorkerHealth`, logs a warning when online count falls below `minOnlineWorkers`.
|
||||
- `handleWorkerFailure(worker, error, config)` — probes health after a failure; flips to `UNHEALTHY` and optionally triggers a restart.
|
||||
- `restartIfNeeded(worker, config)` — bounded restart attempt with stop+start, post-restart health check, and OTel emission.
|
||||
- `WORKER_RESTART_EXHAUSTED` emitted on the OTel counter when the limit is reached.
|
||||
- Per-worker restart counter (`config.restartCounts`) lets the orchestrator share the budget across runs.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
These helpers are internal to `WorkerManager`. A consumer wanting raw access can compose them directly:
|
||||
|
||||
```ts
|
||||
import { checkAllWorkersHealth } from "@suaveplan/queue-distribution/manager/manager-supervisor";
|
||||
|
||||
const results = await checkAllWorkersHealth({
|
||||
pool,
|
||||
healthChecker,
|
||||
autoRestartUnhealthy: true,
|
||||
maxWorkerRestarts: 3,
|
||||
minOnlineWorkers: 1,
|
||||
clock: Date.now,
|
||||
logger: console as never,
|
||||
restartCounts: new Map(),
|
||||
});
|
||||
for (const [id, health] of results) {
|
||||
console.log(`${id}: ${health.isHealthy ? "OK" : "DOWN"}`);
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Drive the supervisor manually for a control-plane component that wants to coordinate restarts itself rather than letting `WorkerManager` schedule them.
|
||||
|
||||
```ts
|
||||
import {
|
||||
handleWorkerFailure,
|
||||
restartIfNeeded,
|
||||
} from "@suaveplan/queue-distribution/manager/manager-supervisor";
|
||||
|
||||
async function escalate(worker, err): Promise<void> {
|
||||
await handleWorkerFailure(worker, err, supervisorConfig);
|
||||
if (worker.status === "UNHEALTHY") {
|
||||
await restartIfNeeded(worker, supervisorConfig);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `interface SupervisorConfig` — `pool`, `healthChecker`, `autoRestartUnhealthy`, `maxWorkerRestarts`, `minOnlineWorkers`, `clock`, `logger`, `restartCounts`.
|
||||
- `async function checkAllWorkersHealth(config, signal?): Promise<Map<string, WorkerHealth>>`.
|
||||
- `async function handleWorkerFailure(worker, error, config): Promise<void>`.
|
||||
- `async function restartIfNeeded(worker, config): Promise<void>`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`restartIfNeeded` swallows pre-restart `stop` errors and post-restart health-check failures by logging rather than throwing — the goal is "best-effort recovery", not "block the supervisor loop on every flake". Counter resetting on successful restart is intentional: a transient failure shouldn't permanently consume the worker's restart budget. The `WORKER_RESTART_EXHAUSTED` log is emitted with the structured error message but not thrown, again so a single saturated worker doesn't poison the whole supervisor scan.
|
||||
@@ -0,0 +1,91 @@
|
||||
# `WorkerManager`
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
`WorkerManager` is the public entry point for distributing tasks across a `WorkerPool`. It owns task assignment (load-balancing, retry, dead-letter), supervision (periodic health checks, auto-restart of unhealthy workers), and option lifecycle (configurable at construction and at runtime via `updateOptions`). Implementation details — the retry-with-backoff loop and the supervisor — live in `manager-dispatch.ts` and `manager-supervisor.ts` so this file stays focused on the surface contract.
|
||||
|
||||
## Features
|
||||
|
||||
- `assignTask<I, O>(task, input, signal?): Promise<O>` — pick a worker, execute, retry up to `maxTaskRetries`, record dead-letter on exhaustion.
|
||||
- `assignTasks<I, O>(tasks[], inputs[], signal?)` — fan-out wrapper over `assignTask`.
|
||||
- Auto health-check loop driven by `setInterval` over `healthCheckIntervalMs`; toggleable at runtime.
|
||||
- Auto-restart of `UNHEALTHY` workers up to `maxWorkerRestarts`.
|
||||
- `failedTasks` Set and per-task retry counter for observability.
|
||||
- Injectable `LoadBalancingStrategy` (default `RoundRobinStrategy`), `WorkerHealthChecker` (default `PeriodicHealthChecker`), `Clock`, `LoggerLike`.
|
||||
- AbortSignal threaded through every public async method.
|
||||
- Throws `InvalidOptionsError` for negative numeric options at construction and on `updateOptions`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { WorkerManager } from "@suaveplan/queue-distribution/manager/manager";
|
||||
import { WorkerPool } from "@suaveplan/queue-distribution/pool/pool";
|
||||
import { LocalWorker } from "@suaveplan/queue-distribution/worker/local/local";
|
||||
|
||||
const pool = new WorkerPool();
|
||||
pool.addWorker(new LocalWorker({}, { id: "a" }));
|
||||
const manager = new WorkerManager(pool, {
|
||||
autoHealthCheck: false,
|
||||
maxTaskRetries: 2,
|
||||
});
|
||||
|
||||
const result = await manager.assignTask(task, input);
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Stand up a production-shaped manager with a least-busy strategy, performance bias, a custom health checker, and a deterministic clock for tests.
|
||||
|
||||
```ts
|
||||
import { WorkerManager } from "@suaveplan/queue-distribution/manager/manager";
|
||||
import { WorkerPool } from "@suaveplan/queue-distribution/pool/pool";
|
||||
import { LeastBusyStrategy } from "@suaveplan/queue-distribution/load-balancing/least-busy/least-busy";
|
||||
import { PeriodicHealthChecker } from "@suaveplan/queue-distribution/health-checker/health-checker";
|
||||
|
||||
const clock = (): number => Date.now();
|
||||
const pool = new WorkerPool();
|
||||
// ... add workers ...
|
||||
|
||||
const manager = new WorkerManager(
|
||||
pool,
|
||||
{
|
||||
balancer: new LeastBusyStrategy({ usePerformanceMetrics: true }),
|
||||
healthChecker: new PeriodicHealthChecker({ unhealthyThreshold: 3 }),
|
||||
autoHealthCheck: true,
|
||||
healthCheckIntervalMs: 10_000,
|
||||
maxTaskRetries: 5,
|
||||
taskRetryDelayMs: 250,
|
||||
autoRestartUnhealthy: true,
|
||||
maxWorkerRestarts: 5,
|
||||
minOnlineWorkers: 2,
|
||||
},
|
||||
{ clock, logger: console as never },
|
||||
);
|
||||
|
||||
const ctrl = new AbortController();
|
||||
setTimeout(() => ctrl.abort(), 30_000); // hard ceiling
|
||||
await manager.assignTasks(tasks, inputs, ctrl.signal);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `interface WorkerManagerOptions` — `healthCheckIntervalMs?`, `autoHealthCheck?`, `taskRetryDelayMs?`, `maxTaskRetries?`, `autoRestartUnhealthy?`, `maxWorkerRestarts?`, `minOnlineWorkers?`.
|
||||
- `interface WorkerManagerDeps` — `clock?`, `logger?`.
|
||||
- `class WorkerManager`.
|
||||
- `constructor(pool, options?: { balancer?, healthChecker?, ...WorkerManagerOptions }, deps?: WorkerManagerDeps)`.
|
||||
- `assignTask<I, O>(task, input, signal?): Promise<O>`.
|
||||
- `assignTasks<I, O>(tasks[], inputs[], signal?): Promise<O[]>`.
|
||||
- `startHealthChecks(): void`, `stopHealthChecks(): void`.
|
||||
- `checkAllWorkersHealth(signal?): Promise<Map<string, WorkerHealth>>`.
|
||||
- `handleWorkerFailure(worker, error): Promise<void>`.
|
||||
- `getWorkerPool(): WorkerPool`, `getLoadBalancer(): LoadBalancingStrategy`, `setLoadBalancer(b): void`, `getHealthChecker(): WorkerHealthChecker`.
|
||||
- `getFailedTasks(): ReadonlySet<DistributedTask>`, `clearFailedTasks(): void`.
|
||||
- `getTaskRetries(taskId): number`.
|
||||
- `getOptions(): Required<WorkerManagerOptions>`, `updateOptions(partial): void`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`updateOptions` performs validate-then-apply: a negative `maxTaskRetries` is rejected before any mutation, so the manager remains in a consistent state across failed updates. Toggling `autoHealthCheck` mid-run actually starts or stops the interval — passing the same value as the current state is a no-op, which lets callers idempotently bounce settings without disrupting an active loop. The dispatch and supervisor configs are recreated on each call to `assignTask` / `checkAllWorkersHealth` so changes via `setLoadBalancer` are picked up immediately without rebuilding the manager.
|
||||
Reference in New Issue
Block a user