Move docs/ package mirror into document-navigator plugin

Relocate the ~6k-file docs tree from the repo root into
plugins/document-navigator/docs/ so the plugin is self-contained and
portable. docs/superpowers/ (specs and plans) stays at the repo root.

Moved via cp+rm rather than git mv because directory rename() fails with
EPERM on this WSL /mnt DrvFs mount; git rename-detection pairs the
identical-content files at commit time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-30 01:10:00 +01:00
parent fe3fedf42e
commit 9bf56a806b
6022 changed files with 12437 additions and 0 deletions

View File

@@ -0,0 +1,67 @@
# `dispatchWithRetry`
## Purpose
`dispatchWithRetry` is the core retry loop of `WorkerManager`. It picks a worker via the configured `LoadBalancingStrategy`, executes the task, and retries up to `maxTaskRetries` on failure, sleeping `taskRetryDelayMs` between attempts via `@suaveplan/retry`'s `sleep`. The loop is bounded (no recursive self-call) and honours `AbortSignal` mid-flight — interrupting the sleep, surfacing `TaskCancelledError` thrown from the worker, and short-circuiting cleanly. Implementation is factored out of `manager.ts` so each file stays under the 300-LOC budget.
## Features
- Re-selects a worker on every iteration so a single bad worker can't capture every retry.
- Emits OTel `queue_dist.manager.assign` span with `task_id` and `worker_id` attributes; histogram + counter per attempt.
- Bounded loop — no recursion stack growth on long retry counts.
- `TaskCancelledError` thrown by the worker propagates immediately without further retries.
- Dead-letter via `config.onFailedTask(task)` callback when retries exhaust.
- Per-attempt retry count tracked via `config.trackRetryCount(taskId, attempts)`.
## Usage: Basic Example
`dispatchWithRetry` is internal to `manager.ts`; consumers use `WorkerManager.assignTask`. Direct usage looks like:
```ts
import { dispatchWithRetry } from "@suaveplan/queue-distribution/manager/manager-dispatch";
const result = await dispatchWithRetry(
task,
input,
{
pool,
balancer,
maxTaskRetries: 3,
taskRetryDelayMs: 250,
clock: Date.now,
onFailedTask: (t) => deadLetter.add(t),
trackRetryCount: (id, n) => retries.set(id, n),
},
);
```
## Usage: Advanced Example
The dispatcher consults `pickWorker` and `runOnce` (helpers in this file) to apply the strategy and run a single attempt with full OTel + balancer-callback emission. To extend behaviour without forking, swap the `LoadBalancingStrategy` or wrap the `onFailedTask` callback.
```ts
import { dispatchWithRetry } from "@suaveplan/queue-distribution/manager/manager-dispatch";
await dispatchWithRetry(task, input, {
pool,
balancer: {
selectWorker: (workers) => workers[0] ?? null,
notifyTaskCompleted: (worker, t, success, durationMs) => metrics.record(success, durationMs),
getName: () => "first",
},
maxTaskRetries: 5,
taskRetryDelayMs: 500,
clock: () => Date.now(),
onFailedTask: (task) => alertingClient.fire(task.id),
trackRetryCount: () => undefined,
}, ctrl.signal);
```
## API Reference
- `interface DispatchConfig` — `pool`, `balancer`, `maxTaskRetries`, `taskRetryDelayMs`, `clock`, `onFailedTask`, `trackRetryCount`.
- `function dispatchWithRetry<I, O>(task, input, config, signal?): Promise<O>` — throws `NoAvailableWorkersError`, `StrategyNoWorkerSelectedError`, `TaskCancelledError`, or `TaskRetriesExhaustedError`.
## Implementation Notes
The sleep between retries is routed through `@suaveplan/retry`'s `sleep(ms, signal)` so aborting the supplied signal cleanly interrupts the wait and rethrows as `TaskCancelledError`. The reason the abort handler uses `.catch(() => throw new TaskCancelledError(...))` rather than checking the rejection type is that `sleep` only ever rejects via `AbortSignal` — so any rejection is by definition a cancellation. This keeps the catch arm single-branch and 100%-coverable, and avoids a dead-code arm for non-existent non-abort rejections.

View File

@@ -0,0 +1,64 @@
# Manager supervisor helpers
## Purpose
The supervisor module owns the worker-health side of `WorkerManager`: periodic health checks across the pool, restart of `UNHEALTHY` workers up to a configurable limit, and structured failure handling. It is split into pure functions taking a `SupervisorConfig` so each unit is testable in isolation and so `manager.ts` stays under the 300-LOC budget.
## Features
- `checkAllWorkersHealth(config, signal?)` — fan-out over every worker, returns map of id → `WorkerHealth`, logs a warning when online count falls below `minOnlineWorkers`.
- `handleWorkerFailure(worker, error, config)` — probes health after a failure; flips to `UNHEALTHY` and optionally triggers a restart.
- `restartIfNeeded(worker, config)` — bounded restart attempt with stop+start, post-restart health check, and OTel emission.
- `WORKER_RESTART_EXHAUSTED` emitted on the OTel counter when the limit is reached.
- Per-worker restart counter (`config.restartCounts`) lets the orchestrator share the budget across runs.
## Usage: Basic Example
These helpers are internal to `WorkerManager`. A consumer wanting raw access can compose them directly:
```ts
import { checkAllWorkersHealth } from "@suaveplan/queue-distribution/manager/manager-supervisor";
const results = await checkAllWorkersHealth({
pool,
healthChecker,
autoRestartUnhealthy: true,
maxWorkerRestarts: 3,
minOnlineWorkers: 1,
clock: Date.now,
logger: console as never,
restartCounts: new Map(),
});
for (const [id, health] of results) {
console.log(`${id}: ${health.isHealthy ? "OK" : "DOWN"}`);
}
```
## Usage: Advanced Example
Drive the supervisor manually for a control-plane component that wants to coordinate restarts itself rather than letting `WorkerManager` schedule them.
```ts
import {
handleWorkerFailure,
restartIfNeeded,
} from "@suaveplan/queue-distribution/manager/manager-supervisor";
async function escalate(worker, err): Promise<void> {
await handleWorkerFailure(worker, err, supervisorConfig);
if (worker.status === "UNHEALTHY") {
await restartIfNeeded(worker, supervisorConfig);
}
}
```
## API Reference
- `interface SupervisorConfig` — `pool`, `healthChecker`, `autoRestartUnhealthy`, `maxWorkerRestarts`, `minOnlineWorkers`, `clock`, `logger`, `restartCounts`.
- `async function checkAllWorkersHealth(config, signal?): Promise<Map<string, WorkerHealth>>`.
- `async function handleWorkerFailure(worker, error, config): Promise<void>`.
- `async function restartIfNeeded(worker, config): Promise<void>`.
## Implementation Notes
`restartIfNeeded` swallows pre-restart `stop` errors and post-restart health-check failures by logging rather than throwing — the goal is "best-effort recovery", not "block the supervisor loop on every flake". Counter resetting on successful restart is intentional: a transient failure shouldn't permanently consume the worker's restart budget. The `WORKER_RESTART_EXHAUSTED` log is emitted with the structured error message but not thrown, again so a single saturated worker doesn't poison the whole supervisor scan.

View File

@@ -0,0 +1,91 @@
# `WorkerManager`
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`WorkerManager` is the public entry point for distributing tasks across a `WorkerPool`. It owns task assignment (load-balancing, retry, dead-letter), supervision (periodic health checks, auto-restart of unhealthy workers), and option lifecycle (configurable at construction and at runtime via `updateOptions`). Implementation details — the retry-with-backoff loop and the supervisor — live in `manager-dispatch.ts` and `manager-supervisor.ts` so this file stays focused on the surface contract.
## Features
- `assignTask<I, O>(task, input, signal?): Promise<O>` — pick a worker, execute, retry up to `maxTaskRetries`, record dead-letter on exhaustion.
- `assignTasks<I, O>(tasks[], inputs[], signal?)` — fan-out wrapper over `assignTask`.
- Auto health-check loop driven by `setInterval` over `healthCheckIntervalMs`; toggleable at runtime.
- Auto-restart of `UNHEALTHY` workers up to `maxWorkerRestarts`.
- `failedTasks` Set and per-task retry counter for observability.
- Injectable `LoadBalancingStrategy` (default `RoundRobinStrategy`), `WorkerHealthChecker` (default `PeriodicHealthChecker`), `Clock`, `LoggerLike`.
- AbortSignal threaded through every public async method.
- Throws `InvalidOptionsError` for negative numeric options at construction and on `updateOptions`.
## Usage: Basic Example
```ts
import { WorkerManager } from "@suaveplan/queue-distribution/manager/manager";
import { WorkerPool } from "@suaveplan/queue-distribution/pool/pool";
import { LocalWorker } from "@suaveplan/queue-distribution/worker/local/local";
const pool = new WorkerPool();
pool.addWorker(new LocalWorker({}, { id: "a" }));
const manager = new WorkerManager(pool, {
autoHealthCheck: false,
maxTaskRetries: 2,
});
const result = await manager.assignTask(task, input);
```
## Usage: Advanced Example
Stand up a production-shaped manager with a least-busy strategy, performance bias, a custom health checker, and a deterministic clock for tests.
```ts
import { WorkerManager } from "@suaveplan/queue-distribution/manager/manager";
import { WorkerPool } from "@suaveplan/queue-distribution/pool/pool";
import { LeastBusyStrategy } from "@suaveplan/queue-distribution/load-balancing/least-busy/least-busy";
import { PeriodicHealthChecker } from "@suaveplan/queue-distribution/health-checker/health-checker";
const clock = (): number => Date.now();
const pool = new WorkerPool();
// ... add workers ...
const manager = new WorkerManager(
pool,
{
balancer: new LeastBusyStrategy({ usePerformanceMetrics: true }),
healthChecker: new PeriodicHealthChecker({ unhealthyThreshold: 3 }),
autoHealthCheck: true,
healthCheckIntervalMs: 10_000,
maxTaskRetries: 5,
taskRetryDelayMs: 250,
autoRestartUnhealthy: true,
maxWorkerRestarts: 5,
minOnlineWorkers: 2,
},
{ clock, logger: console as never },
);
const ctrl = new AbortController();
setTimeout(() => ctrl.abort(), 30_000); // hard ceiling
await manager.assignTasks(tasks, inputs, ctrl.signal);
```
## API Reference
- `interface WorkerManagerOptions` — `healthCheckIntervalMs?`, `autoHealthCheck?`, `taskRetryDelayMs?`, `maxTaskRetries?`, `autoRestartUnhealthy?`, `maxWorkerRestarts?`, `minOnlineWorkers?`.
- `interface WorkerManagerDeps` — `clock?`, `logger?`.
- `class WorkerManager`.
- `constructor(pool, options?: { balancer?, healthChecker?, ...WorkerManagerOptions }, deps?: WorkerManagerDeps)`.
- `assignTask<I, O>(task, input, signal?): Promise<O>`.
- `assignTasks<I, O>(tasks[], inputs[], signal?): Promise<O[]>`.
- `startHealthChecks(): void`, `stopHealthChecks(): void`.
- `checkAllWorkersHealth(signal?): Promise<Map<string, WorkerHealth>>`.
- `handleWorkerFailure(worker, error): Promise<void>`.
- `getWorkerPool(): WorkerPool`, `getLoadBalancer(): LoadBalancingStrategy`, `setLoadBalancer(b): void`, `getHealthChecker(): WorkerHealthChecker`.
- `getFailedTasks(): ReadonlySet<DistributedTask>`, `clearFailedTasks(): void`.
- `getTaskRetries(taskId): number`.
- `getOptions(): Required<WorkerManagerOptions>`, `updateOptions(partial): void`.
## Implementation Notes
`updateOptions` performs validate-then-apply: a negative `maxTaskRetries` is rejected before any mutation, so the manager remains in a consistent state across failed updates. Toggling `autoHealthCheck` mid-run actually starts or stops the interval — passing the same value as the current state is a no-op, which lets callers idempotently bounce settings without disrupting an active loop. The dispatch and supervisor configs are recreated on each call to `assignTask` / `checkAllWorkersHealth` so changes via `setLoadBalancer` are picked up immediately without rebuilding the manager.