first commit

This commit is contained in:
2026-05-29 08:20:56 +01:00
commit 949299f58d
6368 changed files with 527176 additions and 0 deletions

View File

@@ -0,0 +1,75 @@
# @suaveplan/queue-distribution-http - HTTP transport plugin for @suaveplan/queue-distribution: HttpWorkerCommunicator (fetch-backed, Zod-validated wire envelope) and RemoteWorker (AbstractWorker subclass with @suaveplan/retry-backed connect / reconnect / health loop).
## Summary
HTTP transport plugin for `@suaveplan/queue-distribution`. Ships `HttpWorkerCommunicator` (a fetch-backed `WorkerCommunicator` with a Zod-validated wire envelope, injectable `fetch`, structured timeouts, and AbortSignal threading) and `RemoteWorker` (an `AbstractWorker` subclass driving the communicator with bounded exponential-backoff connect retries via `@suaveplan/retry`).
The plugin lives separately from the core distribution package so browser-only and worker-thread-only consumers don't have to carry HTTP transport code they never use.
<!-- begin:auto-docs -->
## Documentation tree
- [base](./src/communicator/base/base.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [base](./src/communicator/base/base.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [http](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [http-communicator](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [errors](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [queue-distribution-http-error](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [observability](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [instruments](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
<!-- end:auto-docs --> - [base](./src/communicator/base/base.md) — <!-- begin:auto-docs --> _(no submodules)_- [http](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_- [http-communicator](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-distribution-http-error](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [observability](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_- [instruments](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_- [base](./src/communicator/base/base.md) — <!-- begin:auto-docs --> _(no submodules)_- [http](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_- [http-communicator](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-distribution-http-error](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [observability](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_- [instruments](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_- [base](./src/communicator/base/base.md) — <!-- begin:auto-docs --> _(no submodules)_- [http](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_- [http-communicator](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-distribution-http-error](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [observability](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_- [instruments](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_- [base](./src/communicator/base/base.md) — <!-- begin:auto-docs --> _(no submodules)_- [http](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_ - [http-communicator](./src/communicator/http/http-communicator.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_ - [queue-distribution-http-error](./src/errors/queue-distribution-http-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [observability](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_ - [instruments](./src/observability/instruments.md) — <!-- begin:auto-docs --> _(no submodules)_- [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_ - [remote](./src/worker/remote/remote.md) — <!-- begin:auto-docs --> _(no submodules)_
## Install
```bash
bun add @suaveplan/queue-distribution-http
```
## Quick start
```ts
import { WorkerPool, WorkerManager } from "@suaveplan/queue-distribution";
import {
HttpWorkerCommunicator,
RemoteWorker,
} from "@suaveplan/queue-distribution-http";
const communicator = new HttpWorkerCommunicator({
defaultTimeoutMs: 5_000,
headers: { Authorization: `Bearer ${token}` },
});
const pool = new WorkerPool();
pool.addWorker(new RemoteWorker("https://worker-a.example", communicator, { id: "a" }));
pool.addWorker(new RemoteWorker("https://worker-b.example", communicator, { id: "b" }));
await pool.startAllWorkers();
const manager = new WorkerManager(pool, {
autoHealthCheck: false,
maxTaskRetries: 3,
});
const result = await manager.assignTask(task, input);
```
## Modules
- [Errors](./src/errors/queue-distribution-http-error.md) — `QueueDistributionHttpError` + 6 stable `HTTP.*` codes.
- [WorkerCommunicator base](./src/communicator/base/base.md) — interface + `AbstractWorkerCommunicator`.
- [HttpWorkerCommunicator](./src/communicator/http/http-communicator.md) — fetch-backed transport.
- [RemoteWorker](./src/worker/remote/remote.md) — connect / reconnect / health loop.
- [Observability](./src/observability/instruments.md) — OTel counters, histograms, tracers.
## Design highlights
- 100% test coverage (statements + branches + functions + lines).
- Injectable `fetch` and `Clock` — deterministic under test, swappable for auth-wrapped fetch in production.
- Connect retries routed through `@suaveplan/retry`'s `sleep`; bounded loop, no recursion.
- Zod-validated wire envelope rejects malformed payloads at the boundary.
- Structured `QueueDistributionHttpError` subclasses with a frozen stable-code registry.
## Status
Part of the rebuild — version `0.1.0` while it stabilises.

View File

@@ -0,0 +1,80 @@
# `WorkerCommunicator` + `AbstractWorkerCommunicator`
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`WorkerCommunicator` is the structural contract every transport implementation in `@suaveplan/queue-distribution-http` (and future RPC transports outside this package) implements. `AbstractWorkerCommunicator` carries the shared bookkeeping — default-timeout handling, header carriage, JSON serialise/deserialise helpers, and the `withTimeout` race used by every send/check call — so concrete subclasses focus on wire-level details. Stateless transports get connect/disconnect no-op defaults.
## Features
- `sendTask<I, O>(endpoint, task, input, signal?)` — preserves `<I, O>` end-to-end.
- `checkStatus`, `checkHealth`, `ping`, `connect`, `disconnect` — full transport surface.
- `getProtocol(): string` — short identifier for logs and capabilities.
- `withTimeout(operation, promise, timeoutMs?)` — Promise.race against an injectable timeout that rejects with `HTTP.TIMEOUT`.
- Configurable `defaultTimeoutMs` and per-request `headers`.
- Injectable `Clock` for tests.
## Usage: Basic Example
```ts
import {
AbstractWorkerCommunicator,
type WorkerCommunicator,
} from "@suaveplan/queue-distribution-http/communicator/base/base";
import type {
DistributedTask,
WorkerHealth,
WorkerStatusValue,
} from "@suaveplan/queue-distribution";
class EchoCommunicator extends AbstractWorkerCommunicator {
async sendTask<I, O>(_e, _t, input: I): Promise<O> {
return input as unknown as O;
}
async checkStatus(): Promise<WorkerStatusValue> {
return "ONLINE";
}
async checkHealth(): Promise<WorkerHealth> {
return {
isHealthy: true,
metrics: { cpu: 0, memory: 0, tasksFailed: 0, tasksSucceeded: 0, responseTime: 0 },
};
}
async ping(): Promise<number> {
return 0;
}
getProtocol(): string {
return "echo";
}
}
```
## Usage: Advanced Example
Subclasses can ride on `withTimeout` to enforce a per-call deadline without duplicating timer plumbing.
```ts
import { AbstractWorkerCommunicator } from "@suaveplan/queue-distribution-http/communicator/base/base";
class SlowCommunicator extends AbstractWorkerCommunicator {
async sendTask(): Promise<unknown> {
return this.withTimeout("sendTask", longRunningRpc(), 2_000);
}
// … rest of interface …
}
```
## API Reference
- `interface WorkerCommunicator` — public surface.
- `abstract class AbstractWorkerCommunicator implements WorkerCommunicator`.
- `constructor(options?: { defaultTimeoutMs?, headers? }, deps?: { clock? })`.
- `protected async withTimeout<T>(operation, promise, timeoutMs?): Promise<T>`.
- `protected serialize(value): string`, `protected deserialize<T>(s): T`.
- `connect`, `disconnect` — overridable no-ops.
## Implementation Notes
`withTimeout` attaches a stateful `then` handler to the source promise so we can `clearTimeout` from the resolution path, and races against a deferred `timeoutSignal`. This avoids the unhandled-rejection trap of using a freestanding `setTimeout(..., reject)` Promise alongside `Promise.race`: the source promise's settlement always clears the timer before the rejection can fire late. Non-Error rejections from `promise` are wrapped in `new Error(String(err))` so the error chain stays well-typed for downstream `SuaveplanError` consumers.

View File

@@ -0,0 +1,65 @@
# `HttpWorkerCommunicator`
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`HttpWorkerCommunicator` is the `WorkerCommunicator` implementation for talking to a remote worker over HTTP(S). It uses an injectable `fetch` (defaulting to `globalThis.fetch`) so tests can drive deterministic request/response sequences without an actual server. Every response is validated against a Zod envelope (`{ success: true, data }` | `{ success: false, error }`) so malformed payloads are rejected at the boundary rather than smuggled in as `unknown` data.
## Features
- Endpoints used: `POST /execute`, `GET /status`, `GET /health`, `GET /ping`.
- Zod-validated wire envelope (`workerApiResponseSchema(dataSchema)`).
- Structured error mapping: `RESPONSE_MALFORMED` for envelope mismatch, `PROTOCOL_ERROR` for upstream-reported failures and non-2xx with a successful envelope, `REQUEST_FAILED` for network rejections including `AbortError`.
- Configurable headers and `defaultTimeoutMs`; injectable `fetch` and `Clock`.
- OTel histogram + counter per request, dimensioned by `outcome`.
- AbortSignal threaded into `fetch.init.signal`.
- `getProtocol()` returns `"http"`.
## Usage: Basic Example
```ts
import { HttpWorkerCommunicator } from "@suaveplan/queue-distribution-http/communicator/http/http-communicator";
const communicator = new HttpWorkerCommunicator({
defaultTimeoutMs: 5_000,
headers: { "X-Tenant": "acme" },
});
const result = await communicator.sendTask("https://worker.example", task, input);
```
## Usage: Advanced Example
Inject a deterministic `fetch` for tests, or wrap it with a custom auth + retry layer in production.
```ts
import { HttpWorkerCommunicator } from "@suaveplan/queue-distribution-http/communicator/http/http-communicator";
const communicator = new HttpWorkerCommunicator(
{ defaultTimeoutMs: 10_000, headers: { Authorization: `Bearer ${token}` } },
{
fetch: async (url, init) => {
const response = await globalThis.fetch(url, {
...init,
credentials: "include",
});
return response;
},
},
);
```
## API Reference
- `class HttpWorkerCommunicator extends AbstractWorkerCommunicator`.
- `constructor(options?: HttpWorkerCommunicatorOptions, deps?: HttpWorkerCommunicatorDeps)`.
- `interface HttpWorkerCommunicatorOptions` — `defaultTimeoutMs?`, `headers?`.
- `interface HttpWorkerCommunicatorDeps` — `clock?`, `fetch?`.
- `type FetchLike` — minimal `fetch` shape (`ok`, `status`, `statusText`, `json()`).
- `function workerApiResponseSchema<T>(dataSchema)` — produces the Zod envelope for any payload shape.
## Implementation Notes
The wire envelope is a discriminated union (`success: true` carries `data`, `success: false` carries `error`). The communicator treats non-2xx HTTP status as a protocol error even when the envelope says `success: true` — that protects against misbehaving servers returning 500 with a "this somehow worked" payload. Headers are spread at request time so per-call overrides are not yet supported; if needed, subclass and override `#makeRequest`.

View File

@@ -0,0 +1,80 @@
# `QueueDistributionHttpError` registry
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`QueueDistributionHttpError` is the base class for every structured error thrown by `@suaveplan/queue-distribution-http`. Each subclass carries a stable code from the registry produced by `defineCodes("HTTP", [...])`. Consumers programmatically match on `error.code` to distinguish network failures from upstream protocol errors from timeouts — the registry is exhaustive and every code is thrown by at least one production code path.
## Features
- 6 stable codes — `REQUEST_FAILED`, `PROTOCOL_ERROR`, `TIMEOUT`, `RESPONSE_MALFORMED`, `CONNECT_FAILED`, `UNREACHABLE`.
- One concrete subclass per code with a typed constructor signature.
- Optional `cause` narrowed to `Error` instances; non-Error causes drop silently.
- Compatible with `SuaveplanError` cause-chain traversal.
## Usage: Basic Example
```ts
import {
HTTP_CODES,
QueueDistributionHttpError,
} from "@suaveplan/queue-distribution-http/errors/queue-distribution-http-error";
try {
await communicator.sendTask("http://w", task, input);
} catch (error: unknown) {
if (
error instanceof QueueDistributionHttpError &&
error.code === HTTP_CODES.PROTOCOL_ERROR
) {
// Upstream worker reported a structured failure — surface it.
}
throw error;
}
```
## Usage: Advanced Example
Combine with `@suaveplan/error` cause-chain helpers to extract underlying network failures.
```ts
import {
HTTP_CODES,
HttpRequestFailedError,
QueueDistributionHttpError,
} from "@suaveplan/queue-distribution-http/errors/queue-distribution-http-error";
function classify(error: unknown): "network" | "protocol" | "unknown" {
if (!(error instanceof QueueDistributionHttpError)) {
return "unknown";
}
if (error.code === HTTP_CODES.REQUEST_FAILED) {
return "network";
}
if (
error.code === HTTP_CODES.PROTOCOL_ERROR ||
error.code === HTTP_CODES.RESPONSE_MALFORMED
) {
return "protocol";
}
return "unknown";
}
```
## API Reference
- `const HTTP_CODES` — frozen registry of `HTTP.*` codes.
- `type QueueDistributionHttpErrorCode` — union of every code.
- `class QueueDistributionHttpError extends SuaveplanError<QueueDistributionHttpErrorCode>`.
- `class HttpRequestFailedError({ url, cause? })`.
- `class HttpProtocolError({ url, status, upstreamCode, upstreamMessage, cause? })`.
- `class HttpTimeoutError({ operation, timeoutMs })`.
- `class HttpResponseMalformedError({ url, cause? })`.
- `class HttpConnectFailedError({ endpoint, attempts, cause? })`.
- `class HttpUnreachableError({ endpoint, cause? })`.
## Implementation Notes
`HttpProtocolError` carries the upstream `code` and `message` in its formatted message so logs and dashboards can group failures by the remote worker's own error vocabulary without parsing JSON. `HttpUnreachableError` exists as a documented "this is the failure-class" signal — the error is allocated and discarded inside `RemoteWorker.#handleCommunicationError` because the worker's reaction (markUnhealthy + reconnect) is the actual recovery; the error class is here so future call sites (e.g. observability dashboards) can match on the registered code.

View File

@@ -0,0 +1,59 @@
# HTTP plugin OTel instruments
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`observability/instruments.ts` centralises every OpenTelemetry counter, histogram, and tracer emitted by `@suaveplan/queue-distribution-http`. Instruments are registered under the meter / tracer name `"@suaveplan/queue-distribution-http"` so consumers configuring an SDK can route them with a single rule. Tracer / meter handles are cached behind `resetInstrumentCache()` so tests can swap providers between assertions.
## Features
- Counters: `queue_dist_http.request_total{outcome}`, `queue_dist_http.connect.attempts_total{outcome}`.
- Histogram: `queue_dist_http.request.duration_ms{outcome}`.
- Tracers: `queue_dist_http.sendTask`, `queue_dist_http.connect`.
- `resetInstrumentCache()` for tests that install a new provider mid-suite.
## Usage: Basic Example
Production code calls the instruments automatically. Consumers add a meter / tracer provider at startup:
```ts
import { MeterProvider } from "@opentelemetry/sdk-metrics";
import { metrics } from "@opentelemetry/api";
metrics.setGlobalMeterProvider(new MeterProvider({/* …readers… */}));
```
## Usage: Advanced Example
In tests, install an in-memory exporter, drive the system, then assert directly on the captured metrics.
```ts
import {
AggregationTemporality,
InMemoryMetricExporter,
MeterProvider,
PeriodicExportingMetricReader,
} from "@opentelemetry/sdk-metrics";
import { metrics } from "@opentelemetry/api";
import { resetInstrumentCache } from "@suaveplan/queue-distribution-http/observability/instruments";
const exporter = new InMemoryMetricExporter(AggregationTemporality.DELTA);
const reader = new PeriodicExportingMetricReader({ exporter, exportIntervalMillis: 100 });
metrics.setGlobalMeterProvider(new MeterProvider({ readers: [reader] }));
resetInstrumentCache();
// … run system under test, then await reader.forceFlush() …
```
## API Reference
- `function resetInstrumentCache(): void`.
- `function recordHttpRequestOutcome(outcome: "success" | "failure"): void`.
- `function recordHttpRequestDuration(durationMs, outcome): void`.
- `function recordConnectAttempt(outcome: "success" | "failure" | "exhausted"): void`.
- `function getSendTaskTracer(): Tracer`, `function getConnectTracer(): Tracer`.
## Implementation Notes
The instruments share a single meter handle so the per-call overhead in the hot path is a single map lookup. `resetInstrumentCache()` is wired exclusively for tests; production code never calls it because the underlying global provider is set once at startup. The `outcome` attribute is constrained to the three documented values for connect attempts and two for requests — keeping the cardinality bounded for backend dashboards.

View File

@@ -0,0 +1,82 @@
# `RemoteWorker`
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`RemoteWorker` extends `AbstractWorker` and delegates execution to a remote endpoint via an injected `WorkerCommunicator`. It manages the connect / reconnect lifecycle (with `@suaveplan/retry`-style exponential backoff routed through `sleep`), runs a periodic health-check loop, and reacts to communication failures by pinging and either reporting the worker still-reachable or marking it `UNHEALTHY`. The connect loop is bounded — no recursion — and every retry attempt is captured in the OTel `queue_dist_http.connect.attempts_total` counter.
## Features
- `start()` calls `#connect` (exponential backoff between attempts) then starts a `setInterval` health-check loop.
- `stop(force, signal?)` halts the loop, disconnects the communicator, and flips `#isConnected` to `false`.
- `doExecuteTask` refuses when not connected (`HTTP.CONNECT_FAILED`).
- Communication-error recovery: ping with a 5-second timeout; reachable → log + propagate; unreachable → `markUnhealthy("Unreachable")` + `#isConnected = false`.
- `getHealth()` proxies to the remote and refreshes `lastHealthCheck` / `lastHealthStatus`; on failure returns a fallback unhealthy payload.
- Configurable `healthCheckIntervalMs`, `maxConnectionAttempts`, `initialBackoffMs`, `maxBackoffMs`.
- `getCapabilities()` advertises `{ isRemote: true, protocol, endpoint }` so strategies can route work.
## Usage: Basic Example
```ts
import { HttpWorkerCommunicator } from "@suaveplan/queue-distribution-http";
import { RemoteWorker } from "@suaveplan/queue-distribution-http/worker/remote/remote";
const communicator = new HttpWorkerCommunicator();
const worker = new RemoteWorker("https://worker.example", communicator, {
healthCheckIntervalMs: 15_000,
maxConnectionAttempts: 5,
});
await worker.start();
const result = await worker.executeTask(task, input);
await worker.stop();
```
## Usage: Advanced Example
For deterministic property-tests, inject a custom communicator and watch connect retries with a fake clock.
```ts
import { RemoteWorker } from "@suaveplan/queue-distribution-http/worker/remote/remote";
import type { WorkerCommunicator } from "@suaveplan/queue-distribution-http";
let connectAttempts = 0;
const flakyCommunicator: WorkerCommunicator = {
sendTask: async (_e, _t, input) => input as never,
connect: async () => {
connectAttempts += 1;
if (connectAttempts < 3) throw new Error("refused");
},
checkStatus: async () => "ONLINE",
checkHealth: async () => ({
isHealthy: true,
metrics: { cpu: 0, memory: 0, tasksFailed: 0, tasksSucceeded: 0, responseTime: 0 },
}),
disconnect: async () => undefined,
ping: async () => 5,
getProtocol: () => "http",
};
const worker = new RemoteWorker("https://worker", flakyCommunicator, {
maxConnectionAttempts: 5,
initialBackoffMs: 0,
healthCheckIntervalMs: 60_000,
});
await worker.start();
console.log(connectAttempts); // 3
```
## API Reference
- `class RemoteWorker extends AbstractWorker`.
- `constructor(endpoint, communicator, options?: RemoteWorkerOptions, deps?: AbstractWorkerDeps)`.
- `interface RemoteWorkerOptions extends AbstractWorkerOptions` — `healthCheckIntervalMs?`, `maxConnectionAttempts?`, `initialBackoffMs?`, `maxBackoffMs?`.
- `getEndpoint(): string`, `getCommunicator(): WorkerCommunicator`, `isConnected(): boolean`.
- `getHealth(signal?): Promise<WorkerHealth>` — overrides the abstract default.
- Protected: `doExecuteTask`, `doStart`, `doStop`, `getType()`, `getCapabilities()`.
## Implementation Notes
The connect loop iterates up to `maxConnectionAttempts` times, doubling the delay each iteration (capped at `maxBackoffMs`). On every retry, the loop also re-checks the remote status — so a worker that responds OK on the third attempt connects without further delay. Communication errors during sendTask are NOT silently swallowed: the worker rethrows so the calling `WorkerManager` can retry against a different worker via its load-balancing strategy. The ping-based reachability probe runs in parallel with the rethrow so the worker's `status` is up-to-date by the time the next strategy invocation observes the pool.