Initial Commit
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: queue-distribution-http (enterprise-queue-distribution)
|
||||
|
||||
**Package**: `@suaveplan/queue-distribution-http`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. (See `openspec/AGENTS.md` Section 13.)
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Worker communicator interface and abstract base
|
||||
|
||||
The package SHALL expose `WorkerCommunicator` with `sendTask<I, O>(endpoint, task, input, signal?): Promise<O>`, `checkStatus(endpoint, signal?): Promise<WorkerStatus>`, `checkHealth(endpoint, signal?): Promise<WorkerHealth>`, `connect(endpoint, signal?): Promise<void>`, `disconnect(endpoint, signal?): Promise<void>`, `ping(endpoint, timeoutMs?, signal?): Promise<number>`, `getProtocol(): string`. `AbstractWorkerCommunicator` SHALL provide constructor-injected `defaultTimeoutMs` and `headers`, plus protected `serialize`/`deserialize`/`createTimeout`/`withTimeout` helpers. The timeout race SHALL use an injected `Clock`, not `setTimeout` at module scope.
|
||||
|
||||
#### Scenario: withTimeout honours injected Clock
|
||||
|
||||
- **WHEN** a `FakeClock` is injected and a never-resolving promise is passed to `withTimeout(promise, 100)`
|
||||
- **THEN** after advancing the clock by 100 ms, `withTimeout` rejects with `QueueDistributionHttpError(HTTP.TIMEOUT)`
|
||||
|
||||
**Verified by:** `src/communicator/base/base.test.ts::"withTimeout rejects with HTTP.TIMEOUT after clock advances past deadline"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: HTTP communicator with injectable fetch and Zod-validated wire envelope
|
||||
|
||||
The package SHALL ship `HttpWorkerCommunicator` extending `AbstractWorkerCommunicator`. The class SHALL accept an injectable `fetch` function (default: `globalThis.fetch`). All responses SHALL be parsed against a Zod schema `WorkerApiResponseSchema<T>` shaped `{ success: boolean; data?: T; error?: { code: string; message: string; details?: unknown } }`. Non-conformant responses SHALL throw `QueueDistributionHttpError(HTTP.RESPONSE_MALFORMED)`. HTTP non-2xx OR `success === false` SHALL throw `QueueDistributionHttpError(HTTP.PROTOCOL_ERROR)` carrying the upstream `error.code` and `error.message` on `cause`. Network-layer `fetch` rejections SHALL throw `QueueDistributionHttpError(HTTP.REQUEST_FAILED)` with the original error on `cause`. The implementation SHALL forward AbortSignal to `fetch`. `getProtocol()` SHALL return `"http"`. `sendTask` SHALL POST `{ taskId, taskMetadata, input }` to `<endpoint>/execute`. `checkStatus` SHALL GET `<endpoint>/status`. `checkHealth` SHALL GET `<endpoint>/health`. `ping` SHALL GET `<endpoint>/ping` and return the round-trip duration via the injected `Clock`.
|
||||
|
||||
#### Scenario: sendTask happy path
|
||||
|
||||
- **WHEN** an injected `fetch` returns `{ success: true, data: <O> }` at `<endpoint>/execute`
|
||||
- **THEN** `sendTask` resolves with the typed `<O>` value AND the `queue_dist_http.request.duration_ms` histogram emits one observation
|
||||
|
||||
#### Scenario: response envelope rejection
|
||||
|
||||
- **WHEN** an injected `fetch` returns `{ foo: 1 }` (no `success` field)
|
||||
- **THEN** `sendTask` rejects with `QueueDistributionHttpError(HTTP.RESPONSE_MALFORMED)`
|
||||
|
||||
#### Scenario: protocol error carries upstream code
|
||||
|
||||
- **WHEN** an injected `fetch` returns 500 with body `{ success: false, error: { code: "INTERNAL", message: "boom" } }`
|
||||
- **THEN** `sendTask` rejects with `QueueDistributionHttpError(HTTP.PROTOCOL_ERROR)` and `error.cause.message === "boom"`
|
||||
|
||||
#### Scenario: AbortSignal forwarded to fetch
|
||||
|
||||
- **WHEN** `sendTask(endpoint, task, input, signal)` is called and `signal.abort()` fires before fetch resolves
|
||||
- **THEN** the call rejects, the injected fetch sees `init.signal.aborted === true`, AND the error code is `HTTP.REQUEST_FAILED` with `cause.name === "AbortError"`
|
||||
|
||||
**Verified by:** `src/communicator/http/http-communicator.test.ts::"sendTask resolves with typed data on success"` AND `src/communicator/http/http-communicator.test.ts::"rejects with HTTP.RESPONSE_MALFORMED when envelope shape is wrong"` AND `src/communicator/http/http-communicator.test.ts::"rejects with HTTP.PROTOCOL_ERROR and carries upstream message on cause"` AND `src/communicator/http/http-communicator.test.ts::"forwards AbortSignal to the injected fetch"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: RemoteWorker — connect/reconnect via `@suaveplan/retry`
|
||||
|
||||
The package SHALL ship `RemoteWorker` extending `AbstractWorker` from `@suaveplan/queue-distribution`. Constructor accepts `endpoint`, `communicator: WorkerCommunicator`, `capacity`, `weight`, optional `id`, optional logger, and `RemoteWorkerOptions { healthCheckIntervalMs?, maxConnectionAttempts? }`. `doStart()` SHALL invoke `connect()` and start a periodic health-check loop on the injected `Clock`. `connect()` SHALL use `@suaveplan/retry` exponential-backoff between attempts (initial delay 1,000 ms, factor 2, max 10,000 ms, max attempts from `maxConnectionAttempts` default 3). The retry SHALL NOT recurse — it MUST use a bounded loop. `doStop()` SHALL stop the health-check loop and `disconnect` the communicator; disconnect errors SHALL be logged but NOT re-thrown. `doExecuteTask` SHALL throw `HTTP.CONNECT_FAILED` when `isConnected === false`. Errors during `communicator.sendTask` SHALL be routed through `handleCommunicationError(error)`: ping with a 5,000 ms timeout; if ping succeeds, the error propagates and the worker stays in current status; if ping fails, the worker is `markUnhealthy("Unreachable")`, `isConnected = false`, and a single non-recursive `attemptConnection()` is scheduled. `getHealth()` SHALL fetch real health from the remote worker; failures SHALL log a warning and return a default unhealthy payload. `getCapabilities()` SHALL return `{ isRemote: true, protocol, endpoint }`. `getType()` SHALL return `"remote"`.
|
||||
|
||||
#### Scenario: connect succeeds on first try
|
||||
|
||||
- **WHEN** `worker.start()` is called and `communicator.connect`/`checkStatus` resolve with `WorkerStatus.ONLINE`
|
||||
- **THEN** `worker.status === ONLINE` AND the connection-attempts counter is back at 0
|
||||
|
||||
#### Scenario: connect retries with exponential backoff and exhausts
|
||||
|
||||
- **WHEN** `communicator.connect` rejects on every call under `maxConnectionAttempts === 3` and the test injects a `FakeClock`
|
||||
- **THEN** `worker.start()` rejects with `QueueDistributionHttpError(HTTP.CONNECT_FAILED)` AND the `FakeClock` recorded sleeps of 1,000 ms then 2,000 ms (no recursion stack growth)
|
||||
|
||||
#### Scenario: unreachable remote marks worker UNHEALTHY
|
||||
|
||||
- **WHEN** `sendTask` throws and a subsequent ping rejects
|
||||
- **THEN** `worker.status === UNHEALTHY`, `isConnected === false`, AND a follow-up `attemptConnection` is scheduled (not awaited)
|
||||
|
||||
**Verified by:** `src/worker/remote/remote.test.ts::"start resolves with ONLINE on first connect"` AND `src/worker/remote/remote.test.ts::"exhausts retries and throws HTTP.CONNECT_FAILED under fake clock"` AND `src/worker/remote/remote.test.ts::"marks worker UNHEALTHY and schedules reconnect when ping fails"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Stable error registry for the HTTP plugin
|
||||
|
||||
The package SHALL register every error code through `defineCodes("HTTP", [...])` from `@suaveplan/error` and expose `QueueDistributionHttpError extends SuaveplanError<HttpErrorCode>`. The exhaustive set is: `REQUEST_FAILED`, `PROTOCOL_ERROR`, `TIMEOUT`, `RESPONSE_MALFORMED`, `CONNECT_FAILED`, `UNREACHABLE`. Every code SHALL be thrown by ≥1 production call path.
|
||||
|
||||
#### Scenario: every registered code is thrown
|
||||
|
||||
- **WHEN** the test suite iterates over every code in the registry
|
||||
- **THEN** a corresponding production throw site exists
|
||||
|
||||
**Verified by:** `src/errors/queue-distribution-http-error.test.ts::"every registered HTTP.* code is thrown by a production call path"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: OpenTelemetry instrumentation
|
||||
|
||||
The package SHALL emit, via `@suaveplan/opentelemetry` under meter name `"@suaveplan/queue-distribution-http"`:
|
||||
|
||||
- histogram: `queue_dist_http.request.duration_ms` (attributes `endpoint`, `operation: "sendTask" | "checkStatus" | "checkHealth" | "ping"`, `outcome: "success" | "failure"`)
|
||||
- counter: `queue_dist_http.request_total` (same attributes)
|
||||
- counter: `queue_dist_http.connect.attempts_total` (attribute `outcome: "success" | "failure" | "exhausted"`)
|
||||
- span: `queue_dist_http.sendTask`, `queue_dist_http.connect`
|
||||
|
||||
#### Scenario: histogram emission
|
||||
|
||||
- **WHEN** `sendTask` resolves successfully under an in-memory `MeterProvider`
|
||||
- **THEN** one observation is recorded on `queue_dist_http.request.duration_ms` with `outcome === "success"` AND `operation === "sendTask"`
|
||||
|
||||
**Verified by:** `src/communicator/http/http-communicator.spec-conformance.test.ts::"emits queue_dist_http.request.duration_ms histogram with operation and outcome attributes"`
|
||||
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
@@ -0,0 +1,83 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: queue-distribution-worker-thread (enterprise-queue-distribution)
|
||||
|
||||
**Package**: `@suaveplan/queue-distribution-worker-thread`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. (See `openspec/AGENTS.md` Section 13.)
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: WorkerFactory-driven communicator covering Web-Worker and `worker_threads`
|
||||
|
||||
The package SHALL expose `WorkerThreadCommunicator` implementing `WorkerCommunicator` from `@suaveplan/queue-distribution-http` (re-exported `WorkerCommunicator` interface). The constructor SHALL accept an injectable `WorkerFactory` of shape `() => { postMessage(msg, transfer?); addEventListener(kind, handler); terminate(); }` so the same class works against `globalThis.Worker` (browser) and a thin `worker_threads` wrapper (Node/Bun) without importing platform APIs at module scope. The implementation SHALL maintain a `Map<requestId, { resolve, reject, timeoutHandle }>` of outstanding requests. Each outgoing message SHALL include an `id` (generated via injected `IdGenerator`), a `kind: "task" | "status" | "health" | "ping"`, and a `payload` validated by Zod. Reply correlation SHALL match `id`; unknown `id` replies SHALL be logged and dropped. Reply payloads SHALL be Zod-validated against per-kind schemas; failures SHALL reject the in-flight request with `QueueDistributionWorkerThreadError(WT.MESSAGE_MALFORMED)`. Each request SHALL be subject to a per-call timeout (default `defaultTimeoutMs`) honoured by the injected `Clock`; expiry SHALL reject with `WT.REQUEST_TIMEOUT` and remove the entry from the map.
|
||||
|
||||
#### Scenario: round-trip correlation
|
||||
|
||||
- **WHEN** `sendTask`/`checkStatus`/`checkHealth`/`ping` are invoked concurrently and the underlying worker replies in arbitrary order
|
||||
- **THEN** each promise resolves with its own reply payload (matched by `id`)
|
||||
|
||||
#### Scenario: malformed reply throws WT.MESSAGE_MALFORMED
|
||||
|
||||
- **WHEN** the underlying worker posts back `{ id: <known>, kind: "task", payload: { something: "bad" } }` not conforming to the task-reply Zod schema
|
||||
- **THEN** the in-flight `sendTask` promise rejects with `QueueDistributionWorkerThreadError(WT.MESSAGE_MALFORMED)`
|
||||
|
||||
#### Scenario: request timeout
|
||||
|
||||
- **WHEN** a `FakeClock` is injected and the worker never replies within `defaultTimeoutMs`
|
||||
- **THEN** after the clock advances past the deadline, the call rejects with `QueueDistributionWorkerThreadError(WT.REQUEST_TIMEOUT)` AND the outstanding-request map no longer contains the entry
|
||||
|
||||
**Verified by:** `src/communicator/worker-thread/worker-thread-communicator.test.ts::"correlates concurrent replies by id"` AND `src/communicator/worker-thread/worker-thread-communicator.test.ts::"rejects with WT.MESSAGE_MALFORMED when reply fails Zod validation"` AND `src/communicator/worker-thread/worker-thread-communicator.test.ts::"rejects with WT.REQUEST_TIMEOUT and clears the outstanding-request map"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: WorkerThreadWorker — `AbstractWorker` subclass dispatching to a backing worker
|
||||
|
||||
The package SHALL ship `WorkerThreadWorker` extending `AbstractWorker` from `@suaveplan/queue-distribution`. Constructor SHALL accept an injectable `WorkerFactory` and instantiate a fresh `WorkerThreadCommunicator` per worker, AND a single backing worker thread. `doStart()` SHALL register error / exit listeners on the backing worker — `worker.exit` events SHALL `markUnhealthy("Worker exited")` and flip `status` to `OFFLINE`; uncaught `error` events SHALL emit the `WT.WORKER_CRASHED` OTel counter. `doStop(force)` SHALL `terminate()` the backing worker, fail every outstanding request with `WT.WORKER_TERMINATED`, and clear the outstanding-request map. `doExecuteTask` SHALL delegate to `communicator.sendTask(<no-endpoint>, task, input, signal)`. `getCapabilities()` SHALL return `{ isRemote: false, isThreaded: true }`. `getType()` SHALL return `"worker-thread"`.
|
||||
|
||||
#### Scenario: backing worker crash promotes to UNHEALTHY
|
||||
|
||||
- **WHEN** the backing worker emits an `error` event
|
||||
- **THEN** the OTel `queue_dist_wt.worker.crashed_total` counter increments and the worker is marked UNHEALTHY
|
||||
|
||||
#### Scenario: terminate fails outstanding requests with WT.WORKER_TERMINATED
|
||||
|
||||
- **WHEN** two `sendTask` calls are in flight and `worker.stop(true)` is invoked
|
||||
- **THEN** both pending promises reject with `QueueDistributionWorkerThreadError(WT.WORKER_TERMINATED)` AND `worker.status === OFFLINE`
|
||||
|
||||
**Verified by:** `src/worker/worker-thread/worker-thread-worker.test.ts::"backing worker crash flips status to UNHEALTHY and increments WT.WORKER_CRASHED counter"` AND `src/worker/worker-thread/worker-thread-worker.test.ts::"terminate fails outstanding requests with WT.WORKER_TERMINATED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Stable error registry for the worker-thread plugin
|
||||
|
||||
The package SHALL register every error code through `defineCodes("WT", [...])` from `@suaveplan/error` and expose `QueueDistributionWorkerThreadError extends SuaveplanError<WtErrorCode>`. The exhaustive set is: `WORKER_CRASHED`, `MESSAGE_MALFORMED`, `REQUEST_TIMEOUT`, `WORKER_TERMINATED`. Every code SHALL be thrown by ≥1 production call path.
|
||||
|
||||
#### Scenario: every registered code is thrown
|
||||
|
||||
- **WHEN** the test suite iterates over every code in the registry
|
||||
- **THEN** a corresponding production throw site exists
|
||||
|
||||
**Verified by:** `src/errors/queue-distribution-worker-thread-error.test.ts::"every registered WT.* code is thrown by a production call path"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: OpenTelemetry instrumentation
|
||||
|
||||
The package SHALL emit, via `@suaveplan/opentelemetry` under meter name `"@suaveplan/queue-distribution-worker-thread"`:
|
||||
|
||||
- histogram: `queue_dist_wt.request.duration_ms` (attributes `kind`, `outcome`)
|
||||
- counter: `queue_dist_wt.request_total` (attributes `kind`, `outcome`)
|
||||
- counter: `queue_dist_wt.worker.crashed_total`
|
||||
- counter: `queue_dist_wt.worker.terminated_total`
|
||||
- span: `queue_dist_wt.sendTask`
|
||||
|
||||
#### Scenario: histogram emission on sendTask round-trip
|
||||
|
||||
- **WHEN** `sendTask` resolves successfully under an in-memory `MeterProvider`
|
||||
- **THEN** one observation is recorded on `queue_dist_wt.request.duration_ms` with `kind === "task"` AND `outcome === "success"`
|
||||
|
||||
**Verified by:** `src/communicator/worker-thread/worker-thread-communicator.spec-conformance.test.ts::"emits queue_dist_wt.request.duration_ms histogram with kind and outcome attributes"`
|
||||
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
@@ -0,0 +1,309 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: queue-distribution (enterprise-queue-distribution)
|
||||
|
||||
**Package**: `@suaveplan/queue-distribution`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. (See `openspec/AGENTS.md` Section 13.)
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Distributed task contract decoupled from `@suaveplan/queue.Job`
|
||||
|
||||
The package SHALL expose `DistributedTask<I, O>` as the unit of work executed by workers, independent of `@suaveplan/queue`'s `Job` / state-machine model. The contract has: `id: string`, `execute(input: I, context: ExecutionContext): Promise<O>`, `getMetadata(): TaskMetadata`, `getStatus(): TaskStatus`, `setStatus(status: TaskStatus): void`. `TaskMetadata` includes `createdAt: Date`, optional `startedAt`/`completedAt`, `attempts: number`, optional `tags: string[]`, and indexed extras. `TaskStatus` is the closed enum `PENDING | RUNNING | COMPLETED | FAILED | CANCELLED | TIMED_OUT`. `ExecutionContext` is caller-defined.
|
||||
|
||||
#### Scenario: typed metadata round-trip
|
||||
|
||||
- **WHEN** a caller constructs a `DistributedTask<I, O>` literal and passes it through `Worker.executeTask`
|
||||
- **THEN** `task.getMetadata()` returns a value validated by `TaskMetadataSchema` and `getStatus()` reflects the lifecycle (`PENDING` → `RUNNING` → `COMPLETED`/`FAILED`)
|
||||
|
||||
#### Scenario: generics are preserved end-to-end
|
||||
|
||||
- **WHEN** a caller writes `const w: Worker; const t: DistributedTask<number, string>; const r = await w.executeTask(t, 42)`
|
||||
- **THEN** the inferred type of `r` is exactly `string` (no `unknown`, no `any`)
|
||||
|
||||
**Verified by:** `src/task/task.test.ts::"validates TaskMetadata via TaskMetadataSchema"` AND `src/task/task.spec-conformance.test.ts::"preserves <I, O> generics through Worker.executeTask"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Worker interface and status model
|
||||
|
||||
The package SHALL expose a `Worker` interface with: `id: string`, `capacity: number`, `weight: number`, `status: WorkerStatus`, `activeTasks: number`, `executeTask<I, O>(task, input, signal?): Promise<O>`, `getHealth(signal?): Promise<WorkerHealth>`, `start(signal?): Promise<void>`, `stop(force?, signal?): Promise<void>`, `canAcceptTask(): boolean`, `getInfo(): WorkerInfo`. `WorkerStatus` is the closed enum `ONLINE | BUSY | OFFLINE | UNHEALTHY`. `WorkerHealth` has `isHealthy: boolean` and a `metrics` object containing at least `cpu`, `memory`, `tasksFailed`, `tasksSucceeded`, `responseTime` plus open-ended string-keyed extras. `WorkerInfo` returns identity, type label, capacity, weight, status, active count, `startedAt?`, `lastHealthCheck?`, `lastHealthStatus?`, and `capabilities?`.
|
||||
|
||||
#### Scenario: health and info shapes parse via Zod
|
||||
|
||||
- **WHEN** a `Worker` implementation returns from `getHealth()` / `getInfo()`
|
||||
- **THEN** the value passes `WorkerHealthSchema` / `WorkerInfoSchema` (failures throw `QueueDistributionError(QUEUE_DIST.INVALID_OPTIONS)`)
|
||||
|
||||
#### Scenario: canAcceptTask honours status and capacity
|
||||
|
||||
- **WHEN** `status === ONLINE` and `activeTasks < capacity`
|
||||
- **THEN** `canAcceptTask()` returns `true`
|
||||
- **WHEN** `status === OFFLINE` or `activeTasks >= capacity`
|
||||
- **THEN** `canAcceptTask()` returns `false`
|
||||
|
||||
**Verified by:** `src/worker/worker.test.ts::"WorkerHealthSchema rejects malformed health"` AND `src/worker/worker.test.ts::"canAcceptTask honours ONLINE+capacity and rejects OFFLINE"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: AbstractWorker lifecycle and metrics, no module-scope clocks
|
||||
|
||||
The package SHALL ship an `AbstractWorker` base class implementing `Worker` with: idempotent `start()` (no-op when not OFFLINE) emitting `WorkerStatus.ONLINE`; `stop(force)` rejecting when `activeTasks > 0 && !force` with `QUEUE_DIST.WORKER_BUSY_STOP`; `executeTask` tracking `activeTasks`, success/failure counters, response-time total, and toggling `status` between ONLINE and BUSY; `markUnhealthy(reason)` flipping `status` to UNHEALTHY; `getInfo()` aggregating all of the above. The class SHALL accept `Clock`, `IdGenerator`, and `LoggerLike` via constructor options — no `Date.now()`, no `Math.random()`, no `uuid` import at module scope.
|
||||
|
||||
#### Scenario: executeTask refuses OFFLINE workers
|
||||
|
||||
- **WHEN** `await worker.executeTask(task, input)` is called while `status === OFFLINE`
|
||||
- **THEN** the call rejects with `QueueDistributionError` whose code is `QUEUE_DIST.WORKER_NOT_AVAILABLE`
|
||||
|
||||
#### Scenario: force stop cancels active tasks
|
||||
|
||||
- **WHEN** `await worker.stop(true)` is called with `activeTasks === 2`
|
||||
- **THEN** `worker.status === OFFLINE` AND `worker.activeTasks === 0`
|
||||
|
||||
#### Scenario: metrics emit through injectable Clock
|
||||
|
||||
- **WHEN** a `FakeClock` returns `1000` at task start and `1750` at completion
|
||||
- **THEN** `getHealth()` returns `metrics.responseTime === 750` (the difference, not wall-clock)
|
||||
|
||||
**Verified by:** `src/worker/abstract/abstract.test.ts::"executeTask refuses OFFLINE worker with WORKER_NOT_AVAILABLE"` AND `src/worker/abstract/abstract.test.ts::"force stop drains activeTasks and transitions to OFFLINE"` AND `src/worker/abstract/abstract.spec-conformance.test.ts::"metrics derive responseTime from injected Clock"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: LocalWorker in-process execution
|
||||
|
||||
The package SHALL ship `LocalWorker<Ctx>` extending `AbstractWorker`. `doExecuteTask` SHALL invoke `task.execute(input, this.context)` where `context` is the `ExecutionContext` passed at construction. `getCapabilities()` SHALL return `{ hasFileSystemAccess: true, isRemote: false, contextType: <ctor name> }`. `getType()` SHALL return `"local"`.
|
||||
|
||||
#### Scenario: task receives the configured context
|
||||
|
||||
- **WHEN** `new LocalWorker(ctx).executeTask(task, input)` is called
|
||||
- **THEN** `task.execute(input, ctx)` is invoked with the exact same `ctx` reference
|
||||
|
||||
**Verified by:** `src/worker/local/local.test.ts::"executeTask passes the configured ExecutionContext to task.execute"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Load-balancing strategy interface
|
||||
|
||||
The package SHALL expose `LoadBalancingStrategy` with `selectWorker(workers, task?): Worker | null`, optional `notifyWorkerSelected(worker, task)`, optional `notifyTaskCompleted(worker, task, success, durationMs)`, optional `getName(): string`. `AbstractLoadBalancingStrategy` provides `filterAvailableWorkers(workers): Worker[]` returning workers with `status === ONLINE || status === BUSY`, `activeTasks < capacity`, and `canAcceptTask() === true`.
|
||||
|
||||
#### Scenario: filterAvailableWorkers excludes OFFLINE and at-capacity workers
|
||||
|
||||
- **WHEN** the input list contains workers in every `WorkerStatus` state and varying `activeTasks`
|
||||
- **THEN** the output contains only those satisfying the three predicates above
|
||||
|
||||
**Verified by:** `src/load-balancing/base/base.test.ts::"filterAvailableWorkers returns only ONLINE/BUSY workers under capacity"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Round-robin strategy
|
||||
|
||||
The package SHALL ship `RoundRobinStrategy` that selects workers in rotation order. `currentIndex` SHALL wrap to 0 when reaching the end of the available-worker list, and SHALL clamp to 0 if the available-worker list shrinks below `currentIndex`. `reset()` SHALL set `currentIndex` to 0. `selectWorker` SHALL return `null` when no workers are available. `getName()` SHALL return `"round-robin"`.
|
||||
|
||||
#### Scenario: rotation order and reset
|
||||
|
||||
- **WHEN** `selectWorker` is invoked 4 times against the same 3-worker list
|
||||
- **THEN** the selections are workers `[0, 1, 2, 0]` in order
|
||||
- **WHEN** `reset()` is then called and `selectWorker` invoked once more
|
||||
- **THEN** the selection is worker `[0]` again
|
||||
|
||||
#### Scenario: pool shrink is safe
|
||||
|
||||
- **WHEN** `currentIndex === 2` and the available-worker list shrinks to `length === 1`
|
||||
- **THEN** the next `selectWorker` returns worker `[0]` (no out-of-range read)
|
||||
|
||||
**Verified by:** `src/load-balancing/round-robin/round-robin.test.ts::"cycles through workers in order and wraps after the end"` AND `src/load-balancing/round-robin/round-robin.test.ts::"clamps currentIndex when the pool shrinks below it"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Least-busy strategy with optional performance bias
|
||||
|
||||
The package SHALL ship `LeastBusyStrategy` selecting the worker with the fewest `activeTasks`. When `usePerformanceMetrics` is `true`, ties on `activeTasks` SHALL break by lower `avgExecutionTime`. Further ties SHALL break by higher `capacity` then higher `weight`. `notifyTaskCompleted(worker, task, success, durationMs)` SHALL update an internal EWMA-style cumulative average for successful runs only. `getWorkerPerformance(id)` SHALL return the stored record or `null`. `resetPerformanceMetrics()` SHALL clear the registry. `setUsePerformanceMetrics(enabled)` SHALL toggle the flag at runtime.
|
||||
|
||||
#### Scenario: tie-break by performance when enabled
|
||||
|
||||
- **WHEN** two workers both have `activeTasks === 0`, perf metrics enabled, and recorded average durations of 100 ms and 200 ms
|
||||
- **THEN** `selectWorker` returns the 100 ms worker
|
||||
|
||||
#### Scenario: tie-break by capacity when perf metrics unavailable
|
||||
|
||||
- **WHEN** two workers both have `activeTasks === 0` and no perf history, with `capacity` 2 and 4
|
||||
- **THEN** `selectWorker` returns the `capacity === 4` worker
|
||||
|
||||
**Verified by:** `src/load-balancing/least-busy/least-busy.test.ts::"ties break by lower average execution time when usePerformanceMetrics is on"` AND `src/load-balancing/least-busy/least-busy.test.ts::"ties break by higher capacity when no performance history exists"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Weighted strategy with injectable RNG
|
||||
|
||||
The package SHALL ship `WeightedStrategy` that selects workers with probability proportional to `weight`. The RNG SHALL be supplied via constructor option (`Rng` from `@suaveplan/types`); the strategy SHALL NOT call `Math.random()`. When `totalWeight <= 0`, the strategy SHALL fall back to uniform selection. `getName()` SHALL return `"weighted"`.
|
||||
|
||||
#### Scenario: deterministic distribution under fixed RNG
|
||||
|
||||
- **WHEN** a deterministic `Rng` yields the sequence `[0.0, 0.5, 0.9]` against weights `[1, 3, 6]` (total 10)
|
||||
- **THEN** `selectWorker` returns workers `[0, 1, 2]` in that order
|
||||
|
||||
#### Scenario: zero-weight pool falls back to uniform
|
||||
|
||||
- **WHEN** every worker has `weight === 0` and the `Rng` yields `0.5`
|
||||
- **THEN** `selectWorker` returns the middle worker in the list
|
||||
|
||||
**Verified by:** `src/load-balancing/weighted/weighted.test.ts::"selects workers proportionally under a deterministic Rng"` AND `src/load-balancing/weighted/weighted.test.ts::"falls back to uniform when total weight is zero"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Periodic health checker
|
||||
|
||||
The package SHALL ship `WorkerHealthChecker` (interface) and `PeriodicHealthChecker` (implementation). `checkHealth(worker)` SHALL race the worker's `getHealth()` against an injectable-`Clock`-driven timeout (default 5,000 ms). On success, results SHALL be cached for `cacheTimeMs` (default 5,000 ms) when `cacheResults` is true. Consecutive failures (`unhealthyThreshold`, default 3) SHALL flip `worker.status` to `UNHEALTHY`; consecutive successes (`healthyThreshold`, default 2) SHALL flip `UNHEALTHY` back to `ONLINE` or `BUSY` based on `activeTasks`. `getHealthCheckCounts(id)`, `resetHealthCheckCounts(id)`, and `clearCache()` SHALL be exposed.
|
||||
|
||||
#### Scenario: unhealthy after threshold consecutive failures
|
||||
|
||||
- **WHEN** `worker.getHealth()` rejects on three consecutive calls (with threshold 3)
|
||||
- **THEN** `worker.status === UNHEALTHY` after the third call, AND `QUEUE_DIST.HEALTH_CHECK_FAILED` is emitted via the OTel counter
|
||||
|
||||
#### Scenario: recovery after threshold consecutive successes
|
||||
|
||||
- **WHEN** a worker previously marked UNHEALTHY returns healthy on two consecutive calls (with threshold 2) and has `activeTasks === 0`
|
||||
- **THEN** `worker.status === ONLINE` after the second call
|
||||
- **WHEN** the same but `activeTasks > 0`
|
||||
- **THEN** `worker.status === BUSY`
|
||||
|
||||
#### Scenario: timeout uses injected Clock
|
||||
|
||||
- **WHEN** the test injects a `FakeClock` and `worker.getHealth()` never resolves
|
||||
- **THEN** after advancing the clock by `healthCheckTimeoutMs`, `checkHealth` returns the fallback `isHealthy: false` payload AND emits the OTel `queue_dist.health_check.duration_ms` histogram with the elapsed time
|
||||
|
||||
**Verified by:** `src/health-checker/health-checker.test.ts::"flips worker to UNHEALTHY after threshold consecutive failures"` AND `src/health-checker/health-checker.test.ts::"recovers worker to ONLINE/BUSY after threshold consecutive successes"` AND `src/health-checker/health-checker.spec-conformance.test.ts::"timeout race honours injected Clock and emits duration histogram"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Worker pool registry and search
|
||||
|
||||
The package SHALL ship `WorkerPool` with: `addWorker(worker)` (throws `QUEUE_DIST.WORKER_DUPLICATE` on duplicate id), `removeWorker(id)`, `getWorkerById(id)`, `getAllWorkers()`, `getWorkersByStatus(status)`, `getAvailableWorkers()`, `findWorkers(criteria)` filtering by status, capacity range, weight range, type, and a custom matcher predicate, `startAllWorkers(signal?)`, `stopAllWorkers(force?, signal?)`, `clear()`, `size()`, `getWorkersInfo()`.
|
||||
|
||||
#### Scenario: duplicate add throws WORKER_DUPLICATE
|
||||
|
||||
- **WHEN** `addWorker(w)` is called twice with the same `w.id`
|
||||
- **THEN** the second call throws `QueueDistributionError(QUEUE_DIST.WORKER_DUPLICATE)`
|
||||
|
||||
#### Scenario: findWorkers matches every criteria field
|
||||
|
||||
- **WHEN** `findWorkers({ status: ONLINE, minCapacity: 2, maxWeight: 3, type: "local", matcher: w => w.activeTasks === 0 })` is called against a mixed pool
|
||||
- **THEN** only workers satisfying every predicate are returned
|
||||
|
||||
**Verified by:** `src/pool/pool.test.ts::"addWorker rejects duplicate ids with WORKER_DUPLICATE"` AND `src/pool/pool.test.ts::"findWorkers filters by every criteria field including custom matcher"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: WorkerManager — assignment, retry, health, restart
|
||||
|
||||
The package SHALL ship `WorkerManager(pool, balancer?, healthChecker?, options?, deps?)` providing:
|
||||
|
||||
- `assignTask<I, O>(task, input, signal?)` — selects via the configured `LoadBalancingStrategy`, executes on the chosen worker, retries up to `maxTaskRetries` (default 3) on failure with backoff via `@suaveplan/retry`, and records exhausted tasks in `failedTasks`. The retry sleep SHALL be driven by the injected `Clock`, NOT `setTimeout` at module scope. The retry loop SHALL be bounded (no recursive self-call). AbortSignal cancellation MUST be honoured.
|
||||
- `assignTasks<I, O>(tasks[], inputs[], signal?)` — `Promise.all` over `assignTask`; mismatch in array lengths throws `QUEUE_DIST.INVALID_OPTIONS`.
|
||||
- `startHealthChecks()` / `stopHealthChecks()` — periodic loop driven by injected `Clock.setInterval`; idempotent.
|
||||
- `checkAllWorkersHealth(signal?)` — fan-out over all workers, returns map of id → `WorkerHealth`; logs a warning when online count falls below `minOnlineWorkers`.
|
||||
- `handleWorkerFailure(worker, error)` — checks isHealthy; if not, flips to `UNHEALTHY` and optionally restarts.
|
||||
- `setLoadBalancer(balancer)`, `getLoadBalancer()`, `getHealthChecker()`, `getWorkerPool()`, `getFailedTasks()`, `clearFailedTasks()`, `getTaskRetries(id)`.
|
||||
- `getOptions()` / `updateOptions(partial)` — option changes that affect the health-check interval SHALL restart the loop.
|
||||
|
||||
The constructor SHALL throw `QUEUE_DIST.INVALID_OPTIONS` for negative `maxTaskRetries`, negative `taskRetryDelayMs`, or `minOnlineWorkers < 0`.
|
||||
|
||||
#### Scenario: no available workers throws NO_AVAILABLE_WORKERS
|
||||
|
||||
- **WHEN** `pool.getAvailableWorkers()` returns `[]` and `assignTask` is called
|
||||
- **THEN** the call rejects with `QueueDistributionError(QUEUE_DIST.NO_AVAILABLE_WORKERS)`
|
||||
|
||||
#### Scenario: balancer returning null throws STRATEGY_NO_WORKER_SELECTED
|
||||
|
||||
- **WHEN** the configured `LoadBalancingStrategy.selectWorker` returns `null` despite a non-empty available pool
|
||||
- **THEN** the call rejects with `QueueDistributionError(QUEUE_DIST.STRATEGY_NO_WORKER_SELECTED)`
|
||||
|
||||
#### Scenario: retries exhausted throws TASK_RETRIES_EXHAUSTED
|
||||
|
||||
- **WHEN** the chosen worker rejects `executeTask` `maxTaskRetries + 1` times in a row
|
||||
- **THEN** the call rejects with `QueueDistributionError(QUEUE_DIST.TASK_RETRIES_EXHAUSTED)` AND `getFailedTasks()` contains the task
|
||||
|
||||
#### Scenario: AbortSignal cancellation between retries
|
||||
|
||||
- **WHEN** an AbortController is aborted between the first failure and the retry sleep
|
||||
- **THEN** the call rejects with `QueueDistributionError(QUEUE_DIST.TASK_CANCELLED)` without sleeping the full `taskRetryDelayMs`
|
||||
|
||||
#### Scenario: restart limit exhaustion throws WORKER_RESTART_EXHAUSTED
|
||||
|
||||
- **WHEN** an unhealthy worker has reached `maxWorkerRestarts` and the supervisor attempts another restart
|
||||
- **THEN** the supervisor emits `QUEUE_DIST.WORKER_RESTART_EXHAUSTED` via the OTel counter and SHALL NOT call `worker.start()` again
|
||||
|
||||
**Verified by:** `src/manager/manager.test.ts::"throws NO_AVAILABLE_WORKERS when pool is empty"` AND `src/manager/manager.test.ts::"throws STRATEGY_NO_WORKER_SELECTED when balancer returns null"` AND `src/manager/manager.test.ts::"throws TASK_RETRIES_EXHAUSTED and records the task in failedTasks"` AND `src/manager/manager.spec-conformance.test.ts::"AbortSignal cancellation between retries throws TASK_CANCELLED"` AND `src/manager/manager.spec-conformance.test.ts::"supervisor stops restarting after WORKER_RESTART_EXHAUSTED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Stable error registry via `defineCodes`
|
||||
|
||||
The package SHALL register every error code through `defineCodes("QUEUE_DIST", [...])` from `@suaveplan/error` and expose `QueueDistributionError extends SuaveplanError<QueueDistributionErrorCode>`. The exhaustive code set is: `WORKER_NOT_AVAILABLE`, `WORKER_DUPLICATE`, `WORKER_BUSY_STOP`, `WORKER_NOT_FOUND`, `WORKER_RESTART_EXHAUSTED`, `NO_AVAILABLE_WORKERS`, `STRATEGY_NO_WORKER_SELECTED`, `TASK_RETRIES_EXHAUSTED`, `HEALTH_CHECK_TIMEOUT`, `HEALTH_CHECK_FAILED`, `INVALID_OPTIONS`, `TASK_CANCELLED`. Every code SHALL be thrown by at least one production call path.
|
||||
|
||||
#### Scenario: every registered code is thrown by a production code path
|
||||
|
||||
- **WHEN** the test suite iterates over every code in the registry
|
||||
- **THEN** a corresponding throw site exists, and the test forces the failure and asserts the code matches
|
||||
|
||||
**Verified by:** `src/errors/queue-distribution-error.test.ts::"every registered code is thrown by a production call path"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: OpenTelemetry instrumentation on every public entrypoint
|
||||
|
||||
The package SHALL emit the following instruments via `@suaveplan/opentelemetry`, registered under meter name `"@suaveplan/queue-distribution"`:
|
||||
|
||||
- histograms: `queue_dist.task.duration_ms`, `queue_dist.health_check.duration_ms`, `queue_dist.worker.restart.duration_ms`
|
||||
- counters: `queue_dist.task.assigned_total`, `queue_dist.task.completed_total` (attribute `outcome: "success" | "failure"`), `queue_dist.task.retried_total`, `queue_dist.worker.restart_total` (attribute `outcome: "success" | "failure" | "exhausted"`), `queue_dist.strategy.no_worker_total`
|
||||
- up/down counter: `queue_dist.workers.active`
|
||||
- spans: `queue_dist.manager.assign`, `queue_dist.worker.executeTask`, `queue_dist.health.check`, `queue_dist.worker.restart`
|
||||
|
||||
#### Scenario: task duration histogram emits with expected attributes
|
||||
|
||||
- **WHEN** `manager.assignTask(task, input)` completes successfully under an in-memory `MeterProvider`
|
||||
- **THEN** the `queue_dist.task.duration_ms` histogram has one recorded value with attribute `outcome === "success"`, AND the `queue_dist.task.completed_total` counter incremented by 1 with the same attribute
|
||||
|
||||
#### Scenario: span emission
|
||||
|
||||
- **WHEN** `manager.assignTask` is invoked
|
||||
- **THEN** a span named `queue_dist.manager.assign` is recorded with attributes `queue_dist.worker_id` and `queue_dist.task_id`
|
||||
|
||||
**Verified by:** `src/manager/manager.spec-conformance.test.ts::"emits queue_dist.task.duration_ms and completed counter on success"` AND `src/manager/manager.spec-conformance.test.ts::"emits queue_dist.manager.assign span with worker_id and task_id attributes"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Universal-tier — zero browser / Node / React deps
|
||||
|
||||
The package SHALL be installable in browser, Bun, and Node with no platform-specific runtime imports. The package SHALL NOT import from `node:*`, `bun:*`, `react`, `react-dom`, `next/*`, or any DOM-only API at module scope. `sideEffects: false` SHALL be declared in `package.json` and enforced via the tree-shake test.
|
||||
|
||||
#### Scenario: dep-graph guard rejects platform imports
|
||||
|
||||
- **WHEN** `tests/dep-graph.test.ts` scans every `.ts` file in `src/` for forbidden imports
|
||||
- **THEN** the count of matches is zero AND the test passes
|
||||
|
||||
**Verified by:** `tests/dep-graph.test.ts::"src/ contains no node:*, bun:*, or React imports"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Test files use the canonical runner wrapper
|
||||
|
||||
Every test file in `src/` SHALL import runner primitives (`describe`, `it`, `expect`, `beforeAll`, `afterAll`, `beforeEach`, `afterEach`, `vi`, `expectTypeOf`) from `@suaveplan/testing/runner`, NOT from `vitest` or `bun:test` directly.
|
||||
|
||||
#### Scenario: scan shows zero direct runner imports
|
||||
|
||||
- **WHEN** `grep -rnE "from ['\"](vitest|bun:test)['\"]" packages/streams/queue-distribution/src/ --include="*.ts" --include="*.tsx"` is run
|
||||
- **THEN** the output is empty
|
||||
|
||||
**Verified by:** `tests/dep-graph.test.ts::"no test file imports vitest or bun:test directly"`
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: `WorkerManager.rebalance()`
|
||||
|
||||
**Reason:** Upstream body was a stub: `this.logger.warn("Rebalance operation not implemented")`. Per CLAUDE.md §Rules ("Complete all implementations — no stubs") and AGENTS.md §2b ("NEVER ship stubs"), stubs do not ship to genesis.
|
||||
|
||||
**Migration:** None today — the method had no real implementation upstream, so no consumer can have relied on its behaviour. If a real rebalance algorithm is needed later, open a fresh change proposal.
|
||||
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
Reference in New Issue
Block a user