Initial Commit
This commit is contained in:
@@ -0,0 +1,191 @@
|
||||
# Enterprise-grade `@suaveplan/queue-distribution` (split into 3 packages)
|
||||
|
||||
**Package**: `@suaveplan/queue-distribution`, `@suaveplan/queue-distribution-http`, `@suaveplan/queue-distribution-worker-thread`
|
||||
**Status**: PROPOSED
|
||||
**Type**: new-package (migration from `../suaveplan/packages/universal/queue-distribution`)
|
||||
|
||||
> This proposal supersedes the placeholder scaffolded into commit `f52ca3d1`, which described a `WorkerPool<T>.scale()` / autoscale design that did NOT match the upstream source layout. The placeholder marked every parity row ✅ without verifying upstream — the listed capabilities (`WorkerPool<T>.scale()`, `JobHandler<T>`, `HashStrategy`, autoscale depth/idle thresholds) do not exist in `../suaveplan/packages/universal/queue-distribution/src/`. Replacing it per `openspec/AGENTS.md §0a` ("fresh build with a different API" is a blocking anti-pattern).
|
||||
|
||||
## Why
|
||||
|
||||
Worker-pool distribution is a recurring need across genesis — anywhere a unit of work must execute concurrently against a finite, health-monitored pool of executors. Upstream shipped a single `@suaveplan/queue-distribution` package coupling four concerns (load-balancing, worker lifecycle, in-process workers, remote-over-HTTP workers) into one universal-tier bundle. That coupling makes the package un-tree-shakeable for browser consumers who never use HTTP, and conversely forces server consumers to carry transport code they don't need. The upstream implementation also predates genesis enterprise standards: no Zod boundaries, no OTel, plain `Error` throws, `Math.random` / `Date.now` baked in at strategy / health-checker / connect-retry layer (no injectable `Rng`/`Clock`), `any` type holes in catch arms, `uuid` runtime dep for worker IDs, every test file imports from `vitest` directly, and every feature barrel uses `export *`.
|
||||
|
||||
This change migrates that capability to genesis at enterprise quality and splits it into a core + two transport plugins so consumers only pay for what they use. The core package is also decoupled from `@suaveplan/queue`'s `Task<I,O>` abstraction (which genesis deliberately replaced with a `Job` / state-machine / Zod-schema model when it migrated `@suaveplan/queue`): distribution defines its own minimal `DistributedTask<I,O>` interface so it can serve queue, batch, RPC, or agent-dispatch consumers via thin adapters. A separate follow-up change can add a `createWorkerForQueueJob(...)` adapter once a concrete genesis consumer of both packages exists.
|
||||
|
||||
## What changes
|
||||
|
||||
### `@suaveplan/queue-distribution` (core, universal tier — `packages/streams/queue-distribution`)
|
||||
|
||||
- ADDED: `DistributedTask<I, O>` — minimal task contract (`id: string`, `execute(input, ctx): Promise<O>`, `getMetadata(): TaskMetadata`, `setStatus(status)`, `getStatus()`).
|
||||
- ADDED: `TaskStatus` enum (`PENDING` | `RUNNING` | `COMPLETED` | `FAILED` | `CANCELLED` | `TIMED_OUT`).
|
||||
- ADDED: `TaskMetadata` shape (`createdAt`, `startedAt?`, `completedAt?`, `attempts`, `tags?`, indexed extras).
|
||||
- ADDED: `ExecutionContext` — opaque, caller-defined context object threaded through `execute`.
|
||||
- ADDED: `Worker` interface — `id`, `capacity`, `weight`, `status`, `activeTasks`, `executeTask<I,O>`, `getHealth`, `start`, `stop`, `canAcceptTask`, `getInfo`. Zod schema at the boundary.
|
||||
- ADDED: `WorkerStatus` enum (`ONLINE` | `BUSY` | `OFFLINE` | `UNHEALTHY`).
|
||||
- ADDED: `WorkerHealth` + `WorkerInfo` interfaces.
|
||||
- ADDED: `AbstractWorker` — base class with lifecycle FSM, metrics, structured `start`/`stop`/`executeTask`/`getHealth`/`markUnhealthy`/`getInfo`. Injectable `Clock`, `IdGenerator`, `LoggerLike`. Source split across `abstract.ts` (public surface, ≤200 LOC) + `abstract-execute.ts` (`executeTask` lifecycle, ≤120 LOC) + `abstract-metrics.ts` (metric accumulation, ≤80 LOC) to stay under the 300-LOC hard limit.
|
||||
- ADDED: `LocalWorker<Ctx>` — in-process executor parameterised by `ExecutionContext`.
|
||||
- ADDED: `LoadBalancingStrategy` interface + `AbstractLoadBalancingStrategy` base with `filterAvailableWorkers`.
|
||||
- ADDED: `RoundRobinStrategy` — rotation index, `reset()`, pool-shrink safe.
|
||||
- ADDED: `LeastBusyStrategy` — fewest-active-tasks; optional `usePerformanceMetrics` flag with EWMA bias, capacity tie-break, weight tie-break; `notifyTaskCompleted` callback updates metrics; `resetPerformanceMetrics`, `getWorkerPerformance`, `setUsePerformanceMetrics`.
|
||||
- ADDED: `WeightedStrategy` — weighted random via injectable `Rng`; total-weight ≤ 0 falls back to uniform.
|
||||
- ADDED: `WorkerHealthChecker` interface + `PeriodicHealthChecker` — consecutive-failure unhealthy promotion, consecutive-success recovery, configurable timeout, optional caching, `getHealthCheckCounts`, `resetHealthCheckCounts`, `clearCache`.
|
||||
- ADDED: `WorkerPool` — `addWorker` (rejects duplicate IDs), `removeWorker`, `getWorkerById`, `getAllWorkers`, `getAvailableWorkers`, `getWorkersByStatus`, `findWorkers(criteria)`, `startAllWorkers`, `stopAllWorkers`, `clear`, `size`, `getWorkersInfo`.
|
||||
- ADDED: `WorkerSearchCriteria` shape (status, min/max capacity, min/max weight, type, custom matcher).
|
||||
- ADDED: `WorkerManager` — `assignTask<I,O>(task, input, signal?)`, `assignTasks<I,O>(tasks[], inputs[], signal?)`, `startHealthChecks`, `stopHealthChecks`, `checkAllWorkersHealth`, `handleWorkerFailure`, `getWorkerPool`, `getLoadBalancer`, `setLoadBalancer`, `getHealthChecker`, `getFailedTasks`, `clearFailedTasks`, `getTaskRetries`, `getOptions`, `updateOptions`. Source split across `manager.ts` (public surface, ≤200 LOC) + `manager-dispatch.ts` (assignTask / retry orchestration, ≤200 LOC) + `manager-supervisor.ts` (health-check loop / restart logic, ≤200 LOC).
|
||||
- ADDED: Retry-with-backoff at the manager layer routed through `@suaveplan/retry` (replaces upstream's hand-rolled `setTimeout` retry loop).
|
||||
- ADDED: `QueueDistributionError` extending `SuaveplanError`, registry of stable codes via `defineCodes("QUEUE_DIST", [...])`:
|
||||
- `WORKER_NOT_AVAILABLE`, `WORKER_DUPLICATE`, `WORKER_BUSY_STOP`, `WORKER_NOT_FOUND`, `WORKER_RESTART_EXHAUSTED`
|
||||
- `NO_AVAILABLE_WORKERS`, `STRATEGY_NO_WORKER_SELECTED`, `TASK_RETRIES_EXHAUSTED`
|
||||
- `HEALTH_CHECK_TIMEOUT`, `HEALTH_CHECK_FAILED`
|
||||
- `INVALID_OPTIONS`, `TASK_CANCELLED`
|
||||
- ADDED: OTel instruments via `@suaveplan/opentelemetry`:
|
||||
- histograms: `queue_dist.task.duration_ms`, `queue_dist.health_check.duration_ms`, `queue_dist.worker.restart.duration_ms`
|
||||
- counters: `queue_dist.task.assigned_total`, `queue_dist.task.completed_total{outcome}`, `queue_dist.task.retried_total`, `queue_dist.worker.restart_total{outcome}`, `queue_dist.strategy.no_worker_total`
|
||||
- up/down counter: `queue_dist.workers.active`
|
||||
- spans: `queue_dist.manager.assign`, `queue_dist.worker.executeTask`, `queue_dist.health.check`, `queue_dist.worker.restart`
|
||||
- ADDED: Zod schemas at every public boundary (`WorkerManagerOptionsSchema`, `HealthCheckerOptionsSchema`, `WorkerSearchCriteriaSchema`, `WorkerHealthSchema`, `WorkerInfoSchema`, `TaskMetadataSchema`).
|
||||
- ADDED: AbortSignal threaded through every async public method (`assignTask`, `assignTasks`, `executeTask`, `getHealth`, `start`, `stop`, retry delay sleeps).
|
||||
|
||||
### `@suaveplan/queue-distribution-http` (plugin — `packages/streams/queue-distribution-http`)
|
||||
|
||||
- ADDED: `WorkerCommunicator` interface (`sendTask`, `checkStatus`, `checkHealth`, `connect`, `disconnect`, `ping`, `getProtocol`).
|
||||
- ADDED: `AbstractWorkerCommunicator` base with default-timeout + serialize/deserialize helpers + `withTimeout` race.
|
||||
- ADDED: `HttpWorkerCommunicator` — fetch-based transport, injectable `fetch`, AbortSignal-aware, Zod-validated wire envelope `WorkerApiResponse<T>`, structured headers.
|
||||
- ADDED: `RemoteWorker` — `AbstractWorker` subclass over a `WorkerCommunicator`. Connect-retry routed through `@suaveplan/retry` (replaces the upstream hand-rolled `Math.min(1000 * attempts, 10000)` ladder + recursive call). Source split across `remote.ts` (public + lifecycle, ≤200 LOC) + `remote-connect.ts` (connect/reconnect, ≤120 LOC) + `remote-health.ts` (periodic health loop, ≤120 LOC).
|
||||
- ADDED: `RemoteWorkerOptions` (healthCheckIntervalMs, maxConnectionAttempts).
|
||||
- ADDED: `QueueDistributionHttpError` with codes `HTTP.REQUEST_FAILED`, `HTTP.PROTOCOL_ERROR`, `HTTP.TIMEOUT`, `HTTP.RESPONSE_MALFORMED`, `HTTP.CONNECT_FAILED`, `HTTP.UNREACHABLE`.
|
||||
|
||||
### `@suaveplan/queue-distribution-worker-thread` (plugin — `packages/streams/queue-distribution-worker-thread`)
|
||||
|
||||
- ADDED: `WorkerThreadCommunicator` — Web-Worker / Node `worker_threads` bridge with caller-injected `WorkerFactory` so the same code runs in browser and server. Message-correlation envelope `{ id, kind: "task" | "status" | "health" | "ping", payload }` validated via Zod.
|
||||
- ADDED: `WorkerThreadWorker` — `AbstractWorker` subclass dispatching to a single backing worker thread, with bounded outstanding-request map, AbortSignal cancellation, structured timeout.
|
||||
- ADDED: `QueueDistributionWorkerThreadError` with codes `WT.WORKER_CRASHED`, `WT.MESSAGE_MALFORMED`, `WT.REQUEST_TIMEOUT`, `WT.WORKER_TERMINATED`.
|
||||
|
||||
### All three packages
|
||||
|
||||
- Test files import runner primitives from `@suaveplan/testing/runner` — never `vitest` or `bun:test` directly.
|
||||
- Feature-module barrels use explicit named re-exports — no `export *` outside the root `src/index.ts`.
|
||||
- mitata benchmarks under `src/__benchmarks__/`.
|
||||
- 100% coverage with `provider: "istanbul"`.
|
||||
- Sibling `.md` for every implementation file; root `README.md` with module index.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- `createWorkerForQueueJob(...)` adapter — deferred to a follow-up change once we have a concrete genesis consumer of both `@suaveplan/queue` and `@suaveplan/queue-distribution`. The proposal commits the core package to a self-contained `DistributedTask` contract so the adapter is purely additive when it lands.
|
||||
- WebSocket / gRPC communicators — neither shipped upstream; not requested.
|
||||
- `WorkerManager.rebalance()` — upstream body was `this.logger.warn("Rebalance operation not implemented")`. Per CLAUDE.md §Rules and AGENTS.md §2b, stubs do not ship to genesis. Removed; reopens via a separate change with a real algorithm. Documented in REMOVED section of spec delta.
|
||||
- Real browser-side CPU/memory metrics — upstream `LocalWorker.getHealth` already documented "could enhance with actual system metrics here" as a future TODO and returned mock values. Same baseline carried over; real metrics is a separate plugin concern.
|
||||
- Autoscale (`scale-up on depth threshold` / `scale-down on idle threshold`) — described in the now-superseded placeholder proposal but absent from upstream. Not migrated; opens via a separate genesis-original change if requested.
|
||||
|
||||
## Source mapping
|
||||
|
||||
This change derives primarily from `../suaveplan/packages/universal/queue-distribution/src/` because `../suaveplan/enterprise/queue-distribution.md` is a stub (Purpose section ends `"Worker pool and load balancing distribution layer fo..."` at L12 — truncated). Source line ranges below cite the upstream source tree.
|
||||
|
||||
| Requirement | Source lines | Notes |
|
||||
|---|---|---|
|
||||
| `Worker` interface (id/capacity/weight/status/activeTasks/executeTask/getHealth/start/stop/canAcceptTask/getInfo) | `workers/types/worker/worker.ts` L62-L126 | verbatim, generalised over `DistributedTask` |
|
||||
| `WorkerStatus` enum (ONLINE/BUSY/OFFLINE/UNHEALTHY) | `workers/types/worker/worker.ts` L7-L11 | verbatim |
|
||||
| `WorkerHealth` + metrics shape | `workers/types/worker/worker.ts` L16-L56 | verbatim |
|
||||
| `WorkerInfo` | `workers/types/worker/worker.ts` L131-L188 | verbatim |
|
||||
| `AbstractWorker` lifecycle + metrics | `workers/types/abstract/abstract.ts` L16-L314 | verbatim semantics; `any` catch → `unknown`; `uuid` → injectable `IdGenerator`; `Date.now()` → injectable `Clock`; logger inlined `ConsoleLogger` → injected `LoggerLike` |
|
||||
| `LocalWorker` in-process execution | `workers/types/local/local.ts` L1-L97 | verbatim |
|
||||
| `LoadBalancingStrategy` interface | `load-balancing/base.ts` L4-L14 | verbatim |
|
||||
| `AbstractLoadBalancingStrategy.filterAvailableWorkers` | `load-balancing/base.ts` L21-L28 | verbatim |
|
||||
| `RoundRobinStrategy` rotation + reset | `load-balancing/round-robin/roundRobin.ts` L10-L55 | verbatim |
|
||||
| `LeastBusyStrategy` (fewest active tasks + perf bias + sort fallback by capacity/weight) | `load-balancing/least-busy/leastBusy.ts` L10-L155 | verbatim |
|
||||
| `WeightedStrategy` weighted random | `load-balancing/weighted/weighted.ts` L10-L64 | `Math.random()` → injectable `Rng` |
|
||||
| `PeriodicHealthChecker` (intervals, thresholds, cache, timeout, recovery, mark-unhealthy) | `workers/health/healthChecker.ts` L62-L265 | verbatim; `Date.now()` → injectable `Clock`; `setTimeout` race → AbortSignal + `Clock.sleep` |
|
||||
| `WorkerPool` (CRUD, search, lifecycle, info) | `workers/management/pool/pool.ts` L13-L201 | verbatim |
|
||||
| `WorkerSearchCriteria` | `workers/management/pool/pool.ts` L206-L241 | verbatim |
|
||||
| `WorkerManager` (assignTask, assignTasks, health loop, restart, options) | `workers/management/manager/manager.ts` L61-L539 | verbatim semantics; retry-with-backoff routed through `@suaveplan/retry`; `setInterval` driven by injectable `Clock`; recursive retry replaced with bounded loop |
|
||||
| `WorkerCommunicator` interface | `workers/communication/worker/workerCommunicator.ts` L10-L61 | verbatim |
|
||||
| `AbstractWorkerCommunicator` | `workers/communication/worker/workerCommunicator.ts` L66-L200 | verbatim |
|
||||
| `HttpWorkerCommunicator` (fetch wrapper, `WorkerApiResponse<T>`, ping/health/status/sendTask) | `workers/communication/http/httpCommunicator.ts` L21-L174 | verbatim; `fetch` injectable; envelope Zod-validated |
|
||||
| `RemoteWorker` (connect/reconnect/health/sendTask + AbstractWorker integration) | `workers/types/remote/remote.ts` L10-L368 | verbatim semantics; backoff via `@suaveplan/retry`; non-recursive |
|
||||
| `RemoteWorkerOptions` | `workers/types/remote/remote.ts` L373-L383 | verbatim |
|
||||
| `Task<I,O>` reference shape (basis for `DistributedTask<I,O>`) | `../suaveplan/packages/universal/queue/src/core/base/task/task.types.ts` L7-L62 | reworded — genesis defines its own `DistributedTask` to decouple from `@suaveplan/queue.Job` model |
|
||||
| `TaskStatus` enum | `../suaveplan/packages/universal/queue/src/core/base/task/task.types.ts` L67-L74 | verbatim |
|
||||
| `TaskMetadata` shape | `../suaveplan/packages/universal/queue/src/core/base/task/task.types.ts` L79-L86 | verbatim |
|
||||
|
||||
### Requirements with no source line
|
||||
|
||||
| Requirement | Justification |
|
||||
|---|---|
|
||||
| 100% test coverage (statements/branches/functions/lines) | Genesis-wide policy (CLAUDE.md §Quality, AGENTS.md §7) |
|
||||
| Zod schemas at every public boundary | Genesis-wide policy (AGENTS.md §0a "design quality bar") |
|
||||
| `SuaveplanError` + stable code registry | Genesis-wide policy (AGENTS.md §0a anti-pattern row: replacing upstream errors with plain `Error`) |
|
||||
| OpenTelemetry counters/histograms/spans | Genesis-wide policy (CLAUDE.md §Quality) — upstream had none |
|
||||
| Injectable `Clock` / `Rng` / `IdGenerator` / `LoggerLike` / `fetch` | Genesis-wide policy: no `Math.random` / `Date.now` at module scope (CLAUDE.md §Rules, AGENTS.md §0a) |
|
||||
| Tests use `@suaveplan/testing/runner` wrapper | CLAUDE.md rule 11 / AGENTS.md §7 — applies repo-wide, not derivable from upstream |
|
||||
| Feature barrels use explicit named exports (no `export *`) | CLAUDE.md rule 10 / AGENTS.md §6a — applies repo-wide, not derivable from upstream |
|
||||
| Split into core + 2 transport plugins | User decision (this conversation): better tree-shaking for browser-only consumers; matches the modular plugin model called out in `enterprise/queue-distribution.md` §4 |
|
||||
| Decouple from `@suaveplan/queue` `Task<I,O>` — define `DistributedTask<I,O>` locally | Genesis `@suaveplan/queue` deliberately replaced upstream `Task` with `Job` / state-machine model when it was migrated; reintroducing `Task` in queue would create two parallel work-unit abstractions. Distribution becomes reusable for queue, batch, RPC, agent-dispatch via thin adapter (deferred). |
|
||||
| AbortSignal everywhere async | Genesis-wide policy (CLAUDE.md §Rules — "Backpressure & cancellation") |
|
||||
| mitata benchmarks under `src/__benchmarks__/` | Genesis-wide policy (AGENTS.md §9) |
|
||||
| `@suaveplan/queue-distribution-worker-thread` (Web-Worker / `worker_threads` transport) | New plugin; upstream had only an `AbstractWorkerCommunicator`+`HttpWorkerCommunicator` pair and never shipped a Web-Worker transport despite the docs mentioning "browser" worker type. Adding it now under the same plugin pattern as HTTP. |
|
||||
|
||||
## Parity Checklist
|
||||
|
||||
| Capability | Upstream | Migrated | Notes |
|
||||
|---|---|---|---|
|
||||
| `Worker` interface + `WorkerStatus` + `WorkerHealth` + `WorkerInfo` | ✅ | ✅ | |
|
||||
| `AbstractWorker` lifecycle, metrics, FSM | ✅ | ⬆️ | Enhanced: injectable `Clock`/`IdGenerator`/`LoggerLike`; `uuid` dep dropped; `any` catches narrowed to `unknown`+`SuaveplanError` |
|
||||
| `LocalWorker` | ✅ | ✅ | |
|
||||
| `LoadBalancingStrategy` interface | ✅ | ✅ | |
|
||||
| `RoundRobinStrategy` | ✅ | ✅ | |
|
||||
| `LeastBusyStrategy` + performance bias | ✅ | ✅ | |
|
||||
| `WeightedStrategy` weighted random | ✅ | ⬆️ | Enhanced: `Math.random` → injectable `Rng` for deterministic tests |
|
||||
| `PeriodicHealthChecker` thresholds + cache + timeout | ✅ | ⬆️ | Enhanced: `setTimeout` race → AbortSignal + injectable `Clock`; OTel histograms; structured errors |
|
||||
| `WorkerPool` CRUD + search + bulk start/stop | ✅ | ✅ | |
|
||||
| `WorkerManager` assign + retry + health loop + restart | ✅ | ⬆️ | Enhanced: retry-with-backoff via `@suaveplan/retry`; OTel spans; recursive retry replaced with bounded loop; AbortSignal threaded |
|
||||
| `WorkerCommunicator` interface | ✅ | ✅ | Moved to `@suaveplan/queue-distribution-http` (transport-specific) |
|
||||
| `AbstractWorkerCommunicator` | ✅ | ✅ | Moved to `@suaveplan/queue-distribution-http` |
|
||||
| `HttpWorkerCommunicator` | ✅ | ⬆️ | Enhanced: injectable `fetch`; Zod-validated envelope; structured `QueueDistributionHttpError`; AbortSignal |
|
||||
| `RemoteWorker` connect/reconnect/health | ✅ | ⬆️ | Enhanced: connect backoff via `@suaveplan/retry`; non-recursive reconnect; moved to `@suaveplan/queue-distribution-http` |
|
||||
| `RemoteWorkerOptions` | ✅ | ✅ | Moved to `@suaveplan/queue-distribution-http` |
|
||||
| Web-Worker / `worker_threads` transport | 🚫 | ⬆️ | Upstream had **no** Web-Worker transport (despite docs referencing "browser" worker type). Genesis adds `@suaveplan/queue-distribution-worker-thread` as a new transport plugin. Marked ⬆️ because it extends the surface upstream documented but never shipped. |
|
||||
| `WorkerManager.rebalance()` | ✅ | 🚫 | Scoped out: upstream body was `this.logger.warn("Rebalance operation not implemented")` — a stub. Per CLAUDE.md §Rules and AGENTS.md §2b, stubs do not ship to genesis. Reopens via a separate change with a real algorithm. |
|
||||
| OTel instrumentation | 🚫 | ⬆️ | Added per genesis enterprise standard. |
|
||||
| Zod boundary schemas | 🚫 | ⬆️ | Added per genesis enterprise standard. |
|
||||
| `SuaveplanError` + code registry | 🚫 | ⬆️ | Replaces plain `throw new Error(...)`. |
|
||||
| AbortSignal-aware async surface | 🚫 | ⬆️ | Added per genesis enterprise standard. |
|
||||
|
||||
## Verification approach
|
||||
|
||||
- `src/load-balancing/round-robin/round-robin.test.ts` — rotation, reset, pool-shrink invariants, empty-pool null return.
|
||||
- `src/load-balancing/least-busy/least-busy.test.ts` — fewest-active-tasks selection, performance-bias tie-break, capacity tie-break, weight tie-break, `notifyTaskCompleted` perf-metric update, `resetPerformanceMetrics`.
|
||||
- `src/load-balancing/weighted/weighted.test.ts` — distribution proportional to weight (with injected deterministic `Rng`), zero-weight fallback to uniform, single-worker fast path, fast-check property suite.
|
||||
- `src/load-balancing/base/base.test.ts` — `filterAvailableWorkers` filters by status and `canAcceptTask`.
|
||||
- `src/worker/abstract/abstract.test.ts` — start/stop/executeTask happy path, OFFLINE/UNHEALTHY refusal, force-stop active tasks, metrics accumulation, `markUnhealthy`, `getInfo` shape.
|
||||
- `src/worker/abstract/abstract.spec-conformance.test.ts` — generic `<I,O>` threading via `expectTypeOf`, OTel histogram + counter emission via real in-memory `MeterProvider`, `Clock`/`IdGenerator` injection.
|
||||
- `src/worker/local/local.test.ts` — task delegation, capability reporting, health pass-through.
|
||||
- `src/health-checker/health-checker.test.ts` — consecutive-failure unhealthy promotion, consecutive-success recovery, cache TTL, timeout via injected `Clock`, error → fallback unhealthy.
|
||||
- `src/pool/pool.test.ts` — add/remove/get/findWorkers with every criteria field, duplicate-add throws `WORKER_DUPLICATE`, bulk start/stop, clear.
|
||||
- `src/manager/manager.test.ts` — assignTask happy path, balancer null → throws `STRATEGY_NO_WORKER_SELECTED`, retry exhaustion → throws `TASK_RETRIES_EXHAUSTED`, failed-tasks set populated, options updates restart health loop, restart limit exhausted → throws `WORKER_RESTART_EXHAUSTED`.
|
||||
- `src/manager/manager.spec-conformance.test.ts` — OTel span + counter assertions, AbortSignal cancellation mid-retry → throws `TASK_CANCELLED`, `Clock.sleep` honoured for retry delay.
|
||||
- `src/errors/queue-distribution-error.test.ts` — every registered code is thrown by at least one production call path.
|
||||
- `tests/dep-graph.test.ts` — core has no transport-package dep; http plugin has core dep; worker-thread plugin has core dep.
|
||||
|
||||
For `@suaveplan/queue-distribution-http`:
|
||||
|
||||
- `src/communicator/http/http-communicator.test.ts` — sendTask happy path, response envelope Zod rejection, fetch throw, AbortSignal, ping latency, headers passthrough.
|
||||
- `src/worker/remote/remote.test.ts` — connect happy path, connect retries with injected `Clock`, max-attempts exhausted → throws `HTTP.CONNECT_FAILED`, periodic health driven by `Clock`, unreachable → `markUnhealthy`.
|
||||
|
||||
For `@suaveplan/queue-distribution-worker-thread`:
|
||||
|
||||
- `src/communicator/worker-thread/worker-thread-communicator.test.ts` — message correlation, timeout, crash-event → `WT.WORKER_CRASHED`, malformed reply → `WT.MESSAGE_MALFORMED`.
|
||||
- `src/worker/worker-thread/worker-thread-worker.test.ts` — task dispatch, bounded outstanding map, terminate cleanup → `WT.WORKER_TERMINATED`.
|
||||
|
||||
## Migration impact
|
||||
|
||||
Additive only — no existing genesis consumers (`grep -rn "@suaveplan/queue-distribution" packages/ apps/` returns zero).
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
`bunx turbo ls --filter=...@suaveplan/queue-distribution` — none on genesis at the time this proposal is opened.
|
||||
|
||||
- No dependents (verified — `@suaveplan/queue-distribution` does not yet exist on genesis).
|
||||
- `@suaveplan/queue-distribution-http` and `@suaveplan/queue-distribution-worker-thread` are new packages introduced in this change.
|
||||
@@ -0,0 +1,105 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: queue-distribution-http (enterprise-queue-distribution)
|
||||
|
||||
**Package**: `@suaveplan/queue-distribution-http`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. (See `openspec/AGENTS.md` Section 13.)
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Worker communicator interface and abstract base
|
||||
|
||||
The package SHALL expose `WorkerCommunicator` with `sendTask<I, O>(endpoint, task, input, signal?): Promise<O>`, `checkStatus(endpoint, signal?): Promise<WorkerStatus>`, `checkHealth(endpoint, signal?): Promise<WorkerHealth>`, `connect(endpoint, signal?): Promise<void>`, `disconnect(endpoint, signal?): Promise<void>`, `ping(endpoint, timeoutMs?, signal?): Promise<number>`, `getProtocol(): string`. `AbstractWorkerCommunicator` SHALL provide constructor-injected `defaultTimeoutMs` and `headers`, plus protected `serialize`/`deserialize`/`createTimeout`/`withTimeout` helpers. The timeout race SHALL use an injected `Clock`, not `setTimeout` at module scope.
|
||||
|
||||
#### Scenario: withTimeout honours injected Clock
|
||||
|
||||
- **WHEN** a `FakeClock` is injected and a never-resolving promise is passed to `withTimeout(promise, 100)`
|
||||
- **THEN** after advancing the clock by 100 ms, `withTimeout` rejects with `QueueDistributionHttpError(HTTP.TIMEOUT)`
|
||||
|
||||
**Verified by:** `src/communicator/base/base.test.ts::"withTimeout rejects with HTTP.TIMEOUT after clock advances past deadline"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: HTTP communicator with injectable fetch and Zod-validated wire envelope
|
||||
|
||||
The package SHALL ship `HttpWorkerCommunicator` extending `AbstractWorkerCommunicator`. The class SHALL accept an injectable `fetch` function (default: `globalThis.fetch`). All responses SHALL be parsed against a Zod schema `WorkerApiResponseSchema<T>` shaped `{ success: boolean; data?: T; error?: { code: string; message: string; details?: unknown } }`. Non-conformant responses SHALL throw `QueueDistributionHttpError(HTTP.RESPONSE_MALFORMED)`. HTTP non-2xx OR `success === false` SHALL throw `QueueDistributionHttpError(HTTP.PROTOCOL_ERROR)` carrying the upstream `error.code` and `error.message` on `cause`. Network-layer `fetch` rejections SHALL throw `QueueDistributionHttpError(HTTP.REQUEST_FAILED)` with the original error on `cause`. The implementation SHALL forward AbortSignal to `fetch`. `getProtocol()` SHALL return `"http"`. `sendTask` SHALL POST `{ taskId, taskMetadata, input }` to `<endpoint>/execute`. `checkStatus` SHALL GET `<endpoint>/status`. `checkHealth` SHALL GET `<endpoint>/health`. `ping` SHALL GET `<endpoint>/ping` and return the round-trip duration via the injected `Clock`.
|
||||
|
||||
#### Scenario: sendTask happy path
|
||||
|
||||
- **WHEN** an injected `fetch` returns `{ success: true, data: <O> }` at `<endpoint>/execute`
|
||||
- **THEN** `sendTask` resolves with the typed `<O>` value AND the `queue_dist_http.request.duration_ms` histogram emits one observation
|
||||
|
||||
#### Scenario: response envelope rejection
|
||||
|
||||
- **WHEN** an injected `fetch` returns `{ foo: 1 }` (no `success` field)
|
||||
- **THEN** `sendTask` rejects with `QueueDistributionHttpError(HTTP.RESPONSE_MALFORMED)`
|
||||
|
||||
#### Scenario: protocol error carries upstream code
|
||||
|
||||
- **WHEN** an injected `fetch` returns 500 with body `{ success: false, error: { code: "INTERNAL", message: "boom" } }`
|
||||
- **THEN** `sendTask` rejects with `QueueDistributionHttpError(HTTP.PROTOCOL_ERROR)` and `error.cause.message === "boom"`
|
||||
|
||||
#### Scenario: AbortSignal forwarded to fetch
|
||||
|
||||
- **WHEN** `sendTask(endpoint, task, input, signal)` is called and `signal.abort()` fires before fetch resolves
|
||||
- **THEN** the call rejects, the injected fetch sees `init.signal.aborted === true`, AND the error code is `HTTP.REQUEST_FAILED` with `cause.name === "AbortError"`
|
||||
|
||||
**Verified by:** `src/communicator/http/http-communicator.test.ts::"sendTask resolves with typed data on success"` AND `src/communicator/http/http-communicator.test.ts::"rejects with HTTP.RESPONSE_MALFORMED when envelope shape is wrong"` AND `src/communicator/http/http-communicator.test.ts::"rejects with HTTP.PROTOCOL_ERROR and carries upstream message on cause"` AND `src/communicator/http/http-communicator.test.ts::"forwards AbortSignal to the injected fetch"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: RemoteWorker — connect/reconnect via `@suaveplan/retry`
|
||||
|
||||
The package SHALL ship `RemoteWorker` extending `AbstractWorker` from `@suaveplan/queue-distribution`. Constructor accepts `endpoint`, `communicator: WorkerCommunicator`, `capacity`, `weight`, optional `id`, optional logger, and `RemoteWorkerOptions { healthCheckIntervalMs?, maxConnectionAttempts? }`. `doStart()` SHALL invoke `connect()` and start a periodic health-check loop on the injected `Clock`. `connect()` SHALL use `@suaveplan/retry` exponential-backoff between attempts (initial delay 1,000 ms, factor 2, max 10,000 ms, max attempts from `maxConnectionAttempts` default 3). The retry SHALL NOT recurse — it MUST use a bounded loop. `doStop()` SHALL stop the health-check loop and `disconnect` the communicator; disconnect errors SHALL be logged but NOT re-thrown. `doExecuteTask` SHALL throw `HTTP.CONNECT_FAILED` when `isConnected === false`. Errors during `communicator.sendTask` SHALL be routed through `handleCommunicationError(error)`: ping with a 5,000 ms timeout; if ping succeeds, the error propagates and the worker stays in current status; if ping fails, the worker is `markUnhealthy("Unreachable")`, `isConnected = false`, and a single non-recursive `attemptConnection()` is scheduled. `getHealth()` SHALL fetch real health from the remote worker; failures SHALL log a warning and return a default unhealthy payload. `getCapabilities()` SHALL return `{ isRemote: true, protocol, endpoint }`. `getType()` SHALL return `"remote"`.
|
||||
|
||||
#### Scenario: connect succeeds on first try
|
||||
|
||||
- **WHEN** `worker.start()` is called and `communicator.connect`/`checkStatus` resolve with `WorkerStatus.ONLINE`
|
||||
- **THEN** `worker.status === ONLINE` AND the connection-attempts counter is back at 0
|
||||
|
||||
#### Scenario: connect retries with exponential backoff and exhausts
|
||||
|
||||
- **WHEN** `communicator.connect` rejects on every call under `maxConnectionAttempts === 3` and the test injects a `FakeClock`
|
||||
- **THEN** `worker.start()` rejects with `QueueDistributionHttpError(HTTP.CONNECT_FAILED)` AND the `FakeClock` recorded sleeps of 1,000 ms then 2,000 ms (no recursion stack growth)
|
||||
|
||||
#### Scenario: unreachable remote marks worker UNHEALTHY
|
||||
|
||||
- **WHEN** `sendTask` throws and a subsequent ping rejects
|
||||
- **THEN** `worker.status === UNHEALTHY`, `isConnected === false`, AND a follow-up `attemptConnection` is scheduled (not awaited)
|
||||
|
||||
**Verified by:** `src/worker/remote/remote.test.ts::"start resolves with ONLINE on first connect"` AND `src/worker/remote/remote.test.ts::"exhausts retries and throws HTTP.CONNECT_FAILED under fake clock"` AND `src/worker/remote/remote.test.ts::"marks worker UNHEALTHY and schedules reconnect when ping fails"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Stable error registry for the HTTP plugin
|
||||
|
||||
The package SHALL register every error code through `defineCodes("HTTP", [...])` from `@suaveplan/error` and expose `QueueDistributionHttpError extends SuaveplanError<HttpErrorCode>`. The exhaustive set is: `REQUEST_FAILED`, `PROTOCOL_ERROR`, `TIMEOUT`, `RESPONSE_MALFORMED`, `CONNECT_FAILED`, `UNREACHABLE`. Every code SHALL be thrown by ≥1 production call path.
|
||||
|
||||
#### Scenario: every registered code is thrown
|
||||
|
||||
- **WHEN** the test suite iterates over every code in the registry
|
||||
- **THEN** a corresponding production throw site exists
|
||||
|
||||
**Verified by:** `src/errors/queue-distribution-http-error.test.ts::"every registered HTTP.* code is thrown by a production call path"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: OpenTelemetry instrumentation
|
||||
|
||||
The package SHALL emit, via `@suaveplan/opentelemetry` under meter name `"@suaveplan/queue-distribution-http"`:
|
||||
|
||||
- histogram: `queue_dist_http.request.duration_ms` (attributes `endpoint`, `operation: "sendTask" | "checkStatus" | "checkHealth" | "ping"`, `outcome: "success" | "failure"`)
|
||||
- counter: `queue_dist_http.request_total` (same attributes)
|
||||
- counter: `queue_dist_http.connect.attempts_total` (attribute `outcome: "success" | "failure" | "exhausted"`)
|
||||
- span: `queue_dist_http.sendTask`, `queue_dist_http.connect`
|
||||
|
||||
#### Scenario: histogram emission
|
||||
|
||||
- **WHEN** `sendTask` resolves successfully under an in-memory `MeterProvider`
|
||||
- **THEN** one observation is recorded on `queue_dist_http.request.duration_ms` with `outcome === "success"` AND `operation === "sendTask"`
|
||||
|
||||
**Verified by:** `src/communicator/http/http-communicator.spec-conformance.test.ts::"emits queue_dist_http.request.duration_ms histogram with operation and outcome attributes"`
|
||||
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
@@ -0,0 +1,83 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: queue-distribution-worker-thread (enterprise-queue-distribution)
|
||||
|
||||
**Package**: `@suaveplan/queue-distribution-worker-thread`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. (See `openspec/AGENTS.md` Section 13.)
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: WorkerFactory-driven communicator covering Web-Worker and `worker_threads`
|
||||
|
||||
The package SHALL expose `WorkerThreadCommunicator` implementing `WorkerCommunicator` from `@suaveplan/queue-distribution-http` (re-exported `WorkerCommunicator` interface). The constructor SHALL accept an injectable `WorkerFactory` of shape `() => { postMessage(msg, transfer?); addEventListener(kind, handler); terminate(); }` so the same class works against `globalThis.Worker` (browser) and a thin `worker_threads` wrapper (Node/Bun) without importing platform APIs at module scope. The implementation SHALL maintain a `Map<requestId, { resolve, reject, timeoutHandle }>` of outstanding requests. Each outgoing message SHALL include an `id` (generated via injected `IdGenerator`), a `kind: "task" | "status" | "health" | "ping"`, and a `payload` validated by Zod. Reply correlation SHALL match `id`; unknown `id` replies SHALL be logged and dropped. Reply payloads SHALL be Zod-validated against per-kind schemas; failures SHALL reject the in-flight request with `QueueDistributionWorkerThreadError(WT.MESSAGE_MALFORMED)`. Each request SHALL be subject to a per-call timeout (default `defaultTimeoutMs`) honoured by the injected `Clock`; expiry SHALL reject with `WT.REQUEST_TIMEOUT` and remove the entry from the map.
|
||||
|
||||
#### Scenario: round-trip correlation
|
||||
|
||||
- **WHEN** `sendTask`/`checkStatus`/`checkHealth`/`ping` are invoked concurrently and the underlying worker replies in arbitrary order
|
||||
- **THEN** each promise resolves with its own reply payload (matched by `id`)
|
||||
|
||||
#### Scenario: malformed reply throws WT.MESSAGE_MALFORMED
|
||||
|
||||
- **WHEN** the underlying worker posts back `{ id: <known>, kind: "task", payload: { something: "bad" } }` not conforming to the task-reply Zod schema
|
||||
- **THEN** the in-flight `sendTask` promise rejects with `QueueDistributionWorkerThreadError(WT.MESSAGE_MALFORMED)`
|
||||
|
||||
#### Scenario: request timeout
|
||||
|
||||
- **WHEN** a `FakeClock` is injected and the worker never replies within `defaultTimeoutMs`
|
||||
- **THEN** after the clock advances past the deadline, the call rejects with `QueueDistributionWorkerThreadError(WT.REQUEST_TIMEOUT)` AND the outstanding-request map no longer contains the entry
|
||||
|
||||
**Verified by:** `src/communicator/worker-thread/worker-thread-communicator.test.ts::"correlates concurrent replies by id"` AND `src/communicator/worker-thread/worker-thread-communicator.test.ts::"rejects with WT.MESSAGE_MALFORMED when reply fails Zod validation"` AND `src/communicator/worker-thread/worker-thread-communicator.test.ts::"rejects with WT.REQUEST_TIMEOUT and clears the outstanding-request map"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: WorkerThreadWorker — `AbstractWorker` subclass dispatching to a backing worker
|
||||
|
||||
The package SHALL ship `WorkerThreadWorker` extending `AbstractWorker` from `@suaveplan/queue-distribution`. Constructor SHALL accept an injectable `WorkerFactory` and instantiate a fresh `WorkerThreadCommunicator` per worker, AND a single backing worker thread. `doStart()` SHALL register error / exit listeners on the backing worker — `worker.exit` events SHALL `markUnhealthy("Worker exited")` and flip `status` to `OFFLINE`; uncaught `error` events SHALL emit the `WT.WORKER_CRASHED` OTel counter. `doStop(force)` SHALL `terminate()` the backing worker, fail every outstanding request with `WT.WORKER_TERMINATED`, and clear the outstanding-request map. `doExecuteTask` SHALL delegate to `communicator.sendTask(<no-endpoint>, task, input, signal)`. `getCapabilities()` SHALL return `{ isRemote: false, isThreaded: true }`. `getType()` SHALL return `"worker-thread"`.
|
||||
|
||||
#### Scenario: backing worker crash promotes to UNHEALTHY
|
||||
|
||||
- **WHEN** the backing worker emits an `error` event
|
||||
- **THEN** the OTel `queue_dist_wt.worker.crashed_total` counter increments and the worker is marked UNHEALTHY
|
||||
|
||||
#### Scenario: terminate fails outstanding requests with WT.WORKER_TERMINATED
|
||||
|
||||
- **WHEN** two `sendTask` calls are in flight and `worker.stop(true)` is invoked
|
||||
- **THEN** both pending promises reject with `QueueDistributionWorkerThreadError(WT.WORKER_TERMINATED)` AND `worker.status === OFFLINE`
|
||||
|
||||
**Verified by:** `src/worker/worker-thread/worker-thread-worker.test.ts::"backing worker crash flips status to UNHEALTHY and increments WT.WORKER_CRASHED counter"` AND `src/worker/worker-thread/worker-thread-worker.test.ts::"terminate fails outstanding requests with WT.WORKER_TERMINATED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Stable error registry for the worker-thread plugin
|
||||
|
||||
The package SHALL register every error code through `defineCodes("WT", [...])` from `@suaveplan/error` and expose `QueueDistributionWorkerThreadError extends SuaveplanError<WtErrorCode>`. The exhaustive set is: `WORKER_CRASHED`, `MESSAGE_MALFORMED`, `REQUEST_TIMEOUT`, `WORKER_TERMINATED`. Every code SHALL be thrown by ≥1 production call path.
|
||||
|
||||
#### Scenario: every registered code is thrown
|
||||
|
||||
- **WHEN** the test suite iterates over every code in the registry
|
||||
- **THEN** a corresponding production throw site exists
|
||||
|
||||
**Verified by:** `src/errors/queue-distribution-worker-thread-error.test.ts::"every registered WT.* code is thrown by a production call path"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: OpenTelemetry instrumentation
|
||||
|
||||
The package SHALL emit, via `@suaveplan/opentelemetry` under meter name `"@suaveplan/queue-distribution-worker-thread"`:
|
||||
|
||||
- histogram: `queue_dist_wt.request.duration_ms` (attributes `kind`, `outcome`)
|
||||
- counter: `queue_dist_wt.request_total` (attributes `kind`, `outcome`)
|
||||
- counter: `queue_dist_wt.worker.crashed_total`
|
||||
- counter: `queue_dist_wt.worker.terminated_total`
|
||||
- span: `queue_dist_wt.sendTask`
|
||||
|
||||
#### Scenario: histogram emission on sendTask round-trip
|
||||
|
||||
- **WHEN** `sendTask` resolves successfully under an in-memory `MeterProvider`
|
||||
- **THEN** one observation is recorded on `queue_dist_wt.request.duration_ms` with `kind === "task"` AND `outcome === "success"`
|
||||
|
||||
**Verified by:** `src/communicator/worker-thread/worker-thread-communicator.spec-conformance.test.ts::"emits queue_dist_wt.request.duration_ms histogram with kind and outcome attributes"`
|
||||
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
@@ -0,0 +1,309 @@
|
||||
<!-- markdownlint-disable MD024 -->
|
||||
|
||||
# Capability Spec Delta: queue-distribution (enterprise-queue-distribution)
|
||||
|
||||
**Package**: `@suaveplan/queue-distribution`
|
||||
**Status**: PROPOSED
|
||||
|
||||
> Every requirement below carries a `**Verified by:**` line at authoring time. (See `openspec/AGENTS.md` Section 13.)
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Distributed task contract decoupled from `@suaveplan/queue.Job`
|
||||
|
||||
The package SHALL expose `DistributedTask<I, O>` as the unit of work executed by workers, independent of `@suaveplan/queue`'s `Job` / state-machine model. The contract has: `id: string`, `execute(input: I, context: ExecutionContext): Promise<O>`, `getMetadata(): TaskMetadata`, `getStatus(): TaskStatus`, `setStatus(status: TaskStatus): void`. `TaskMetadata` includes `createdAt: Date`, optional `startedAt`/`completedAt`, `attempts: number`, optional `tags: string[]`, and indexed extras. `TaskStatus` is the closed enum `PENDING | RUNNING | COMPLETED | FAILED | CANCELLED | TIMED_OUT`. `ExecutionContext` is caller-defined.
|
||||
|
||||
#### Scenario: typed metadata round-trip
|
||||
|
||||
- **WHEN** a caller constructs a `DistributedTask<I, O>` literal and passes it through `Worker.executeTask`
|
||||
- **THEN** `task.getMetadata()` returns a value validated by `TaskMetadataSchema` and `getStatus()` reflects the lifecycle (`PENDING` → `RUNNING` → `COMPLETED`/`FAILED`)
|
||||
|
||||
#### Scenario: generics are preserved end-to-end
|
||||
|
||||
- **WHEN** a caller writes `const w: Worker; const t: DistributedTask<number, string>; const r = await w.executeTask(t, 42)`
|
||||
- **THEN** the inferred type of `r` is exactly `string` (no `unknown`, no `any`)
|
||||
|
||||
**Verified by:** `src/task/task.test.ts::"validates TaskMetadata via TaskMetadataSchema"` AND `src/task/task.spec-conformance.test.ts::"preserves <I, O> generics through Worker.executeTask"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Worker interface and status model
|
||||
|
||||
The package SHALL expose a `Worker` interface with: `id: string`, `capacity: number`, `weight: number`, `status: WorkerStatus`, `activeTasks: number`, `executeTask<I, O>(task, input, signal?): Promise<O>`, `getHealth(signal?): Promise<WorkerHealth>`, `start(signal?): Promise<void>`, `stop(force?, signal?): Promise<void>`, `canAcceptTask(): boolean`, `getInfo(): WorkerInfo`. `WorkerStatus` is the closed enum `ONLINE | BUSY | OFFLINE | UNHEALTHY`. `WorkerHealth` has `isHealthy: boolean` and a `metrics` object containing at least `cpu`, `memory`, `tasksFailed`, `tasksSucceeded`, `responseTime` plus open-ended string-keyed extras. `WorkerInfo` returns identity, type label, capacity, weight, status, active count, `startedAt?`, `lastHealthCheck?`, `lastHealthStatus?`, and `capabilities?`.
|
||||
|
||||
#### Scenario: health and info shapes parse via Zod
|
||||
|
||||
- **WHEN** a `Worker` implementation returns from `getHealth()` / `getInfo()`
|
||||
- **THEN** the value passes `WorkerHealthSchema` / `WorkerInfoSchema` (failures throw `QueueDistributionError(QUEUE_DIST.INVALID_OPTIONS)`)
|
||||
|
||||
#### Scenario: canAcceptTask honours status and capacity
|
||||
|
||||
- **WHEN** `status === ONLINE` and `activeTasks < capacity`
|
||||
- **THEN** `canAcceptTask()` returns `true`
|
||||
- **WHEN** `status === OFFLINE` or `activeTasks >= capacity`
|
||||
- **THEN** `canAcceptTask()` returns `false`
|
||||
|
||||
**Verified by:** `src/worker/worker.test.ts::"WorkerHealthSchema rejects malformed health"` AND `src/worker/worker.test.ts::"canAcceptTask honours ONLINE+capacity and rejects OFFLINE"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: AbstractWorker lifecycle and metrics, no module-scope clocks
|
||||
|
||||
The package SHALL ship an `AbstractWorker` base class implementing `Worker` with: idempotent `start()` (no-op when not OFFLINE) emitting `WorkerStatus.ONLINE`; `stop(force)` rejecting when `activeTasks > 0 && !force` with `QUEUE_DIST.WORKER_BUSY_STOP`; `executeTask` tracking `activeTasks`, success/failure counters, response-time total, and toggling `status` between ONLINE and BUSY; `markUnhealthy(reason)` flipping `status` to UNHEALTHY; `getInfo()` aggregating all of the above. The class SHALL accept `Clock`, `IdGenerator`, and `LoggerLike` via constructor options — no `Date.now()`, no `Math.random()`, no `uuid` import at module scope.
|
||||
|
||||
#### Scenario: executeTask refuses OFFLINE workers
|
||||
|
||||
- **WHEN** `await worker.executeTask(task, input)` is called while `status === OFFLINE`
|
||||
- **THEN** the call rejects with `QueueDistributionError` whose code is `QUEUE_DIST.WORKER_NOT_AVAILABLE`
|
||||
|
||||
#### Scenario: force stop cancels active tasks
|
||||
|
||||
- **WHEN** `await worker.stop(true)` is called with `activeTasks === 2`
|
||||
- **THEN** `worker.status === OFFLINE` AND `worker.activeTasks === 0`
|
||||
|
||||
#### Scenario: metrics emit through injectable Clock
|
||||
|
||||
- **WHEN** a `FakeClock` returns `1000` at task start and `1750` at completion
|
||||
- **THEN** `getHealth()` returns `metrics.responseTime === 750` (the difference, not wall-clock)
|
||||
|
||||
**Verified by:** `src/worker/abstract/abstract.test.ts::"executeTask refuses OFFLINE worker with WORKER_NOT_AVAILABLE"` AND `src/worker/abstract/abstract.test.ts::"force stop drains activeTasks and transitions to OFFLINE"` AND `src/worker/abstract/abstract.spec-conformance.test.ts::"metrics derive responseTime from injected Clock"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: LocalWorker in-process execution
|
||||
|
||||
The package SHALL ship `LocalWorker<Ctx>` extending `AbstractWorker`. `doExecuteTask` SHALL invoke `task.execute(input, this.context)` where `context` is the `ExecutionContext` passed at construction. `getCapabilities()` SHALL return `{ hasFileSystemAccess: true, isRemote: false, contextType: <ctor name> }`. `getType()` SHALL return `"local"`.
|
||||
|
||||
#### Scenario: task receives the configured context
|
||||
|
||||
- **WHEN** `new LocalWorker(ctx).executeTask(task, input)` is called
|
||||
- **THEN** `task.execute(input, ctx)` is invoked with the exact same `ctx` reference
|
||||
|
||||
**Verified by:** `src/worker/local/local.test.ts::"executeTask passes the configured ExecutionContext to task.execute"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Load-balancing strategy interface
|
||||
|
||||
The package SHALL expose `LoadBalancingStrategy` with `selectWorker(workers, task?): Worker | null`, optional `notifyWorkerSelected(worker, task)`, optional `notifyTaskCompleted(worker, task, success, durationMs)`, optional `getName(): string`. `AbstractLoadBalancingStrategy` provides `filterAvailableWorkers(workers): Worker[]` returning workers with `status === ONLINE || status === BUSY`, `activeTasks < capacity`, and `canAcceptTask() === true`.
|
||||
|
||||
#### Scenario: filterAvailableWorkers excludes OFFLINE and at-capacity workers
|
||||
|
||||
- **WHEN** the input list contains workers in every `WorkerStatus` state and varying `activeTasks`
|
||||
- **THEN** the output contains only those satisfying the three predicates above
|
||||
|
||||
**Verified by:** `src/load-balancing/base/base.test.ts::"filterAvailableWorkers returns only ONLINE/BUSY workers under capacity"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Round-robin strategy
|
||||
|
||||
The package SHALL ship `RoundRobinStrategy` that selects workers in rotation order. `currentIndex` SHALL wrap to 0 when reaching the end of the available-worker list, and SHALL clamp to 0 if the available-worker list shrinks below `currentIndex`. `reset()` SHALL set `currentIndex` to 0. `selectWorker` SHALL return `null` when no workers are available. `getName()` SHALL return `"round-robin"`.
|
||||
|
||||
#### Scenario: rotation order and reset
|
||||
|
||||
- **WHEN** `selectWorker` is invoked 4 times against the same 3-worker list
|
||||
- **THEN** the selections are workers `[0, 1, 2, 0]` in order
|
||||
- **WHEN** `reset()` is then called and `selectWorker` invoked once more
|
||||
- **THEN** the selection is worker `[0]` again
|
||||
|
||||
#### Scenario: pool shrink is safe
|
||||
|
||||
- **WHEN** `currentIndex === 2` and the available-worker list shrinks to `length === 1`
|
||||
- **THEN** the next `selectWorker` returns worker `[0]` (no out-of-range read)
|
||||
|
||||
**Verified by:** `src/load-balancing/round-robin/round-robin.test.ts::"cycles through workers in order and wraps after the end"` AND `src/load-balancing/round-robin/round-robin.test.ts::"clamps currentIndex when the pool shrinks below it"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Least-busy strategy with optional performance bias
|
||||
|
||||
The package SHALL ship `LeastBusyStrategy` selecting the worker with the fewest `activeTasks`. When `usePerformanceMetrics` is `true`, ties on `activeTasks` SHALL break by lower `avgExecutionTime`. Further ties SHALL break by higher `capacity` then higher `weight`. `notifyTaskCompleted(worker, task, success, durationMs)` SHALL update an internal EWMA-style cumulative average for successful runs only. `getWorkerPerformance(id)` SHALL return the stored record or `null`. `resetPerformanceMetrics()` SHALL clear the registry. `setUsePerformanceMetrics(enabled)` SHALL toggle the flag at runtime.
|
||||
|
||||
#### Scenario: tie-break by performance when enabled
|
||||
|
||||
- **WHEN** two workers both have `activeTasks === 0`, perf metrics enabled, and recorded average durations of 100 ms and 200 ms
|
||||
- **THEN** `selectWorker` returns the 100 ms worker
|
||||
|
||||
#### Scenario: tie-break by capacity when perf metrics unavailable
|
||||
|
||||
- **WHEN** two workers both have `activeTasks === 0` and no perf history, with `capacity` 2 and 4
|
||||
- **THEN** `selectWorker` returns the `capacity === 4` worker
|
||||
|
||||
**Verified by:** `src/load-balancing/least-busy/least-busy.test.ts::"ties break by lower average execution time when usePerformanceMetrics is on"` AND `src/load-balancing/least-busy/least-busy.test.ts::"ties break by higher capacity when no performance history exists"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Weighted strategy with injectable RNG
|
||||
|
||||
The package SHALL ship `WeightedStrategy` that selects workers with probability proportional to `weight`. The RNG SHALL be supplied via constructor option (`Rng` from `@suaveplan/types`); the strategy SHALL NOT call `Math.random()`. When `totalWeight <= 0`, the strategy SHALL fall back to uniform selection. `getName()` SHALL return `"weighted"`.
|
||||
|
||||
#### Scenario: deterministic distribution under fixed RNG
|
||||
|
||||
- **WHEN** a deterministic `Rng` yields the sequence `[0.0, 0.5, 0.9]` against weights `[1, 3, 6]` (total 10)
|
||||
- **THEN** `selectWorker` returns workers `[0, 1, 2]` in that order
|
||||
|
||||
#### Scenario: zero-weight pool falls back to uniform
|
||||
|
||||
- **WHEN** every worker has `weight === 0` and the `Rng` yields `0.5`
|
||||
- **THEN** `selectWorker` returns the middle worker in the list
|
||||
|
||||
**Verified by:** `src/load-balancing/weighted/weighted.test.ts::"selects workers proportionally under a deterministic Rng"` AND `src/load-balancing/weighted/weighted.test.ts::"falls back to uniform when total weight is zero"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Periodic health checker
|
||||
|
||||
The package SHALL ship `WorkerHealthChecker` (interface) and `PeriodicHealthChecker` (implementation). `checkHealth(worker)` SHALL race the worker's `getHealth()` against an injectable-`Clock`-driven timeout (default 5,000 ms). On success, results SHALL be cached for `cacheTimeMs` (default 5,000 ms) when `cacheResults` is true. Consecutive failures (`unhealthyThreshold`, default 3) SHALL flip `worker.status` to `UNHEALTHY`; consecutive successes (`healthyThreshold`, default 2) SHALL flip `UNHEALTHY` back to `ONLINE` or `BUSY` based on `activeTasks`. `getHealthCheckCounts(id)`, `resetHealthCheckCounts(id)`, and `clearCache()` SHALL be exposed.
|
||||
|
||||
#### Scenario: unhealthy after threshold consecutive failures
|
||||
|
||||
- **WHEN** `worker.getHealth()` rejects on three consecutive calls (with threshold 3)
|
||||
- **THEN** `worker.status === UNHEALTHY` after the third call, AND `QUEUE_DIST.HEALTH_CHECK_FAILED` is emitted via the OTel counter
|
||||
|
||||
#### Scenario: recovery after threshold consecutive successes
|
||||
|
||||
- **WHEN** a worker previously marked UNHEALTHY returns healthy on two consecutive calls (with threshold 2) and has `activeTasks === 0`
|
||||
- **THEN** `worker.status === ONLINE` after the second call
|
||||
- **WHEN** the same but `activeTasks > 0`
|
||||
- **THEN** `worker.status === BUSY`
|
||||
|
||||
#### Scenario: timeout uses injected Clock
|
||||
|
||||
- **WHEN** the test injects a `FakeClock` and `worker.getHealth()` never resolves
|
||||
- **THEN** after advancing the clock by `healthCheckTimeoutMs`, `checkHealth` returns the fallback `isHealthy: false` payload AND emits the OTel `queue_dist.health_check.duration_ms` histogram with the elapsed time
|
||||
|
||||
**Verified by:** `src/health-checker/health-checker.test.ts::"flips worker to UNHEALTHY after threshold consecutive failures"` AND `src/health-checker/health-checker.test.ts::"recovers worker to ONLINE/BUSY after threshold consecutive successes"` AND `src/health-checker/health-checker.spec-conformance.test.ts::"timeout race honours injected Clock and emits duration histogram"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Worker pool registry and search
|
||||
|
||||
The package SHALL ship `WorkerPool` with: `addWorker(worker)` (throws `QUEUE_DIST.WORKER_DUPLICATE` on duplicate id), `removeWorker(id)`, `getWorkerById(id)`, `getAllWorkers()`, `getWorkersByStatus(status)`, `getAvailableWorkers()`, `findWorkers(criteria)` filtering by status, capacity range, weight range, type, and a custom matcher predicate, `startAllWorkers(signal?)`, `stopAllWorkers(force?, signal?)`, `clear()`, `size()`, `getWorkersInfo()`.
|
||||
|
||||
#### Scenario: duplicate add throws WORKER_DUPLICATE
|
||||
|
||||
- **WHEN** `addWorker(w)` is called twice with the same `w.id`
|
||||
- **THEN** the second call throws `QueueDistributionError(QUEUE_DIST.WORKER_DUPLICATE)`
|
||||
|
||||
#### Scenario: findWorkers matches every criteria field
|
||||
|
||||
- **WHEN** `findWorkers({ status: ONLINE, minCapacity: 2, maxWeight: 3, type: "local", matcher: w => w.activeTasks === 0 })` is called against a mixed pool
|
||||
- **THEN** only workers satisfying every predicate are returned
|
||||
|
||||
**Verified by:** `src/pool/pool.test.ts::"addWorker rejects duplicate ids with WORKER_DUPLICATE"` AND `src/pool/pool.test.ts::"findWorkers filters by every criteria field including custom matcher"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: WorkerManager — assignment, retry, health, restart
|
||||
|
||||
The package SHALL ship `WorkerManager(pool, balancer?, healthChecker?, options?, deps?)` providing:
|
||||
|
||||
- `assignTask<I, O>(task, input, signal?)` — selects via the configured `LoadBalancingStrategy`, executes on the chosen worker, retries up to `maxTaskRetries` (default 3) on failure with backoff via `@suaveplan/retry`, and records exhausted tasks in `failedTasks`. The retry sleep SHALL be driven by the injected `Clock`, NOT `setTimeout` at module scope. The retry loop SHALL be bounded (no recursive self-call). AbortSignal cancellation MUST be honoured.
|
||||
- `assignTasks<I, O>(tasks[], inputs[], signal?)` — `Promise.all` over `assignTask`; mismatch in array lengths throws `QUEUE_DIST.INVALID_OPTIONS`.
|
||||
- `startHealthChecks()` / `stopHealthChecks()` — periodic loop driven by injected `Clock.setInterval`; idempotent.
|
||||
- `checkAllWorkersHealth(signal?)` — fan-out over all workers, returns map of id → `WorkerHealth`; logs a warning when online count falls below `minOnlineWorkers`.
|
||||
- `handleWorkerFailure(worker, error)` — checks isHealthy; if not, flips to `UNHEALTHY` and optionally restarts.
|
||||
- `setLoadBalancer(balancer)`, `getLoadBalancer()`, `getHealthChecker()`, `getWorkerPool()`, `getFailedTasks()`, `clearFailedTasks()`, `getTaskRetries(id)`.
|
||||
- `getOptions()` / `updateOptions(partial)` — option changes that affect the health-check interval SHALL restart the loop.
|
||||
|
||||
The constructor SHALL throw `QUEUE_DIST.INVALID_OPTIONS` for negative `maxTaskRetries`, negative `taskRetryDelayMs`, or `minOnlineWorkers < 0`.
|
||||
|
||||
#### Scenario: no available workers throws NO_AVAILABLE_WORKERS
|
||||
|
||||
- **WHEN** `pool.getAvailableWorkers()` returns `[]` and `assignTask` is called
|
||||
- **THEN** the call rejects with `QueueDistributionError(QUEUE_DIST.NO_AVAILABLE_WORKERS)`
|
||||
|
||||
#### Scenario: balancer returning null throws STRATEGY_NO_WORKER_SELECTED
|
||||
|
||||
- **WHEN** the configured `LoadBalancingStrategy.selectWorker` returns `null` despite a non-empty available pool
|
||||
- **THEN** the call rejects with `QueueDistributionError(QUEUE_DIST.STRATEGY_NO_WORKER_SELECTED)`
|
||||
|
||||
#### Scenario: retries exhausted throws TASK_RETRIES_EXHAUSTED
|
||||
|
||||
- **WHEN** the chosen worker rejects `executeTask` `maxTaskRetries + 1` times in a row
|
||||
- **THEN** the call rejects with `QueueDistributionError(QUEUE_DIST.TASK_RETRIES_EXHAUSTED)` AND `getFailedTasks()` contains the task
|
||||
|
||||
#### Scenario: AbortSignal cancellation between retries
|
||||
|
||||
- **WHEN** an AbortController is aborted between the first failure and the retry sleep
|
||||
- **THEN** the call rejects with `QueueDistributionError(QUEUE_DIST.TASK_CANCELLED)` without sleeping the full `taskRetryDelayMs`
|
||||
|
||||
#### Scenario: restart limit exhaustion throws WORKER_RESTART_EXHAUSTED
|
||||
|
||||
- **WHEN** an unhealthy worker has reached `maxWorkerRestarts` and the supervisor attempts another restart
|
||||
- **THEN** the supervisor emits `QUEUE_DIST.WORKER_RESTART_EXHAUSTED` via the OTel counter and SHALL NOT call `worker.start()` again
|
||||
|
||||
**Verified by:** `src/manager/manager.test.ts::"throws NO_AVAILABLE_WORKERS when pool is empty"` AND `src/manager/manager.test.ts::"throws STRATEGY_NO_WORKER_SELECTED when balancer returns null"` AND `src/manager/manager.test.ts::"throws TASK_RETRIES_EXHAUSTED and records the task in failedTasks"` AND `src/manager/manager.spec-conformance.test.ts::"AbortSignal cancellation between retries throws TASK_CANCELLED"` AND `src/manager/manager.spec-conformance.test.ts::"supervisor stops restarting after WORKER_RESTART_EXHAUSTED"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Stable error registry via `defineCodes`
|
||||
|
||||
The package SHALL register every error code through `defineCodes("QUEUE_DIST", [...])` from `@suaveplan/error` and expose `QueueDistributionError extends SuaveplanError<QueueDistributionErrorCode>`. The exhaustive code set is: `WORKER_NOT_AVAILABLE`, `WORKER_DUPLICATE`, `WORKER_BUSY_STOP`, `WORKER_NOT_FOUND`, `WORKER_RESTART_EXHAUSTED`, `NO_AVAILABLE_WORKERS`, `STRATEGY_NO_WORKER_SELECTED`, `TASK_RETRIES_EXHAUSTED`, `HEALTH_CHECK_TIMEOUT`, `HEALTH_CHECK_FAILED`, `INVALID_OPTIONS`, `TASK_CANCELLED`. Every code SHALL be thrown by at least one production call path.
|
||||
|
||||
#### Scenario: every registered code is thrown by a production code path
|
||||
|
||||
- **WHEN** the test suite iterates over every code in the registry
|
||||
- **THEN** a corresponding throw site exists, and the test forces the failure and asserts the code matches
|
||||
|
||||
**Verified by:** `src/errors/queue-distribution-error.test.ts::"every registered code is thrown by a production call path"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: OpenTelemetry instrumentation on every public entrypoint
|
||||
|
||||
The package SHALL emit the following instruments via `@suaveplan/opentelemetry`, registered under meter name `"@suaveplan/queue-distribution"`:
|
||||
|
||||
- histograms: `queue_dist.task.duration_ms`, `queue_dist.health_check.duration_ms`, `queue_dist.worker.restart.duration_ms`
|
||||
- counters: `queue_dist.task.assigned_total`, `queue_dist.task.completed_total` (attribute `outcome: "success" | "failure"`), `queue_dist.task.retried_total`, `queue_dist.worker.restart_total` (attribute `outcome: "success" | "failure" | "exhausted"`), `queue_dist.strategy.no_worker_total`
|
||||
- up/down counter: `queue_dist.workers.active`
|
||||
- spans: `queue_dist.manager.assign`, `queue_dist.worker.executeTask`, `queue_dist.health.check`, `queue_dist.worker.restart`
|
||||
|
||||
#### Scenario: task duration histogram emits with expected attributes
|
||||
|
||||
- **WHEN** `manager.assignTask(task, input)` completes successfully under an in-memory `MeterProvider`
|
||||
- **THEN** the `queue_dist.task.duration_ms` histogram has one recorded value with attribute `outcome === "success"`, AND the `queue_dist.task.completed_total` counter incremented by 1 with the same attribute
|
||||
|
||||
#### Scenario: span emission
|
||||
|
||||
- **WHEN** `manager.assignTask` is invoked
|
||||
- **THEN** a span named `queue_dist.manager.assign` is recorded with attributes `queue_dist.worker_id` and `queue_dist.task_id`
|
||||
|
||||
**Verified by:** `src/manager/manager.spec-conformance.test.ts::"emits queue_dist.task.duration_ms and completed counter on success"` AND `src/manager/manager.spec-conformance.test.ts::"emits queue_dist.manager.assign span with worker_id and task_id attributes"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Universal-tier — zero browser / Node / React deps
|
||||
|
||||
The package SHALL be installable in browser, Bun, and Node with no platform-specific runtime imports. The package SHALL NOT import from `node:*`, `bun:*`, `react`, `react-dom`, `next/*`, or any DOM-only API at module scope. `sideEffects: false` SHALL be declared in `package.json` and enforced via the tree-shake test.
|
||||
|
||||
#### Scenario: dep-graph guard rejects platform imports
|
||||
|
||||
- **WHEN** `tests/dep-graph.test.ts` scans every `.ts` file in `src/` for forbidden imports
|
||||
- **THEN** the count of matches is zero AND the test passes
|
||||
|
||||
**Verified by:** `tests/dep-graph.test.ts::"src/ contains no node:*, bun:*, or React imports"`
|
||||
|
||||
---
|
||||
|
||||
### Requirement: Test files use the canonical runner wrapper
|
||||
|
||||
Every test file in `src/` SHALL import runner primitives (`describe`, `it`, `expect`, `beforeAll`, `afterAll`, `beforeEach`, `afterEach`, `vi`, `expectTypeOf`) from `@suaveplan/testing/runner`, NOT from `vitest` or `bun:test` directly.
|
||||
|
||||
#### Scenario: scan shows zero direct runner imports
|
||||
|
||||
- **WHEN** `grep -rnE "from ['\"](vitest|bun:test)['\"]" packages/streams/queue-distribution/src/ --include="*.ts" --include="*.tsx"` is run
|
||||
- **THEN** the output is empty
|
||||
|
||||
**Verified by:** `tests/dep-graph.test.ts::"no test file imports vitest or bun:test directly"`
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: `WorkerManager.rebalance()`
|
||||
|
||||
**Reason:** Upstream body was a stub: `this.logger.warn("Rebalance operation not implemented")`. Per CLAUDE.md §Rules ("Complete all implementations — no stubs") and AGENTS.md §2b ("NEVER ship stubs"), stubs do not ship to genesis.
|
||||
|
||||
**Migration:** None today — the method had no real implementation upstream, so no consumer can have relied on its behaviour. If a real rebalance algorithm is needed later, open a fresh change proposal.
|
||||
|
||||
<!-- markdownlint-enable MD024 -->
|
||||
@@ -0,0 +1,99 @@
|
||||
# Tasks — enterprise-queue-distribution
|
||||
|
||||
> Every checkbox MUST reflect work actually done in this change. Do not retroactively tick boxes for work that was already in main when this change opened. Retroactive ticking is the failure mode this template guards against.
|
||||
|
||||
## 1. Spec authoring
|
||||
|
||||
- [x] 1.1 Read `../suaveplan/enterprise/queue-distribution.md` end-to-end (stub — only L1-L12 are populated; recorded in proposal)
|
||||
- [x] 1.2 Read upstream source under `../suaveplan/packages/universal/queue-distribution/src/`
|
||||
- [x] 1.3 Read `openspec/AGENTS.md` §0a (parity), §0b (status), §6a (barrels), §7 (test runner), §13 (pre-archive)
|
||||
- [x] 1.4 Replace placeholder `proposal.md` with upstream-faithful design
|
||||
- [x] 1.5 Fill `## Source mapping` table — every requirement cites a line range in upstream source OR appears in the "no source line" table with justification
|
||||
- [x] 1.6 Draft `specs/queue-distribution/spec.md`, `specs/queue-distribution-http/spec.md`, `specs/queue-distribution-worker-thread/spec.md` from `openspec/templates/spec-delta.md`
|
||||
- [x] 1.7 Every requirement in each spec delta carries a `**Verified by:**` line naming concrete `it(...)` titles
|
||||
|
||||
## 2. Implementation — `@suaveplan/queue-distribution` (core)
|
||||
|
||||
- [ ] 2.1 Scaffold `packages/streams/queue-distribution/` (`package.json`, `tsconfig.lib.json`, `vite.config.ts`, `vitest.config.ts`, `README.md`)
|
||||
- [ ] 2.2 `src/errors/queue-distribution-error.ts` — `QueueDistributionError` + `defineCodes("QUEUE_DIST", [...])` registry (12 codes)
|
||||
- [ ] 2.3 `src/task/task.ts` — `DistributedTask<I,O>`, `TaskStatus`, `TaskMetadata`, `TaskMetadataSchema`, `ExecutionContext`
|
||||
- [ ] 2.4 `src/worker/worker.ts` — `Worker` interface, `WorkerStatus`, `WorkerHealth`, `WorkerInfo`, `WorkerHealthSchema`, `WorkerInfoSchema`
|
||||
- [ ] 2.5 `src/worker/abstract/abstract.ts` + `abstract-execute.ts` + `abstract-metrics.ts` — `AbstractWorker` split across ≤300-LOC files
|
||||
- [ ] 2.6 `src/worker/local/local.ts` — `LocalWorker<Ctx>`
|
||||
- [ ] 2.7 `src/load-balancing/base/base.ts` — `LoadBalancingStrategy`, `AbstractLoadBalancingStrategy.filterAvailableWorkers`
|
||||
- [ ] 2.8 `src/load-balancing/round-robin/round-robin.ts` — `RoundRobinStrategy`
|
||||
- [ ] 2.9 `src/load-balancing/least-busy/least-busy.ts` — `LeastBusyStrategy` with optional EWMA perf bias
|
||||
- [ ] 2.10 `src/load-balancing/weighted/weighted.ts` — `WeightedStrategy` with injectable `Rng`
|
||||
- [ ] 2.11 `src/health-checker/health-checker.ts` — `WorkerHealthChecker` interface + `PeriodicHealthChecker` (Clock-driven timeout race)
|
||||
- [ ] 2.12 `src/pool/pool.ts` — `WorkerPool` + `WorkerSearchCriteria` + `WorkerSearchCriteriaSchema`
|
||||
- [ ] 2.13 `src/manager/manager.ts` + `manager-dispatch.ts` + `manager-supervisor.ts` — `WorkerManager` split across ≤300-LOC files
|
||||
- [ ] 2.14 `src/observability/instruments.ts` — `@suaveplan/opentelemetry` counter/histogram/span definitions
|
||||
- [ ] 2.15 `src/index.ts` — explicit named re-exports (no `export *`)
|
||||
- [ ] 2.16 Subpath exports in `package.json` per feature module (one entry per `src/<category>/<feature>/`)
|
||||
|
||||
## 3. Implementation — `@suaveplan/queue-distribution-http`
|
||||
|
||||
- [ ] 3.1 Scaffold `packages/streams/queue-distribution-http/`
|
||||
- [ ] 3.2 `src/errors/queue-distribution-http-error.ts` — 6 codes
|
||||
- [ ] 3.3 `src/communicator/base/base.ts` — `WorkerCommunicator` interface + `AbstractWorkerCommunicator` (Clock-driven `withTimeout`)
|
||||
- [ ] 3.4 `src/communicator/http/http-communicator.ts` — `HttpWorkerCommunicator` with injectable `fetch`, Zod-validated wire envelope `WorkerApiResponse<T>`
|
||||
- [ ] 3.5 `src/worker/remote/remote.ts` + `remote-connect.ts` + `remote-health.ts` — `RemoteWorker` split across ≤300-LOC files; backoff via `@suaveplan/retry`
|
||||
- [ ] 3.6 `src/index.ts` — explicit named re-exports
|
||||
- [ ] 3.7 Subpath exports per feature module
|
||||
|
||||
## 4. Implementation — `@suaveplan/queue-distribution-worker-thread`
|
||||
|
||||
- [ ] 4.1 Scaffold `packages/streams/queue-distribution-worker-thread/`
|
||||
- [ ] 4.2 `src/errors/queue-distribution-worker-thread-error.ts` — 4 codes
|
||||
- [ ] 4.3 `src/communicator/worker-thread/worker-thread-communicator.ts` — message-correlation map, Zod envelope, AbortSignal
|
||||
- [ ] 4.4 `src/worker/worker-thread/worker-thread-worker.ts` — `WorkerThreadWorker` subclass, injectable `WorkerFactory`
|
||||
- [ ] 4.5 `src/index.ts` — explicit named re-exports
|
||||
- [ ] 4.6 Subpath exports per feature module
|
||||
|
||||
## 5. Tests (all three packages)
|
||||
|
||||
- [ ] 5.1 Per-feature `*.test.ts` covering happy path + every failure branch
|
||||
- [ ] 5.2 Per-feature `*.spec-conformance.test.ts` pinning type-level / OTel / strict-mode / lifecycle scenarios
|
||||
- [ ] 5.3 `src/errors/queue-distribution-error.test.ts` — every registered code thrown by ≥1 production path
|
||||
- [ ] 5.4 `tests/dep-graph.test.ts` — runtime + dev deps respect plugin model
|
||||
- [ ] 5.5 fast-check property suite for `WeightedStrategy` distribution
|
||||
- [ ] 5.6 Every `it(...)` carries `{ timeout: 5000 }` (or 30000 for genuinely async)
|
||||
- [ ] 5.7 All test files import runner primitives from `@suaveplan/testing/runner` — verified with the grep in §13 step 4b
|
||||
- [ ] 5.8 OTel emission asserted with real in-memory `MeterProvider` (no `vi.mock`)
|
||||
|
||||
## 6. Documentation
|
||||
|
||||
- [ ] 6.1 Sibling `.md` for every implementation `.ts` (≥200 words, basic + advanced examples, full API reference)
|
||||
- [ ] 6.2 Each package's `README.md` with module index + Quick-start
|
||||
- [ ] 6.3 Update `CLAUDE.md` package listing (3 new packages under streams plugin)
|
||||
- [ ] 6.4 Update `STATUS.md` — flip `queue-distribution` row to ✅ and add 2 new plugin rows
|
||||
- [ ] 6.5 Update `SUMMARY.md` — paragraph per package
|
||||
- [ ] 6.6 mitata benchmarks under `src/__benchmarks__/` for each package
|
||||
|
||||
## 7. Pre-archive verification (the 5-gate triple per package)
|
||||
|
||||
- [ ] 7.1 `bun run check` — Biome 0/0/0 across whole monorepo
|
||||
- [ ] 7.2 `bunx turbo lint typecheck test --filter=@suaveplan/queue-distribution{,-http,-worker-thread}` — green
|
||||
- [ ] 7.3 `bunx turbo test:coverage --filter=@suaveplan/queue-distribution{,-http,-worker-thread}` — 100/100/100/100
|
||||
- [ ] 7.4 `bun scripts/openspec-conformance.ts @suaveplan/queue-distribution` (+ both plugins) — exit 0, no `error` findings
|
||||
- [ ] 7.5 `bunx turbo build --filter=@suaveplan/queue-distribution{,-http,-worker-thread}` — green
|
||||
|
||||
## 8. Pre-archive 9-row checklist (skill / AGENTS.md §13)
|
||||
|
||||
- [ ] 8.1 Generics threaded; `expectTypeOf` assertions on every literal-type claim
|
||||
- [ ] 8.2 Zod schemas wired at every declared boundary
|
||||
- [ ] 8.3 Every error code thrown by ≥1 production code path
|
||||
- [ ] 8.4 Strict-mode behaviors tested both on and off
|
||||
- [ ] 8.5 OTel instruments emitted from production paths and asserted with a real in-memory `MeterProvider`
|
||||
- [ ] 8.6 Lifecycle dedup tested where claimed (in-flight Promise, idempotent start/stop)
|
||||
- [ ] 8.7 No retroactive tick — every box reflects work done in this change
|
||||
- [ ] 8.8 No `export *` in any feature-module barrel — verified with the grep in §13 step 4a
|
||||
- [ ] 8.9 No direct `vitest` / `bun:test` imports — verified with the grep in §13 step 4b
|
||||
|
||||
## 9. Archive
|
||||
|
||||
- [ ] 9.1 Fold spec deltas into `openspec/specs/queue-distribution/spec.md`, `openspec/specs/queue-distribution-http/spec.md`, `openspec/specs/queue-distribution-worker-thread/spec.md`
|
||||
- [ ] 9.2 Move `openspec/changes/enterprise-queue-distribution/` → `openspec/archive/2026-05-11-enterprise-queue-distribution/`
|
||||
- [ ] 9.3 Refresh `MANIFEST.md` (`bun scripts/folder-reorg/generate-manifest.ts`)
|
||||
- [ ] 9.4 Refresh `MISSING.yml` (`bun scripts/generate-missing.ts`)
|
||||
- [ ] 9.5 `bunx markdownlint-cli2 --fix` and verify exit 0 on second pass
|
||||
Reference in New Issue
Block a user