suaveplan/openspec/archive/2026-05-11-enterprise-queue-distribution/proposal.md

Enterprise-grade @suaveplan/queue-distribution (split into 3 packages)

Package: @suaveplan/queue-distribution, @suaveplan/queue-distribution-http, @suaveplan/queue-distribution-worker-thread Status: PROPOSED Type: new-package (migration from ../suaveplan/packages/universal/queue-distribution)

This proposal supersedes the placeholder scaffolded into commit f52ca3d1, which described a WorkerPool<T>.scale() / autoscale design that did NOT match the upstream source layout. The placeholder marked every parity row ✅ without verifying upstream — the listed capabilities (WorkerPool<T>.scale(), JobHandler<T>, HashStrategy, autoscale depth/idle thresholds) do not exist in ../suaveplan/packages/universal/queue-distribution/src/. Replacing it per openspec/AGENTS.md §0a ("fresh build with a different API" is a blocking anti-pattern).

Why

Worker-pool distribution is a recurring need across genesis — anywhere a unit of work must execute concurrently against a finite, health-monitored pool of executors. Upstream shipped a single @suaveplan/queue-distribution package coupling four concerns (load-balancing, worker lifecycle, in-process workers, remote-over-HTTP workers) into one universal-tier bundle. That coupling makes the package un-tree-shakeable for browser consumers who never use HTTP, and conversely forces server consumers to carry transport code they don't need. The upstream implementation also predates genesis enterprise standards: no Zod boundaries, no OTel, plain Error throws, Math.random / Date.now baked in at strategy / health-checker / connect-retry layer (no injectable Rng/Clock), any type holes in catch arms, uuid runtime dep for worker IDs, every test file imports from vitest directly, and every feature barrel uses export *.

This change migrates that capability to genesis at enterprise quality and splits it into a core + two transport plugins so consumers only pay for what they use. The core package is also decoupled from @suaveplan/queue's Task<I,O> abstraction (which genesis deliberately replaced with a Job / state-machine / Zod-schema model when it migrated @suaveplan/queue): distribution defines its own minimal DistributedTask<I,O> interface so it can serve queue, batch, RPC, or agent-dispatch consumers via thin adapters. A separate follow-up change can add a createWorkerForQueueJob(...) adapter once a concrete genesis consumer of both packages exists.

What changes

@suaveplan/queue-distribution (core, universal tier — packages/streams/queue-distribution)

• ADDED: DistributedTask<I, O> — minimal task contract (id: string, execute(input, ctx): Promise<O>, getMetadata(): TaskMetadata, setStatus(status), getStatus()).
• ADDED: TaskStatus enum (PENDING | RUNNING | COMPLETED | FAILED | CANCELLED | TIMED_OUT).
• ADDED: TaskMetadata shape (createdAt, startedAt?, completedAt?, attempts, tags?, indexed extras).
• ADDED: ExecutionContext — opaque, caller-defined context object threaded through execute.
• ADDED: Worker interface — id, capacity, weight, status, activeTasks, executeTask<I,O>, getHealth, start, stop, canAcceptTask, getInfo. Zod schema at the boundary.
• ADDED: WorkerStatus enum (ONLINE | BUSY | OFFLINE | UNHEALTHY).
• ADDED: WorkerHealth + WorkerInfo interfaces.
• ADDED: AbstractWorker — base class with lifecycle FSM, metrics, structured start/stop/executeTask/getHealth/markUnhealthy/getInfo. Injectable Clock, IdGenerator, LoggerLike. Source split across abstract.ts (public surface, ≤200 LOC) + abstract-execute.ts (executeTask lifecycle, ≤120 LOC) + abstract-metrics.ts (metric accumulation, ≤80 LOC) to stay under the 300-LOC hard limit.
• ADDED: LocalWorker<Ctx> — in-process executor parameterised by ExecutionContext.
• ADDED: LoadBalancingStrategy interface + AbstractLoadBalancingStrategy base with filterAvailableWorkers.
• ADDED: RoundRobinStrategy — rotation index, reset(), pool-shrink safe.
• ADDED: LeastBusyStrategy — fewest-active-tasks; optional usePerformanceMetrics flag with EWMA bias, capacity tie-break, weight tie-break; notifyTaskCompleted callback updates metrics; resetPerformanceMetrics, getWorkerPerformance, setUsePerformanceMetrics.
• ADDED: WeightedStrategy — weighted random via injectable Rng; total-weight ≤ 0 falls back to uniform.
• ADDED: WorkerHealthChecker interface + PeriodicHealthChecker — consecutive-failure unhealthy promotion, consecutive-success recovery, configurable timeout, optional caching, getHealthCheckCounts, resetHealthCheckCounts, clearCache.
• ADDED: WorkerPool — addWorker (rejects duplicate IDs), removeWorker, getWorkerById, getAllWorkers, getAvailableWorkers, getWorkersByStatus, findWorkers(criteria), startAllWorkers, stopAllWorkers, clear, size, getWorkersInfo.
• ADDED: WorkerSearchCriteria shape (status, min/max capacity, min/max weight, type, custom matcher).
• ADDED: WorkerManager — assignTask<I,O>(task, input, signal?), assignTasks<I,O>(tasks[], inputs[], signal?), startHealthChecks, stopHealthChecks, checkAllWorkersHealth, handleWorkerFailure, getWorkerPool, getLoadBalancer, setLoadBalancer, getHealthChecker, getFailedTasks, clearFailedTasks, getTaskRetries, getOptions, updateOptions. Source split across manager.ts (public surface, ≤200 LOC) + manager-dispatch.ts (assignTask / retry orchestration, ≤200 LOC) + manager-supervisor.ts (health-check loop / restart logic, ≤200 LOC).
• ADDED: Retry-with-backoff at the manager layer routed through @suaveplan/retry (replaces upstream's hand-rolled setTimeout retry loop).
• ADDED: QueueDistributionError extending SuaveplanError, registry of stable codes via defineCodes("QUEUE_DIST", [...]):
  • WORKER_NOT_AVAILABLE, WORKER_DUPLICATE, WORKER_BUSY_STOP, WORKER_NOT_FOUND, WORKER_RESTART_EXHAUSTED
  • NO_AVAILABLE_WORKERS, STRATEGY_NO_WORKER_SELECTED, TASK_RETRIES_EXHAUSTED
  • HEALTH_CHECK_TIMEOUT, HEALTH_CHECK_FAILED
  • INVALID_OPTIONS, TASK_CANCELLED
• ADDED: OTel instruments via @suaveplan/opentelemetry:
  • histograms: queue_dist.task.duration_ms, queue_dist.health_check.duration_ms, queue_dist.worker.restart.duration_ms
  • counters: queue_dist.task.assigned_total, queue_dist.task.completed_total{outcome}, queue_dist.task.retried_total, queue_dist.worker.restart_total{outcome}, queue_dist.strategy.no_worker_total
  • up/down counter: queue_dist.workers.active
  • spans: queue_dist.manager.assign, queue_dist.worker.executeTask, queue_dist.health.check, queue_dist.worker.restart
• ADDED: Zod schemas at every public boundary (WorkerManagerOptionsSchema, HealthCheckerOptionsSchema, WorkerSearchCriteriaSchema, WorkerHealthSchema, WorkerInfoSchema, TaskMetadataSchema).
• ADDED: AbortSignal threaded through every async public method (assignTask, assignTasks, executeTask, getHealth, start, stop, retry delay sleeps).

@suaveplan/queue-distribution-http (plugin — packages/streams/queue-distribution-http)

• ADDED: WorkerCommunicator interface (sendTask, checkStatus, checkHealth, connect, disconnect, ping, getProtocol).
• ADDED: AbstractWorkerCommunicator base with default-timeout + serialize/deserialize helpers + withTimeout race.
• ADDED: HttpWorkerCommunicator — fetch-based transport, injectable fetch, AbortSignal-aware, Zod-validated wire envelope WorkerApiResponse<T>, structured headers.
• ADDED: RemoteWorker — AbstractWorker subclass over a WorkerCommunicator. Connect-retry routed through @suaveplan/retry (replaces the upstream hand-rolled Math.min(1000 * attempts, 10000) ladder + recursive call). Source split across remote.ts (public + lifecycle, ≤200 LOC) + remote-connect.ts (connect/reconnect, ≤120 LOC) + remote-health.ts (periodic health loop, ≤120 LOC).
• ADDED: RemoteWorkerOptions (healthCheckIntervalMs, maxConnectionAttempts).
• ADDED: QueueDistributionHttpError with codes HTTP.REQUEST_FAILED, HTTP.PROTOCOL_ERROR, HTTP.TIMEOUT, HTTP.RESPONSE_MALFORMED, HTTP.CONNECT_FAILED, HTTP.UNREACHABLE.

@suaveplan/queue-distribution-worker-thread (plugin — packages/streams/queue-distribution-worker-thread)

• ADDED: WorkerThreadCommunicator — Web-Worker / Node worker_threads bridge with caller-injected WorkerFactory so the same code runs in browser and server. Message-correlation envelope { id, kind: "task" | "status" | "health" | "ping", payload } validated via Zod.
• ADDED: WorkerThreadWorker — AbstractWorker subclass dispatching to a single backing worker thread, with bounded outstanding-request map, AbortSignal cancellation, structured timeout.
• ADDED: QueueDistributionWorkerThreadError with codes WT.WORKER_CRASHED, WT.MESSAGE_MALFORMED, WT.REQUEST_TIMEOUT, WT.WORKER_TERMINATED.

All three packages

• Test files import runner primitives from @suaveplan/testing/runner — never vitest or bun:test directly.
• Feature-module barrels use explicit named re-exports — no export * outside the root src/index.ts.
• mitata benchmarks under src/__benchmarks__/.
• 100% coverage with provider: "istanbul".
• Sibling .md for every implementation file; root README.md with module index.

Out of scope

• createWorkerForQueueJob(...) adapter — deferred to a follow-up change once we have a concrete genesis consumer of both @suaveplan/queue and @suaveplan/queue-distribution. The proposal commits the core package to a self-contained DistributedTask contract so the adapter is purely additive when it lands.
• WebSocket / gRPC communicators — neither shipped upstream; not requested.
• WorkerManager.rebalance() — upstream body was this.logger.warn("Rebalance operation not implemented"). Per CLAUDE.md §Rules and AGENTS.md §2b, stubs do not ship to genesis. Removed; reopens via a separate change with a real algorithm. Documented in REMOVED section of spec delta.
• Real browser-side CPU/memory metrics — upstream LocalWorker.getHealth already documented "could enhance with actual system metrics here" as a future TODO and returned mock values. Same baseline carried over; real metrics is a separate plugin concern.
• Autoscale (scale-up on depth threshold / scale-down on idle threshold) — described in the now-superseded placeholder proposal but absent from upstream. Not migrated; opens via a separate genesis-original change if requested.

Source mapping

This change derives primarily from ../suaveplan/packages/universal/queue-distribution/src/ because ../suaveplan/enterprise/queue-distribution.md is a stub (Purpose section ends "Worker pool and load balancing distribution layer fo..." at L12 — truncated). Source line ranges below cite the upstream source tree.

Requirement	Source lines	Notes
Worker interface (id/capacity/weight/status/activeTasks/executeTask/getHealth/start/stop/canAcceptTask/getInfo)	workers/types/worker/worker.ts L62-L126	verbatim, generalised over DistributedTask
WorkerStatus enum (ONLINE/BUSY/OFFLINE/UNHEALTHY)	workers/types/worker/worker.ts L7-L11	verbatim
WorkerHealth + metrics shape	workers/types/worker/worker.ts L16-L56	verbatim
WorkerInfo	workers/types/worker/worker.ts L131-L188	verbatim
AbstractWorker lifecycle + metrics	workers/types/abstract/abstract.ts L16-L314	verbatim semantics; any catch → unknown; uuid → injectable IdGenerator; Date.now() → injectable Clock; logger inlined ConsoleLogger → injected LoggerLike
LocalWorker in-process execution	workers/types/local/local.ts L1-L97	verbatim
LoadBalancingStrategy interface	load-balancing/base.ts L4-L14	verbatim
AbstractLoadBalancingStrategy.filterAvailableWorkers	load-balancing/base.ts L21-L28	verbatim
RoundRobinStrategy rotation + reset	load-balancing/round-robin/roundRobin.ts L10-L55	verbatim
LeastBusyStrategy (fewest active tasks + perf bias + sort fallback by capacity/weight)	load-balancing/least-busy/leastBusy.ts L10-L155	verbatim
WeightedStrategy weighted random	load-balancing/weighted/weighted.ts L10-L64	Math.random() → injectable Rng
PeriodicHealthChecker (intervals, thresholds, cache, timeout, recovery, mark-unhealthy)	workers/health/healthChecker.ts L62-L265	verbatim; Date.now() → injectable Clock; setTimeout race → AbortSignal + Clock.sleep
WorkerPool (CRUD, search, lifecycle, info)	workers/management/pool/pool.ts L13-L201	verbatim
WorkerSearchCriteria	workers/management/pool/pool.ts L206-L241	verbatim
WorkerManager (assignTask, assignTasks, health loop, restart, options)	workers/management/manager/manager.ts L61-L539	verbatim semantics; retry-with-backoff routed through @suaveplan/retry; setInterval driven by injectable Clock; recursive retry replaced with bounded loop
WorkerCommunicator interface	workers/communication/worker/workerCommunicator.ts L10-L61	verbatim
AbstractWorkerCommunicator	workers/communication/worker/workerCommunicator.ts L66-L200	verbatim
HttpWorkerCommunicator (fetch wrapper, WorkerApiResponse<T>, ping/health/status/sendTask)	workers/communication/http/httpCommunicator.ts L21-L174	verbatim; fetch injectable; envelope Zod-validated
RemoteWorker (connect/reconnect/health/sendTask + AbstractWorker integration)	workers/types/remote/remote.ts L10-L368	verbatim semantics; backoff via @suaveplan/retry; non-recursive
RemoteWorkerOptions	workers/types/remote/remote.ts L373-L383	verbatim
Task<I,O> reference shape (basis for DistributedTask<I,O>)	../suaveplan/packages/universal/queue/src/core/base/task/task.types.ts L7-L62	reworded — genesis defines its own DistributedTask to decouple from @suaveplan/queue.Job model
TaskStatus enum	../suaveplan/packages/universal/queue/src/core/base/task/task.types.ts L67-L74	verbatim
TaskMetadata shape	../suaveplan/packages/universal/queue/src/core/base/task/task.types.ts L79-L86	verbatim

Requirements with no source line

Requirement	Justification
100% test coverage (statements/branches/functions/lines)	Genesis-wide policy (CLAUDE.md §Quality, AGENTS.md §7)
Zod schemas at every public boundary	Genesis-wide policy (AGENTS.md §0a "design quality bar")
SuaveplanError + stable code registry	Genesis-wide policy (AGENTS.md §0a anti-pattern row: replacing upstream errors with plain Error)
OpenTelemetry counters/histograms/spans	Genesis-wide policy (CLAUDE.md §Quality) — upstream had none
Injectable Clock / Rng / IdGenerator / LoggerLike / fetch	Genesis-wide policy: no Math.random / Date.now at module scope (CLAUDE.md §Rules, AGENTS.md §0a)
Tests use @suaveplan/testing/runner wrapper	CLAUDE.md rule 11 / AGENTS.md §7 — applies repo-wide, not derivable from upstream
Feature barrels use explicit named exports (no export *)	CLAUDE.md rule 10 / AGENTS.md §6a — applies repo-wide, not derivable from upstream
Split into core + 2 transport plugins	User decision (this conversation): better tree-shaking for browser-only consumers; matches the modular plugin model called out in enterprise/queue-distribution.md §4
Decouple from @suaveplan/queue Task<I,O> — define DistributedTask<I,O> locally	Genesis @suaveplan/queue deliberately replaced upstream Task with Job / state-machine model when it was migrated; reintroducing Task in queue would create two parallel work-unit abstractions. Distribution becomes reusable for queue, batch, RPC, agent-dispatch via thin adapter (deferred).
AbortSignal everywhere async	Genesis-wide policy (CLAUDE.md §Rules — "Backpressure & cancellation")
mitata benchmarks under src/__benchmarks__/	Genesis-wide policy (AGENTS.md §9)
@suaveplan/queue-distribution-worker-thread (Web-Worker / worker_threads transport)	New plugin; upstream had only an AbstractWorkerCommunicator+HttpWorkerCommunicator pair and never shipped a Web-Worker transport despite the docs mentioning "browser" worker type. Adding it now under the same plugin pattern as HTTP.

Parity Checklist

Capability	Upstream	Migrated	Notes
Worker interface + WorkerStatus + WorkerHealth + WorkerInfo	✅	✅
AbstractWorker lifecycle, metrics, FSM	✅	⬆️	Enhanced: injectable Clock/IdGenerator/LoggerLike; uuid dep dropped; any catches narrowed to unknown+SuaveplanError
LocalWorker	✅	✅
LoadBalancingStrategy interface	✅	✅
RoundRobinStrategy	✅	✅
LeastBusyStrategy + performance bias	✅	✅
WeightedStrategy weighted random	✅	⬆️	Enhanced: Math.random → injectable Rng for deterministic tests
PeriodicHealthChecker thresholds + cache + timeout	✅	⬆️	Enhanced: setTimeout race → AbortSignal + injectable Clock; OTel histograms; structured errors
WorkerPool CRUD + search + bulk start/stop	✅	✅
WorkerManager assign + retry + health loop + restart	✅	⬆️	Enhanced: retry-with-backoff via @suaveplan/retry; OTel spans; recursive retry replaced with bounded loop; AbortSignal threaded
WorkerCommunicator interface	✅	✅	Moved to @suaveplan/queue-distribution-http (transport-specific)
AbstractWorkerCommunicator	✅	✅	Moved to @suaveplan/queue-distribution-http
HttpWorkerCommunicator	✅	⬆️	Enhanced: injectable fetch; Zod-validated envelope; structured QueueDistributionHttpError; AbortSignal
RemoteWorker connect/reconnect/health	✅	⬆️	Enhanced: connect backoff via @suaveplan/retry; non-recursive reconnect; moved to @suaveplan/queue-distribution-http
RemoteWorkerOptions	✅	✅	Moved to @suaveplan/queue-distribution-http
Web-Worker / worker_threads transport	🚫	⬆️	Upstream had no Web-Worker transport (despite docs referencing "browser" worker type). Genesis adds @suaveplan/queue-distribution-worker-thread as a new transport plugin. Marked ⬆️ because it extends the surface upstream documented but never shipped.
WorkerManager.rebalance()	✅	🚫	Scoped out: upstream body was this.logger.warn("Rebalance operation not implemented") — a stub. Per CLAUDE.md §Rules and AGENTS.md §2b, stubs do not ship to genesis. Reopens via a separate change with a real algorithm.
OTel instrumentation	🚫	⬆️	Added per genesis enterprise standard.
Zod boundary schemas	🚫	⬆️	Added per genesis enterprise standard.
SuaveplanError + code registry	🚫	⬆️	Replaces plain throw new Error(...).
AbortSignal-aware async surface	🚫	⬆️	Added per genesis enterprise standard.

Verification approach

• src/load-balancing/round-robin/round-robin.test.ts — rotation, reset, pool-shrink invariants, empty-pool null return.
• src/load-balancing/least-busy/least-busy.test.ts — fewest-active-tasks selection, performance-bias tie-break, capacity tie-break, weight tie-break, notifyTaskCompleted perf-metric update, resetPerformanceMetrics.
• src/load-balancing/weighted/weighted.test.ts — distribution proportional to weight (with injected deterministic Rng), zero-weight fallback to uniform, single-worker fast path, fast-check property suite.
• src/load-balancing/base/base.test.ts — filterAvailableWorkers filters by status and canAcceptTask.
• src/worker/abstract/abstract.test.ts — start/stop/executeTask happy path, OFFLINE/UNHEALTHY refusal, force-stop active tasks, metrics accumulation, markUnhealthy, getInfo shape.
• src/worker/abstract/abstract.spec-conformance.test.ts — generic <I,O> threading via expectTypeOf, OTel histogram + counter emission via real in-memory MeterProvider, Clock/IdGenerator injection.
• src/worker/local/local.test.ts — task delegation, capability reporting, health pass-through.
• src/health-checker/health-checker.test.ts — consecutive-failure unhealthy promotion, consecutive-success recovery, cache TTL, timeout via injected Clock, error → fallback unhealthy.
• src/pool/pool.test.ts — add/remove/get/findWorkers with every criteria field, duplicate-add throws WORKER_DUPLICATE, bulk start/stop, clear.
• src/manager/manager.test.ts — assignTask happy path, balancer null → throws STRATEGY_NO_WORKER_SELECTED, retry exhaustion → throws TASK_RETRIES_EXHAUSTED, failed-tasks set populated, options updates restart health loop, restart limit exhausted → throws WORKER_RESTART_EXHAUSTED.
• src/manager/manager.spec-conformance.test.ts — OTel span + counter assertions, AbortSignal cancellation mid-retry → throws TASK_CANCELLED, Clock.sleep honoured for retry delay.
• src/errors/queue-distribution-error.test.ts — every registered code is thrown by at least one production call path.
• tests/dep-graph.test.ts — core has no transport-package dep; http plugin has core dep; worker-thread plugin has core dep.

For @suaveplan/queue-distribution-http:

• src/communicator/http/http-communicator.test.ts — sendTask happy path, response envelope Zod rejection, fetch throw, AbortSignal, ping latency, headers passthrough.
• src/worker/remote/remote.test.ts — connect happy path, connect retries with injected Clock, max-attempts exhausted → throws HTTP.CONNECT_FAILED, periodic health driven by Clock, unreachable → markUnhealthy.

For @suaveplan/queue-distribution-worker-thread:

• src/communicator/worker-thread/worker-thread-communicator.test.ts — message correlation, timeout, crash-event → WT.WORKER_CRASHED, malformed reply → WT.MESSAGE_MALFORMED.
• src/worker/worker-thread/worker-thread-worker.test.ts — task dispatch, bounded outstanding map, terminate cleanup → WT.WORKER_TERMINATED.

Migration impact

Additive only — no existing genesis consumers (grep -rn "@suaveplan/queue-distribution" packages/ apps/ returns zero).

Impact

Affected dependents

bunx turbo ls --filter=...@suaveplan/queue-distribution — none on genesis at the time this proposal is opened.

• No dependents (verified — @suaveplan/queue-distribution does not yet exist on genesis).
• @suaveplan/queue-distribution-http and @suaveplan/queue-distribution-worker-thread are new packages introduced in this change.