Move docs/ package mirror into document-navigator plugin
Relocate the ~6k-file docs tree from the repo root into plugins/document-navigator/docs/ so the plugin is self-contained and portable. docs/superpowers/ (specs and plans) stays at the repo root. Moved via cp+rm rather than git mv because directory rename() fails with EPERM on this WSL /mnt DrvFs mount; git rename-detection pairs the identical-content files at commit time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
# @suaveplan/queue - Enterprise-grade queue management: memory/priority/delayed queues, concurrent workers with retry and dead-letter, job state machines, OTel instrumentation
|
||||
|
||||
## Summary
|
||||
|
||||
Enterprise-grade queue management: memory/priority/delayed queues, concurrent workers with retry and dead-letter forwarding, job state machines, and OTel instrumentation.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [errors](./src/errors/queue-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [queue-error](./src/errors/queue-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [delayed-queue](./src/queue/delayed-queue.md) — `DelayedQueue<T>` is an explicit alias for a delay-aware FIFO queue. It extends `MemoryQueue<T>` to make the delay inte…
|
||||
- [memory-queue](./src/queue/memory-queue.md) — `MemoryQueue<T>` is the foundational FIFO queue for `@suaveplan/queue`. It stores jobs in memory in insertion order and…
|
||||
- [priority-queue](./src/queue/priority-queue.md) — `PriorityQueue<T>` extends `MemoryQueue<T>` to provide priority-ordered job scheduling. Jobs with higher `priority` val…
|
||||
- [types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [queue-types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [queue-worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
<!-- end:auto-docs --> - [queue-error](./src/errors/queue-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_- [delayed-queue](./src/queue/delayed-queue.md) — `DelayedQueue<T>` is an explicit alias for a delay-aware FIFO queue. It extends `MemoryQueue<T>` to make the delay inte…
|
||||
- [memory-queue](./src/queue/memory-queue.md) — `MemoryQueue<T>` is the foundational FIFO queue for `@suaveplan/queue`. It stores jobs in memory in insertion order and…
|
||||
- [priority-queue](./src/queue/priority-queue.md) — `PriorityQueue<T>` extends `MemoryQueue<T>` to provide priority-ordered job scheduling. Jobs with higher `priority` val…
|
||||
- [types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_- [worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-error](./src/errors/queue-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_- [delayed-queue](./src/queue/delayed-queue.md) — `DelayedQueue<T>` is an explicit alias for a delay-aware FIFO queue. It extends `MemoryQueue<T>` to make the delay inte…
|
||||
- [memory-queue](./src/queue/memory-queue.md) — `MemoryQueue<T>` is the foundational FIFO queue for `@suaveplan/queue`. It stores jobs in memory in insertion order and…
|
||||
- [priority-queue](./src/queue/priority-queue.md) — `PriorityQueue<T>` extends `MemoryQueue<T>` to provide priority-ordered job scheduling. Jobs with higher `priority` val…
|
||||
- [types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_- [worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-error](./src/errors/queue-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_- [delayed-queue](./src/queue/delayed-queue.md) — `DelayedQueue<T>` is an explicit alias for a delay-aware FIFO queue. It extends `MemoryQueue<T>` to make the delay inte…
|
||||
- [memory-queue](./src/queue/memory-queue.md) — `MemoryQueue<T>` is the foundational FIFO queue for `@suaveplan/queue`. It stores jobs in memory in insertion order and…
|
||||
- [priority-queue](./src/queue/priority-queue.md) — `PriorityQueue<T>` extends `MemoryQueue<T>` to provide priority-ordered job scheduling. Jobs with higher `priority` val…
|
||||
- [types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_- [worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_- [queue-error](./src/errors/queue-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_ - [job](./src/job/job.md) — <!-- begin:auto-docs --> _(no submodules)_- [delayed-queue](./src/queue/delayed-queue.md) — `DelayedQueue<T>` is an explicit alias for a delay-aware FIFO queue. It extends `MemoryQueue<T>` to make the delay inte…
|
||||
- [memory-queue](./src/queue/memory-queue.md) — `MemoryQueue<T>` is the foundational FIFO queue for `@suaveplan/queue`. It stores jobs in memory in insertion order and…
|
||||
- [priority-queue](./src/queue/priority-queue.md) — `PriorityQueue<T>` extends `MemoryQueue<T>` to provide priority-ordered job scheduling. Jobs with higher `priority` val…
|
||||
- [types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_ - [queue-types](./src/types/queue-types.md) — <!-- begin:auto-docs --> _(no submodules)_- [worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_ - [queue-worker](./src/worker/queue-worker.md) — <!-- begin:auto-docs --> _(no submodules)_
|
||||
## Packages
|
||||
|
||||
| Module | Description |
|
||||
|--------|-------------|
|
||||
| [errors/queue-error](src/errors/queue-error.md) | Error hierarchy with stable codes |
|
||||
| [job/job](src/job/job.md) | Job state machine |
|
||||
| [queue/memory-queue](src/queue/memory-queue.md) | FIFO in-memory queue |
|
||||
| [queue/priority-queue](src/queue/priority-queue.md) | Priority-ordered queue |
|
||||
| [queue/delayed-queue](src/queue/delayed-queue.md) | Delay-aware queue |
|
||||
| [worker/queue-worker](src/worker/queue-worker.md) | Concurrent job worker with retry |
|
||||
| [types/queue-types](src/types/queue-types.md) | Shared types and Zod schemas |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```ts
|
||||
import { createMemoryQueue } from "@suaveplan/queue/queue/memory-queue";
|
||||
import { runWorker } from "@suaveplan/queue/worker/queue-worker";
|
||||
|
||||
const queue = createMemoryQueue<{ userId: string }>();
|
||||
await queue.enqueue({ userId: "user_1" });
|
||||
await queue.enqueue({ userId: "user_2" });
|
||||
|
||||
const stats = await runWorker(queue, async (job) => {
|
||||
console.log("Processing user:", job.payload.userId);
|
||||
}, {
|
||||
concurrency: 2,
|
||||
retryPolicy: { maxAttempts: 3, backoffMs: 1000, backoffMultiplier: 2 },
|
||||
});
|
||||
console.log(stats); // { processed: 2, succeeded: 2, failed: 0, dead: 0, retried: 0 }
|
||||
```
|
||||
@@ -0,0 +1,95 @@
|
||||
# queue-error
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Defines the error hierarchy for `@suaveplan/queue`. All errors extend `SuaveplanError` via a shared `QueueError` base class, giving them stable numeric codes, cause-chain traversal, and OTel span-event integration from `@suaveplan/error`.
|
||||
|
||||
Error codes are registered once via `defineCodes("queue", {...})` so each error can be identified numerically in distributed tracing, log pipelines, and wire protocols without relying on class names that may be minified.
|
||||
|
||||
## Features
|
||||
|
||||
- `QueueError` — base class for the entire queue error family
|
||||
- `QueueFullError` — capacity exceeded on enqueue
|
||||
- `QueueJobNotFoundError` — job ID lookup failure (includes optional `jobId` in message)
|
||||
- `QueueWorkerAbortedError` — worker stopped via `AbortSignal`
|
||||
- `QueueHandlerFailedError` — job handler threw an unhandled error
|
||||
- `QueueInvalidStateError` — illegal job state transition (includes `from`/`to` in message)
|
||||
- `QueueDeadLetterFailedError` — dead-letter enqueue failure
|
||||
- All codes under the `"queue"` domain namespace via `defineCodes`
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { QueueFullError, QueueHandlerFailedError } from "@suaveplan/queue/errors/queue-error";
|
||||
|
||||
try {
|
||||
throw new QueueFullError({ message: "Memory queue at capacity" });
|
||||
} catch (err) {
|
||||
if (err instanceof QueueFullError) {
|
||||
console.error("Queue full, code:", err.code); // stable numeric code
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
QueueInvalidStateError,
|
||||
QueueJobNotFoundError,
|
||||
QUEUE_CODES,
|
||||
} from "@suaveplan/queue/errors/queue-error";
|
||||
|
||||
// State transition with context
|
||||
throw new QueueInvalidStateError({
|
||||
from: "completed",
|
||||
to: "processing",
|
||||
});
|
||||
|
||||
// Job lookup failure with ID
|
||||
const err = new QueueJobNotFoundError({ jobId: "job_abc123" });
|
||||
console.log(err.message); // "Job not found: job_abc123"
|
||||
console.log(err.code === QUEUE_CODES.QUEUE_JOB_NOT_FOUND); // true
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `QUEUE_CODES`
|
||||
|
||||
Stable code map: `QUEUE_FULL=1`, `QUEUE_JOB_NOT_FOUND=2`, `QUEUE_WORKER_ABORTED=3`, `QUEUE_HANDLER_FAILED=4`, `QUEUE_INVALID_STATE=5`, `QUEUE_DEAD_LETTER_FAILED=6`.
|
||||
|
||||
### `QueueError`
|
||||
|
||||
Base class. Extends `SuaveplanError`, domain `"queue"`.
|
||||
|
||||
### `QueueFullError(options?)`
|
||||
|
||||
- `options.message?` — override default message
|
||||
- `options.cause?` — wrapped cause
|
||||
|
||||
### `QueueJobNotFoundError(options?)`
|
||||
|
||||
- `options.jobId?` — included in default message
|
||||
|
||||
### `QueueWorkerAbortedError(options?)`
|
||||
|
||||
Thrown when `AbortSignal` fires during worker processing.
|
||||
|
||||
### `QueueHandlerFailedError(options?)`
|
||||
|
||||
Wraps the original handler error as `cause`.
|
||||
|
||||
### `QueueInvalidStateError(options?)`
|
||||
|
||||
- `options.from?` / `options.to?` — state labels included in default message
|
||||
|
||||
### `QueueDeadLetterFailedError(options?)`
|
||||
|
||||
Thrown when the dead-letter enqueue itself fails.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`defineCodes` from `@suaveplan/error` allocates a namespace-prefixed numeric code map. This approach prevents code collisions across packages and survives minification. Each error class declares its code as both a static and instance property to support `instanceof` checks and code-based switching.
|
||||
@@ -0,0 +1,111 @@
|
||||
# job
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
The `Job<T>` class is the fundamental unit of work in `@suaveplan/queue`. It manages a job's lifecycle through a strict state machine — `pending → processing → completed | failed → dead` — and records timing information and error context at each transition.
|
||||
|
||||
All state transitions are guarded: attempting an invalid transition (e.g., marking a `"completed"` job as `"processing"`) throws `QueueInvalidStateError`, preventing silent data corruption in concurrent workers.
|
||||
|
||||
## Features
|
||||
|
||||
- Strict state machine with `QueueInvalidStateError` on invalid transitions
|
||||
- Tracks `attempts`, `processedAt`, `completedAt`, `failedAt`, and `error`
|
||||
- Injectable `now` factory on every transition for deterministic testing
|
||||
- Priority and metadata stored at construction time
|
||||
- Delay-based `eligibleAt` timestamp computed from `delayMs` option
|
||||
- `isEligible()` for delayed-queue comparison
|
||||
- `toJSON()` for structured serialisation / logging
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { Job } from "@suaveplan/queue/job/job";
|
||||
|
||||
const job = new Job("job_1", { userId: 42 });
|
||||
job.markProcessing();
|
||||
job.markCompleted();
|
||||
console.log(job.state); // "completed"
|
||||
console.log(job.attempts); // 1
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { Job } from "@suaveplan/queue/job/job";
|
||||
import { QueueInvalidStateError } from "@suaveplan/queue/errors/queue-error";
|
||||
|
||||
// Deterministic timestamps in tests
|
||||
const ts = ["2024-01-01T00:00:00Z", "2024-01-01T00:00:01Z", "2024-01-01T00:00:02Z"];
|
||||
let i = 0;
|
||||
const now = () => ts[i++]!;
|
||||
|
||||
const job = new Job("j1", "payload", { delayMs: 5000 }, now);
|
||||
console.log(job.isEligible(() => Date.now())); // false (not yet eligible)
|
||||
|
||||
job.markProcessing(now);
|
||||
job.markFailed(new Error("connection reset"), now);
|
||||
console.log(job.state); // "failed"
|
||||
console.log(job.attempts); // 1
|
||||
|
||||
// Retry
|
||||
job.markProcessing(now);
|
||||
job.markCompleted(now);
|
||||
console.log(job.state); // "completed"
|
||||
|
||||
// Invalid transition throws
|
||||
try {
|
||||
job.markDead(now);
|
||||
} catch (err) {
|
||||
console.log(err instanceof QueueInvalidStateError); // true
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `new Job(id, payload, options?, now?)`
|
||||
|
||||
- `id: string` — unique identifier
|
||||
- `payload: T` — the work data
|
||||
- `options.priority?: number` — default `0`
|
||||
- `options.delayMs?: number` — delay before eligible, default `0`
|
||||
- `options.metadata?: Record<string, unknown>` — attached metadata
|
||||
- `now?: () => string` — injectable ISO timestamp factory
|
||||
|
||||
### `job.state: JobState`
|
||||
|
||||
Read-only current state: `"pending" | "processing" | "completed" | "failed" | "dead"`.
|
||||
|
||||
### `job.attempts: number`
|
||||
|
||||
Number of `markProcessing()` calls made.
|
||||
|
||||
### `job.markProcessing(now?): void`
|
||||
|
||||
Transitions `pending | failed → processing`. Increments `attempts`.
|
||||
|
||||
### `job.markCompleted(now?): void`
|
||||
|
||||
Transitions `processing → completed`.
|
||||
|
||||
### `job.markFailed(err, now?): void`
|
||||
|
||||
Transitions `processing → failed`. Wraps `err` in `QueueHandlerFailedError`.
|
||||
|
||||
### `job.markDead(now?): void`
|
||||
|
||||
Transitions `failed → dead`.
|
||||
|
||||
### `job.isEligible(now?): boolean`
|
||||
|
||||
Returns `true` when `now() >= eligibleAt`. Defaults to `Date.now()`.
|
||||
|
||||
### `job.toJSON(): Record<string, unknown>`
|
||||
|
||||
Plain object representation for serialisation.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The state machine uses `protected` fields so that test subclasses or extensions can inspect internal state without public API pollution. The `now` parameter is injected to every transition method rather than captured at construction time, allowing fine-grained control in tests where each transition needs a distinct timestamp.
|
||||
@@ -0,0 +1,65 @@
|
||||
# delayed-queue
|
||||
|
||||
## Purpose
|
||||
|
||||
`DelayedQueue<T>` is an explicit alias for a delay-aware FIFO queue. It extends `MemoryQueue<T>` to make the delay intent clear at the call site. Jobs enqueued with a `delayMs` option are held back from `dequeue()` until that many milliseconds have elapsed since enqueue time.
|
||||
|
||||
The core delay logic lives in `Job.isEligible()` and `MemoryQueue.dequeue()`. `DelayedQueue` is a thin semantic wrapper that documents intent in the type system and provides a dedicated factory.
|
||||
|
||||
## Features
|
||||
|
||||
- Inherits all `MemoryQueue` capabilities: FIFO ordering, `maxSize`, counters
|
||||
- Jobs with `delayMs > 0` are skipped by `dequeue()` until eligible
|
||||
- Named class export makes delayed semantics explicit in application code
|
||||
- `createDelayedQueue` factory for ergonomic creation
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createDelayedQueue } from "@suaveplan/queue/queue/delayed-queue";
|
||||
|
||||
const queue = createDelayedQueue<string>();
|
||||
await queue.enqueue("immediate");
|
||||
await queue.enqueue("delayed", { delayMs: 60_000 }); // 1 minute delay
|
||||
|
||||
const now = await queue.dequeue();
|
||||
console.log(now?.payload); // "immediate"
|
||||
const notYet = await queue.dequeue();
|
||||
console.log(notYet); // undefined (still delayed)
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { createDelayedQueue } from "@suaveplan/queue/queue/delayed-queue";
|
||||
|
||||
// Scheduled retry: re-enqueue with exponential backoff delay
|
||||
async function scheduleRetry(payload: { taskId: string }, attempt: number) {
|
||||
const delayMs = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s, 8s...
|
||||
const queue = createDelayedQueue<typeof payload>();
|
||||
const jobId = await queue.enqueue(payload, { delayMs, metadata: { attempt } });
|
||||
console.log(`Retrying task ${payload.taskId} in ${delayMs}ms, job ${jobId}`);
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `createDelayedQueue<T>(options?): DelayedQueue<T>`
|
||||
|
||||
- `options.maxSize?` — capacity cap; throws `QueueFullError` when exceeded
|
||||
|
||||
### `DelayedQueue<T>`
|
||||
|
||||
Extends `MemoryQueue<T>`. All inherited methods available.
|
||||
|
||||
#### `enqueue(payload, options?): Promise<string>`
|
||||
|
||||
Delegates to `MemoryQueue.enqueue`. `options.delayMs` activates delay logic.
|
||||
|
||||
#### `dequeue(): Promise<Job<T> | undefined>`
|
||||
|
||||
Inherited from `MemoryQueue`. Returns first job where `job.isEligible()` is true.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`DelayedQueue` does not need a polling timer or scheduler. The caller is responsible for calling `dequeue()` at appropriate intervals; the queue simply filters by eligibility at dequeue time. This keeps the implementation stateless and easily testable — pass a synthetic clock to `Job` at enqueue time and control `Date.now()` in tests with a stub.
|
||||
@@ -0,0 +1,86 @@
|
||||
# memory-queue
|
||||
|
||||
## Purpose
|
||||
|
||||
`MemoryQueue<T>` is the foundational FIFO queue for `@suaveplan/queue`. It stores jobs in memory in insertion order and dequeues the first eligible job (respecting `delayMs`). An optional `maxSize` cap enforces bounded operation, throwing `QueueFullError` when the limit is reached.
|
||||
|
||||
This queue is suitable for single-process scenarios — worker pools, test harnesses, and local development. For distributed systems, wrap it behind an adapter that delegates to a persistent store.
|
||||
|
||||
## Features
|
||||
|
||||
- Pure in-memory FIFO ordering
|
||||
- Optional `maxSize` cap with `QueueFullError` on overflow
|
||||
- Delay support via `Job.isEligible()` — dequeues skip ineligible jobs
|
||||
- `enqueueCount` / `dequeueCount` counters for observability
|
||||
- `clear()` for draining between tests
|
||||
- Implements `QueueLike<T>` for compatibility with `QueueWorker`
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createMemoryQueue } from "@suaveplan/queue/queue/memory-queue";
|
||||
|
||||
const queue = createMemoryQueue<string>();
|
||||
await queue.enqueue("task-1");
|
||||
await queue.enqueue("task-2");
|
||||
|
||||
const job = await queue.dequeue();
|
||||
console.log(job?.payload); // "task-1"
|
||||
console.log(queue.size()); // 1
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { createMemoryQueue } from "@suaveplan/queue/queue/memory-queue";
|
||||
import { QueueFullError } from "@suaveplan/queue/errors/queue-error";
|
||||
|
||||
const bounded = createMemoryQueue<number>({ maxSize: 2 });
|
||||
await bounded.enqueue(1);
|
||||
await bounded.enqueue(2);
|
||||
try {
|
||||
await bounded.enqueue(3); // throws QueueFullError
|
||||
} catch (err) {
|
||||
console.log(err instanceof QueueFullError); // true
|
||||
}
|
||||
|
||||
// Delayed job: enqueue with 100ms delay
|
||||
await bounded.clear();
|
||||
await bounded.enqueue(42, { delayMs: 100 });
|
||||
const notYet = await bounded.dequeue();
|
||||
console.log(notYet); // undefined (not yet eligible)
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `createMemoryQueue<T>(options?): MemoryQueue<T>`
|
||||
|
||||
Factory function. `options.maxSize` defaults to unbounded.
|
||||
|
||||
### `MemoryQueue<T>`
|
||||
|
||||
Implements `QueueLike<T>`.
|
||||
|
||||
#### `enqueue(payload, options?): Promise<string>`
|
||||
|
||||
Returns the generated job ID. Throws `QueueFullError` when at capacity.
|
||||
|
||||
#### `dequeue(): Promise<Job<T> | undefined>`
|
||||
|
||||
Returns the first eligible job, or `undefined` if none.
|
||||
|
||||
#### `size(): number`
|
||||
|
||||
Current number of jobs (including delayed ones).
|
||||
|
||||
#### `clear(): void`
|
||||
|
||||
Removes all jobs.
|
||||
|
||||
#### `enqueueCount: number` / `dequeueCount: number`
|
||||
|
||||
Monotonically increasing counters.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The `QueueFullError` import is dynamic (`await import(...)`) to avoid a top-level circular import issue between the error module and the queue module. Job IDs are generated with `Date.now()` + a random suffix — suitable for in-memory uniqueness but not globally unique across processes; replace `_generateId()` in a subclass for UUID-based IDs.
|
||||
@@ -0,0 +1,70 @@
|
||||
# priority-queue
|
||||
|
||||
## Purpose
|
||||
|
||||
`PriorityQueue<T>` extends `MemoryQueue<T>` to provide priority-ordered job scheduling. Jobs with higher `priority` values (set via `EnqueueOptions.priority`) are placed ahead of lower-priority jobs in the queue. When priorities are equal, insertion order (FIFO) is preserved.
|
||||
|
||||
This is useful for latency-sensitive systems where some job categories should skip ahead of background work without requiring a separate queue.
|
||||
|
||||
## Features
|
||||
|
||||
- Descending-priority ordering: higher value = dequeued first
|
||||
- FIFO tie-breaking among equal-priority jobs
|
||||
- Inherits all `MemoryQueue` capabilities: `maxSize`, delay support, counters
|
||||
- O(n) insertion — optimal for typical in-memory queue sizes
|
||||
- `createPriorityQueue` factory for ergonomic creation
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createPriorityQueue } from "@suaveplan/queue/queue/priority-queue";
|
||||
|
||||
const queue = createPriorityQueue<string>();
|
||||
await queue.enqueue("low", { priority: 1 });
|
||||
await queue.enqueue("high", { priority: 10 });
|
||||
await queue.enqueue("medium", { priority: 5 });
|
||||
|
||||
const first = await queue.dequeue();
|
||||
console.log(first?.payload); // "high"
|
||||
const second = await queue.dequeue();
|
||||
console.log(second?.payload); // "medium"
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { createPriorityQueue } from "@suaveplan/queue/queue/priority-queue";
|
||||
|
||||
// Real-time + batch processing: real-time tasks get priority 100, batch get 0
|
||||
const queue = createPriorityQueue<{ type: string; data: unknown }>({ maxSize: 1000 });
|
||||
|
||||
await queue.enqueue({ type: "batch", data: null }, { priority: 0 });
|
||||
await queue.enqueue({ type: "batch", data: null }, { priority: 0 });
|
||||
await queue.enqueue({ type: "realtime", data: "urgent" }, { priority: 100 });
|
||||
|
||||
const job = await queue.dequeue();
|
||||
console.log(job?.payload.type); // "realtime"
|
||||
|
||||
// FIFO among same priority
|
||||
const j2 = await queue.dequeue();
|
||||
const j3 = await queue.dequeue();
|
||||
console.log(j2?.payload.type, j3?.payload.type); // "batch", "batch"
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `createPriorityQueue<T>(options?): PriorityQueue<T>`
|
||||
|
||||
- `options.maxSize?` — capacity cap; throws `QueueFullError` when exceeded
|
||||
|
||||
### `PriorityQueue<T>`
|
||||
|
||||
Extends `MemoryQueue<T>`. All inherited methods available.
|
||||
|
||||
#### `enqueue(payload, options?): Promise<string>`
|
||||
|
||||
Inserts at the correct sorted position. `options.priority` defaults to `0`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Insertion uses a linear scan to find the first position where the stored job has a lower priority than the new job, then splices it in. For typical queue sizes (hundreds to low thousands of jobs), this is performant and avoids the complexity of a binary heap. If queue sizes routinely exceed tens of thousands, a heap-based implementation would improve amortised insertion from O(n) to O(log n).
|
||||
@@ -0,0 +1,81 @@
|
||||
# queue-types
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Shared type definitions and Zod schemas for the `@suaveplan/queue` package. This module is the single source of truth for all queue-related interfaces, ensuring consistent type safety across job, queue, and worker implementations.
|
||||
|
||||
These types are deliberately kept in their own module so that any consumer can import just the schemas for runtime validation without pulling in the full queue implementation.
|
||||
|
||||
## Features
|
||||
|
||||
- `JobState` union type and Zod schema for job lifecycle state machine
|
||||
- `RetryPolicy` interface and schema for configurable backoff strategies
|
||||
- `EnqueueOptions` for attaching priority, delay, and metadata to jobs
|
||||
- `WorkerOptions` for configuring concurrency and retry behaviour
|
||||
- `QueueLike<T>` structural interface enabling duck-typed queue implementations
|
||||
- All Zod schemas use strict validation (no `.passthrough()`)
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type { JobState, RetryPolicy } from "@suaveplan/queue/types/queue-types";
|
||||
import { RetryPolicySchema } from "@suaveplan/queue/types/queue-types";
|
||||
|
||||
const policy: RetryPolicy = { maxAttempts: 3, backoffMs: 500, backoffMultiplier: 2 };
|
||||
const parsed = RetryPolicySchema.parse(policy);
|
||||
console.log(parsed.maxAttempts); // 3
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { EnqueueOptionsSchema, WorkerOptionsSchema } from "@suaveplan/queue/types/queue-types";
|
||||
|
||||
// Validate worker config from an untrusted source
|
||||
const rawConfig = { concurrency: 4, retryPolicy: { maxAttempts: 5, backoffMs: 200, backoffMultiplier: 1.5 } };
|
||||
const workerOpts = WorkerOptionsSchema.parse(rawConfig);
|
||||
|
||||
// Validate enqueue options at runtime
|
||||
const rawEnqueue = { priority: 10, delayMs: 2000, metadata: { source: "api" } };
|
||||
const enqueueOpts = EnqueueOptionsSchema.parse(rawEnqueue);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `JobState`
|
||||
|
||||
Union of `"pending" | "processing" | "completed" | "failed" | "dead"`.
|
||||
|
||||
### `JobStateSchema`
|
||||
|
||||
`z.enum([...])` — runtime validation for `JobState`.
|
||||
|
||||
### `RetryPolicy`
|
||||
|
||||
- `maxAttempts: number` — retries after initial failure (0 = no retries)
|
||||
- `backoffMs: number` — initial delay in ms
|
||||
- `backoffMultiplier: number` — multiplier applied to delay each retry
|
||||
|
||||
### `EnqueueOptions`
|
||||
|
||||
- `priority?: number` — higher values processed first in priority queues
|
||||
- `delayMs?: number` — milliseconds before job becomes eligible
|
||||
- `metadata?: Record<string, unknown>` — arbitrary job metadata
|
||||
|
||||
### `WorkerOptions`
|
||||
|
||||
- `concurrency: number` — max simultaneous jobs
|
||||
- `retryPolicy: RetryPolicy` — retry configuration
|
||||
- `deadLetterQueue?: QueueLike<unknown>` — receives exhausted jobs
|
||||
- `signal?: AbortSignal` — cancels the worker
|
||||
|
||||
### `QueueLike<T>`
|
||||
|
||||
Structural interface: `enqueue`, `dequeue`, `size`, `clear`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`QueueLike<T>` uses a structural interface instead of an abstract class to allow any object with the right shape to act as a queue. The `Job` import inside `QueueLike` is a type-only dynamic import to break the circular dependency between `queue-types.ts` and `job.ts`.
|
||||
@@ -0,0 +1,100 @@
|
||||
# queue-worker
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
`QueueWorker<T>` drains a `QueueLike<T>` queue by calling a user-supplied handler for each job. It manages concurrency (up to N simultaneous jobs), implements exponential-backoff retry, forwards exhausted jobs to an optional dead-letter queue, and honours `AbortSignal` for graceful shutdown.
|
||||
|
||||
The worker is intentionally decoupled from any specific queue implementation — it accepts any object that satisfies `QueueLike<T>`, allowing `MemoryQueue`, `PriorityQueue`, `DelayedQueue`, or a custom adapter to be swapped in.
|
||||
|
||||
## Features
|
||||
|
||||
- Configurable concurrency: processes up to N jobs simultaneously
|
||||
- Exponential-backoff retry with configurable `maxAttempts`, `backoffMs`, and `backoffMultiplier`
|
||||
- Dead-letter queue forwarding when all retries are exhausted
|
||||
- `AbortSignal` cancellation: throws `QueueWorkerAbortedError` on abort
|
||||
- `WorkerStats` snapshot (processed, succeeded, failed, dead, retried)
|
||||
- `runWorker()` convenience wrapper for fire-and-forget usage
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createMemoryQueue } from "@suaveplan/queue/queue/memory-queue";
|
||||
import { QueueWorker } from "@suaveplan/queue/worker/queue-worker";
|
||||
|
||||
const queue = createMemoryQueue<string>();
|
||||
await queue.enqueue("hello");
|
||||
await queue.enqueue("world");
|
||||
|
||||
const worker = new QueueWorker(queue, async (job) => {
|
||||
console.log("Processing:", job.payload);
|
||||
});
|
||||
await worker.run();
|
||||
console.log(worker.stats); // { processed: 2, succeeded: 2, failed: 0, dead: 0, retried: 0 }
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { createMemoryQueue } from "@suaveplan/queue/queue/memory-queue";
|
||||
import { runWorker } from "@suaveplan/queue/worker/queue-worker";
|
||||
|
||||
const mainQueue = createMemoryQueue<{ taskId: string }>();
|
||||
const deadLetterQueue = createMemoryQueue<unknown>();
|
||||
|
||||
await mainQueue.enqueue({ taskId: "a" });
|
||||
await mainQueue.enqueue({ taskId: "b" });
|
||||
|
||||
let callCount = 0;
|
||||
const stats = await runWorker(
|
||||
mainQueue,
|
||||
async (job) => {
|
||||
callCount++;
|
||||
if (callCount <= 4) throw new Error("transient failure"); // triggers retries
|
||||
// succeeds on later attempts
|
||||
},
|
||||
{
|
||||
concurrency: 2,
|
||||
retryPolicy: { maxAttempts: 2, backoffMs: 10, backoffMultiplier: 2 },
|
||||
deadLetterQueue,
|
||||
}
|
||||
);
|
||||
|
||||
console.log(stats.dead); // jobs that exhausted all retries
|
||||
console.log(deadLetterQueue.size()); // dead-letter queue depth
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `QueueWorker<T>`
|
||||
|
||||
#### `new QueueWorker(queue, handler, options?)`
|
||||
|
||||
- `queue: QueueLike<T>` — source queue
|
||||
- `handler: (job: Job<T>) => Promise<void>` — async job processor; throw to fail the job
|
||||
- `options.concurrency` — default `1`
|
||||
- `options.retryPolicy` — default `{ maxAttempts: 3, backoffMs: 1000, backoffMultiplier: 2 }`
|
||||
- `options.deadLetterQueue?` — receives exhausted jobs
|
||||
- `options.signal?` — `AbortSignal` for cancellation
|
||||
|
||||
#### `worker.run(): Promise<void>`
|
||||
|
||||
Drains the queue. Resolves when empty. Throws `QueueWorkerAbortedError` on abort.
|
||||
|
||||
#### `worker.stats: WorkerStats`
|
||||
|
||||
Snapshot: `{ processed, succeeded, failed, dead, retried }`.
|
||||
|
||||
#### `worker.running: boolean`
|
||||
|
||||
`true` while `run()` has not resolved.
|
||||
|
||||
### `runWorker<T>(queue, handler, options?): Promise<WorkerStats>`
|
||||
|
||||
Convenience factory: creates a worker, runs it, and returns stats.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The drain loop maintains a `Set<Promise<void>>` of in-flight job promises. When `slots.size` reaches `concurrency`, it awaits `Promise.race(slots)` to free up a slot before dequeuing the next job. This achieves true concurrent processing without spawning unbound promises. Dead-letter forwarding errors wrap the original error as `QueueDeadLetterFailedError` so callers can distinguish delivery failures from job failures.
|
||||
Reference in New Issue
Block a user