first commit
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
# MeshCircuitBreaker
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Per-path circuit-breaker FSM that walks `CLOSED → OPEN → HALF_OPEN → CLOSED`. Stops repeated wire-level retries against a known-dead path so retries don't cascade into healthy paths. Each `(localHostId, nextHopHostId)` pair gets its own breaker instance, identified by `pathId`.
|
||||
|
||||
## Features
|
||||
|
||||
- Three-state FSM with deterministic transitions.
|
||||
- Configurable `failureThreshold` (default 5), `successThreshold` (default 3), `openDurationMs` (default 30 s), `halfOpenMaxCalls` (default 3).
|
||||
- Auto-transition `OPEN → HALF_OPEN` after the cooldown elapses (lazy-evaluated on `getState`/`canExecute`).
|
||||
- Throws `INVALID_TRANSITION` on `recordSuccess()` from `OPEN` to surface programmer errors loudly.
|
||||
- Bus-free: state-change listeners installed via `on(handler)` for downstream observability without coupling to a specific bus.
|
||||
- Injectable `clock` for deterministic tests.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { MeshCircuitBreaker } from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
const cb = new MeshCircuitBreaker({ pathId: "meshA:host1→meshB:host2" });
|
||||
|
||||
if (cb.canExecute()) {
|
||||
try {
|
||||
await sendThroughPath();
|
||||
cb.recordSuccess();
|
||||
} catch (err) {
|
||||
cb.recordFailure();
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { MeshCircuitBreaker } from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
let now = 0;
|
||||
const cb = new MeshCircuitBreaker({
|
||||
pathId: "p",
|
||||
failureThreshold: 2,
|
||||
successThreshold: 1,
|
||||
openDurationMs: 100,
|
||||
halfOpenMaxCalls: 1,
|
||||
clock: () => now,
|
||||
});
|
||||
|
||||
cb.on((state) => {
|
||||
console.log("breaker", cb.pathId, "→", state);
|
||||
});
|
||||
|
||||
cb.recordFailure();
|
||||
cb.recordFailure(); // OPEN
|
||||
now += 200;
|
||||
// canExecute lazily transitions to HALF_OPEN
|
||||
if (cb.canExecute()) {
|
||||
cb.recordSuccess(); // CLOSED
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `type CircuitBreakerState = "CLOSED" | "OPEN" | "HALF_OPEN"`.
|
||||
- `interface MeshCircuitBreakerOptions { pathId, failureThreshold?, successThreshold?, openDurationMs?, halfOpenMaxCalls?, clock? }`.
|
||||
- `getState(): CircuitBreakerState` — lazy `OPEN → HALF_OPEN` if cooled down.
|
||||
- `canExecute(): boolean` — gating check.
|
||||
- `recordSuccess()` / `recordFailure()` — feedback hooks.
|
||||
- `on(handler) → () => void` — state-change subscription.
|
||||
- `pathId: string` — read-only public path identifier.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The auto-transition from `OPEN → HALF_OPEN` is lazy because we don't run a clock-driven timer inside the breaker — every `getState`/`canExecute` call is the moment we recompute the state if cooldown has elapsed. This avoids leaking timers and keeps the breaker fully synchronous from the caller's perspective. `recordSuccess()` from `OPEN` is a programmer error (the FSM should be `HALF_OPEN` first), so the breaker throws `INVALID_TRANSITION` to surface the bug rather than silently accepting it. In `HALF_OPEN`, `halfOpenMaxCalls` is the cap on trial executions; once exceeded, `canExecute` returns false until the next successful close.
|
||||
@@ -0,0 +1,57 @@
|
||||
# CB_CODES
|
||||
|
||||
## Purpose
|
||||
|
||||
Frozen error-code registry for the mesh circuit-breaker plugin. Two codes cover the full surface: `PATH_OPEN` when a request is rejected by an open breaker, and `INVALID_TRANSITION` when the FSM is asked to do something unrepresentable (e.g. `recordSuccess()` while still in `OPEN`). The `CircuitBreakerError` class extends the cross-package `SuaveplanError` so consumers can pattern-match on `code` without parsing message strings.
|
||||
|
||||
## Features
|
||||
|
||||
- Two frozen codes registered under the `CB` namespace.
|
||||
- `CircuitBreakerError extends SuaveplanError<CbErrorCode>`.
|
||||
- `name: "CircuitBreakerError"` on every instance for log filtering and dashboard alerting.
|
||||
- `cause` and `context` propagation via `SuaveplanErrorOptions`.
|
||||
- Codes are stable across versions — once registered they never disappear, only new codes are appended.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
CB_CODES,
|
||||
CircuitBreakerError,
|
||||
} from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
throw new CircuitBreakerError(CB_CODES.PATH_OPEN, "breaker.path-open", {
|
||||
context: { pathId: "meshA:host1→meshB:host2" },
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
CB_CODES,
|
||||
CircuitBreakerError,
|
||||
} from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
try {
|
||||
sendThroughRelay(envelope);
|
||||
} catch (err) {
|
||||
if (
|
||||
err instanceof CircuitBreakerError &&
|
||||
err.code === CB_CODES.PATH_OPEN
|
||||
) {
|
||||
return sendThroughBackup(envelope);
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `CB_CODES` — frozen `{ PATH_OPEN, INVALID_TRANSITION }`.
|
||||
- `type CbErrorCode` — union of the registered code values.
|
||||
- `class CircuitBreakerError extends SuaveplanError<CbErrorCode>` — typed throwable for every failure mode the plugin surfaces.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The registry is intentionally tight — two codes are enough because every other failure mode bubbles up as the underlying transport error (the breaker just wraps the wire-level error in `recordFailure`). `INVALID_TRANSITION` is the only "programmer-error" code; it should never appear in production unless someone calls `recordSuccess()` from `OPEN` state, which the FSM forbids. The package barrel side-effect-imports `./error-codes.js` so subpath consumers register the codes too — relying on a tree-shaken root import is unsafe when consumers may import deeper paths only.
|
||||
@@ -0,0 +1,75 @@
|
||||
# createCircuitBreakerMiddleware
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Factory that produces the relay-middleware glue between a `CircuitBreakerRegistry` and any next-hop send pipeline. The middleware fast-fails with `PATH_OPEN` when the relevant breaker is open and otherwise records the outcome of `next()` so the breaker accumulates the right success/failure counts.
|
||||
|
||||
## Features
|
||||
|
||||
- `buildPathId` is injectable; default is `${local}→${next}`.
|
||||
- Throws `CircuitBreakerError(CB.PATH_OPEN)` on the open path, with `context: { pathId }`.
|
||||
- Records success on `next()` resolve, failure on `next()` reject (and re-throws so callers see the original error).
|
||||
- Pure function: returns a middleware closure that holds no mutable state of its own — all state lives in the `registry`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
CircuitBreakerRegistry,
|
||||
createCircuitBreakerMiddleware,
|
||||
} from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
const registry = new CircuitBreakerRegistry();
|
||||
const middleware = createCircuitBreakerMiddleware({
|
||||
registry,
|
||||
localHostId: "host1",
|
||||
});
|
||||
|
||||
await middleware(envelope, "host2", async () => {
|
||||
await rawSend(envelope);
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
CB_CODES,
|
||||
CircuitBreakerError,
|
||||
CircuitBreakerRegistry,
|
||||
createCircuitBreakerMiddleware,
|
||||
} from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
const middleware = createCircuitBreakerMiddleware({
|
||||
registry,
|
||||
localHostId: "host1",
|
||||
buildPathId: (local, next) => `mesh:${local}::${next}`,
|
||||
});
|
||||
|
||||
try {
|
||||
await middleware(envelope, "host2", () => realSend(envelope));
|
||||
} catch (err) {
|
||||
if (
|
||||
err instanceof CircuitBreakerError &&
|
||||
err.code === CB_CODES.PATH_OPEN
|
||||
) {
|
||||
await sendThroughBackup(envelope);
|
||||
return;
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `type PathIdBuilder = (localHostId, nextHopHostId) => string`.
|
||||
- `interface CircuitBreakerMiddlewareOptions { registry, localHostId, buildPathId? }`.
|
||||
- `type RelayNext = () => Promise<void>`.
|
||||
- `createCircuitBreakerMiddleware(options): (envelope, nextHopHostId, next) => Promise<void>`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The middleware is a free function (rather than a class) because it has no internal state — the registry owns all the breaker state and the closure only needs `localHostId` + `buildPathId`. This makes the middleware trivially safe to share across many concurrent sends. The thrown `CircuitBreakerError` always carries `pathId` in its context so log filters and dashboards can group breaker rejections by path. Callers MUST treat the thrown error as a hard fast-fail signal — retrying immediately defeats the entire point of the breaker.
|
||||
@@ -0,0 +1,80 @@
|
||||
# HalfOpenProbe
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Periodically issues lightweight probe envelopes against every relay path the registry reports as `HALF_OPEN`, then records success/failure into the corresponding breaker. Decoupling the probe from real traffic means a flaky path can be re-tested without risking application payload — if the probe fails the path stays open and real callers continue to fast-fail.
|
||||
|
||||
## Features
|
||||
|
||||
- Configurable `probeIntervalMs` (default 5 s).
|
||||
- Pluggable `probeMessageFactory` so callers can attach signing/auth metadata.
|
||||
- Injectable `send`, `setIntervalFn`, `clearIntervalFn` for deterministic tests.
|
||||
- Idempotent `start()` and `stop()`; no leaked timers.
|
||||
- `tick()` is exposed publicly so test harnesses can step the loop manually.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
CircuitBreakerRegistry,
|
||||
HalfOpenProbe,
|
||||
} from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
const registry = new CircuitBreakerRegistry();
|
||||
const probe = new HalfOpenProbe({
|
||||
registry,
|
||||
probeMessageFactory: (pathId) => ({
|
||||
id: crypto.randomUUID(),
|
||||
type: "relay:probe",
|
||||
senderId: "self",
|
||||
payload: null,
|
||||
timestamp: Date.now(),
|
||||
headers: { pathId },
|
||||
ttl: 16,
|
||||
visitedNodes: [],
|
||||
}),
|
||||
send: async (pathId, envelope) => {
|
||||
await sendThroughRelay(pathId, envelope);
|
||||
},
|
||||
});
|
||||
probe.start();
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
CircuitBreakerRegistry,
|
||||
HalfOpenProbe,
|
||||
} from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
const registry = new CircuitBreakerRegistry();
|
||||
const probe = new HalfOpenProbe({
|
||||
registry,
|
||||
probeIntervalMs: 1_000,
|
||||
probeMessageFactory: () => makeProbeEnvelope(),
|
||||
send: realSend,
|
||||
setIntervalFn: testHarness.setInterval,
|
||||
clearIntervalFn: testHarness.clearInterval,
|
||||
});
|
||||
|
||||
probe.start();
|
||||
testHarness.advance(1_000);
|
||||
await probe.tick(); // explicit tick if the harness doesn't auto-fire
|
||||
probe.stop();
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `type ProbeSender = (pathId, envelope) => Promise<void>`.
|
||||
- `interface HalfOpenProbeOptions { registry, probeIntervalMs?, probeMessageFactory, send, setIntervalFn?, clearIntervalFn? }`.
|
||||
- `start()` — idempotent.
|
||||
- `stop()` — idempotent; cancels the timer.
|
||||
- `tick()` — runs one probe pass; resolves once every half-open path has a result.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`tick()` lists `OPEN` paths from the registry and then re-queries `getState(pathId)` to filter for `HALF_OPEN` — this catches the lazy `OPEN → HALF_OPEN` transition without forcing the registry to expose a separate "ready to probe" query. A failed probe trips the breaker back to `OPEN` via `recordFailure`; a successful probe accumulates toward `successThreshold` and eventually closes the breaker. The probe never throws — every send error is swallowed into `recordFailure` so a transient probe rejection doesn't crash the periodic loop.
|
||||
@@ -0,0 +1,68 @@
|
||||
# CircuitBreakerRegistry
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Owns one `MeshCircuitBreaker` per relay path and exposes a thin facade for cross-path queries (`listOpenPaths`, `getState(pathId)`). Centralising the breakers means there's exactly one place that decides "is this path healthy?" and middleware/probes can both consult it without state-sharing.
|
||||
|
||||
## Features
|
||||
|
||||
- Lazy construction: `getOrCreate` makes a breaker on first reference, returns the existing instance on subsequent calls.
|
||||
- Per-call config overrides on `getOrCreate` so different paths can have different thresholds.
|
||||
- Shared `defaults` config applied to every breaker that doesn't override.
|
||||
- `listOpenPaths()` for observability dashboards / health endpoints.
|
||||
- `remove(pathId)` for explicit teardown when a peer leaves the mesh.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { CircuitBreakerRegistry } from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
const registry = new CircuitBreakerRegistry();
|
||||
const cb = registry.getOrCreate("meshA:host1→meshB:host2");
|
||||
if (cb.canExecute()) {
|
||||
/* ... */
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { CircuitBreakerRegistry } from "@suaveplan/network-plugin-circuit-breaker";
|
||||
|
||||
const registry = new CircuitBreakerRegistry({
|
||||
defaults: {
|
||||
failureThreshold: 3,
|
||||
successThreshold: 2,
|
||||
openDurationMs: 10_000,
|
||||
halfOpenMaxCalls: 2,
|
||||
},
|
||||
});
|
||||
|
||||
// per-path override for a known-flaky link
|
||||
registry.getOrCreate("meshA:host1→meshC:host5", {
|
||||
failureThreshold: 10,
|
||||
});
|
||||
|
||||
const open = registry.listOpenPaths();
|
||||
if (open.length > 0) {
|
||||
metrics.gauge("mesh.open_breakers", open.length);
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `interface CircuitBreakerRegistryOptions { defaults? }`.
|
||||
- `getOrCreate(pathId, options?): MeshCircuitBreaker`.
|
||||
- `getState(pathId): CircuitBreakerState | undefined`.
|
||||
- `remove(pathId): boolean`.
|
||||
- `listOpenPaths(): readonly string[]`.
|
||||
- `paths(): readonly string[]` — every known path id, regardless of state.
|
||||
- `size(): number`.
|
||||
- `clear()`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
`getOrCreate` is the canonical constructor — the registry never returns a raw `new` breaker, so observers don't have to wonder whether they're looking at a registry-owned or one-off instance. Per-call options shallow-merge over `defaults`, with caller options winning. `listOpenPaths` consults `getState()` (not the cached state) so paths that have lazy-transitioned to `HALF_OPEN` are excluded from the open list — matching what callers expect from "which paths are dead right now?". The companion `paths()` accessor returns every known path so probe loops can re-query each one's live state without missing entries that have already moved to `HALF_OPEN`. This split keeps health-dashboard semantics ("just the dead ones") separate from probe-driver semantics ("everything we might want to retry").
|
||||
Reference in New Issue
Block a user