first commit

This commit is contained in:
2026-05-29 08:20:56 +01:00
commit 949299f58d
6368 changed files with 527176 additions and 0 deletions

View File

@@ -0,0 +1,24 @@
# @suaveplan/network-plugin-failover - Mesh failover: heartbeat manager, health monitor (alive → suspect → dead → recovering), promoter, coordinator, failover plugin
## Summary
Mesh failover plugin: per-peer heartbeat manager, alive→suspect→dead→recovering health monitor, debounced promoter that triggers emergency elections, and a coordinator that wires everything to the bus.
<!-- begin:auto-docs -->
## Documentation tree
- [failover](./src/failover/failover-coordinator.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [failover-coordinator](./src/failover/failover-coordinator.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [health](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [health-monitor](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [heartbeat](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [heartbeat-manager](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [failover-plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [promoter](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [promoter-manager](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [error-codes](./src/error-codes.md) — Frozen registry of error codes for the network-plugin-failover plugin. Two codes cover the surface: `INVALID_HEALTH_TRA…
<!-- end:auto-docs --> - [failover-coordinator](./src/failover/failover-coordinator.md) — <!-- begin:auto-docs --> _(no submodules)_- [health](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_- [health-monitor](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_- [heartbeat](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [heartbeat-manager](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_- [failover-plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_- [promoter](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [promoter-manager](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Frozen registry of error codes for the network-plugin-failover plugin. Two codes cover the surface: `INVALID_HEALTH_TRA…
- [failover-coordinator](./src/failover/failover-coordinator.md) — <!-- begin:auto-docs --> _(no submodules)_- [health](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_- [health-monitor](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_- [heartbeat](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [heartbeat-manager](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_- [failover-plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_- [promoter](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [promoter-manager](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Frozen registry of error codes for the network-plugin-failover plugin. Two codes cover the surface: `INVALID_HEALTH_TRA…
- [failover-coordinator](./src/failover/failover-coordinator.md) — <!-- begin:auto-docs --> _(no submodules)_- [health](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_- [health-monitor](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_- [heartbeat](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [heartbeat-manager](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_- [failover-plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_- [promoter](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [promoter-manager](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Frozen registry of error codes for the network-plugin-failover plugin. Two codes cover the surface: `INVALID_HEALTH_TRA…
- [failover-coordinator](./src/failover/failover-coordinator.md) — <!-- begin:auto-docs --> _(no submodules)_- [health](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_ - [health-monitor](./src/health/health-monitor.md) — <!-- begin:auto-docs --> _(no submodules)_- [heartbeat](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_ - [heartbeat-manager](./src/heartbeat/heartbeat-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_ - [failover-plugin](./src/plugin/failover-plugin.md) — <!-- begin:auto-docs --> _(no submodules)_- [promoter](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_ - [promoter-manager](./src/promoter/promoter-manager.md) — <!-- begin:auto-docs --> _(no submodules)_- [error-codes](./src/error-codes.md) — Frozen registry of error codes for the network-plugin-failover plugin. Two codes cover the surface: `INVALID_HEALTH_TRA…

View File

@@ -0,0 +1,60 @@
# NETWORK_FO_CODES
## Purpose
Frozen registry of error codes for the network-plugin-failover plugin. Two codes cover the surface: `INVALID_HEALTH_TRANSITION` when the health monitor is asked to make an unrepresentable state move, and `HEARTBEAT_NOT_RUNNING` when callers try to send a heartbeat after the manager has been stopped. The companion `NetworkFoError` extends `SuaveplanError<NetworkFoErrorCode>` so consumers pattern-match on `code` rather than parse messages.
## Features
- Two frozen codes registered under the `NET_FO` namespace.
- `NetworkFoError extends SuaveplanError<NetworkFoErrorCode>`.
- `name: "NetworkFoError"` on every instance — useful for log filters and dashboard alerts.
- `cause` and `context` propagation via `SuaveplanErrorOptions`.
- Codes are stable across versions; the registry only ever grows, never shrinks.
## Usage: Basic Example
```ts
import {
NETWORK_FO_CODES,
NetworkFoError,
} from "@suaveplan/network-plugin-failover";
throw new NetworkFoError(
NETWORK_FO_CODES.INVALID_HEALTH_TRANSITION,
"health.invalid-transition",
{ context: { from: "alive", to: "recovering" } }
);
```
## Usage: Advanced Example
```ts
import {
NETWORK_FO_CODES,
NetworkFoError,
} from "@suaveplan/network-plugin-failover";
try {
heartbeat.send("alice");
} catch (err) {
if (
err instanceof NetworkFoError &&
err.code === NETWORK_FO_CODES.HEARTBEAT_NOT_RUNNING
) {
await heartbeat.start();
return heartbeat.send("alice");
}
throw err;
}
```
## API Reference
- `NETWORK_FO_CODES` — frozen `{ INVALID_HEALTH_TRANSITION, HEARTBEAT_NOT_RUNNING }`.
- `type NetworkFoErrorCode` — union of registered code values.
- `class NetworkFoError extends SuaveplanError<NetworkFoErrorCode>` — typed throwable.
## Implementation Notes
`INVALID_HEALTH_TRANSITION` is a programmer-error code: the FSM only permits a fixed graph of moves and any other input is a bug. `HEARTBEAT_NOT_RUNNING` is recoverable — callers can `start()` and retry. Routine outcomes like "peer went from alive to suspect" do not throw at all; they are dispatched to the bus as `health:state-change` events. The package barrel side-effect-imports `./error-codes.js` so subpath consumers register the codes too. Future failure modes should be added as new `NET_FO.*` codes rather than overloading existing ones.

View File

@@ -0,0 +1,78 @@
# FailoverCoordinator
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Wires `HeartbeatManager`, `HealthMonitor`, and `PromoterManager` together via the bus. Subscribes to peer-lifecycle and pong events, sweeps stale peers on demand, and triggers `failover:emergency-election` when a peer transitions to dead. Centralising the wiring here means the surrounding plugin only has to construct one coordinator rather than re-implement the bus integration.
## Features
- Subscribes to four bus topics: `peer:connected`, `peer:disconnected`, `peer:pong`, and `health:state-change`.
- `start()` returns the wired-up coordinator; `stop()` unsubscribes every listener.
- `sweep()` finds stale peers from the heartbeat manager and feeds them to the health monitor.
- `peer:pong` is the canonical signal that a peer is alive — feeds both the heartbeat RTT and the health monitor's `recordSeen`.
- `health:state-change → dead` automatically fires the promoter's emergency-election trigger.
- All inter-component wiring happens through the bus, never through direct method calls between sibling components.
## Usage: Basic Example
```ts
import {
FailoverCoordinator,
HealthMonitor,
HeartbeatManager,
PromoterManager,
} from "@suaveplan/network-plugin-failover";
const heartbeat = new HeartbeatManager({ sender });
const health = new HealthMonitor(bus);
const promoter = new PromoterManager({ bus });
const coordinator = new FailoverCoordinator({
heartbeat,
health,
promoter,
bus,
});
coordinator.start();
```
## Usage: Advanced Example
```ts
import {
FailoverCoordinator,
HealthMonitor,
HeartbeatManager,
PromoterManager,
} from "@suaveplan/network-plugin-failover";
const coordinator = new FailoverCoordinator({
heartbeat,
health,
promoter,
bus,
});
coordinator.start();
heartbeat.start();
// Driven sweep — tests step the loop manually rather than running a
// timer inside the coordinator.
setInterval(() => {
const stale = coordinator.sweep();
metrics.gauge("failover.stale_peers", stale.length);
}, 1_000);
```
## API Reference
- `interface FailoverCoordinatorConfig { heartbeat, health, promoter, bus }`.
- `interface HealthBusListener` — subset of the bus surface the coordinator needs (`on` for the four topics).
- `start()` / `stop()` — wire / unwire every listener.
- `sweep(): readonly string[]` — feeds heartbeat-driven stale peers into the health monitor.
## Implementation Notes
`stop()` unsubscribes every listener via the unsubscribe-returning `on()` API, so multiple `start`/`stop` cycles do not leak listeners. `sweep` is intentionally not driven by a clock-loop inside the coordinator — that lets test harnesses step at deterministic intervals and prevents the coordinator from owning a teardown the consumer might forget about. The `peer:pong` listener calls `recordSeen` only when the heartbeat RTT was successfully matched (sequence-correlation guard) so spoofed pongs cannot upgrade a `dead` peer back to `alive`.

View File

@@ -0,0 +1,66 @@
# HealthMonitor
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Tracks per-peer health states (`alive | suspect | dead | recovering`) and publishes `health:state-change` on every transition. Push-driven via explicit calls — there is no internal clock loop. Decoupling the FSM from the cadence lets consumers (typically `FailoverCoordinator`) decide when to call `markStale`, which in turn keeps the monitor leak-free in tests.
## Features
- Four-state FSM with allowed-transitions matrix: `alive ↔ suspect`, `suspect → dead`, `dead → recovering`, `recovering → alive | dead`.
- Configurable thresholds: `suspectAfterMs` (default 3 s), `deadAfterMs` (default 10 s), `recoverAfterMs` (default 5 s).
- Pluggable bus (`emit("health:state-change", payload)`).
- Pluggable `clock` for deterministic tests.
- Throws `INVALID_HEALTH_TRANSITION` on illegal moves so bugs surface loudly rather than silently mutating state.
## Usage: Basic Example
```ts
import { HealthMonitor } from "@suaveplan/network-plugin-failover";
const monitor = new HealthMonitor(bus);
monitor.recordSeen("alice");
// later, after a missed heartbeat sweep:
monitor.markStale("alice");
```
## Usage: Advanced Example
```ts
import { HealthMonitor } from "@suaveplan/network-plugin-failover";
let now = 0;
const monitor = new HealthMonitor(bus, {
suspectAfterMs: 100,
deadAfterMs: 500,
clock: () => now,
});
monitor.recordSeen("alice");
now += 200;
monitor.markStale("alice"); // → suspect
now += 600;
monitor.markStale("alice"); // → dead
// recovery flow
monitor.markRecovering("alice"); // dead → recovering
monitor.markRecovered("alice"); // recovering → alive
```
## API Reference
- `type HealthState = "alive" | "suspect" | "dead" | "recovering"`.
- `interface HealthMonitorConfig { suspectAfterMs?, deadAfterMs?, recoverAfterMs?, clock? }`.
- `interface HealthBus { emit("health:state-change", { peerId, from, to }) }`.
- `getState(peerId): HealthState`.
- `recordSeen(peerId)` — fresh activity; restores `suspect` / `recovering` to `alive`.
- `markStale(peerId)` — checks elapsed time against thresholds and steps the FSM.
- `markRecovering(peerId)` / `markRecovered(peerId)` — explicit recovery flow.
- `clear()` — drop all state.
## Implementation Notes
`recordSeen` upgrades `suspect` directly to `alive` rather than walking through `recovering`, because seen activity is the strongest possible signal of liveness. `recovering` is reserved for the operator-driven case where a previously-dead peer is being deliberately reintroduced (e.g. after a manual restart). The allowed-transition matrix is enforced at the private `transition` helper, which means even internal callers cannot accidentally short-cut the FSM.

View File

@@ -0,0 +1,70 @@
# HeartbeatManager
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Sends periodic heartbeats to a set of peers, correlates pongs by sequence number, and surfaces missed beats via `findStalePeers`. The manager only handles the wire-level concern; the `HealthMonitor` decides what `alive`/`suspect`/`dead` mean. Decoupling them means the heartbeat cadence and the health policy can evolve independently.
## Features
- Configurable `intervalMs` (default 1 s) and `timeoutMs` (default 3 s).
- Pluggable `HeartbeatSender` so consumers can route through any transport.
- Injectable `setIntervalFn` / `clearIntervalFn` / `clock` for deterministic tests.
- `addPeer` / `removePeer` for explicit lifecycle.
- `tick()` is exposed publicly so test harnesses can step the loop manually.
- Sequence-number correlation prevents stale pongs from masking missed heartbeats.
- `markPong(peerId, sequence)` returns the RTT in ms or `undefined` if the sequence didn't match.
## Usage: Basic Example
```ts
import { HeartbeatManager } from "@suaveplan/network-plugin-failover";
const heartbeat = new HeartbeatManager({
sender: async (peerId, seq) => {
await transport.send(peerId, { type: "ping", seq });
},
});
heartbeat.addPeer("alice");
heartbeat.start();
```
## Usage: Advanced Example
```ts
import { HeartbeatManager } from "@suaveplan/network-plugin-failover";
let now = 0;
const heartbeat = new HeartbeatManager({
sender: async () => {},
intervalMs: 100,
timeoutMs: 500,
clock: () => now,
});
heartbeat.addPeer("alice");
heartbeat.start();
await heartbeat.tick();
now += 1_000;
const stale = heartbeat.findStalePeers();
// stale === ["alice"]
heartbeat.stop();
```
## API Reference
- `type HeartbeatSender = (peerId, sequence) => void | Promise<void>`.
- `interface HeartbeatManagerConfig { sender, intervalMs?, timeoutMs?, setIntervalFn?, clearIntervalFn?, clock? }`.
- `addPeer(peerId)` / `removePeer(peerId)`.
- `start()` / `stop()` — idempotent.
- `tick(): Promise<void>` — throws `HEARTBEAT_NOT_RUNNING` when called before `start()`.
- `markPong(peerId, sequence): number | undefined`.
- `findStalePeers(): readonly string[]`.
- `getPeers(): readonly string[]`.
## Implementation Notes
The pending map is keyed by `peerId` rather than `(peerId, sequence)` because we only ever care about the _latest_ outbound ping per peer — a pong for an older sequence is treated as stale and dropped via the sequence check inside `markPong`. This keeps memory bounded by peer count rather than tick count. `findStalePeers` is read-only and does not mutate the pending map; the consumer (typically `HealthMonitor`) is responsible for transitioning the peer's state and removing it from the heartbeat set.

View File

@@ -0,0 +1,70 @@
# FailoverPlugin
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
`NetworkPlugin` that owns the failover stack: heartbeat manager, health monitor, promoter, and coordinator. Each sub-component is exposed as a public property so callers can inspect or extend behaviour without going through the plugin abstraction. The plugin only wires the components together; per-component behaviour lives in the dedicated module docs.
## Features
- Lifecycle: `install(node)` builds health + promoter + coordinator against the node's bus; `start()` starts heartbeat + coordinator subscriptions; `stop()` reverses both; `dispose()` clears state.
- Heartbeat is constructed eagerly so callers can `addPeer` before `start()`.
- Health and promoter are constructed lazily inside `install` because they need the node's bus.
- Public `heartbeat` / `health` / `promoter` / `coordinator` properties for observability and test harnesses.
- Configurable via `FailoverPluginConfig` — heartbeat is required, health and promoter inherit defaults.
## Usage: Basic Example
```ts
import { FailoverPlugin } from "@suaveplan/network-plugin-failover";
const plugin = new FailoverPlugin({
heartbeat: {
sender: async (peerId, seq) =>
transport.send(peerId, { type: "ping", seq }),
},
});
node.use(plugin);
await node.start();
plugin.heartbeat.addPeer("alice");
```
## Usage: Advanced Example
```ts
import { FailoverPlugin } from "@suaveplan/network-plugin-failover";
const plugin = new FailoverPlugin({
heartbeat: {
sender: async (peerId, seq) =>
transport.send(peerId, { type: "ping", seq }),
intervalMs: 500,
timeoutMs: 2_000,
},
health: {
suspectAfterMs: 1_000,
deadAfterMs: 4_000,
},
promoter: {
debounceMs: 500,
},
});
node.use(plugin);
await node.start();
setInterval(() => plugin.coordinator?.sweep(), 1_000);
```
## API Reference
- `interface FailoverPluginConfig { heartbeat, health?, promoter? }`.
- `name: "failover"`.
- `install(node) / start() / stop() / dispose()` — standard `NetworkPlugin` lifecycle.
- Public properties: `heartbeat: HeartbeatManager`, `health: HealthMonitor | null`, `promoter: PromoterManager | null`, `coordinator: FailoverCoordinator | null`.
## Implementation Notes
`health`, `promoter`, and `coordinator` are typed nullable because they only exist after `install` runs — callers that need to access them before the plugin is installed must use the always-present `heartbeat` reference. `dispose` clears state but does not re-construct components, so a re-installed plugin instance must be a fresh `FailoverPlugin`. The bus type assertion (`as unknown as HealthBusListener & ...`) is necessary because `NetworkBus` is generically typed and the failover-specific topics are not in its compile-time map.

View File

@@ -0,0 +1,62 @@
# PromoterManager
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Fires `failover:emergency-election` bus events when a peer transitions to dead, while debouncing duplicate triggers within a configurable window. Decouples failover from the governance plugin — communication is bus-only, never a direct import — so either side can be swapped without touching the other.
## Features
- `onPeerDead(peerId, reason): boolean` — returns `true` if the trigger fired, `false` if it was suppressed by debounce.
- Configurable `debounceMs` (default 1 s).
- Pluggable `clock` for deterministic tests.
- `reset(peerId?)` clears one peer's debounce window or all of them.
- Per-peer last-fired timestamps so peers can independently re-trigger after their windows elapse.
## Usage: Basic Example
```ts
import { PromoterManager } from "@suaveplan/network-plugin-failover";
const promoter = new PromoterManager({ bus });
const fired = promoter.onPeerDead("alice", "heartbeat-timeout");
```
## Usage: Advanced Example
```ts
import { PromoterManager } from "@suaveplan/network-plugin-failover";
let now = 0;
const promoter = new PromoterManager({
bus,
debounceMs: 500,
clock: () => now,
});
bus.on("health:state-change", ({ peerId, to }) => {
if (to === "dead") {
promoter.onPeerDead(peerId, "health:state-change");
}
});
// rapid duplicate transitions are de-duplicated
promoter.onPeerDead("alice", "first");
now += 100;
promoter.onPeerDead("alice", "second"); // suppressed by debounce
now += 600;
promoter.onPeerDead("alice", "third"); // window elapsed → fires again
```
## API Reference
- `interface PromoterBus { emit("failover:emergency-election", { deadPeerId, reason }) }`.
- `interface PromoterManagerConfig { bus, debounceMs?, clock? }`.
- `onPeerDead(peerId, reason): boolean`.
- `reset(peerId?)`.
## Implementation Notes
The debounce is per-peer rather than global because two peers dying at once is a normal cascading-failure scenario that operators want to see two events for, not one. Suppressing the second event by accident would mask compound outages. `reset` is exposed mostly for tests; in production, the lastFiredAt map is bounded by peer count and never grows without bound. The `onPeerDead` boolean return is intentional — callers logging promotion attempts can distinguish "fired" from "suppressed" without parsing event-bus state. The default debounce of 1 s is a reasonable middle-ground; aggressive meshes that need faster re-triggering can reduce it, while slow networks that want fewer false positives can stretch it.