Initial Commit
This commit is contained in:
@@ -0,0 +1,195 @@
|
||||
# OpenSpec Change: enterprise-network-failover
|
||||
|
||||
- **Change ID**: enterprise-network-failover
|
||||
- **Package**: `@suaveplan/network-failover`
|
||||
- **Category**: universal/network
|
||||
- **Target path**: `universal/network/network-failover`
|
||||
- **Status**: SUPERSEDED-BY-GENESIS-WAVE
|
||||
- **Date**: 2026-04-24
|
||||
- **Reference spec**: `/mnt/s/web/suaveplan/enterprise/network-failover.md`
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Deliver `@suaveplan/network-failover` on the genesis branch from scratch at enterprise quality. The package provides network failover and recovery: health-checked connection groups, automatic failover on primary failure, configurable failover strategies (round-robin, fastest, weighted), recovery detection and fail-back, and in-flight request replay after failover. Genesis adds structured errors via `@suaveplan/error`, Zod schemas for all failover configuration, OTel instrumentation on failover events, `AbortSignal` on recovery operations, injectable `Clock` for health-check intervals, event-bus integration for failover state changes, and property-based tests for strategy correctness.
|
||||
|
||||
---
|
||||
|
||||
## Current-state audit (genesis)
|
||||
|
||||
The package does not exist on genesis yet — this is a fresh build.
|
||||
|
||||
---
|
||||
|
||||
## Available workspace deps on genesis
|
||||
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| `@suaveplan/error` (`workspace:^`) | Base class for `NetworkFailoverError`; stable error codes |
|
||||
| `@suaveplan/types` (`workspace:^`) | `LoggerLike`, `Clock`, `EventBusLike` |
|
||||
| `@suaveplan/events` (`workspace:^`) | Event bus for failover events |
|
||||
| `@suaveplan/logger` (`workspace:^`) | Structured logging |
|
||||
| `@suaveplan/retry` (`workspace:^`) | Retry wired into health checks and recovery probes |
|
||||
| `@suaveplan/timing` (`workspace:^`) | Interval timer for health checks |
|
||||
| `zod` (`catalog:forms`) | Zod schemas |
|
||||
| `@opentelemetry/api` (`catalog:observability`) | Spans + counters on failover |
|
||||
| `tslib` (`catalog:build`) | Runtime helpers |
|
||||
|
||||
Dev deps: `vitest`, `@vitest/coverage-istanbul`, `fast-check` (all `catalog:testing`).
|
||||
|
||||
---
|
||||
|
||||
## Public API surface
|
||||
|
||||
### Core
|
||||
|
||||
```ts
|
||||
function createFailoverGroup(options: FailoverGroupOptions): FailoverGroup
|
||||
|
||||
interface FailoverGroup {
|
||||
readonly primary: FailoverTarget | null
|
||||
readonly targets: ReadonlyArray<FailoverTarget>
|
||||
readonly state: FailoverState
|
||||
addTarget(target: FailoverTarget): void
|
||||
removeTarget(targetId: string): void
|
||||
execute<T>(fn: (target: FailoverTarget, signal?: AbortSignal) => Promise<T>, signal?: AbortSignal): Promise<T>
|
||||
probe(targetId: string, signal?: AbortSignal): Promise<HealthStatus>
|
||||
on(event: FailoverEvent, handler: (data: FailoverEventData) => void): () => void
|
||||
}
|
||||
```
|
||||
|
||||
### Target types
|
||||
|
||||
```ts
|
||||
interface FailoverTarget {
|
||||
id: string
|
||||
address: string
|
||||
port?: number
|
||||
weight?: number
|
||||
priority?: number
|
||||
tags?: string[]
|
||||
healthCheck?: HealthCheckConfig
|
||||
}
|
||||
|
||||
interface HealthCheckConfig {
|
||||
intervalMs?: number
|
||||
timeoutMs?: number
|
||||
successThreshold?: number
|
||||
failureThreshold?: number
|
||||
probe?: (target: FailoverTarget, signal?: AbortSignal) => Promise<boolean>
|
||||
}
|
||||
|
||||
type HealthStatus = {
|
||||
healthy: boolean
|
||||
latencyMs?: number
|
||||
checkedAt: number
|
||||
consecutiveFailures: number
|
||||
consecutiveSuccesses: number
|
||||
}
|
||||
```
|
||||
|
||||
### Failover types
|
||||
|
||||
```ts
|
||||
type FailoverState = 'healthy' | 'failing-over' | 'degraded' | 'recovering' | 'no-targets'
|
||||
type FailoverStrategy = 'round-robin' | 'fastest' | 'weighted' | 'priority'
|
||||
type FailoverEvent = 'failover' | 'recovery' | 'target-failed' | 'target-recovered' | 'state-change'
|
||||
|
||||
interface FailoverEventData {
|
||||
previousPrimary?: FailoverTarget
|
||||
newPrimary?: FailoverTarget
|
||||
target?: FailoverTarget
|
||||
reason?: string
|
||||
}
|
||||
|
||||
interface FailoverGroupOptions {
|
||||
strategy?: FailoverStrategy
|
||||
failoverDelayMs?: number
|
||||
failBackDelayMs?: number
|
||||
maxRetries?: number
|
||||
logger?: LoggerLike
|
||||
clock?: Clock
|
||||
eventBus?: EventBusLike
|
||||
}
|
||||
```
|
||||
|
||||
### Error codes
|
||||
|
||||
```ts
|
||||
type NetworkFailoverErrorCode =
|
||||
| 'FAILOVER_NO_TARGETS'
|
||||
| 'FAILOVER_ALL_UNHEALTHY'
|
||||
| 'FAILOVER_EXECUTE_FAILED'
|
||||
| 'FAILOVER_PROBE_FAILED'
|
||||
| 'FAILOVER_ABORT'
|
||||
|
||||
function registerNetworkFailoverErrorCodes(): void
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Genesis additions
|
||||
|
||||
1. **Structured `NetworkFailoverError`** with stable `FAILOVER.*` codes
|
||||
2. **Zod schemas** for `FailoverGroupOptions`, `FailoverTarget`, `HealthCheckConfig`
|
||||
3. **Four failover strategies**: round-robin, fastest (lowest latency), weighted, priority
|
||||
4. **Automatic health checking** via injectable interval timer and `@suaveplan/timing`
|
||||
5. **Fail-back with delay** — prevents oscillation; configurable `failBackDelayMs`
|
||||
6. **In-flight request retry** on failover — `execute()` replays on new primary with `AbortSignal`
|
||||
7. **Injectable `Clock`** for health-check interval testing
|
||||
8. **OTel instrumentation** — span `failover.execute`, counter `failover.events_total`, histogram `failover.latency_ms`
|
||||
9. **Property-based tests** — failover always selects a healthy target when available; strategy returns consistent ordering
|
||||
|
||||
---
|
||||
|
||||
## File layout
|
||||
|
||||
```text
|
||||
universal/network/network-failover/
|
||||
package.json
|
||||
tsconfig.json
|
||||
vitest.config.ts
|
||||
src/
|
||||
index.ts
|
||||
error-codes.ts
|
||||
constants/
|
||||
index.ts
|
||||
types/
|
||||
common.ts
|
||||
common.md
|
||||
core/
|
||||
failover-group/
|
||||
failover-group.ts
|
||||
failover-group.md
|
||||
failover-group.types.ts
|
||||
failover-group.test.ts
|
||||
index.ts
|
||||
health-checker/
|
||||
health-checker.ts
|
||||
health-checker.md
|
||||
health-checker.test.ts
|
||||
index.ts
|
||||
strategies/
|
||||
round-robin.ts
|
||||
round-robin.md
|
||||
round-robin.test.ts
|
||||
fastest.ts
|
||||
fastest.md
|
||||
fastest.test.ts
|
||||
weighted.ts
|
||||
weighted.md
|
||||
weighted.test.ts
|
||||
priority.ts
|
||||
priority.md
|
||||
priority.test.ts
|
||||
tests/
|
||||
integration.test.ts
|
||||
property.test.ts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Impact
|
||||
|
||||
**Workspace dep prerequisites**: `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/events`, `@suaveplan/logger`, `@suaveplan/retry`, `@suaveplan/timing` must be published.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Tasks: enterprise-network-failover
|
||||
|
||||
**Change ID**: enterprise-network-failover
|
||||
**Package**: `@suaveplan/network-failover`
|
||||
**Target path**: `universal/network/network-failover`
|
||||
**Prerequisite packages on Verdaccio**: `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/events`, `@suaveplan/logger`, `@suaveplan/retry`, `@suaveplan/timing`
|
||||
|
||||
---
|
||||
|
||||
## 1. Scaffold
|
||||
|
||||
- [x] Create `universal/network/network-failover/` with `src/`, `tests/`, and all feature subdirectories.
|
||||
- [x] Write `package.json` at version `0.1.0`: `"sideEffects": false`, exports map, deps per `proposal.md`.
|
||||
- [x] Write `tsconfig.json`, `vitest.config.ts`, scripts.
|
||||
|
||||
---
|
||||
|
||||
## 2. Types and error codes
|
||||
|
||||
- [x] Define all public types in `src/types/common.ts`: `FailoverTarget`, `HealthCheckConfig`, `HealthStatus`, `FailoverState`, `FailoverStrategy`, `FailoverEvent`, `FailoverEventData`, `FailoverGroupOptions`.
|
||||
- [x] Write `src/types/common.md` (≥200 words, simple + advanced examples).
|
||||
- [x] Define error codes in `src/error-codes.ts`.
|
||||
- [x] Write Zod schemas in `src/types/schemas.ts` + `src/types/schemas.md`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Failover strategies (`src/strategies/`)
|
||||
|
||||
### 3a. Round-robin
|
||||
|
||||
- [x] Implement `roundRobin(targets: FailoverTarget[], lastIndex: number): FailoverTarget | null`:
|
||||
- Returns next healthy target in round-robin order; wraps around.
|
||||
- [x] Write `round-robin.test.ts` + `round-robin.md`.
|
||||
|
||||
### 3b. Fastest
|
||||
|
||||
- [x] Implement `fastest(targets: FailoverTarget[]): FailoverTarget | null`:
|
||||
- Returns healthy target with lowest `health.latencyMs`; falls back to any healthy target.
|
||||
- [x] Write `fastest.test.ts` + `fastest.md`.
|
||||
|
||||
### 3c. Weighted
|
||||
|
||||
- [x] Implement `weighted(targets: FailoverTarget[], rng: Rng): FailoverTarget | null`:
|
||||
- Normalised-weight random selection among healthy targets only.
|
||||
- [x] Write `weighted.test.ts` + `weighted.md`.
|
||||
|
||||
### 3d. Priority
|
||||
|
||||
- [x] Implement `priority(targets: FailoverTarget[]): FailoverTarget | null`:
|
||||
- Returns healthy target with highest `priority` value.
|
||||
- [x] Write `priority.test.ts` + `priority.md`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Health checker (`src/core/health-checker/`)
|
||||
|
||||
- [x] Implement `createHealthChecker(target, options, clock)`:
|
||||
- Runs `options.healthCheck.probe ?? defaultProbe` on `intervalMs` interval.
|
||||
- Track `consecutiveFailures` / `consecutiveSuccesses`.
|
||||
- Emit `target-failed` after `failureThreshold` consecutive failures.
|
||||
- Emit `target-recovered` after `successThreshold` consecutive successes.
|
||||
- [x] Write `health-checker.test.ts` (≤500 lines): failure threshold fires, recovery fires, intervals.
|
||||
- [x] Write `health-checker.md` (≥200 words).
|
||||
|
||||
---
|
||||
|
||||
## 5. Failover group (`src/core/failover-group/`)
|
||||
|
||||
- [x] Implement `createFailoverGroup(options)`:
|
||||
- Validates options with Zod.
|
||||
- Maintains `primary` and `targets[]`; starts health checkers for all.
|
||||
- On `target-failed` for primary: wait `failoverDelayMs` then run strategy to select new primary.
|
||||
- On `target-recovered` for non-primary: wait `failBackDelayMs`; consider fail-back only if score is higher.
|
||||
- OTel: span `failover.execute`, counter `failover.events_total`, histogram `failover.latency_ms`.
|
||||
- [x] Implement `group.execute(fn, signal?)`:
|
||||
- Call `fn(primary, signal)`.
|
||||
- On failure, immediately promote next target via strategy and retry `fn` once.
|
||||
- Throw `NetworkFailoverError(FAILOVER_EXECUTE_FAILED)` after all targets exhausted.
|
||||
- [x] Implement `group.probe(targetId, signal?)` — runs single health probe.
|
||||
- [x] Write `failover-group.test.ts` (≤500 lines): execute on healthy primary, primary fails → failover, all fail throws, probe.
|
||||
- [x] Write `failover-group.md` (≥200 words).
|
||||
|
||||
---
|
||||
|
||||
## 6. Property-based tests (`tests/property.test.ts`)
|
||||
|
||||
- [x] Strategy completeness: when all targets are healthy, strategy always returns a target.
|
||||
- [x] Failover selects next healthy target after primary failure.
|
||||
|
||||
---
|
||||
|
||||
## 7. Docs and verify
|
||||
|
||||
- [x] All per-file `.md` files per layout in `proposal.md`.
|
||||
- [x] `universal/network/network-failover/README.md`.
|
||||
- [x] `bunx turbo lint typecheck test test:coverage --filter=@suaveplan/network-failover` — all green, 100%.
|
||||
- [x] `bun scripts/publish.ts --package @suaveplan/network-failover`.
|
||||
- [x] Update `CLAUDE.md`. Archive change folder.
|
||||
Reference in New Issue
Block a user