Initial Commit
This commit is contained in:
@@ -0,0 +1,183 @@
|
||||
# OpenSpec Change: network-plugin-partition
|
||||
|
||||
- **Change ID**: network-plugin-partition
|
||||
- **Package**: `@suaveplan/network-plugin-partition` (new)
|
||||
- **Status**: PROPOSED
|
||||
- **Date**: 2026-05-01
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Implement split-brain detection and resolution for the mesh. The `network-plugin-failover` proposal explicitly excludes network partition detection: *"Network partition detection (split-brain) → future hardening change."* This plugin fills that exclusion. Without it, a network partition can cause:
|
||||
|
||||
- Duplicate hosts in the same mesh
|
||||
- Messages routed to the wrong partition
|
||||
- Infinite relay loops across partitioned meshes
|
||||
|
||||
---
|
||||
|
||||
## Why
|
||||
|
||||
**Current state**: `network-plugin-failover` handles host death (alive → suspect → dead) but does not detect when the mesh splits into two or more isolated groups, each believing it is the authoritative partition.
|
||||
|
||||
**Why this matters**: In a multi-mesh system with relay, split-brain can cause:
|
||||
|
||||
- Two Tier 0 hosts both claiming to be the primary host for the same mesh
|
||||
- `network-plugin-relay` forwarding messages into a partition that has diverged
|
||||
- `network-plugin-registry` peers replicating to hosts that are no longer reachable
|
||||
|
||||
---
|
||||
|
||||
## Core components
|
||||
|
||||
### 1. `PartitionDetector`
|
||||
|
||||
Detects network partitions using vector clocks or epoch numbers.
|
||||
|
||||
```ts
|
||||
class PartitionDetector {
|
||||
constructor(options: {
|
||||
heartbeatSource: HeartbeatSource; // consumes from failover plugin
|
||||
meshId: string;
|
||||
quorumRatio: number; // default 0.5 (majority)
|
||||
epochIntervalMs: number; // default 5000
|
||||
});
|
||||
start(): void;
|
||||
stop(): void;
|
||||
getCurrentEpoch(): number;
|
||||
getPartitionView(): PartitionView;
|
||||
}
|
||||
|
||||
interface PartitionView {
|
||||
epoch: number;
|
||||
knownPeers: string[];
|
||||
reachablePeers: string[];
|
||||
isPartitioned: boolean;
|
||||
partitionId: string; // deterministic hash of reachable set
|
||||
primary: boolean; // true if this partition believes it is the primary
|
||||
}
|
||||
```
|
||||
|
||||
**Detection strategy**:
|
||||
|
||||
- Each peer emits an `epoch` heartbeat every `epochIntervalMs`.
|
||||
- Peers track the highest epoch they've seen from every other peer.
|
||||
- If a peer misses epochs from a majority of known peers for `suspectThreshold * 2`, a partition is suspected.
|
||||
- `PartitionView.partitionId` is a deterministic hash (e.g. BLAKE3) of the sorted reachable peer set.
|
||||
- Two peers with different `partitionId` values are in different partitions.
|
||||
|
||||
### 2. `QuorumResolver`
|
||||
|
||||
Determines which partition is authoritative when split-brain is detected.
|
||||
|
||||
```ts
|
||||
class QuorumResolver {
|
||||
constructor(options: {
|
||||
strategy: QuorumStrategy; // "size" | "performance" | "age" | "hybrid"
|
||||
});
|
||||
resolve(partitions: PartitionView[]): PartitionView | null; // null = cannot resolve
|
||||
}
|
||||
|
||||
type QuorumStrategy = "size" | "performance" | "age" | "hybrid";
|
||||
```
|
||||
|
||||
**Strategies**:
|
||||
|
||||
- `size`: the partition with the most peers wins.
|
||||
- `performance`: the partition with the lowest aggregate latency / highest bandwidth wins.
|
||||
- `age`: the partition containing the oldest-established host wins.
|
||||
- `hybrid`: weighted composite of size (0.4), performance (0.4), age (0.2).
|
||||
|
||||
### 3. `MergeProtocol`
|
||||
|
||||
Reconciles diverged state when partitions heal.
|
||||
|
||||
```ts
|
||||
class MergeProtocol {
|
||||
constructor(options: {
|
||||
registryReconciler: RegistryReconciler;
|
||||
topologyReconciler: TopologyReconciler;
|
||||
auditReconciler: AuditReconciler;
|
||||
});
|
||||
initiateMerge(localView: PartitionView, remoteView: PartitionView): Promise<MergeResult>;
|
||||
}
|
||||
|
||||
interface MergeResult {
|
||||
success: boolean;
|
||||
mergedRegistry: PeerRegistrySnapshot;
|
||||
mergedTopology: TopologySnapshot;
|
||||
conflicts: MergeConflict[];
|
||||
}
|
||||
```
|
||||
|
||||
**Merge steps**:
|
||||
|
||||
1. Exchange `PartitionView` and validate epoch sequences.
|
||||
2. Compare registry snapshots: add missing peers, remove stale entries.
|
||||
3. Compare topology: re-run host election if both partitions had a Tier 0 host.
|
||||
4. Replay audit logs from the non-authoritative partition into the authoritative one.
|
||||
5. Emit `partition:healed` on the `NetworkBus`.
|
||||
|
||||
---
|
||||
|
||||
## Integration with failover
|
||||
|
||||
`network-plugin-partition` consumes heartbeat data from `network-plugin-failover` via the `NetworkBus`:
|
||||
|
||||
- Listens to `failover:triggered` events — if multiple hosts trigger failover simultaneously, partition is suspected.
|
||||
- Does NOT replace failover; it complements it. Failover handles single-host death; partition handles group isolation.
|
||||
|
||||
---
|
||||
|
||||
## Error codes
|
||||
|
||||
```ts
|
||||
const PARTITION_CODES = defineCodes("PARTITION", [
|
||||
"SPLIT_BRAIN_DETECTED",
|
||||
"QUORUM_NOT_REACHED",
|
||||
"MERGE_CONFLICT",
|
||||
"MERGE_FAILED",
|
||||
"EPOCH_SYNC_ERROR",
|
||||
"INVALID_PARTITION_VIEW",
|
||||
"PARTITION_HEAL_TIMEOUT",
|
||||
] as const);
|
||||
```
|
||||
|
||||
`PartitionError extends SuaveplanError<PartitionErrorCode>`.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **Consumes**: `@suaveplan/network` (`NetworkNode`, `NetworkBus`), `@suaveplan/network-plugin-failover` (heartbeat events, FSM states), `@suaveplan/network-plugin-registry` (registry snapshots), `@suaveplan/network-plugin-topology` (topology snapshots), `@suaveplan/network-plugin-audit-mesh` (audit log replay), `@suaveplan/error`, `@suaveplan/types`, `@opentelemetry/api`.
|
||||
- **Blocked on**: `network-core` (shipped), `network-plugin-failover`, `network-plugin-registry`, `network-plugin-topology`.
|
||||
|
||||
---
|
||||
|
||||
## Verification approach
|
||||
|
||||
- `src/detector/detector.spec-conformance.test.ts` — pins epoch tracking, partition detection, reachable-set hashing.
|
||||
- `src/quorum/quorum.spec-conformance.test.ts` — pins all four quorum strategies, tie-breaking.
|
||||
- `src/merge/merge.spec-conformance.test.ts` — pins registry merge, topology merge, conflict reporting.
|
||||
- `tests/integration/partition.spec-conformance.test.ts` — end-to-end: simulate partition, detect, resolve, heal.
|
||||
- 100/100/100/100 Istanbul coverage.
|
||||
|
||||
---
|
||||
|
||||
## Conformance requirements (proposed)
|
||||
|
||||
- `NET-78` — `PartitionDetector` tracks epochs from all known peers and detects partition when a majority is unreachable.
|
||||
- `NET-79` — `PartitionView.partitionId` is a deterministic hash of the sorted reachable peer set.
|
||||
- `NET-80` — `QuorumResolver` supports `size`, `performance`, `age`, and `hybrid` strategies.
|
||||
- `NET-81` — `MergeProtocol` reconciles registry, topology, and audit state when partitions heal.
|
||||
- `NET-82` — Non-authoritative partition peers rejoin the authoritative partition after merge.
|
||||
- `NET-83` — `partition:detected` and `partition:healed` events are emitted on the `NetworkBus`.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Byzantine fault tolerance** → v2; v1 assumes honest peers.
|
||||
- **Automatic partition creation** → v2; v1 pauses new mesh operations during partition.
|
||||
- **Cross-mesh partition detection** → handled by `network-plugin-relay` loop prevention; this plugin focuses on intra-mesh splits.
|
||||
Reference in New Issue
Block a user