Initial Commit
This commit is contained in:
205
docs/network/topology-and-failover.md
Normal file
205
docs/network/topology-and-failover.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# Topology, Tiers, Meshed Pockets, and Failover
|
||||
|
||||
The user's framing — verbatim — was: *"a resilient tiered network system where if the server goes down, it's all in place with meshed pockets of users with their nominated hosts / those hosts on a tier nominate their tier host."*
|
||||
|
||||
This is the document for that. It is the load-bearing piece of the network design.
|
||||
|
||||
## The tier model (NET-11, NET-49, NET-122)
|
||||
|
||||
Every peer occupies exactly one of three tiers at any moment:
|
||||
|
||||
| Tier | Role | Authority | Sibling mesh |
|
||||
| --- | --- | --- | --- |
|
||||
| **Tier 0** — Host | Top of the local mesh. Sends/receives from the central server (when present). | Token issuance, peer admission, log finalisation. | Other Tier 0 hosts (cross-mesh peers). |
|
||||
| **Tier 1** — Sub-host | Aggregator for a group of leaves. Relay between Tier 0 and Tier 2. | Vote in elections, relay messages, no token issuance. | Other Tier 1 sub-hosts in the same mesh. |
|
||||
| **Tier 2** — Leaf | Regular client. Speaks only to its assigned Tier 1 (and laterally to other Tier 2 in its session). | Read-only for sensitive ops. | Other Tier 2 in the same session pocket. |
|
||||
|
||||
`network-plugin-topology` owns this. Promotion/demotion is automatic, based on capability scores (latency, uptime, reliability, device profile from NET-49) and group-size limits. Peers MUST meet a stability threshold and a cooldown (`network-plugin-topology` NET-52) to prevent thrashing.
|
||||
|
||||
**Tier depth is dynamic.** A 5-node mesh runs flat (everyone Tier 0/Tier 2; no sub-host). A 5,000-node mesh runs three-tier with hundreds of Tier 1 sub-hosts. The topology plugin adapts depth based on size; collapsing or expanding tiers as needed (`AdaptiveTierDepth`).
|
||||
|
||||
## The four hosting models (NET-56)
|
||||
|
||||
`network-plugin-server-bridge` lets a deployment pick how clients and the central server interact:
|
||||
|
||||
| Model | Behaviour |
|
||||
| --- | --- |
|
||||
| **`PURE_CLIENT`** | No central server. Mesh is fully autonomous. A user creates a session and acts as Tier 0 host. Subsequent users join, get assigned Tier 1/Tier 2 by topology, and the host slot transfers via election when the host leaves. |
|
||||
| **`SERVER_BASED`** | Server is always Tier 0. All clients are Tier 1 or Tier 2. Server enforces permissions, holds authoritative state, issues all tokens. If the server disconnects, the mesh **stops** (this model trades resilience for auditability). |
|
||||
| **`SERVER_TO_CLIENT_FAILOVER`** | Server is Tier 0 while online. On disconnect, an election promotes the strongest Tier 1 to acting Tier 0, the mesh continues, and on server reconnect the server smoothly reintegrates as Tier 0. **This is the model that matches your user request.** |
|
||||
| **`CLIENT_WITH_SERVER_BACKUP`** | A user is Tier 0; the server joins as a participant (typically Tier 1). If the user disconnects or becomes unsuitable, hosting can transfer to the server. Hybrid authority — server provides persistence, client provides agency. |
|
||||
|
||||
The model is selected at mesh creation and affects election behaviour, token authority, and reconciliation direction. The mesh token (NET-29) carries it.
|
||||
|
||||
## Meshed pockets
|
||||
|
||||
A "pocket" is a group of peers that can talk to each other without going through Tier 0. Pockets emerge from two structural primitives:
|
||||
|
||||
1. **Sibling links at every tier.** A Tier 1 sub-host has up to 10 sibling Tier 1 sub-hosts in the same mesh. A Tier 2 leaf has up to N session-mates (capped per-session, NET-22). When a tier above goes silent, a pocket continues to function via siblings.
|
||||
2. **Session trees** (NET-18..NET-22). The mesh organises peers into sessions — typed groupings (a video call, a shared drive, a multiplayer game). A pocket is, concretely, the peers in one leaf session plus that session's host. Sessions can nest up to depth 4; child sessions inherit read-only state from parents.
|
||||
|
||||
If the central server vanishes:
|
||||
|
||||
- Each Tier 0 host keeps running its mesh autonomously (modulo authority limits — see below).
|
||||
- Each Tier 1 keeps relaying for its Tier 2 leaves.
|
||||
- Each session pocket keeps gossiping among its members.
|
||||
- Cross-pocket traffic is brokered by `network-plugin-relay` over the Tier 0 backbone (or, if Tier 0 is also gone in a partition, via the elected acting Tier 0).
|
||||
|
||||
Nothing about pocket operation requires the server. The server is needed only for the *authoritative* operations the mesh declines to do without it.
|
||||
|
||||
## Host election — how peers nominate hosts (NET-08, NET-118, NET-119)
|
||||
|
||||
`network-plugin-governance` runs elections. The algorithm is:
|
||||
|
||||
1. **Collect candidates** — peers eligible by capability/stability threshold.
|
||||
2. **Score each candidate** — `HostSelector` (NET-118) does weighted multi-factor evaluation:
|
||||
- Latency (EMA of measured RTT to other peers).
|
||||
- Stability (jitter + packet loss).
|
||||
- Uptime (since last reconnect).
|
||||
- Capabilities (CPU, bandwidth, NAT type).
|
||||
- **Trust score** — historical performance from `HostHistory` (NET-119), which keeps an append-only record of successful/unsuccessful hosting sessions per peer.
|
||||
3. **Vote** — `VotingManager` collects votes via the bus; configurable participation threshold and consensus requirement.
|
||||
4. **Tie-break** — lowest lexicographic peerId wins.
|
||||
5. **Publish** — `election:completed { winnerId, term }` on the bus.
|
||||
|
||||
Three modes (NET-08): `metric` (pure score), `direct` (assigned by Tier 0), `hybrid` (score-suggested + Tier 0-confirmed).
|
||||
|
||||
Each session also runs its own host election independently of the mesh Tier 0 host (NET-20). One peer can host multiple sessions.
|
||||
|
||||
## Server failover — what happens when the server goes down
|
||||
|
||||
This is the resilience pipeline (`SERVER_TO_CLIENT_FAILOVER` model):
|
||||
|
||||
```text
|
||||
1. Heartbeat timeout
|
||||
`network-plugin-failover` heartbeat (NET-10): alive → suspect → dead.
|
||||
`network-plugin-server-bridge` (NET-31) detects server disconnect within 2× heartbeat interval.
|
||||
|
||||
2. Verification
|
||||
Multiple Tier 1 peers re-ping the server through alternate paths.
|
||||
Avoid declaring the server dead on a single network blip.
|
||||
|
||||
3. Consensus
|
||||
Bus topic 'server:dead' published when N Tier 1 peers agree.
|
||||
|
||||
4. Election
|
||||
`network-plugin-governance` runs `HOST_SELECTION` among eligible Tier 1 peers.
|
||||
Winning peer is promoted to Tier 0 (acting host).
|
||||
|
||||
5. Authority handover
|
||||
Tokens issued before server-down remain valid.
|
||||
New token issuance is paused (NET-32) — see "graceful degradation" below.
|
||||
|
||||
6. Operation continues
|
||||
Each Tier 0 host keeps running its mesh autonomously.
|
||||
Cross-mesh traffic uses `network-plugin-relay` Tier 0 backbone (NET-62).
|
||||
|
||||
7. Server returns
|
||||
`network-plugin-server-bridge` reconciliation (NET-33):
|
||||
- Buffered signed log entries replayed in sequence.
|
||||
- Server validates signatures, detects conflicts, broadcasts reconciled state.
|
||||
- Server resumes Tier 0 (election may run to confirm — depends on hosting model).
|
||||
Acting Tier 0 demotes back to Tier 1.
|
||||
```
|
||||
|
||||
## Graceful degradation — server-vs-autonomous authority split
|
||||
|
||||
When the server is offline (NET-32), the mesh continues with reduced authority:
|
||||
|
||||
| Operation | Server present | Server offline |
|
||||
| --- | --- | --- |
|
||||
| Send/receive messages | ✓ | ✓ |
|
||||
| Read existing files (drive layer) | ✓ | ✓ |
|
||||
| Write existing files | ✓ | ✓ (signed by drive identity, verifiable later) |
|
||||
| Vote in elections | ✓ | ✓ |
|
||||
| Reorganise topology | ✓ | ✓ |
|
||||
| Issue new mesh / session tokens | ✓ | ✗ (queued; replayed on reconnect) |
|
||||
| Admit a brand-new peer | ✓ | ✗ (unless server-pre-issued capability token still valid) |
|
||||
| Cross-mesh trust establishment | ✓ | ✗ (existing trust links remain valid) |
|
||||
| Audit-log finalisation | ✓ | ✗ (entries are buffered, signed, replayed on reconnect) |
|
||||
|
||||
`network-plugin-server-bridge` extends token grace periods (NET-32) so existing tokens don't expire mid-outage. New session creation is paused. Signed log entries are buffered locally and replayed on reconnect (NET-33).
|
||||
|
||||
## Cross-mesh relay — connecting pockets across meshes (NET-40, NET-61..NET-70)
|
||||
|
||||
When a peer in Mesh A needs to send to a peer in Mesh B, `network-plugin-relay` handles it.
|
||||
|
||||
- `RelayRouter` maintains a routing table `(destMeshId, destPeerId) → nextHopHostId`.
|
||||
- `HostBackboneTransport` connects Tier 0 hosts of different meshes (capability-negotiated).
|
||||
- `CrossMeshEnvelope` extends `MessageEnvelope` with `senderMeshId`, `targetMeshId`, `hopCount`, `maxHops` (default 8), `visitedNodes`, `ttl`.
|
||||
- Loop prevention is strict (NET-64..NET-66): drop on `hopCount >= maxHops`, `ttl <= 0`, or `visitedNodes` already contains current peer.
|
||||
- Cross-mesh broadcast is deduplicated at mesh boundaries (NET-68).
|
||||
- Relay chooses paths via `LowestLatency` / `HighestReliability` / `FewestHops` / `TierOptimized` / `LoadBalanced` strategies (NET-40).
|
||||
|
||||
This is what makes "meshed pockets" actually composable: a phone-mesh, a corporate-mesh, and a public-server-mesh can interoperate by establishing trust between Tier 0 hosts (NET-110, NET-112) without merging into one big mesh.
|
||||
|
||||
## Partition handling (NET-50, NET-78..NET-83)
|
||||
|
||||
A network partition splits the mesh into two or more disjoint groups, each unable to reach the others. `network-plugin-partition` handles it:
|
||||
|
||||
1. **Detection** — `PartitionDetector` tracks epochs from all known peers; partition declared when a majority is unreachable. Deterministic `partitionId = hash(sorted reachable peer set)`.
|
||||
2. **Autonomous operation** — each partition operates with its local hierarchy. New elections within the partition fill any gaps.
|
||||
3. **Quorum resolution** — when partitions heal, `QuorumResolver` (NET-80) decides the authoritative partition using one of: `size`, `performance`, `age`, `hybrid`.
|
||||
4. **Merge** — `MergeProtocol` (NET-81) reconciles registry, topology, and audit state. Non-authoritative peers rejoin (NET-82). Bus events `partition:detected` / `partition:healed`.
|
||||
|
||||
## Resource awareness (NET-49, NET-120..NET-124)
|
||||
|
||||
The topology plugin adapts to device profile (`CONSTRAINED`, `STANDARD`, `POWERFUL`, `SERVER`):
|
||||
|
||||
- Constrained devices use `MessagePool` (NET-120) to reuse message objects.
|
||||
- Zero-copy passing (NET-121) where the platform allows.
|
||||
- `BatteryAwareMode` (NET-122) reduces heartbeat frequency, defers non-urgent reorganisations, lowers transport retry aggression.
|
||||
- `BandwidthConservator` (NET-124) coalesces small messages, increases batching, pauses redundant CRITICAL sends.
|
||||
|
||||
So a phone running on battery in a constrained network drops to a lighter footprint without being kicked out of the mesh.
|
||||
|
||||
## Putting it all together — the resilience picture
|
||||
|
||||
```text
|
||||
Central Server (Tier 0 of meshes M1, M2, M3)
|
||||
│ (signed tokens, audit log, authoritative state)
|
||||
│
|
||||
┌───────────┼───────────┐
|
||||
│ │ │
|
||||
mesh M1 mesh M2 mesh M3
|
||||
│ │ │
|
||||
┌────┴────┐ ┌────┴────┐ ┌────┴────┐
|
||||
│ Tier 0 │ │ Tier 0 │ │ Tier 0 │
|
||||
│ host │ │ host │ │ host │
|
||||
└────┬────┘ └────┬────┘ └────┬────┘
|
||||
│ │ │
|
||||
┌────┴────┐ ┌────┴────┐ ┌────┴────┐
|
||||
│ Tier 1 │ │ Tier 1 │ │ Tier 1 │
|
||||
│ sub-hosts ......
|
||||
└────┬────┘ └────┬────┘ └────┬────┘
|
||||
│ │ │
|
||||
[pockets of Tier 2 leaves, organised into sessions]
|
||||
|
||||
|
||||
When the server vanishes:
|
||||
|
||||
╳ Central Server ╳
|
||||
(offline)
|
||||
┌─────────────────────────────────┐
|
||||
│ │
|
||||
mesh M1 ─── relay ─── mesh M2 ─── relay ─── mesh M3
|
||||
│ │
|
||||
each mesh elects an acting Tier 0
|
||||
each Tier 0 keeps running its mesh
|
||||
each session pocket keeps gossiping
|
||||
tokens issued before outage stay valid (grace period)
|
||||
new token issuance queued; replayed on reconnect
|
||||
signed audit-log entries buffered locally
|
||||
│
|
||||
server returns
|
||||
│
|
||||
buffered logs replayed; conflicts reconciled
|
||||
acting Tier 0s demote to Tier 1
|
||||
server resumes as Tier 0 of all meshes
|
||||
```
|
||||
|
||||
## Reading further
|
||||
|
||||
- [`security-and-tokens.md`](./security-and-tokens.md) — the cryptographic basis for token grace periods and signed log replay.
|
||||
- [`offline-and-reintegration.md`](./offline-and-reintegration.md) — the offline-mode buffering and reconciliation in detail.
|
||||
- [`plugins-catalog.md`](./plugins-catalog.md) — every plugin called out above, with one-paragraph descriptions.
|
||||
Reference in New Issue
Block a user