suaveplan/docs/network/topology-and-failover.md

Topology, Tiers, Meshed Pockets, and Failover

The user's framing — verbatim — was: "a resilient tiered network system where if the server goes down, it's all in place with meshed pockets of users with their nominated hosts / those hosts on a tier nominate their tier host."

This is the document for that. It is the load-bearing piece of the network design.

The tier model (NET-11, NET-49, NET-122)

Every peer occupies exactly one of three tiers at any moment:

Tier	Role	Authority	Sibling mesh
Tier 0 — Host	Top of the local mesh. Sends/receives from the central server (when present).	Token issuance, peer admission, log finalisation.	Other Tier 0 hosts (cross-mesh peers).
Tier 1 — Sub-host	Aggregator for a group of leaves. Relay between Tier 0 and Tier 2.	Vote in elections, relay messages, no token issuance.	Other Tier 1 sub-hosts in the same mesh.
Tier 2 — Leaf	Regular client. Speaks only to its assigned Tier 1 (and laterally to other Tier 2 in its session).	Read-only for sensitive ops.	Other Tier 2 in the same session pocket.

network-plugin-topology owns this. Promotion/demotion is automatic, based on capability scores (latency, uptime, reliability, device profile from NET-49) and group-size limits. Peers MUST meet a stability threshold and a cooldown (network-plugin-topology NET-52) to prevent thrashing.

Tier depth is dynamic. A 5-node mesh runs flat (everyone Tier 0/Tier 2; no sub-host). A 5,000-node mesh runs three-tier with hundreds of Tier 1 sub-hosts. The topology plugin adapts depth based on size; collapsing or expanding tiers as needed (AdaptiveTierDepth).

The four hosting models (NET-56)

network-plugin-server-bridge lets a deployment pick how clients and the central server interact:

Model	Behaviour
PURE_CLIENT	No central server. Mesh is fully autonomous. A user creates a session and acts as Tier 0 host. Subsequent users join, get assigned Tier 1/Tier 2 by topology, and the host slot transfers via election when the host leaves.
SERVER_BASED	Server is always Tier 0. All clients are Tier 1 or Tier 2. Server enforces permissions, holds authoritative state, issues all tokens. If the server disconnects, the mesh stops (this model trades resilience for auditability).
SERVER_TO_CLIENT_FAILOVER	Server is Tier 0 while online. On disconnect, an election promotes the strongest Tier 1 to acting Tier 0, the mesh continues, and on server reconnect the server smoothly reintegrates as Tier 0. This is the model that matches your user request.
CLIENT_WITH_SERVER_BACKUP	A user is Tier 0; the server joins as a participant (typically Tier 1). If the user disconnects or becomes unsuitable, hosting can transfer to the server. Hybrid authority — server provides persistence, client provides agency.

The model is selected at mesh creation and affects election behaviour, token authority, and reconciliation direction. The mesh token (NET-29) carries it.

Meshed pockets

A "pocket" is a group of peers that can talk to each other without going through Tier 0. Pockets emerge from two structural primitives:

1. Sibling links at every tier. A Tier 1 sub-host has up to 10 sibling Tier 1 sub-hosts in the same mesh. A Tier 2 leaf has up to N session-mates (capped per-session, NET-22). When a tier above goes silent, a pocket continues to function via siblings.
2. Session trees (NET-18..NET-22). The mesh organises peers into sessions — typed groupings (a video call, a shared drive, a multiplayer game). A pocket is, concretely, the peers in one leaf session plus that session's host. Sessions can nest up to depth 4; child sessions inherit read-only state from parents.

If the central server vanishes:

• Each Tier 0 host keeps running its mesh autonomously (modulo authority limits — see below).
• Each Tier 1 keeps relaying for its Tier 2 leaves.
• Each session pocket keeps gossiping among its members.
• Cross-pocket traffic is brokered by network-plugin-relay over the Tier 0 backbone (or, if Tier 0 is also gone in a partition, via the elected acting Tier 0).

Nothing about pocket operation requires the server. The server is needed only for the authoritative operations the mesh declines to do without it.

Host election — how peers nominate hosts (NET-08, NET-118, NET-119)

network-plugin-governance runs elections. The algorithm is:

1. Collect candidates — peers eligible by capability/stability threshold.
2. Score each candidate — HostSelector (NET-118) does weighted multi-factor evaluation:
   • Latency (EMA of measured RTT to other peers).
   • Stability (jitter + packet loss).
   • Uptime (since last reconnect).
   • Capabilities (CPU, bandwidth, NAT type).
   • Trust score — historical performance from HostHistory (NET-119), which keeps an append-only record of successful/unsuccessful hosting sessions per peer.
3. Vote — VotingManager collects votes via the bus; configurable participation threshold and consensus requirement.
4. Tie-break — lowest lexicographic peerId wins.
5. Publish — election:completed { winnerId, term } on the bus.

Three modes (NET-08): metric (pure score), direct (assigned by Tier 0), hybrid (score-suggested + Tier 0-confirmed).

Each session also runs its own host election independently of the mesh Tier 0 host (NET-20). One peer can host multiple sessions.

Server failover — what happens when the server goes down

This is the resilience pipeline (SERVER_TO_CLIENT_FAILOVER model):

1. Heartbeat timeout
   `network-plugin-failover` heartbeat (NET-10): alive → suspect → dead.
   `network-plugin-server-bridge` (NET-31) detects server disconnect within 2× heartbeat interval.

2. Verification
   Multiple Tier 1 peers re-ping the server through alternate paths.
   Avoid declaring the server dead on a single network blip.

3. Consensus
   Bus topic 'server:dead' published when N Tier 1 peers agree.

4. Election
   `network-plugin-governance` runs `HOST_SELECTION` among eligible Tier 1 peers.
   Winning peer is promoted to Tier 0 (acting host).

5. Authority handover
   Tokens issued before server-down remain valid.
   New token issuance is paused (NET-32) — see "graceful degradation" below.

6. Operation continues
   Each Tier 0 host keeps running its mesh autonomously.
   Cross-mesh traffic uses `network-plugin-relay` Tier 0 backbone (NET-62).

7. Server returns
   `network-plugin-server-bridge` reconciliation (NET-33):
     - Buffered signed log entries replayed in sequence.
     - Server validates signatures, detects conflicts, broadcasts reconciled state.
     - Server resumes Tier 0 (election may run to confirm — depends on hosting model).
   Acting Tier 0 demotes back to Tier 1.

Graceful degradation — server-vs-autonomous authority split

When the server is offline (NET-32), the mesh continues with reduced authority:

Operation	Server present	Server offline
Send/receive messages	✓	✓
Read existing files (drive layer)	✓	✓
Write existing files	✓	✓ (signed by drive identity, verifiable later)
Vote in elections	✓	✓
Reorganise topology	✓	✓
Issue new mesh / session tokens	✓	✗ (queued; replayed on reconnect)
Admit a brand-new peer	✓	✗ (unless server-pre-issued capability token still valid)
Cross-mesh trust establishment	✓	✗ (existing trust links remain valid)
Audit-log finalisation	✓	✗ (entries are buffered, signed, replayed on reconnect)

network-plugin-server-bridge extends token grace periods (NET-32) so existing tokens don't expire mid-outage. New session creation is paused. Signed log entries are buffered locally and replayed on reconnect (NET-33).

Cross-mesh relay — connecting pockets across meshes (NET-40, NET-61..NET-70)

When a peer in Mesh A needs to send to a peer in Mesh B, network-plugin-relay handles it.

• RelayRouter maintains a routing table (destMeshId, destPeerId) → nextHopHostId.
• HostBackboneTransport connects Tier 0 hosts of different meshes (capability-negotiated).
• CrossMeshEnvelope extends MessageEnvelope with senderMeshId, targetMeshId, hopCount, maxHops (default 8), visitedNodes, ttl.
• Loop prevention is strict (NET-64..NET-66): drop on hopCount >= maxHops, ttl <= 0, or visitedNodes already contains current peer.
• Cross-mesh broadcast is deduplicated at mesh boundaries (NET-68).
• Relay chooses paths via LowestLatency / HighestReliability / FewestHops / TierOptimized / LoadBalanced strategies (NET-40).

This is what makes "meshed pockets" actually composable: a phone-mesh, a corporate-mesh, and a public-server-mesh can interoperate by establishing trust between Tier 0 hosts (NET-110, NET-112) without merging into one big mesh.

Partition handling (NET-50, NET-78..NET-83)

A network partition splits the mesh into two or more disjoint groups, each unable to reach the others. network-plugin-partition handles it:

1. Detection — PartitionDetector tracks epochs from all known peers; partition declared when a majority is unreachable. Deterministic partitionId = hash(sorted reachable peer set).
2. Autonomous operation — each partition operates with its local hierarchy. New elections within the partition fill any gaps.
3. Quorum resolution — when partitions heal, QuorumResolver (NET-80) decides the authoritative partition using one of: size, performance, age, hybrid.
4. Merge — MergeProtocol (NET-81) reconciles registry, topology, and audit state. Non-authoritative peers rejoin (NET-82). Bus events partition:detected / partition:healed.

Resource awareness (NET-49, NET-120..NET-124)

The topology plugin adapts to device profile (CONSTRAINED, STANDARD, POWERFUL, SERVER):

• Constrained devices use MessagePool (NET-120) to reuse message objects.
• Zero-copy passing (NET-121) where the platform allows.
• BatteryAwareMode (NET-122) reduces heartbeat frequency, defers non-urgent reorganisations, lowers transport retry aggression.
• BandwidthConservator (NET-124) coalesces small messages, increases batching, pauses redundant CRITICAL sends.

So a phone running on battery in a constrained network drops to a lighter footprint without being kicked out of the mesh.

Putting it all together — the resilience picture

                    Central Server (Tier 0 of meshes M1, M2, M3)
                    │  (signed tokens, audit log, authoritative state)
                    │
        ┌───────────┼───────────┐
        │           │           │
     mesh M1     mesh M2     mesh M3
        │           │           │
   ┌────┴────┐ ┌────┴────┐ ┌────┴────┐
   │ Tier 0  │ │ Tier 0  │ │ Tier 0  │
   │ host    │ │ host    │ │ host    │
   └────┬────┘ └────┬────┘ └────┬────┘
        │           │           │
   ┌────┴────┐ ┌────┴────┐ ┌────┴────┐
   │ Tier 1  │ │ Tier 1  │ │ Tier 1  │
   │ sub-hosts ......
   └────┬────┘ └────┬────┘ └────┬────┘
        │           │           │
     [pockets of Tier 2 leaves, organised into sessions]


When the server vanishes:

                     ╳ Central Server ╳
                     (offline)
        ┌─────────────────────────────────┐
        │                                 │
     mesh M1 ─── relay ─── mesh M2 ─── relay ─── mesh M3
        │                                 │
   each mesh elects an acting Tier 0
   each Tier 0 keeps running its mesh
   each session pocket keeps gossiping
   tokens issued before outage stay valid (grace period)
   new token issuance queued; replayed on reconnect
   signed audit-log entries buffered locally
        │
   server returns
        │
   buffered logs replayed; conflicts reconciled
   acting Tier 0s demote to Tier 1
   server resumes as Tier 0 of all meshes

Reading further

• security-and-tokens.md — the cryptographic basis for token grace periods and signed log replay.
• offline-and-reintegration.md — the offline-mode buffering and reconciliation in detail.
• plugins-catalog.md — every plugin called out above, with one-paragraph descriptions.