127 lines
7.2 KiB
Markdown
127 lines
7.2 KiB
Markdown
# Offline Mode and Reintegration
|
||
|
||
The mesh keeps running when the central server is unreachable. This document is the reference for how that works in detail.
|
||
|
||
The central plugin is `network-plugin-server-bridge`, governed by NET-31..NET-34 and NET-56.
|
||
|
||
## Detection (NET-31)
|
||
|
||
Tier 0 hosts maintain a persistent WebSocket to the central server with bidirectional heartbeats. The server-bridge declares the server `dead` within 2× the heartbeat interval after the last successful heartbeat. Standard verification rules apply (re-ping through alternate paths; consensus among multiple Tier 0 hosts) to avoid declaring a dead server on a transient blip.
|
||
|
||
## Autonomous operation (NET-32)
|
||
|
||
Once the server is declared dead, the mesh transitions to autonomous mode. The transitions are:
|
||
|
||
| Subsystem | Online → Offline |
|
||
| --- | --- |
|
||
| Token grace periods | extended by configurable factor (e.g., expiresAt += 24h) |
|
||
| New mesh / session creation | paused; new requests rejected |
|
||
| New peer admission | requires a still-valid pre-issued `CapabilityToken`; raw onboarding paused |
|
||
| Audit-log finalisation | local append continues; finalisation deferred |
|
||
| Audit-log replication to server | buffered |
|
||
| Cross-mesh trust | existing `MeshTrustToken`s remain valid; new ones cannot be issued |
|
||
| Topology, failover, elections | unchanged — fully client-side |
|
||
| Drive read/write | unchanged — every operation is signed by drive identity, verifiable later |
|
||
|
||
The bus emits `server:disconnected` so plugins can react. Most plugins do nothing — they don't depend on the server. Plugins that *do* depend on the server (server-bridge, auth, audit-mesh) flip into offline mode.
|
||
|
||
## What gets buffered (NET-32)
|
||
|
||
Three buffer streams, all in local storage (typically `OpfsStorage` in the browser, `FileStorage` on a server peer):
|
||
|
||
1. **Pending audit-log entries** — every signed log entry that would have been finalised by the server.
|
||
2. **Pending token-issuance requests** — operations the user attempted that need a server-signed token.
|
||
3. **Pending mesh state deltas** — bitpacker-encoded registry/topology deltas that the server would have authoritatively reconciled.
|
||
|
||
All three streams are append-only and signed by the issuing peer. Storage is bounded; on overflow, the oldest *non-CRITICAL* entries are dropped first. CRITICAL entries are never dropped — if the buffer is full of CRITICAL, the system surfaces an error and refuses to accept more until the user clears space (or the server returns).
|
||
|
||
## Reconnection (NET-33)
|
||
|
||
When the server-bridge re-establishes the WebSocket, the reconciliation pipeline runs:
|
||
|
||
```text
|
||
1. Handshake
|
||
Server-bridge sends its mesh-bridge identity and the highest seq the server has.
|
||
Server replies with the seq it expects.
|
||
|
||
2. Replay buffered entries in seq order
|
||
For each:
|
||
a. Server verifies Ed25519 signature against the issuing peer's pubkey.
|
||
b. Server checks for conflicts with other partitions' replays.
|
||
c. On accept: append to authoritative log, advance seq.
|
||
d. On conflict: invoke partition merge protocol (NET-80, NET-81).
|
||
|
||
3. Process pending token requests
|
||
For each queued request:
|
||
a. Re-validate authorization (the issuer may no longer be eligible).
|
||
b. Issue server-signed token.
|
||
c. Notify the requesting peer via bus.
|
||
|
||
4. Reconcile state
|
||
Server broadcasts `state:reconciled { meshState, sessionStates, ... }`.
|
||
Each plugin updates its local view to match.
|
||
|
||
5. Hosting transition (model-dependent)
|
||
In SERVER_TO_CLIENT_FAILOVER, the acting Tier 0 demotes; server resumes Tier 0.
|
||
In CLIENT_WITH_SERVER_BACKUP, the user remains Tier 0; the server returns to its
|
||
participant role.
|
||
|
||
6. Bus emits `server:reconnected` and `mesh:reconciled`.
|
||
Suspended operations (new mesh creation, etc.) resume.
|
||
```
|
||
|
||
The replay is idempotent: replaying an already-finalised entry is a no-op (the server tracks `(peerId, seq)` to dedupe). This means a flaky reconnect that drops mid-replay can simply retry from the last acknowledged seq.
|
||
|
||
## Conflict resolution
|
||
|
||
Two partitions running autonomously can produce conflicting state. Examples:
|
||
|
||
- Both elected a different acting Tier 0.
|
||
- Both wrote different state to the same path.
|
||
- Both admitted a peer the other doesn't know about.
|
||
|
||
The merge protocol (NET-81) reconciles using the partition `QuorumResolver` strategy (NET-80) — `size`, `performance`, `age`, or `hybrid`. Whichever partition wins becomes the authoritative source; the other partition's peers rejoin (NET-82).
|
||
|
||
For state-level conflicts that don't involve elections, `network-plugin-state` reconciliation runs (NET-42) with one of five strategies: `latest-wins`, `remote-wins`, `local-wins`, `deep-merge`, or `custom`. The strategy is configurable per state path via `PathStrategyRegistry`.
|
||
|
||
## Audit-log compaction (NET-39)
|
||
|
||
After reintegration, the audit log can grow unbounded. Compaction folds old entries into signed checkpoints:
|
||
|
||
```text
|
||
checkpoint {
|
||
seqRange: [from, to],
|
||
stateSnapshot: bitpacker-encoded snapshot at seq=to,
|
||
hostSignature: Ed25519 over (seqRange ‖ stateHash),
|
||
}
|
||
```
|
||
|
||
Replicas use the latest checkpoint to skip the entire prefix when bringing up a fresh peer. The full history remains available for forensic queries; the on-the-wire replay path skips it.
|
||
|
||
## Hosting-model interactions
|
||
|
||
| Model | What happens at server-down | What happens at server-up |
|
||
| --- | --- | --- |
|
||
| `PURE_CLIENT` | No-op — there's no server to lose. | No-op. |
|
||
| `SERVER_BASED` | Mesh stops. Operations queue locally; users see "offline" UI. | Replay queue drains; mesh resumes. |
|
||
| `SERVER_TO_CLIENT_FAILOVER` | Election promotes Tier 1 → acting Tier 0. Mesh continues with full operations except new-token issuance. | Acting Tier 0 demotes; server resumes. Buffered log replays. |
|
||
| `CLIENT_WITH_SERVER_BACKUP` | User remains Tier 0; mesh continues normally. The server is "missing" but it wasn't authoritative anyway. | Server rejoins as participant. Buffered log replays. |
|
||
|
||
The mesh token (NET-29) carries the model, so the server-bridge knows which transitions to run.
|
||
|
||
## Common pitfalls and how the design avoids them
|
||
|
||
| Pitfall | Avoidance |
|
||
| --- | --- |
|
||
| Replay stuck on a single bad entry | Server processes entries individually; rejects-but-continues. |
|
||
| Buffer overflow loses history | CRITICAL entries are never dropped. Surface error before silent loss. |
|
||
| Two acting Tier 0s in different partitions both finalising tokens | Tokens issued in autonomous mode have a special bit set; on merge, one partition's tokens are voided per the quorum strategy and re-issued. |
|
||
| Clock drift causes apparent expiry mid-outage | `extendedExpiresAt` is computed from the *peer's* clock, not the server's, so lack of server time doesn't cause spurious expiry. |
|
||
| Long outage ⇒ everyone's CapabilityToken expires | Grace-period extension is automatic; if the outage exceeds the configured maximum, the mesh degrades to read-only. |
|
||
|
||
## Reading further
|
||
|
||
- [`topology-and-failover.md`](./topology-and-failover.md) — election + Tier-0 promotion.
|
||
- [`security-and-tokens.md`](./security-and-tokens.md) — the cryptographic chain that makes replay safe.
|
||
- [`plugins-catalog.md`](./plugins-catalog.md) — `network-plugin-server-bridge`, `network-plugin-audit-mesh`, `network-plugin-partition`.
|