suaveplan/docs/old-network-planning/PHASE_5.md

Phase 5: Tier Management

Overview

Phase 5 focuses on implementing the tier management system, responsible for organizing and maintaining the hierarchical structure of the network. The TierManager handles complex reorganization scenarios, responds to topology changes, and ensures the network structure remains optimal as nodes join, leave, or change roles. This phase builds upon the previous components to create a self-healing, adaptive network hierarchy.

Key Components

1. TierManager Implementation (src/tier/manager.ts)

The TierManager class orchestrates the overall hierarchy structure. It should:

• Manage tier creation, deletion, and modification
• Handle network reorganization during topology changes
• Implement strategies for various edge cases (empty tiers, partitions, etc.)
• Ensure the network structure remains balanced and efficient

Core Features

• Tier Structure Management: Create, modify, and remove tiers
• Reorganization Logic: Handle various scenarios requiring hierarchy changes
• Balance Maintenance: Ensure the hierarchy remains properly balanced
• Edge Case Handling: Address special situations like empty tiers
• Hierarchy Optimization: Optimize the network structure for efficiency

Implementation Requirements

• Support dynamic tier insertion and removal
• Implement safe reorganization transactions
• Provide event notifications for hierarchy changes
• Support configurable tier properties and constraints
• Handle concurrent topology changes

Edge Cases

• Empty tier detection and handling
• Network partition recovery
• Cascading failures prevention
• Client surge/exodus adaptation
• Node rejoining placement

2. Reorganization Logic (src/tier/reorganization.ts)

This component handles the complex logic of reorganizing the network structure. It should:

• Implement algorithms for different reorganization scenarios
• Ensure consistent hierarchy during changes
• Minimize disruption during reorganization
• Support transaction-like operations with rollback capability

Core Features

• Empty Tier Handling: Remove empty tiers and reconnect adjacent levels
• Tier Insertion: Add new tiers between existing levels
• Node Redistribution: Reassign children when parents are demoted/removed
• Partition Recovery: Merge separated network partitions
• Balancing Algorithms: Rebalance the hierarchy for optimal performance

Implementation Requirements

• Support phased reorganization to minimize disruption
• Implement rollback mechanisms for failed reorganizations
• Provide progress tracking for long-running operations
• Ensure hierarchy integrity during all operations

Edge Cases

• Interrupted reorganizations
• Concurrent reorganization attempts
• Partial failures during reorganization
• Conflicting hierarchy views after partition recovery

3. Node Promotion/Demotion (src/tier/level-change.ts)

This component manages the promotion and demotion of nodes between tiers. It should:

• Implement criteria for node level changes
• Handle the logistics of moving nodes between levels
• Manage child redistribution during level changes
• Track performance metrics to guide promotion/demotion decisions

Core Features

• Promotion Criteria: Determine when nodes should move up the hierarchy
• Demotion Criteria: Determine when nodes should move down the hierarchy
• Child Redistribution: Reassign children when node levels change
• Level Change Execution: Handle the logistics of changing a node's level

Implementation Requirements

• Support configurable promotion/demotion thresholds
• Implement performance scoring for level change decisions
• Provide event notifications for level changes
• Ensure smooth transitions during level changes

Edge Cases

• Multiple concurrent level changes
• Failed level changes
• Orphaned children during level changes
• Promotion/demotion thrashing prevention

Critical Scenario: Server Failover and Reintegration

A key capability that must be implemented in this phase is handling server disconnection and reintegration as detailed in SERVER_FAILOVER.md. This scenario requires specific tier management capabilities:

Server Disconnection Handling

When the server (top-level host) disconnects:

1. Detection: The TierManager must quickly detect the empty top tier
2. Emergency Reorganization: Trigger immediate reorganization to establish client-side hosting
3. Hierarchical Continuity: Maintain as much of the existing hierarchy as possible while adapting to server absence
4. Minimal Disruption: Limit changes to only what's necessary for continued operation

Client-Based Hierarchy

The TierManager must support a fully client-side hierarchy:

1. Adaptive Tier Structure: Adjust tier depth based on network size and client capabilities
2. Role Flexibility: Support client nodes functioning as hosts at any tier level
3. Resource Awareness: Consider device capabilities when assigning hosting responsibilities
4. Tier Balance: Distribute clients optimally across hosts to balance load

Server Reintegration

When the server reconnects:

1. Recognition: Detect returning server node
2. Role Restoration: Reestablish server as top-tier host
3. Hierarchy Adjustment: Adapt client-side hierarchy to reincorporate the server
4. Smooth Transition: Minimize disruption during this reintegration process

Implementation of this scenario must be thoroughly tested to ensure the network can seamlessly transition between server-hosted and client-hosted modes.

Tier Structure Management

The tier system organizes nodes into a hierarchical structure:

1. Tier Configuration

Each tier (level) has specific configuration parameters:

interface LevelConfig {
  level: number;          // Numerical level (0, 1, 2, etc.)
  name: string;           // Human-readable name ("Client", "Host", etc.)
  maxChildren: number;    // Maximum children per node
  maxSiblings: number;    // Maximum siblings at this level
  minPromotionScore: number; // Threshold for promotion to next level
  demotionThreshold: number; // Threshold below which demotion occurs
  // Other tier-specific settings
}

2. Tier Relationships

The hierarchical structure follows these rules:

• Level 0 (Client): No children, connects to Level 1
• Level 1 (Host): Manages clients, connects to Level 2
• Level 2 (Tier Host): Manages hosts, connects to Level 3
• And so on, up to the highest level

3. Dynamic Tier Adjustments

The structure can change dynamically:

• Tiers can be added when the network grows
• Tiers can be removed when no longer needed
• Tier properties can be adjusted based on network conditions

Network Reorganization Scenarios

The TierManager must handle various reorganization scenarios:

1. Empty Tier Handling

When all nodes from a tier disappear:

1. Detection: Monitor node counts per tier
2. Evaluation: Determine if tier should be removed
3. Reconnection: Connect adjacent tiers to maintain hierarchy
4. Level Renumbering: Update level numbers for all affected nodes
5. Configuration Adjustment: Update level configurations

Example:

Before: L0 → L1 → L2 → L3 → L4
If L2 becomes empty:
After: L0 → L1 → L2* → L3*
(where L2* was L3 and L3* was L4)

2. Network Partition Recovery

When the network splits and then reconnects:

1. Partition Detection: Identify disconnected segments
2. Partition Evaluation: Determine primary partition
3. Structure Reconciliation: Merge hierarchy structures
4. Conflict Resolution: Resolve conflicting roles
5. State Synchronization: Reconcile state differences

Conflict Resolution Strategy:

• Size-based: Larger partition's structure prevails
• Performance-based: Better-performing structure prevails
• Age-based: Older structure prevails
• Hybrid: Combine aspects of multiple strategies

3. Client Surge Handling

When many clients join simultaneously:

1. Detection: Monitor client-to-host ratios
2. Capacity Evaluation: Determine if more hosts are needed
3. Candidate Selection: Identify clients suitable for promotion
4. Promotion Execution: Promote selected clients to hosts
5. Distribution: Assign clients to new hosts

Promotion Criteria:

• Performance metrics exceed thresholds
• Stable connection history
• Appropriate device capabilities
• Geographic distribution considerations

4. Client Exodus Handling

When many clients leave simultaneously:

1. Detection: Monitor host utilization
2. Capacity Evaluation: Determine excess host capacity
3. Candidate Selection: Identify hosts for demotion
4. Child Redistribution: Reassign remaining clients
5. Demotion Execution: Demote selected hosts to clients

Demotion Selection Criteria:

• Lower performance metrics
• Fewer assigned clients
• Higher resource constraints
• Geographic redundancy

5. Cascading Failure Prevention

When multiple nodes fail in sequence:

1. Detection: Monitor failure patterns
2. Impact Assessment: Evaluate hierarchy stability
3. Circuit Breaking: Pause reorganizations if necessary
4. Controlled Recovery: Implement phased recovery plan
5. Stabilization: Rebalance once failures subside

Circuit Breaking Strategy:

• Time-based: Suspend reorganizations for a period
• Count-based: Limit number of reorganizations in a time window
• Impact-based: Evaluate disruption before proceeding
• Hierarchical: Prioritize higher-level stability

Implementation Details

Reorganization Transaction Model

Reorganizations should follow a transaction-like model:

1. Preparation: Gather necessary information
2. Validation: Verify the operation is safe
3. Execution: Make the changes in a specific order
4. Verification: Validate the new structure
5. Commitment: Finalize the changes
6. Rollback: Revert to previous state if needed

Tier Balancing Algorithm

The balancing algorithm should consider:

1. Distribution Metrics:

   • Nodes per tier
   • Children per parent
   • Geographic distribution
   • Performance distribution

2. Balance Operations:

   • Node promotion/demotion
   • Child redistribution
   • Sibling relationship adjustments
   • Tier creation/removal

3. Optimization Goals:

   • Minimize path lengths
   • Distribute load evenly
   • Improve geographic locality
   • Maximize overall performance

Node Level Change Process

The level change process should:

1. Before Promotion:

   • Verify node meets promotion criteria
   • Ensure target level has capacity
   • Prepare node for new responsibilities

2. During Promotion:

   • Update node's level
   • Apply new level configuration
   • Establish appropriate relationships
   • Initialize level-specific components

3. Before Demotion:

   • Identify new homes for children
   • Prepare for role reduction
   • Notify affected nodes

4. During Demotion:

   • Redistribute children
   • Update node's level
   • Apply new level configuration
   • Adjust relationship with former peers

Testing Strategy

TierManager Tests

1. Configuration Tests:

   • Test with various tier configurations
   • Test configuration validation
   • Test dynamic configuration changes

2. Tier Structure Tests:

   • Test tier creation and removal
   • Test level renumbering
   • Test hierarchy validation

3. Integration Tests:

   • Test interaction with NetworkSystem
   • Test interaction with NetworkNode
   • Test event propagation

Reorganization Tests

1. Empty Tier Tests:

   • Test detection of empty tiers
   • Test tier removal and reconnection
   • Test level renumbering
   • Test edge cases (multiple empty tiers, all tiers empty)

2. Partition Tests:

   • Test partition detection
   • Test partition recovery
   • Test conflict resolution
   • Test with various partition scenarios

3. Edge Case Tests:

   • Test with varying network sizes
   • Test with extreme scenarios (massive joins/leaves)
   • Test with simulated network failures
   • Test interrupted reorganizations

Level Change Tests

1. Promotion Tests:

   • Test promotion criteria evaluation
   • Test promotion execution
   • Test post-promotion state
   • Test failed promotions

2. Demotion Tests:

   • Test demotion criteria evaluation
   • Test child redistribution
   • Test demotion execution
   • Test forced demotions

3. Performance Tests:

   • Test promotion/demotion under load
   • Test with large numbers of level changes
   • Test impact on network performance

Server Failover Tests

1. Disconnection Tests:

   • Test server disconnection detection
   • Test emergency reorganization
   • Test client-side hierarchy formation
   • Test message routing without server

2. Reintegration Tests:

   • Test server reconnection detection
   • Test server role restoration
   • Test hierarchy adjustment
   • Test state reconciliation

3. Stability Tests:

   • Test with repeated server disconnection/reconnection
   • Test long-duration server absence
   • Test with state changes during server absence
   • Test with concurrent client changes during reintegration

Edge Cases and Considerations

1. Reorganization Collisions

Consider what happens when multiple reorganizations occur simultaneously:

• Implement locking or sequencing mechanisms
• Prioritize reorganizations based on impact
• Merge compatible reorganizations
• Detect and resolve conflicts

2. Hierarchical Instability

Prevent thrashing or oscillation in the hierarchy:

• Implement hysteresis in promotion/demotion thresholds
• Add cooldown periods after level changes
• Track historical stability to inform decisions
• Use dampening factors for rapid changes

3. Deep Hierarchies

Handle very deep hierarchical structures:

• Optimize path calculations for deep hierarchies
• Consider hierarchy flattening when appropriate
• Implement level skipping for efficiency
• Monitor and limit maximum depth

4. Sparse Hierarchies

Handle hierarchies with nearly-empty tiers:

• Implement minimum utilization thresholds
• Consider tier merging for underutilized levels
• Balance between hierarchy depth and breadth
• Adapt tier configurations to utilization

5. Incomplete Information

Deal with decisions based on partial information:

• Implement conservative defaults for unknown values
• Use probabilistic models when information is limited
• Weight decisions based on information confidence
• Adapt as more information becomes available

Integration Points

The tier management system integrates with:

1. NetworkSystem: For overall network structure
2. NetworkNode: For node level changes
3. VotingManager: For host selection during reorganization
4. MessageRouter: For adapting routing to hierarchy changes
5. StateManager: For state synchronization during reorganization

Performance Considerations

Optimize the tier management system for:

1. Reorganization Efficiency: Minimize disruption during changes
2. Decision Speed: Make timely decisions with available information
3. Memory Usage: Limit state tracking to necessary information
4. Scalability: Support networks from small to very large

Success Criteria

This phase is complete when:

1. The TierManager correctly manages the network hierarchy
2. Empty tiers are properly detected and handled
3. Network partitions can be recovered from
4. Client surges and exoduses are handled gracefully
5. Cascading failures are prevented
6. Node promotion and demotion work correctly
7. The network structure remains optimized
8. Server disconnection and reintegration work seamlessly
9. All tests pass with 100% code coverage

The tier management system is essential for maintaining an efficient, self-healing network structure. A well-implemented tier manager ensures the network can adapt to changing conditions, recover from failures, and maintain optimal performance even as the network evolves.