HeliosDB Replication Architecture

Overview

HeliosDB implements a primary-mirror replication system with witness-based quorum for high availability and automatic failover. The system ensures data consistency through synchronous replication and provides automatic recovery when nodes fail or return to service.

Architecture Components

Node Roles

Primary Node
- Handles all write operations
- Replicates writes synchronously to mirrors
- Sends periodic heartbeats to all nodes
- Acknowledges writes only after quorum confirmation
Mirror Node
- Receives replicated writes from primary
- Can be promoted to primary on failure
- Participates in leader elections
- Monitors primary health through heartbeats
Witness Node
- Does not store data
- Participates in quorum decisions
- Votes in leader elections
- Helps prevent split-brain scenarios

Cluster Configuration

Standard 3-Node Setup:

1 Primary node
1 Mirror node
1 Witness node
Quorum requirement: 2 out of 3 nodes (majority)

Synchronous Replication Protocol

Write Flow

Client → Primary → Mirror(s) → Quorum Check → Client ACK

Detailed Steps:

Client sends write request to primary
Primary validates and sequences the write operation
Primary sends replication request to all active mirrors
Mirrors apply the write and send acknowledgment
Primary checks if quorum is achieved (majority of mirrors)
If quorum achieved, primary acknowledges to client
If quorum fails, primary returns error to client

Replication Message Format

ReplicationRequest {
    primary_id: NodeId,
    sequence: u64,           // Monotonically increasing sequence number
    operations: Vec<WriteOperation>,
    term: u64,               // Election term for consistency
}

ReplicationAck {
    mirror_id: NodeId,
    sequence: u64,
    success: bool,
    lag_ms: u64,            // Time taken to apply write
}

Failure Detection and Heartbeats

Heartbeat Protocol

Interval: 100ms (configurable)
Timeout: 500ms (configurable)
Failure Detection: 1500ms (3x timeout)

Heartbeat Message:

Heartbeat {
    node_id: NodeId,
    role: NodeRole,
    term: u64,
    last_sequence: u64,
    timestamp: u64,
}

Node States

Active: Receiving regular heartbeats
Suspected: No heartbeat for 1x timeout
Failed: No heartbeat for 3x timeout
Recovering: Previously failed node rejoining

Leader Election

Election Trigger

Leader election is triggered when:

Mirror node detects primary failure (3x heartbeat timeout)
Manual failover is initiated

Election Algorithm

Based on Raft consensus algorithm:

Pre-Election Phase:
- Mirror increments its term number
- Mirror transitions to candidate state
- Random election timeout: 1000-2000ms (prevents conflicts)

Vote Request:

VoteRequest {
    candidate_id: NodeId,
    term: u64,
    last_sequence: u64,  // Log completeness check
}

Voting Rules:
- Nodes grant vote if:
  - Request term ≥ current term
  - Haven’t voted in this term, OR voted for same candidate
  - Candidate’s log is at least as up-to-date
- Each node votes once per term
Election Outcome:
- Win: Candidate receives majority votes (≥2 in 3-node cluster)
- Lose: Another node wins or term expires
- Retry: Start new election with higher term after timeout

Split-Brain Prevention

Term-Based Consensus:

Each term has at most one leader
Nodes with lower term automatically step down
Higher term numbers take precedence

Example Scenario:

Old Primary (Term 1) ← Network Partition → Mirror (Term 2, New Primary)
When partition heals:
  - Old primary receives heartbeat with term 2
  - Old primary steps down to mirror role
  - System maintains single primary

Automatic Failover

Failover Sequence

Phase 1: Failure Detection

t=0ms:    Primary stops sending heartbeats
t=500ms:  Mirror marks primary as SUSPECTED
t=1500ms: Mirror marks primary as FAILED

Phase 2: Election

t=1500ms: Mirror waits random timeout (1000-2000ms)
t=2500ms: Mirror starts election, increments term
t=2600ms: Mirror requests votes from all nodes
t=2700ms: Mirror receives majority votes

Phase 3: Promotion

t=2700ms: Mirror promotes self to PRIMARY
t=2750ms: New primary sends heartbeat to all nodes
t=2800ms: Witness acknowledges new primary
t=2850ms: System operational with new primary

Total Failover Time: ~2.5-3.5 seconds

Replication Lag Monitoring

Lag Metrics

ReplicationLag {
    mirror_id: NodeId,
    sequence_lag: u64,      // Number of operations behind
    time_lag_ms: u64,       // Time to apply last operation
    last_update: Instant,
}

Monitoring

Primary tracks lag for each mirror
Lag updated on every replication acknowledgment
Metrics exposed through get_replication_lag() API

Alert Thresholds

Warning: sequence_lag > 100 operations
Critical: sequence_lag > 1000 operations
Action Required: time_lag_ms > 1000ms consistently

Catch-Up Mechanism

When Catch-Up Triggers

Mirror falls behind by > max_lag_threshold sequences
Mirror reconnects after network partition
New mirror joins cluster

Catch-Up Process

Gap Detection:
- Primary compares mirror’s last_sequence with its own
- Calculates number of missing operations
Batch Transmission:
- Primary fetches operations from persistent log
- Sends operations in batches (default: 100 ops/batch)
- Mirror applies operations in order
Completion:
- Mirror caught up when sequence_lag < threshold
- Mirror returns to normal replication mode

Configuration

ReplicationConfig {
    max_lag_threshold: 1000,      // Trigger catch-up
    catch_up_batch_size: 100,     // Operations per batch
    replication_timeout_ms: 5000, // Timeout for catch-up requests
}

Recovery Scenarios

Scenario 1: Primary Failure and Recovery

Timeline:

t=0: Primary fails, mirror promoted to new primary
t=1: System operating with new primary (Mirror → Primary)
t=2: Old primary recovers, receives heartbeat with higher term
t=3: Old primary steps down to mirror role
t=4: Old primary catches up on missed operations
t=5: System has 1 primary + 1 mirror + 1 witness (normal state)

Scenario 2: Mirror Failure

Timeline:

t=0: Mirror fails
t=1: Primary marks mirror as FAILED
t=2: Primary continues serving writes (witness provides quorum)
t=3: Mirror recovers
t=4: Primary detects mirror recovery via heartbeat
t=5: Primary initiates catch-up for mirror
t=6: Mirror fully synchronized, marked as ACTIVE

Scenario 3: Network Partition

Partition Occurs:

[Primary] ←✗→ [Mirror + Witness]

During Partition:

Primary cannot reach quorum (1/3 nodes)
Primary rejects write operations
Mirror detects primary failure, starts election
Mirror elected with witness vote (2/3 quorum)
Mirror serves writes

Partition Heals:

Old primary receives heartbeat from new primary (higher term)
Old primary steps down automatically
System recovers with single primary

Scenario 4: Witness Failure

Impact:

System remains operational
Quorum still achievable with primary + mirror (2/2 data nodes)
Elections possible without witness
Recovery: Witness rejoins, no catch-up needed (no data)

Quorum Rules

Write Quorum

3-Node Cluster (1 Primary + 1 Mirror + 1 Witness):

Required acknowledgments: 1 out of 1 mirrors
Witness does not participate in write quorum
Primary counts as 1, need majority of mirrors

5-Node Cluster (1 Primary + 3 Mirrors + 1 Witness):

Required acknowledgments: 2 out of 3 mirrors
More fault-tolerant configuration

Election Quorum

3-Node Cluster:

Required votes: 2 out of 3 (including self-vote)
Witness participates in elections
Prevents split-brain in network partitions

Performance Characteristics

Latency

Local Write (Primary): ~0.1ms
Replication (Primary → Mirror): ~10ms (network + storage)
Total Write Latency: ~10-15ms
Heartbeat Overhead: Minimal (~100 bytes every 100ms)

Throughput

Synchronous Replication: Limited by mirror acknowledgment
Expected Throughput: 10,000-50,000 writes/sec
Bottleneck: Network latency and mirror storage speed

Availability

Configuration: 3-node cluster (1P + 1M + 1W)
Tolerate: 1 node failure
Availability: 99.95%+ (with proper infrastructure)
RPO (Recovery Point Objective): 0 (synchronous replication)
RTO (Recovery Time Objective): 2.5-3.5 seconds (automatic failover)

Configuration Reference

Default Configuration

ReplicationConfig {
    heartbeat_interval_ms: 100,
    heartbeat_timeout_ms: 500,
    election_timeout_min_ms: 1000,
    election_timeout_max_ms: 2000,
    max_lag_threshold: 1000,
    catch_up_batch_size: 100,
    replication_timeout_ms: 5000,
}

Tuning Guidelines

Low Latency (LAN):

ReplicationConfig {
    heartbeat_interval_ms: 50,
    heartbeat_timeout_ms: 200,
    election_timeout_min_ms: 500,
    election_timeout_max_ms: 1000,
    ..Default::default()
}

High Latency (WAN):

ReplicationConfig {
    heartbeat_interval_ms: 500,
    heartbeat_timeout_ms: 2000,
    election_timeout_min_ms: 5000,
    election_timeout_max_ms: 10000,
    ..Default::default()
}

API Reference

Core APIs

// Create replication manager
let manager = ReplicationManager::new(
    node_id: String,
    initial_role: NodeRole,
    config: ReplicationConfig,
);

// Register peer nodes
manager.register_node(node_info).await;

// Start background tasks (heartbeat, failure detection, catch-up)
manager.start(shutdown_rx).await?;

// Primary: Replicate write
manager.replicate_write(operation).await?;

// Mirror: Apply replicated write
let ack = manager.apply_replication(request).await?;

// Handle heartbeat from peer
manager.handle_heartbeat(heartbeat).await?;

// Participate in election
let response = manager.request_vote(vote_request).await?;

// Start election (mirror → primary promotion)
let won = manager.start_election().await?;

// Monitor replication lag
let lag_map = manager.get_replication_lag().await;

Testing

Unit Tests

9 unit tests covering:

Node registration
Vote request/response handling
Heartbeat processing
Replication application
Term-based consensus
Lag tracking

Run unit tests:

cargo test -p heliosdb-network --lib replication

Integration Tests

5 integration tests covering:

Complete failover scenario (6 phases)
Quorum requirement verification
Split-brain prevention
Catch-up mechanism
Witness voting behavior

Run integration tests:

cargo test -p heliosdb-network --test replication_failover_test

Test Coverage

Scenarios Covered:

✓ Normal operation with synchronous replication
✓ Primary failure detection
✓ Leader election with witness
✓ Automatic mirror promotion
✓ Old primary recovery and demotion
✓ Replication lag monitoring
✓ Split-brain prevention
✓ Quorum calculations
✓ Term-based consensus
✓ Witness voting rules

Implementation Status

Completed Features

Synchronous replication protocol
Witness-based quorum system
Heartbeat monitoring
Failure detection
Leader election algorithm
Automatic failover
Mirror promotion to primary
Automatic recovery on node return
Replication lag monitoring
Catch-up mechanism framework
Split-brain prevention
Term-based consensus
Comprehensive test suite

Future Enhancements

References

Raft Consensus Algorithm: https://raft.github.io/
Primary-Mirror Replication Pattern
Quorum-Based Consensus Systems
Split-Brain Problem and Solutions

File Locations

Implementation: /home/claude/DM-Databases/HeliosDB/heliosdb-network/src/replication.rs
Integration Tests: /home/claude/DM-Databases/HeliosDB/heliosdb-network/tests/replication_failover_test.rs
Documentation: /home/claude/DM-Databases/HeliosDB/REPLICATION_ARCHITECTURE.md