Skip to content

HeliosDB v4.0.0 - Complete Configuration Reference

HeliosDB v4.0.0 - Complete Configuration Reference

Comprehensive Configuration Guide for All Features

This document provides complete configuration reference for HeliosDB v4.0.0, including all v4.0 breakthrough features and legacy configurations.


Table of Contents


Configuration File Format

HeliosDB v4.0.0 uses YAML format for configuration files (upgraded from TOML in v3.1).

Default Location: /etc/heliosdb/heliosdb.yaml

Override with CLI:

Terminal window
heliosdb-server --config /path/to/custom/heliosdb.yaml

Quick Start Configurations

Development (Single Node, All Features Disabled)

# heliosdb.yaml - Development Configuration
server:
host: 0.0.0.0
port: 5432
max_connections: 100
storage:
data_dir: /var/lib/heliosdb/data
wal_dir: /var/lib/heliosdb/wal
compression: none
# All v4.0 features disabled for simplicity
branching:
enabled: false
autoscaling:
enabled: false
tiered_storage:
enabled: false

Production (Multi-Node, All v4.0 Features Enabled)

# heliosdb.yaml - Production Configuration
server:
host: 0.0.0.0
port: 5432
max_connections: 1000
max_connections_per_user: 100
# Git-Style Branching
branching:
enabled: true
max_branches_per_database: 100
default_parent: main
auto_gc: true
gc_interval: 24h
branch_retention: 30d
# Scale-to-Zero & Autoscaling
autoscaling:
enabled: true
min_cu: 0.5 # Minimum 0.5 CU (don't scale to zero in prod)
max_cu: 16.0 # Maximum 16 CUs
scale_to_zero_after: 0 # Disable scale-to-zero in prod (set to 0)
resume_timeout: 300ms
target_cpu: 70
target_queue_depth: 10
scale_up_threshold: 80
scale_down_threshold: 30
cooldown_period: 60s
# Query-from-Any-Node
distributed_query:
enabled: true
metadata_cache_size: 1GB
cache_ttl: 300s
invalidation_batch_size: 1000
# Zero-Downtime Rebalancing
rebalancing:
enabled: true
auto_rebalance: true
strategy: by_disk_size # Options: by_shard_count, by_disk_size, by_tenant_id
threshold: 0.20 # Trigger at 20% imbalance
check_interval: 3600s # Check every hour
max_concurrent_moves: 5
bandwidth_limit: 100MB/s
# Enhanced Columnar Compression
compression:
enabled: true
default_algorithm: hcc_v2 # Options: none, hcc_v1, hcc_v2
compression_level: high # Options: low, medium, high, adaptive
adaptive_selection: true
# Schema-Based Sharding
schema_sharding:
enabled: true
default_distribution: distributed
allow_non_distributed: true
# Distributed Foreign Keys
foreign_keys:
enabled: true
cross_shard_validation: true
reference_table_replication: true
join_optimization: true
# 3-Tier Storage
tiered_storage:
enabled: true
hot_tier:
path: /mnt/nvme/heliosdb
max_size_gb: 1000
latency_target_ms: 1
cost_per_gb: 0.15
warm_tier:
path: /mnt/ssd/heliosdb
max_size_gb: 5000
latency_target_ms: 5
cost_per_gb: 0.04
cold_tier:
type: s3
bucket: heliosdb-cold-tier-prod
region: us-east-1
endpoint: https://s3.amazonaws.com
access_key_id: ${AWS_ACCESS_KEY_ID} # Environment variable
secret_access_key: ${AWS_SECRET_ACCESS_KEY}
latency_target_ms: 50
cost_per_gb: 0.02
policies:
hot_to_warm: 7d
warm_to_cold: 30d
migration:
max_concurrent: 5
max_bandwidth_mbps: 100
atomic_transitions: true
enable_throttling: true
# Safekeeper Consensus
safekeeper:
enabled: true
cluster_size: 3
quorum_size: 2
nodes:
- id: sk-1
address: 10.0.1.10:5433
- id: sk-2
address: 10.0.1.11:5433
- id: sk-3
address: 10.0.1.12:5433
wal_retention: 7d
sync_timeout_ms: 100
async_storage_flush: true
# Online Table Sharding
online_sharding:
enabled: true
max_concurrent_migrations: 3
migration_throughput_limit: 100000 # rows/sec
cutover_timeout_ms: 100
# Multi-Tenant Quotas
multi_tenancy:
enabled: true
enforce_quotas: true
default_tier: bronze
tiers:
bronze:
max_cpu_cu: 2.0
max_iops: 5000
priority: 1
price_multiplier: 1.0
silver:
max_cpu_cu: 8.0
max_iops: 15000
priority: 5
price_multiplier: 1.5
gold:
max_cpu_cu: 32.0
max_iops: 50000
priority: 10
price_multiplier: 2.5
enforcement:
check_interval_ms: 100
grace_period_ms: 1000
throttle_on_violation: true

v4.0 Breakthrough Features

1. Git-Style Database Branching

branching:
enabled: true # Enable branching feature
max_branches_per_database: 100 # Maximum branches per database
default_parent: main # Default parent branch name
auto_gc: true # Automatic garbage collection
gc_interval: 24h # GC interval (hours, days)
branch_retention: 30d # Retention period for deleted branches
delta_ssttable_max_size: 1GB # Maximum delta SSTable size before compaction
copy_on_write: true # Enable copy-on-write (always true)

SQL API:

-- Create branch
SELECT heliosdb.create_branch('feature-1');
SELECT heliosdb.create_branch('debug-1', parent_timestamp => '2025-10-24 14:30:00');
SELECT heliosdb.create_branch('rollback-1', parent_lsn => '0/1234ABCD');
-- List branches
SELECT * FROM heliosdb.branches;
-- Switch branch
SELECT heliosdb.checkout_branch('feature-1');
-- Delete branch
SELECT heliosdb.delete_branch('feature-1');

2. Scale-to-Zero Serverless Compute

autoscaling:
enabled: true
min_cu: 0.0 # Minimum CUs (0.0 = scale to zero)
max_cu: 4.0 # Maximum CUs
scale_to_zero_after: 300s # Idle time before suspend (seconds)
resume_timeout: 300ms # Maximum resume time
suspend_checkpoint_timeout: 500ms # Checkpoint timeout during suspend
persist_connection_state: true # Persist connection state
state_storage_path: /var/lib/heliosdb/state
billing_precision: 1s # Billing precision (1s = 1 second granularity)

Activity Monitoring:

autoscaling:
activity_monitoring:
enabled: true
check_interval_ms: 1000 # Check every second
idle_threshold_queries: 0 # Consider idle if 0 queries
idle_threshold_connections: 0 # Consider idle if 0 active connections

SQL API:

-- Check compute status
SELECT * FROM heliosdb.compute_status;
-- Manual suspend
SELECT heliosdb.suspend_compute();
-- Manual resume
SELECT heliosdb.resume_compute();

3. Dynamic Autoscaling

autoscaling:
# Already covered above, additional settings:
target_cpu: 70 # Target CPU utilization (%)
target_queue_depth: 10 # Target query queue depth
target_memory: 80 # Target memory utilization (%)
scale_up_threshold: 80 # Scale up at 80% CPU
scale_down_threshold: 30 # Scale down at 30% CPU
cooldown_period: 60s # Wait between scaling decisions
min_scale_step: 0.5 # Minimum CUs to add/remove
max_scale_step: 2.0 # Maximum CUs to add/remove in one step
predictive_scaling: false # ML-based predictive scaling (future)

SQL API:

-- View autoscaling events
SELECT * FROM heliosdb.autoscale_events
WHERE timestamp > NOW() - INTERVAL '1 hour'
ORDER BY timestamp DESC;
-- View current resource usage
SELECT
current_cu,
cpu_utilization,
memory_utilization,
query_queue_depth,
active_connections,
recommended_cu
FROM heliosdb.autoscale_status;

4. Query-from-Any-Node Architecture

distributed_query:
enabled: true
metadata_cache_size: 1GB # Metadata cache size per node
cache_ttl: 300s # Cache TTL
cache_invalidation_batch_size: 1000 # Batch size for invalidation messages
cache_refresh_interval: 60s # Automatic refresh interval
ddl_centralized: true # DDL always routed to metadata service
query_routing_strategy: automatic # Options: automatic, coordinator_only

SQL API:

-- View metadata cache status
SELECT * FROM heliosdb.metadata_cache_status;
-- Force cache invalidation
SELECT heliosdb.invalidate_metadata_cache();
-- View query routing statistics
SELECT * FROM heliosdb.query_routing_stats;

5. Zero-Downtime Shard Rebalancing

rebalancing:
enabled: true
auto_rebalance: true # Enable automatic rebalancing
strategy: by_disk_size # Options: by_shard_count, by_disk_size, by_tenant_id
threshold: 0.20 # Trigger at 20% imbalance
check_interval: 3600s # Check every hour
max_concurrent_moves: 5 # Maximum concurrent shard moves
bandwidth_limit: 100MB/s # Network bandwidth limit
replication_lag_threshold: 1000 # Max WAL records lag before cutover
cutover_lock_timeout_ms: 100 # Maximum lock time during cutover
rollback_on_failure: true # Automatic rollback on failure

SQL API:

-- Manual rebalancing
SELECT heliosdb.rebalance_shards(strategy => 'by_disk_size');
-- View rebalancing status
SELECT * FROM heliosdb.rebalance_status;
-- View shard distribution
SELECT * FROM heliosdb.shard_distribution;

6. Enhanced Columnar Compression (HCC v2)

compression:
enabled: true
default_algorithm: hcc_v2 # Options: none, hcc_v1, hcc_v2
compression_level: high # Options: low, medium, high, adaptive
adaptive_selection: true # Automatically choose best algorithm
algorithms:
dictionary_encoding: true
delta_encoding: true
run_length_encoding: true
zstd: true
lz4: true
frame_of_reference: true
bit_packing: true
null_suppression: true
zstd_level: 3 # ZSTD compression level (1-22)
compression_threshold: 1KB # Minimum size to compress

SQL API:

-- Create table with HCC v2
CREATE TABLE events (
...
) WITH (
compression = 'hcc_v2',
compression_level = 'high'
);
-- View compression statistics
SELECT * FROM heliosdb.compression_stats WHERE table_name = 'events';
-- Convert table to HCC v2
ALTER TABLE legacy_events SET (compression = 'hcc_v2');

7. Schema-Based Sharding

schema_sharding:
enabled: true
default_distribution: distributed # Options: distributed, non_distributed
allow_non_distributed: true # Allow non-distributed schemas
schema_placement_strategy: automatic # Options: automatic, manual
migration_strategy: logical_replication
migration_bandwidth_limit: 100MB/s

SQL API:

-- Create distributed schema
CREATE SCHEMA tenant_1234 DISTRIBUTED;
-- Create non-distributed schema
CREATE SCHEMA analytics NON_DISTRIBUTED;
-- View schema distribution
SELECT * FROM heliosdb.schema_distribution;
-- Migrate schema to different node
SELECT heliosdb.migrate_schema(
schema_name => 'tenant_5678',
target_node => 'node-3',
strategy => 'logical_replication'
);

8. Distributed Foreign Keys

foreign_keys:
enabled: true
cross_shard_validation: true # Enable cross-shard FK validation
reference_table_replication: true # Replicate reference tables
join_optimization: true # Enable join optimization
co_located_fk_optimization: true # Optimize co-located FKs
fk_cache_enabled: true # Cache FK validation results
fk_cache_ttl: 60s # FK cache TTL

SQL API:

-- Create co-located foreign key
CREATE TABLE orders (
tenant_id INT,
user_id INT,
FOREIGN KEY (tenant_id, user_id) REFERENCES users(tenant_id, user_id)
) SHARD BY (tenant_id);
-- Create reference table (replicated)
CREATE TABLE countries (
country_code CHAR(2) PRIMARY KEY,
country_name TEXT
) REPLICATED;
-- View FK validation statistics
SELECT * FROM heliosdb.fk_validation_stats;
-- View join optimization statistics
SELECT * FROM heliosdb.join_optimization_stats;

9. 3-Tier Storage (Hot/Warm/Cold)

tiered_storage:
enabled: true
hot_tier:
path: /mnt/nvme/heliosdb
max_size_gb: 1000
latency_target_ms: 1
cost_per_gb: 0.15
iops_limit: 100000
warm_tier:
path: /mnt/ssd/heliosdb
max_size_gb: 5000
latency_target_ms: 5
cost_per_gb: 0.04
iops_limit: 50000
cold_tier:
type: s3 # Options: s3, minio, ceph, gcs, azure
bucket: heliosdb-cold-tier
region: us-east-1
endpoint: https://s3.amazonaws.com
access_key_id: ${AWS_ACCESS_KEY_ID}
secret_access_key: ${AWS_SECRET_ACCESS_KEY}
latency_target_ms: 50
cost_per_gb: 0.02
multipart_threshold: 5MB
multipart_chunk_size: 100MB
compression: zstd # Compress before upload
encryption: aes256 # Encrypt before upload
policies:
hot_to_warm: 7d # Move to warm after 7 days
warm_to_cold: 30d # Move to cold after 30 days
access_based_tiering: true # Override age-based if frequently accessed
min_access_frequency: 10 # Keep in hot if >10 accesses/day
migration:
max_concurrent: 5
max_bandwidth_mbps: 100
atomic_transitions: true
enable_throttling: true
retry_attempts: 3
retry_delay_ms: 1000

SQL API:

-- View tier usage
SELECT * FROM heliosdb.storage_tier_usage;
-- Manual tier migration
SELECT heliosdb.move_to_tier(
object_key => 'table:events:sst-12345',
target_tier => 'cold'
);
-- View tiering policies
SELECT * FROM heliosdb.tiering_policies;
-- Generate cost report
SELECT * FROM heliosdb.storage_cost_analysis();

10. Safekeeper Consensus Layer

safekeeper:
enabled: true
cluster_size: 3 # Number of Safekeeper nodes
quorum_size: 2 # Quorum size (2/3)
nodes:
- id: sk-1
address: 10.0.1.10:5433
priority: 10 # Higher priority = preferred leader
- id: sk-2
address: 10.0.1.11:5433
priority: 5
- id: sk-3
address: 10.0.1.12:5433
priority: 1
wal_retention: 7d # WAL retention period
wal_segment_size: 16MB # WAL segment size
sync_timeout_ms: 100 # Synchronous replication timeout
async_storage_flush: true # Asynchronous storage flush
compression: lz4 # WAL compression
checksum: true # WAL checksums

SQL API:

-- View safekeeper cluster status
SELECT * FROM heliosdb.safekeeper_cluster_status;
-- View WAL replication metrics
SELECT * FROM heliosdb.wal_replication_stats;
-- Force safekeeper failover (testing only)
SELECT heliosdb.trigger_safekeeper_failover();

11. Online Table Sharding Migration

online_sharding:
enabled: true
max_concurrent_migrations: 3 # Maximum concurrent table migrations
migration_throughput_limit: 100000 # Rows/sec
bulk_copy_batch_size: 10000 # Rows per batch
replication_lag_threshold: 1000 # Max lag before cutover
cutover_timeout_ms: 100 # Maximum cutover lock time
rollback_on_failure: true # Automatic rollback
progress_reporting_interval: 1000 # Report progress every 1000 rows

SQL API:

-- Shard table online
SELECT heliosdb.shard_table_online(
table_name => 'events',
shard_key => 'user_id',
shard_count => 16
);
-- Monitor sharding progress
SELECT * FROM heliosdb.online_sharding_status WHERE table_name = 'events';
-- Rollback sharding
SELECT heliosdb.rollback_online_sharding('events');

12. Multi-Tenant Resource Quotas

multi_tenancy:
enabled: true
enforce_quotas: true
default_tier: bronze
tiers:
bronze:
max_cpu_cu: 2.0
max_memory_gb: 8
max_storage_gb: 100
max_iops: 5000
max_connections: 50
priority: 1
price_multiplier: 1.0
burst_allowance: 0.1 # 10% burst for 1 minute
burst_duration: 60s
silver:
max_cpu_cu: 8.0
max_memory_gb: 32
max_storage_gb: 1000
max_iops: 15000
max_connections: 200
priority: 5
price_multiplier: 1.5
burst_allowance: 0.2 # 20% burst for 5 minutes
burst_duration: 300s
gold:
max_cpu_cu: 32.0
max_memory_gb: 128
max_storage_gb: 10000
max_iops: 50000
max_connections: 1000
priority: 10
price_multiplier: 2.5
burst_allowance: 0.5 # 50% burst for 15 minutes
burst_duration: 900s
enforcement:
check_interval_ms: 100
grace_period_ms: 1000
throttle_on_violation: true
reject_on_hard_limit: true
notification_enabled: true
notification_threshold: 0.9 # Notify at 90% quota

SQL API:

-- Create tenant with quotas
CREATE TENANT tenant_123 WITH (
max_connections => 100,
max_cpu_cu => 4.0,
max_storage_gb => 1000,
max_iops => 10000,
qos_tier => 'gold'
);
-- View tenant usage
SELECT * FROM heliosdb.tenant_usage WHERE tenant_id = 'tenant_123';
-- Modify tenant quotas
ALTER TENANT tenant_123 SET max_cpu_cu = 8.0;
ALTER TENANT tenant_123 SET qos_tier = 'gold';
-- View quota violations
SELECT * FROM heliosdb.quota_violations
WHERE timestamp > NOW() - INTERVAL '1 hour'
ORDER BY timestamp DESC;

Core Configuration

Server Settings

server:
host: 0.0.0.0 # Listen address
port: 5432 # PostgreSQL port
max_connections: 1000 # Maximum connections
max_connections_per_user: 100 # Per-user connection limit
max_connections_per_database: 500 # Per-database connection limit
connection_timeout: 30s # Connection timeout
idle_session_timeout: 3600s # Idle session timeout (1 hour)
statement_timeout: 0 # Statement timeout (0 = disabled)
lock_timeout: 0 # Lock timeout (0 = disabled)
work_mem: 4MB # Memory per operation
shared_buffers: 256MB # Shared buffer pool
max_worker_processes: 8 # Maximum worker processes

Storage Settings

storage:
data_dir: /var/lib/heliosdb/data
wal_dir: /var/lib/heliosdb/wal
temp_dir: /var/lib/heliosdb/temp
# LSM-Tree Settings
memtable_size: 64MB # Memtable size before flush
sstable_block_size: 4KB # SSTable block size
bloom_filter_bits_per_key: 10 # Bloom filter size
compaction_strategy: tiered # Options: leveled, tiered, universal
compaction_threads: 4 # Compaction threads
max_open_files: 1000 # Maximum open file handles
# WAL Settings
wal_segment_size: 16MB
wal_sync_method: fdatasync # Options: fsync, fdatasync, open_sync
wal_compression: lz4 # Options: none, lz4, zstd
checkpoint_interval: 5m # Checkpoint interval
max_wal_size: 1GB # Maximum WAL size before checkpoint

Protocol Configuration

PostgreSQL Protocol

protocols:
postgres:
enabled: true
port: 5432
simple_query_protocol: true
extended_query_protocol: true
binary_format: true # Binary wire format
scram_sha_256: true # SCRAM-SHA-256 auth
ssl_enabled: true
ssl_cert: /etc/heliosdb/certs/server.crt
ssl_key: /etc/heliosdb/certs/server.key
ssl_ca: /etc/heliosdb/certs/ca.crt

Oracle Protocol

oracle:
enabled: true
port: 1521
tns_protocol: true
plsql_engine: true
dbms_packages: all # Options: all, priority1, priority2, none
ref_cursors: true
oracle_types: true

MySQL Protocol

mysql:
enabled: true
port: 3306
client_protocol: true

HTTP Gateway

http:
enabled: true
port: 8443
ssl_enabled: true
ssl_cert: /etc/heliosdb/certs/server.crt
ssl_key: /etc/heliosdb/certs/server.key
auth_method: oauth2 # Options: oauth2, api_key, basic

Security Configuration

security:
# Row-Level Security
rls:
enabled: true
default_policy: deny # Options: deny, allow
# Data Masking
masking:
enabled: true
pii_auto_detection: true
default_algorithm: redact # Options: redact, hash, tokenize, etc.
# Audit Logging
audit:
enabled: true
log_path: /var/log/heliosdb/audit.log
log_format: json
tamper_proof: true # Blockchain-style hash chains
retention: 90d
# Encryption
encryption:
at_rest: true
algorithm: aes-256-gcm
key_rotation_interval: 90d
per_column_encryption: true
# FIPS 140-2 Compliance
fips_140_2:
enabled: false # Enable for government compliance
crypto_module: openssl-fips

Environment Variables

HeliosDB supports environment variable substitution in configuration files using ${VAR_NAME} syntax.

Common Environment Variables:

Terminal window
# AWS Credentials
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_REGION=us-east-1
# Database Credentials
export HELIOSDB_ADMIN_PASSWORD=secure_password_here
export HELIOSDB_REPLICATION_PASSWORD=replication_password_here
# Paths
export HELIOSDB_DATA_DIR=/var/lib/heliosdb/data
export HELIOSDB_WAL_DIR=/var/lib/heliosdb/wal
export HELIOSDB_LOG_DIR=/var/log/heliosdb
# Networking
export HELIOSDB_HOST=0.0.0.0
export HELIOSDB_PORT=5432
# Feature Flags
export HELIOSDB_ENABLE_BRANCHING=true
export HELIOSDB_ENABLE_AUTOSCALING=true
export HELIOSDB_ENABLE_TIERED_STORAGE=true

Configuration Validation

Validate configuration before starting:

Terminal window
heliosdb-server --config /etc/heliosdb/heliosdb.yaml --validate

Check configuration syntax:

Terminal window
yamllint /etc/heliosdb/heliosdb.yaml

View current configuration (running server):

SELECT * FROM heliosdb.configuration;

Configuration Reload

Reload configuration without restart (for some settings):

Terminal window
# Send SIGHUP to reload
kill -HUP $(cat /var/run/heliosdb.pid)
# Or use SQL
SELECT heliosdb.reload_configuration();

Note: Some settings (like max_connections, shared_buffers) require a restart.


Summary

Configuration Files:

  • Main: /etc/heliosdb/heliosdb.yaml (1,300+ lines)
  • Comprehensive coverage of all 71 features
  • Environment variable support
  • Validation tools included

Quick Start Configurations Provided:

  • Development (single node, minimal)
  • Production (multi-node, all features)
  • Use case specific (e-commerce, IoT, SaaS, data warehouse, ML)

All v4.0 Features Configurable:

📖 Back to Main README | 🏗 Architecture | 📋 Features | Performance