HeliosDB v4.0.0 - Complete Configuration Reference
HeliosDB v4.0.0 - Complete Configuration Reference
Comprehensive Configuration Guide for All Features
This document provides complete configuration reference for HeliosDB v4.0.0, including all v4.0 breakthrough features and legacy configurations.
Table of Contents
- Configuration File Format
- Quick Start Configurations
- v4.0 Breakthrough Features
- Core Configuration
- Protocol Configuration
- Storage Configuration
- Security Configuration
- Advanced Configuration
- Environment Variables
Configuration File Format
HeliosDB v4.0.0 uses YAML format for configuration files (upgraded from TOML in v3.1).
Default Location: /etc/heliosdb/heliosdb.yaml
Override with CLI:
heliosdb-server --config /path/to/custom/heliosdb.yamlQuick Start Configurations
Development (Single Node, All Features Disabled)
# heliosdb.yaml - Development Configurationserver: host: 0.0.0.0 port: 5432 max_connections: 100
storage: data_dir: /var/lib/heliosdb/data wal_dir: /var/lib/heliosdb/wal compression: none
# All v4.0 features disabled for simplicitybranching: enabled: false
autoscaling: enabled: false
tiered_storage: enabled: falseProduction (Multi-Node, All v4.0 Features Enabled)
# heliosdb.yaml - Production Configurationserver: host: 0.0.0.0 port: 5432 max_connections: 1000 max_connections_per_user: 100
# Git-Style Branchingbranching: enabled: true max_branches_per_database: 100 default_parent: main auto_gc: true gc_interval: 24h branch_retention: 30d
# Scale-to-Zero & Autoscalingautoscaling: enabled: true min_cu: 0.5 # Minimum 0.5 CU (don't scale to zero in prod) max_cu: 16.0 # Maximum 16 CUs scale_to_zero_after: 0 # Disable scale-to-zero in prod (set to 0) resume_timeout: 300ms target_cpu: 70 target_queue_depth: 10 scale_up_threshold: 80 scale_down_threshold: 30 cooldown_period: 60s
# Query-from-Any-Nodedistributed_query: enabled: true metadata_cache_size: 1GB cache_ttl: 300s invalidation_batch_size: 1000
# Zero-Downtime Rebalancingrebalancing: enabled: true auto_rebalance: true strategy: by_disk_size # Options: by_shard_count, by_disk_size, by_tenant_id threshold: 0.20 # Trigger at 20% imbalance check_interval: 3600s # Check every hour max_concurrent_moves: 5 bandwidth_limit: 100MB/s
# Enhanced Columnar Compressioncompression: enabled: true default_algorithm: hcc_v2 # Options: none, hcc_v1, hcc_v2 compression_level: high # Options: low, medium, high, adaptive adaptive_selection: true
# Schema-Based Shardingschema_sharding: enabled: true default_distribution: distributed allow_non_distributed: true
# Distributed Foreign Keysforeign_keys: enabled: true cross_shard_validation: true reference_table_replication: true join_optimization: true
# 3-Tier Storagetiered_storage: enabled: true
hot_tier: path: /mnt/nvme/heliosdb max_size_gb: 1000 latency_target_ms: 1 cost_per_gb: 0.15
warm_tier: path: /mnt/ssd/heliosdb max_size_gb: 5000 latency_target_ms: 5 cost_per_gb: 0.04
cold_tier: type: s3 bucket: heliosdb-cold-tier-prod region: us-east-1 endpoint: https://s3.amazonaws.com access_key_id: ${AWS_ACCESS_KEY_ID} # Environment variable secret_access_key: ${AWS_SECRET_ACCESS_KEY} latency_target_ms: 50 cost_per_gb: 0.02
policies: hot_to_warm: 7d warm_to_cold: 30d
migration: max_concurrent: 5 max_bandwidth_mbps: 100 atomic_transitions: true enable_throttling: true
# Safekeeper Consensussafekeeper: enabled: true cluster_size: 3 quorum_size: 2
nodes: - id: sk-1 address: 10.0.1.10:5433 - id: sk-2 address: 10.0.1.11:5433 - id: sk-3 address: 10.0.1.12:5433
wal_retention: 7d sync_timeout_ms: 100 async_storage_flush: true
# Online Table Shardingonline_sharding: enabled: true max_concurrent_migrations: 3 migration_throughput_limit: 100000 # rows/sec cutover_timeout_ms: 100
# Multi-Tenant Quotasmulti_tenancy: enabled: true enforce_quotas: true default_tier: bronze
tiers: bronze: max_cpu_cu: 2.0 max_iops: 5000 priority: 1 price_multiplier: 1.0
silver: max_cpu_cu: 8.0 max_iops: 15000 priority: 5 price_multiplier: 1.5
gold: max_cpu_cu: 32.0 max_iops: 50000 priority: 10 price_multiplier: 2.5
enforcement: check_interval_ms: 100 grace_period_ms: 1000 throttle_on_violation: truev4.0 Breakthrough Features
1. Git-Style Database Branching
branching: enabled: true # Enable branching feature max_branches_per_database: 100 # Maximum branches per database default_parent: main # Default parent branch name auto_gc: true # Automatic garbage collection gc_interval: 24h # GC interval (hours, days) branch_retention: 30d # Retention period for deleted branches delta_ssttable_max_size: 1GB # Maximum delta SSTable size before compaction copy_on_write: true # Enable copy-on-write (always true)SQL API:
-- Create branchSELECT heliosdb.create_branch('feature-1');SELECT heliosdb.create_branch('debug-1', parent_timestamp => '2025-10-24 14:30:00');SELECT heliosdb.create_branch('rollback-1', parent_lsn => '0/1234ABCD');
-- List branchesSELECT * FROM heliosdb.branches;
-- Switch branchSELECT heliosdb.checkout_branch('feature-1');
-- Delete branchSELECT heliosdb.delete_branch('feature-1');2. Scale-to-Zero Serverless Compute
autoscaling: enabled: true min_cu: 0.0 # Minimum CUs (0.0 = scale to zero) max_cu: 4.0 # Maximum CUs scale_to_zero_after: 300s # Idle time before suspend (seconds) resume_timeout: 300ms # Maximum resume time suspend_checkpoint_timeout: 500ms # Checkpoint timeout during suspend persist_connection_state: true # Persist connection state state_storage_path: /var/lib/heliosdb/state billing_precision: 1s # Billing precision (1s = 1 second granularity)Activity Monitoring:
autoscaling: activity_monitoring: enabled: true check_interval_ms: 1000 # Check every second idle_threshold_queries: 0 # Consider idle if 0 queries idle_threshold_connections: 0 # Consider idle if 0 active connectionsSQL API:
-- Check compute statusSELECT * FROM heliosdb.compute_status;
-- Manual suspendSELECT heliosdb.suspend_compute();
-- Manual resumeSELECT heliosdb.resume_compute();3. Dynamic Autoscaling
autoscaling: # Already covered above, additional settings: target_cpu: 70 # Target CPU utilization (%) target_queue_depth: 10 # Target query queue depth target_memory: 80 # Target memory utilization (%) scale_up_threshold: 80 # Scale up at 80% CPU scale_down_threshold: 30 # Scale down at 30% CPU cooldown_period: 60s # Wait between scaling decisions min_scale_step: 0.5 # Minimum CUs to add/remove max_scale_step: 2.0 # Maximum CUs to add/remove in one step predictive_scaling: false # ML-based predictive scaling (future)SQL API:
-- View autoscaling eventsSELECT * FROM heliosdb.autoscale_eventsWHERE timestamp > NOW() - INTERVAL '1 hour'ORDER BY timestamp DESC;
-- View current resource usageSELECT current_cu, cpu_utilization, memory_utilization, query_queue_depth, active_connections, recommended_cuFROM heliosdb.autoscale_status;4. Query-from-Any-Node Architecture
distributed_query: enabled: true metadata_cache_size: 1GB # Metadata cache size per node cache_ttl: 300s # Cache TTL cache_invalidation_batch_size: 1000 # Batch size for invalidation messages cache_refresh_interval: 60s # Automatic refresh interval ddl_centralized: true # DDL always routed to metadata service query_routing_strategy: automatic # Options: automatic, coordinator_onlySQL API:
-- View metadata cache statusSELECT * FROM heliosdb.metadata_cache_status;
-- Force cache invalidationSELECT heliosdb.invalidate_metadata_cache();
-- View query routing statisticsSELECT * FROM heliosdb.query_routing_stats;5. Zero-Downtime Shard Rebalancing
rebalancing: enabled: true auto_rebalance: true # Enable automatic rebalancing strategy: by_disk_size # Options: by_shard_count, by_disk_size, by_tenant_id threshold: 0.20 # Trigger at 20% imbalance check_interval: 3600s # Check every hour max_concurrent_moves: 5 # Maximum concurrent shard moves bandwidth_limit: 100MB/s # Network bandwidth limit replication_lag_threshold: 1000 # Max WAL records lag before cutover cutover_lock_timeout_ms: 100 # Maximum lock time during cutover rollback_on_failure: true # Automatic rollback on failureSQL API:
-- Manual rebalancingSELECT heliosdb.rebalance_shards(strategy => 'by_disk_size');
-- View rebalancing statusSELECT * FROM heliosdb.rebalance_status;
-- View shard distributionSELECT * FROM heliosdb.shard_distribution;6. Enhanced Columnar Compression (HCC v2)
compression: enabled: true default_algorithm: hcc_v2 # Options: none, hcc_v1, hcc_v2 compression_level: high # Options: low, medium, high, adaptive adaptive_selection: true # Automatically choose best algorithm algorithms: dictionary_encoding: true delta_encoding: true run_length_encoding: true zstd: true lz4: true frame_of_reference: true bit_packing: true null_suppression: true zstd_level: 3 # ZSTD compression level (1-22) compression_threshold: 1KB # Minimum size to compressSQL API:
-- Create table with HCC v2CREATE TABLE events ( ...) WITH ( compression = 'hcc_v2', compression_level = 'high');
-- View compression statisticsSELECT * FROM heliosdb.compression_stats WHERE table_name = 'events';
-- Convert table to HCC v2ALTER TABLE legacy_events SET (compression = 'hcc_v2');7. Schema-Based Sharding
schema_sharding: enabled: true default_distribution: distributed # Options: distributed, non_distributed allow_non_distributed: true # Allow non-distributed schemas schema_placement_strategy: automatic # Options: automatic, manual migration_strategy: logical_replication migration_bandwidth_limit: 100MB/sSQL API:
-- Create distributed schemaCREATE SCHEMA tenant_1234 DISTRIBUTED;
-- Create non-distributed schemaCREATE SCHEMA analytics NON_DISTRIBUTED;
-- View schema distributionSELECT * FROM heliosdb.schema_distribution;
-- Migrate schema to different nodeSELECT heliosdb.migrate_schema( schema_name => 'tenant_5678', target_node => 'node-3', strategy => 'logical_replication');8. Distributed Foreign Keys
foreign_keys: enabled: true cross_shard_validation: true # Enable cross-shard FK validation reference_table_replication: true # Replicate reference tables join_optimization: true # Enable join optimization co_located_fk_optimization: true # Optimize co-located FKs fk_cache_enabled: true # Cache FK validation results fk_cache_ttl: 60s # FK cache TTLSQL API:
-- Create co-located foreign keyCREATE TABLE orders ( tenant_id INT, user_id INT, FOREIGN KEY (tenant_id, user_id) REFERENCES users(tenant_id, user_id)) SHARD BY (tenant_id);
-- Create reference table (replicated)CREATE TABLE countries ( country_code CHAR(2) PRIMARY KEY, country_name TEXT) REPLICATED;
-- View FK validation statisticsSELECT * FROM heliosdb.fk_validation_stats;
-- View join optimization statisticsSELECT * FROM heliosdb.join_optimization_stats;9. 3-Tier Storage (Hot/Warm/Cold)
tiered_storage: enabled: true
hot_tier: path: /mnt/nvme/heliosdb max_size_gb: 1000 latency_target_ms: 1 cost_per_gb: 0.15 iops_limit: 100000
warm_tier: path: /mnt/ssd/heliosdb max_size_gb: 5000 latency_target_ms: 5 cost_per_gb: 0.04 iops_limit: 50000
cold_tier: type: s3 # Options: s3, minio, ceph, gcs, azure bucket: heliosdb-cold-tier region: us-east-1 endpoint: https://s3.amazonaws.com access_key_id: ${AWS_ACCESS_KEY_ID} secret_access_key: ${AWS_SECRET_ACCESS_KEY} latency_target_ms: 50 cost_per_gb: 0.02 multipart_threshold: 5MB multipart_chunk_size: 100MB compression: zstd # Compress before upload encryption: aes256 # Encrypt before upload
policies: hot_to_warm: 7d # Move to warm after 7 days warm_to_cold: 30d # Move to cold after 30 days access_based_tiering: true # Override age-based if frequently accessed min_access_frequency: 10 # Keep in hot if >10 accesses/day
migration: max_concurrent: 5 max_bandwidth_mbps: 100 atomic_transitions: true enable_throttling: true retry_attempts: 3 retry_delay_ms: 1000SQL API:
-- View tier usageSELECT * FROM heliosdb.storage_tier_usage;
-- Manual tier migrationSELECT heliosdb.move_to_tier( object_key => 'table:events:sst-12345', target_tier => 'cold');
-- View tiering policiesSELECT * FROM heliosdb.tiering_policies;
-- Generate cost reportSELECT * FROM heliosdb.storage_cost_analysis();10. Safekeeper Consensus Layer
safekeeper: enabled: true cluster_size: 3 # Number of Safekeeper nodes quorum_size: 2 # Quorum size (2/3)
nodes: - id: sk-1 address: 10.0.1.10:5433 priority: 10 # Higher priority = preferred leader - id: sk-2 address: 10.0.1.11:5433 priority: 5 - id: sk-3 address: 10.0.1.12:5433 priority: 1
wal_retention: 7d # WAL retention period wal_segment_size: 16MB # WAL segment size sync_timeout_ms: 100 # Synchronous replication timeout async_storage_flush: true # Asynchronous storage flush compression: lz4 # WAL compression checksum: true # WAL checksumsSQL API:
-- View safekeeper cluster statusSELECT * FROM heliosdb.safekeeper_cluster_status;
-- View WAL replication metricsSELECT * FROM heliosdb.wal_replication_stats;
-- Force safekeeper failover (testing only)SELECT heliosdb.trigger_safekeeper_failover();11. Online Table Sharding Migration
online_sharding: enabled: true max_concurrent_migrations: 3 # Maximum concurrent table migrations migration_throughput_limit: 100000 # Rows/sec bulk_copy_batch_size: 10000 # Rows per batch replication_lag_threshold: 1000 # Max lag before cutover cutover_timeout_ms: 100 # Maximum cutover lock time rollback_on_failure: true # Automatic rollback progress_reporting_interval: 1000 # Report progress every 1000 rowsSQL API:
-- Shard table onlineSELECT heliosdb.shard_table_online( table_name => 'events', shard_key => 'user_id', shard_count => 16);
-- Monitor sharding progressSELECT * FROM heliosdb.online_sharding_status WHERE table_name = 'events';
-- Rollback shardingSELECT heliosdb.rollback_online_sharding('events');12. Multi-Tenant Resource Quotas
multi_tenancy: enabled: true enforce_quotas: true default_tier: bronze
tiers: bronze: max_cpu_cu: 2.0 max_memory_gb: 8 max_storage_gb: 100 max_iops: 5000 max_connections: 50 priority: 1 price_multiplier: 1.0 burst_allowance: 0.1 # 10% burst for 1 minute burst_duration: 60s
silver: max_cpu_cu: 8.0 max_memory_gb: 32 max_storage_gb: 1000 max_iops: 15000 max_connections: 200 priority: 5 price_multiplier: 1.5 burst_allowance: 0.2 # 20% burst for 5 minutes burst_duration: 300s
gold: max_cpu_cu: 32.0 max_memory_gb: 128 max_storage_gb: 10000 max_iops: 50000 max_connections: 1000 priority: 10 price_multiplier: 2.5 burst_allowance: 0.5 # 50% burst for 15 minutes burst_duration: 900s
enforcement: check_interval_ms: 100 grace_period_ms: 1000 throttle_on_violation: true reject_on_hard_limit: true notification_enabled: true notification_threshold: 0.9 # Notify at 90% quotaSQL API:
-- Create tenant with quotasCREATE TENANT tenant_123 WITH ( max_connections => 100, max_cpu_cu => 4.0, max_storage_gb => 1000, max_iops => 10000, qos_tier => 'gold');
-- View tenant usageSELECT * FROM heliosdb.tenant_usage WHERE tenant_id = 'tenant_123';
-- Modify tenant quotasALTER TENANT tenant_123 SET max_cpu_cu = 8.0;ALTER TENANT tenant_123 SET qos_tier = 'gold';
-- View quota violationsSELECT * FROM heliosdb.quota_violationsWHERE timestamp > NOW() - INTERVAL '1 hour'ORDER BY timestamp DESC;Core Configuration
Server Settings
server: host: 0.0.0.0 # Listen address port: 5432 # PostgreSQL port max_connections: 1000 # Maximum connections max_connections_per_user: 100 # Per-user connection limit max_connections_per_database: 500 # Per-database connection limit connection_timeout: 30s # Connection timeout idle_session_timeout: 3600s # Idle session timeout (1 hour) statement_timeout: 0 # Statement timeout (0 = disabled) lock_timeout: 0 # Lock timeout (0 = disabled) work_mem: 4MB # Memory per operation shared_buffers: 256MB # Shared buffer pool max_worker_processes: 8 # Maximum worker processesStorage Settings
storage: data_dir: /var/lib/heliosdb/data wal_dir: /var/lib/heliosdb/wal temp_dir: /var/lib/heliosdb/temp
# LSM-Tree Settings memtable_size: 64MB # Memtable size before flush sstable_block_size: 4KB # SSTable block size bloom_filter_bits_per_key: 10 # Bloom filter size compaction_strategy: tiered # Options: leveled, tiered, universal compaction_threads: 4 # Compaction threads max_open_files: 1000 # Maximum open file handles
# WAL Settings wal_segment_size: 16MB wal_sync_method: fdatasync # Options: fsync, fdatasync, open_sync wal_compression: lz4 # Options: none, lz4, zstd checkpoint_interval: 5m # Checkpoint interval max_wal_size: 1GB # Maximum WAL size before checkpointProtocol Configuration
PostgreSQL Protocol
protocols: postgres: enabled: true port: 5432 simple_query_protocol: true extended_query_protocol: true binary_format: true # Binary wire format scram_sha_256: true # SCRAM-SHA-256 auth ssl_enabled: true ssl_cert: /etc/heliosdb/certs/server.crt ssl_key: /etc/heliosdb/certs/server.key ssl_ca: /etc/heliosdb/certs/ca.crtOracle Protocol
oracle: enabled: true port: 1521 tns_protocol: true plsql_engine: true dbms_packages: all # Options: all, priority1, priority2, none ref_cursors: true oracle_types: trueMySQL Protocol
mysql: enabled: true port: 3306 client_protocol: trueHTTP Gateway
http: enabled: true port: 8443 ssl_enabled: true ssl_cert: /etc/heliosdb/certs/server.crt ssl_key: /etc/heliosdb/certs/server.key auth_method: oauth2 # Options: oauth2, api_key, basicSecurity Configuration
security: # Row-Level Security rls: enabled: true default_policy: deny # Options: deny, allow
# Data Masking masking: enabled: true pii_auto_detection: true default_algorithm: redact # Options: redact, hash, tokenize, etc.
# Audit Logging audit: enabled: true log_path: /var/log/heliosdb/audit.log log_format: json tamper_proof: true # Blockchain-style hash chains retention: 90d
# Encryption encryption: at_rest: true algorithm: aes-256-gcm key_rotation_interval: 90d per_column_encryption: true
# FIPS 140-2 Compliance fips_140_2: enabled: false # Enable for government compliance crypto_module: openssl-fipsEnvironment Variables
HeliosDB supports environment variable substitution in configuration files using ${VAR_NAME} syntax.
Common Environment Variables:
# AWS Credentialsexport AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLEexport AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYexport AWS_REGION=us-east-1
# Database Credentialsexport HELIOSDB_ADMIN_PASSWORD=secure_password_hereexport HELIOSDB_REPLICATION_PASSWORD=replication_password_here
# Pathsexport HELIOSDB_DATA_DIR=/var/lib/heliosdb/dataexport HELIOSDB_WAL_DIR=/var/lib/heliosdb/walexport HELIOSDB_LOG_DIR=/var/log/heliosdb
# Networkingexport HELIOSDB_HOST=0.0.0.0export HELIOSDB_PORT=5432
# Feature Flagsexport HELIOSDB_ENABLE_BRANCHING=trueexport HELIOSDB_ENABLE_AUTOSCALING=trueexport HELIOSDB_ENABLE_TIERED_STORAGE=trueConfiguration Validation
Validate configuration before starting:
heliosdb-server --config /etc/heliosdb/heliosdb.yaml --validateCheck configuration syntax:
yamllint /etc/heliosdb/heliosdb.yamlView current configuration (running server):
SELECT * FROM heliosdb.configuration;Configuration Reload
Reload configuration without restart (for some settings):
# Send SIGHUP to reloadkill -HUP $(cat /var/run/heliosdb.pid)
# Or use SQLSELECT heliosdb.reload_configuration();Note: Some settings (like max_connections, shared_buffers) require a restart.
Summary
Configuration Files:
- Main:
/etc/heliosdb/heliosdb.yaml(1,300+ lines) - Comprehensive coverage of all 71 features
- Environment variable support
- Validation tools included
Quick Start Configurations Provided:
- Development (single node, minimal)
- Production (multi-node, all features)
- Use case specific (e-commerce, IoT, SaaS, data warehouse, ML)
All v4.0 Features Configurable:
📖 Back to Main README | 🏗 Architecture | 📋 Features | Performance