CDC Webhooks

CDC Webhooks

Status: Production-Ready (85% Complete) Last Updated: January 4, 2026

---

Overview

HeliosDB’s CDC-to-Webhook integration enables real-time event streaming from database changes (INSERT, UPDATE, DELETE) to external systems via HTTP webhooks. The system provides exactly-once delivery, circuit breaker protection, and comprehensive monitoring.

Key Capabilities

Capability	Description	Status
Event Capture	WAL-based Change Data Capture	Production
Event Delivery	HTTP webhooks with retry	Production
Delivery Guarantee	Exactly-once semantics	Production
Event Filtering	Table/operation/column filters	Production
Event Transformation	JSONPath, Jinja2 templates	Planned
Circuit Breaker	Automatic failure detection	Production
Monitoring	Prometheus metrics, dashboards	Production

Performance Characteristics

Metric	Value
Event Capture Rate	100K events/sec
Webhook Delivery Latency (P50)	5-10ms
Webhook Delivery Latency (P99)	30-50ms
Delivery Success Rate	99.9%
Exactly-Once Guarantee	100% (within 24hr window)
Max Concurrent Webhooks	10,000
Queue Capacity	Unlimited (disk-backed)

Architecture

                    CDC-to-Webhook Pipeline

  ┌─────────────┐     ┌─────────────────┐     ┌──────────────┐
  │  Database   │ --> │   WAL/CDC       │ --> │   Event      │
  │  Changes    │     │   Processor     │     │   Queue      │
  └─────────────┘     └─────────────────┘     └──────────────┘
                                                     │
                                                     v
  ┌─────────────┐     ┌─────────────────┐     ┌──────────────┐
  │  External   │ <-- │   Webhook       │ <-- │   Worker     │
  │  Endpoints  │     │   Delivery      │     │   Pool       │
  └─────────────┘     └─────────────────┘     └──────────────┘
        │                    │                      │
        v                    v                      v
  ┌─────────────────────────────────────────────────────────┐
  │              Monitoring & Observability                  │
  │  (Prometheus metrics, Grafana dashboards, Audit logs)   │
  └─────────────────────────────────────────────────────────┘

Feature Highlights

1. WAL-Based Event Capture

Capture database changes directly from the Write-Ahead Log for maximum performance and minimal overhead.

2. Exactly-Once Delivery

Idempotency keys ensure each event is delivered exactly once, even in the presence of retries and failures.

3. Circuit Breaker Protection

Automatic failure detection prevents cascading failures when webhook endpoints are unhealthy.

4. Flexible Filtering

Filter events by:

• Table name
• Operation type (INSERT, UPDATE, DELETE)
• Column changes
• Custom predicates

5. High Throughput

Process 100K+ events per second with low latency and high reliability.

Quick Example

SQL Interface

-- Create a CDC stream to Kafka
CREATE CHANGE DATA CAPTURE ON users
  TO KAFKA 'localhost:9092' TOPIC 'users-changes'
  FORMAT JSON;

-- Monitor the stream
SHOW CDC STREAMS;
SHOW CDC STREAM STATUS users_cdc_stream;

-- Pause during maintenance
ALTER CDC STREAM users_cdc_stream PAUSE;

-- Resume
ALTER CDC STREAM users_cdc_stream RESUME;

Rust API

use heliosdb_cdc::{CdcConfig, EventProcessor};
use heliosdb_webhooks::{WebhookServer, WebhookConfig};

#[tokio::main]
async fn main() -> Result<()> {
    // Create CDC processor
    let cdc_config = CdcConfig::default();
    let processor = EventProcessor::new(cdc_config).await?;

    // Create webhook server
    let webhook_config = WebhookConfig {
        bind_address: "0.0.0.0:8081".to_string(),
        max_retries: 3,
        timeout_secs: 30,
        ..Default::default()
    };
    let server = WebhookServer::new(webhook_config).await?;

    // Register webhook endpoints
    server.register("https://api.example.com/webhooks/db-changes").await?;

    // Start processing
    processor.start().await?;

    Ok(())
}

Use Cases

Use Case	Description
Real-time Analytics	Stream changes to analytics systems
Event-Driven Architecture	Trigger downstream services on data changes
Data Sync	Keep external systems in sync with database
Audit Logging	Capture all changes for compliance
Cache Invalidation	Invalidate caches when data changes
Search Indexing	Update search indexes in real-time

Related Documentation

• User Guide - Comprehensive CDC webhook guide
• Troubleshooting - Common issues and solutions
• Design Document - Technical architecture
• SQL CDC Commands - SQL interface

API Modules

Module	Description
heliosdb_cdc::EventProcessor	CDC event processor
heliosdb_webhooks::WebhookServer	Webhook delivery server
heliosdb_webhooks::CircuitBreaker	Circuit breaker manager
heliosdb_webhooks::ExactlyOnce	Exactly-once delivery

---

See Also: HeliosDB Feature Index