redborder-alarm-engine

redborder-alarm-engine is a dedicated, high-concurrency Go microservice designed to evaluate redBorder system alarms at scale. It replaces the traditional Rails Delayed::Job polling architecture with a multi-node, parallel Goroutine worker engine querying Apache Druid.

---

Key Features

• Extreme Concurrency: Evaluates hundreds/thousands of alarms in parallel using a Goroutine worker pool with HTTP connection pooling to Apache Druid.
• Multi-Node Automatic Sharding: Operates in a distributed multi-node cluster using a Redis Node Ring (redborder:alarm_engine:node:<node_id>:heartbeat). Workload is dynamically partitioned via (alarm_id % total_nodes) == my_node_index.
• Self-Healing & Automatic Failover: If any node goes down, its Redis heartbeat key expires after 15 seconds, and active nodes seamlessly claim its workload.
• Zero Database Row-Locking: Interacts with redborder-webui exclusively via lightweight internal REST API endpoints (/api/v1/internal/alarms/due and /api/v1/internal/alarms/results).
• Flexible Configuration: Supports YAML/JSON configuration files, command-line flags (-config), and Environment Variables (12-Factor app pattern).

---

Architecture Overview

                                    ┌────────────────────────┐
                                    │    redborder-webui     │
                                    │   (Rails / Postgres)   │
                                    └──────────┬──▲──────────┘
               GET /api/v1/internal/alarms/due │  │ POST /api/v1/internal/alarms/results
                                               ▼  │
                         ┌────────────────────────┴────────────────────────┐
                         │                   Shared Redis                   │
                         │   - Node Heartbeats: node:<node_id>:heartbeat   │
                         └───────────────▲─────────────────▲───────────────┘
                                         │                 │
                        Heartbeat / Ring │                 │ Heartbeat / Ring
                                         │                 │
                  ┌──────────────────────┴┐               ┌┴──────────────────────┐
                  │ redborder-alarm-engine│               │ redborder-alarm-engine│
                  │        Node 1         │               │        Node 2         │
                  │  Worker Pool (Go)     │               │  Worker Pool (Go)     │
                  └───────────┬───────────┘               └───────────┬───────────┘
                              │ Query Druid                           │ Query Druid
                              ▼                                       ▼
                         ┌─────────────────────────────────────────────────┐
                         │               Apache Druid Broker               │
                         └─────────────────────────────────────────────────┘

---

Directory Structure

redborder-alarm-engine/
├── bin/                       # Compiled binary output directory
├── client/                    # WebUI REST API client
│   ├── webui.go
│   └── webui_test.go
├── cluster/                   # Multi-node Redis hash ring & heartbeats
│   ├── ring.go
│   └── ring_test.go
├── druid/                     # High-performance Apache Druid HTTP client
│   ├── client.go
│   └── client_test.go
├── engine/                    # Core alarm rule & condition evaluator
│   ├── evaluator.go
│   └── evaluator_test.go
├── config.go                  # Configuration loader (YAML, JSON, Env vars)
├── config.yml.example         # Example configuration file template
├── go.mod                     # Go module definitions
├── go.sum                     # Go module checksums
├── main.go                    # Main daemon entrypoint & signal handler
└── README.md                  # Engine documentation

---

Building and Testing

Prerequisites

• Go 1.22+ installed (go version)
• Access to Redis and redborder-webui internal API

Run Tests

go test -v ./...

Build Binary

make build

Build RPM Package

make rpm

This produces redborder-alarm-engine-1.0.0-1.el9.x86_64.rpm in the project root directory.

---

Configuration Reference

redborder-alarm-engine supports 3 configuration modes (in order of precedence):

1. Environment Variables (Highest precedence)
2. Config File (Passed via -config flag or default config.yml)
3. Hardcoded Defaults (Safe fallback)

Configuration Options

Parameter	Config File Key	Environment Variable	Default	Description
Redis URL	redis_url	REDIS_URL	localhost:6379	Redis server address for node heartbeats and sharding ring. Supports redis://:pass@host:port format.
Redis Password	redis_password	REDIS_PASSWORD	""	Optional authentication password for Redis server.
WebUI URL	webui_url	WEBUI_URL	http://localhost:3000	WebUI base URL for API interaction.
Druid URL	druid_url	DRUID_URL	http://localhost:8082	Apache Druid Broker REST endpoint.
Internal Token	internal_api_token	INTERNAL_API_TOKEN	secret-token	Shared secret header (X-Internal-Token) with Rails WebUI.
Poll Interval	poll_interval_sec	POLL_INTERVAL_SEC	10	Evaluation cycle tick interval (seconds).
Heartbeat	heartbeat_interval_sec	HEARTBEAT_INTERVAL_SEC	5	Node presence heartbeat interval in Redis (seconds).
Worker Limit	worker_limit	WORKER_LIMIT	50	Maximum parallel Goroutines evaluating alarms per cycle.
Verbose Mode	verbose	VERBOSE	false	Enable detailed evaluation logs per alarm (-v or -verbose flag).
Insecure TLS	insecure_skip_verify	INSECURE_SKIP_VERIFY	false	Disable TLS certificate verification for HTTPS (-insecure or -k flag).
Stall Protection	ingestion_stall_protection	INGESTION_STALL_PROTECTION	true	Suppress bottom-limit false alarms during Druid outages or segment reloads.
Missing Threshold	missing_data_threshold_pct	MISSING_DATA_THRESHOLD_PCT	25.0	Percentage threshold of missing data in a single cycle to trigger Druid restart suppression.
Recovery Cooldown	druid_recovery_cooldown_sec	DRUID_RECOVERY_COOLDOWN_SEC	180	Cooldown duration (seconds) after Druid recovers to healthy status before resuming normal alarm triggers.

---

Configuration Examples

Option A: Using config.yml

Create a config.yml file:

redis_url: "redis.cluster.local:6379"
redis_password: "your-redis-password"
webui_url: "http://webui.cluster.local:3000"
druid_url: "http://druid-broker.cluster.local:8082"
internal_api_token: "your-production-secret-token"
poll_interval_sec: 10
heartbeat_interval_sec: 5
worker_limit: 100
missing_data_threshold_pct: 25.0
druid_recovery_cooldown_sec: 180

Start the engine pointing to the configuration file:

./bin/redborder-alarm-engine -config config.yml

Option B: Using Environment Variables (Docker / Kubernetes)

export REDIS_URL="10.0.0.15:6379"
export REDIS_PASSWORD="your-redis-password"
export WEBUI_URL="http://10.0.0.20:3000"
export DRUID_URL="http://10.0.0.30:8082"
export INTERNAL_API_TOKEN="your-production-secret-token"
export WORKER_LIMIT="100"

./bin/redborder-alarm-engine

Option C: Mixed (Config File with Env Overrides)

WORKER_LIMIT=200 ./bin/redborder-alarm-engine -config config.yml

---

CLI Commands & IPC Status Socket

redborder-alarm-engine provides a built-in Unix socket / TCP IPC interface for querying running status and statistics:

CLI Commands

# Query daemon status and JSON metrics
./bin/redborder-alarm-engine status

# Ping daemon health
./bin/redborder-alarm-engine ping

# Reset evaluation counters
./bin/redborder-alarm-engine reset

IPC Statistics Payload Example

{
  "version": "1.0.0",
  "node_id": "node-1",
  "cluster_nodes": 2,
  "uptime_sec": 391.89,
  "start_time": "2026-08-04T12:19:47Z",
  "workers_in_use": 0,
  "peak_workers_in_use": 13,
  "worker_limit": 20,
  "total_evaluations": 1610,
  "total_fired": 904,
  "total_ok": 706,
  "total_errors": 0,
  "total_druid_queries": 1610,
  "last_evaluation_time": "2026-08-04T12:25:54Z",
  "last_cycle_duration_sec": 48.70,
  "last_cycle_time": "2026-08-04T12:25:54Z",
  "last_cycle_alarms": 230,
  "last_cycle_peak_workers": 1,
  "druid_healthy": true,
  "druid_status_message": "healthy"
}

---

Running in Multi-Node Mode

To scale horizontally across multiple nodes or containers:

1. Deploy redborder-alarm-engine on Node 1, Node 2, ..., Node N.
2. Ensure all nodes point to the same Shared Redis instance (REDIS_URL).
3. Each node registers its unique heartbeat in Redis.
4. Dependency Group Sharding: Connected alarm dependency graphs (precursor and blocked alarms) are grouped into single units via Union-Find and distributed evenly across cluster nodes using rank-based round-robin sharding (group_rank % N). Independent alarms without dependencies are balanced uniformly across all active nodes.

If any node fails, Redis automatically expires its heartbeat key within 15 seconds. Surviving nodes detect the updated cluster state and claim the missing node's share seamlessly.

---

Druid Health & Ingestion Stall Protection

redborder-alarm-engine includes a multi-layered protection mechanism to prevent false bottom-limit alarm triggers during Druid cluster restarts, segment loading, or ingestion stalls:

Detection Method

1. Native Node Readiness (GET /status/health): Ensures the local/remote Druid Broker endpoint is online and ready (HTTP 200 OK with true).

2. Segment Load Status (GET /druid/broker/v1/loadstatus): Monitors segment loading percentage across active telemetry datasources (rb_monitor, rb_flow, rb_vault, rb_ips, rb_ap_state, rb_wireless, rb_malware, rb_location). If segments are < 100.0% loaded (e.g. during a Druid service reboot), bottom-limit alarms are suppressed.

3. Overlord Supervisor & Indexing Task Status (GET /druid/indexer/v1/supervisor/<datasource>/status): Queries the Druid Overlord endpoint for supervisor health (healthy: true, state RUNNING) and verifies task status (/pendingTasks vs /runningTasks) matching redborder-webui's monitor controller.

4. Per-Cycle Missing Data Anomaly Suppression: In every evaluation cycle, the engine calculates the percentage of evaluated alarms returning missing data (null) from Druid.

   • Configuration Parameter: missing_data_threshold_pct (Default: 25.0%).
   • Trigger: If > 25% (or configured threshold) of alarms in a single cycle return null, the engine detects a cluster restart or segment loading state.
   • Behavior: All affected bottom-limit false alarms in that cycle are automatically marked SUPPRESSED (Druid cluster loading) and skipped from WebUI updates.

5. Recovery Cooldown Buffer (druid_recovery_cooldown_sec): When Druid transitions from unhealthy / restarting back to healthy, Druid's realtime workers require time to catch up on the backlog of events queued in Kafka / Logstash during the restart.

   • Configuration Parameter: druid_recovery_cooldown_sec (Default: 180 seconds).
   • Behavior: Bottom-limit false alarms remain SUPPRESSED for 180 seconds after Druid becomes healthy, preventing false alarms while ingestion queues catch up to real time.
   • Self-Healing: Once the 180s recovery buffer completes and realtime data has caught up, normal alarm evaluation resumes cleanly.

---

WebUI Requirements

The Rails WebUI must have the internal API endpoints configured:

• GET /api/v1/internal/alarms/due
• POST /api/v1/internal/alarms/results

Ensure INTERNAL_API_TOKEN matches the secret key configured in the Rails WebUI environment variables (ENV['INTERNAL_API_TOKEN']).