dfe-suite

Control plane for the HyperI Data Fusion Engine. GitOps at scale by design from ground up.

Loads Kafka and gRPC streams into ClickHouse, batching small writes into large inserts. Runs standalone or as part of the HyperI Data Fusion Engine.

Archives event streams to S3, GCS or Azure object storage, spooling to local disk when the sink is unreachable. Runs standalone or as part of the HyperI Data Fusion Engine.

Multi-cloud infrastructure deployment for the HyperI Data Fusion Engine (OpenTofu, Helm, Argo CD).

Transform pipelines, running Vector.dev as a managed subprocess. Runs standalone or as part of the HyperI Data Fusion Engine.

Per-environment deploy repo for the HyperI Data Fusion Engine, instantiated once per deployment.

Docker Compose deployment packaging for HyperI Data Fusion Engine

Web console for the HyperI Data Fusion Engine. Talks only to the dfe-engine control plane API, extended dfe-hyperdx for data discovery.

Shared schema definitions and tools for the HyperI Data Fusion Engine. Includes common header profiles.

Kafka-to-Kafka transform engine with VRL embedded in-process. Runs standalone or as part of the HyperI Data Fusion Engine.

Heavily Rust-optimised transform pipeline for Elastic Stack data (Beats + Elastic Agent). Runs standalone or as part of the HyperI Data Fusion Engine. Equivalent in JSON output to that of elastic beats/agent + server ingest painless scripting, only fast and CPU lean.

Fetch ingest (pull) for a wide selection of data sources and cloud providers such as AWS, Azure, M365, GCP . Lean and at scale by design from ground up. Runs standalone or as part of the HyperI Data Fusion Engine.

Receive ingest endpoint for a wide selection of data sources and protocols. Lean and at scale by design from ground up. Runs standalone or as part of the HyperI Data Fusion Engine.

DFE fork of HyperDX extended with DFE 2.1 Discovery, JSON capabilities and dfe-ui integration. rerere for added functionality, security fixes post-applied.