feat: Implement multi-engine provider support (Native + CNPG Hybrid)

[devantler]

🤖 Generated by the Agentic Engineer

Description

We need to design and implement a provider-based architecture for the data-product-controller. The controller should support provisioning high-performance native database engines for production workloads, while also offering a lightweight, cost-effective hybrid alternative using CloudNativePG (CNPG) with extensions for development or smaller environments.

Proposed Architecture

The controller's core DataProduct CRD should expose a schema that lets the user choose between type (storage model) and provider (the underlying infrastructure controller).

Supported Matrix:

• SQL: Managed via CNPG (Core).
• Document (NoSQL):
  • native: Percona Operator for MongoDB.
  • cnpg-hybrid: Core PostgreSQL utilizing JSONB column types.
• Graph:
  • native: Neo4j (via official operator/Helm) or ArangoDB deployment.
  • cnpg-hybrid: CNPG cluster utilizing a custom image with the Apache AGE extension pre-installed and loaded via shared_preload_libraries.

Example Target CRD Schema

apiVersion: data.devantler.tech/v1alpha1
kind: DataProduct
metadata:
  name: user-analytics-graph
spec:
  type: graph # options: sql, document, graph
  provider: native # options: native, cnpg-hybrid
  storage:
    size: 50Gi

Tasks

• Define the interface/structs representing the decoupled Provider.
• Implement the native driver for Document workloads (Percona MongoDB).
• Implement the native driver for Graph workloads (Neo4j / ArangoDB).
• Implement the cnpg-hybrid driver that injects extensions like age and configures custom images for Postgres.
• Add integration tests in local dev environment (KSail).

---

Notes added when filing

Everything above is the maintainer's specification, kept verbatim apart from one correction. The rest of this section is what filing it surfaced; it is context for whoever picks this up, not a change to the ask.

Correction

The example used apiVersion: devantler.tech/v1alpha1. This repository's group is data.devantler.tech/v1alpha1 (see the README contract and the installed CRD), so the example has been corrected. Flagging it because a copy-pasted manifest with the wrong group fails with a confusing "no matches for kind" rather than a schema error.

🔴 This overlaps #3, and the two propose different architectures

#3 — Provision new data sources through pluggable resource references — already owns this capability, and its third acceptance criterion is "provider-specific logic is isolated behind a versioned adapter contract with a reference Crossplane example." That is the same decoupled-provider interface this epic's first task asks for.

Where they diverge is the part worth deciding deliberately:

	#3 as written	This epic as written
Where engine logic lives	delegated to a provisioner (Crossplane reference example), "without baking cloud or database logic into this controller"	in-controller drivers per engine (Percona, Neo4j/ArangoDB, CNPG+AGE)
What the CR carries	a reference to a provisioner-owned resource + a Secret-backed connection contract	provisioning intent directly (type, provider, storage.size)

The README also currently states that "the custom resource is control-plane metadata" and that product data and credentials do not belong in the Kubernetes API. Putting storage.size in spec is a deliberate move away from that, which may well be the intent — but it should be an explicit decision rather than something discovered mid-implementation.

This is a maintainer call, and it gates the decomposition below. The children are written to be correct under either outcome where possible: the interface child is the one that must resolve it first. If the answer is "supersede #3", #3 should be closed with that reason recorded; if it is "implement #3's adapter contract and express engines as adapters", this epic becomes its concrete first consumers and the CRD example needs reworking.

Not yet established

Per the repo's issue conventions I would normally record evidence, target audience, and a success signal here. I have none for this and will not invent them. Worth adding before implementation starts:

• who is asking for Document and Graph workloads, and which of the two is needed first;
• whether cnpg-hybrid is aimed at cost, at operational simplicity, or at local-dev parity — that decides whether it ships before or after the native drivers;
• what "success" looks like (a workload actually running against a provisioned Graph store, presumably) so the epic can be closed on evidence.

The Neo4j-vs-ArangoDB choice is left open in the spec and is a real decision with different operator maturity and licensing implications; it is called out in that child rather than guessed at here.

Repo conventions this inherits

New non-trivial capability lands feature-flag-first, default-off, tested in both states, and the release flag gets its own removal issue — the same pattern the registry-ui feature already follows in this repository, and which #3 requires explicitly.

[devantler]

🤖 Generated by the Agentic Engineer

Draft #42 implements the existing #3 reference contract: an optional same-namespace provisioner reference, the versioned crossplane/v1 observer, connection publication ownership, and a default-off release flag. Its ADR and guide document external lifecycle ownership, scoped reads, and the limits of readiness evidence. #3 stays open for real provisioner acceptance, tracked with removal child #41.

This foundation does not implement the type/provider/storage matrix or settle the maintainer's engine-driver direction. #34 remains the decision and interface step for that work; #35–#38 retain their existing dependencies. #42 can support delegated engines if that direction is chosen, without making that choice on behalf of this epic.