data-product-controller

A cloud-native control plane for self-describing, composable data products.

data.devantler.tech/v1alpha1 describes a data capability through ownership, inputs, outputs, standard machine-readable contracts, and an optional independently deployed user interface. The controller resolves product-to-product dependencies and reports readiness. A read-only registry and reference UI discover products from Kubernetes without compiling product-specific code into the catalogue.

Foundation

The current foundation provides:

• a namespaced DataProduct CRD with guarded HTTPS interfaces and stable URI identity;
• composition through named output references, with dependency-aware readiness conditions;
• a default-off provisioned-sources feature that observes a provisioner-owned resource and its published connection Secret metadata;
• default-off connector Deployment observation, with full current-generation availability included in product and registry readiness;
• a portable JSON descriptor registry at /api/v1/products;
• a default-off registry-ui feature that renders product descriptors and embeds product UIs in a restricted sandbox;
• an independently deployed harbour-observations example with its own OpenAPI contract, query API, and UI;
• an opt-in, Secret-backed HTTPS JSON export connector with a read-only API, OpenAPI contract, probes, and metrics;
• a Helm chart containing CRDs, least-privilege RBAC, hardened workloads, services, and optional Gateway API routing.

Tagged releases publish the Helm chart plus a controller image and manifest artifact signed by the portfolio's trusted keyless release workflow. Platform deployments should pin the released chart and immutable image digest.

The chart publication job signs the digest returned by Helm and verifies its signature before succeeding. Chart signatures use the issuer https://token.actions.githubusercontent.com and the identity https://github.com/devantler-tech/data-product-controller/.github/workflows/publish-chart.yaml@refs/tags/<tag>. Verify a released chart using its published digest and tag:

cosign verify \
  --certificate-identity 'https://github.com/devantler-tech/data-product-controller/.github/workflows/publish-chart.yaml@refs/tags/<tag>' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  'ghcr.io/devantler-tech/charts/data-product-controller@sha256:<digest>'

Deployment policy must match this chart identity separately from the shared controller-image and manifest publisher. Platform configuration owns signature enforcement and rollout.

Provisioned sources use delegated provisioning: an external controller owns infrastructure and credentials. The versioned crossplane/v1 observer checks readiness and connection publication without creating resources or reading Secret values. See the provisioned-source guide for its contract, scoped access, enablement, and limitations.

Connector workloads remain independently owned. The deployment/v1 observer reads one named Deployment in the product's namespace and publishes ConnectorReady alongside aggregate readiness. See the connector-readiness guide for flag enablement, narrowly scoped RBAC, an authored product example, and rollout semantics.

The HTTP source guide describes the independently deployed reference connector, its credential and network boundaries, and its default-off release gate. It does not register a DataProduct or imply controller-observed connector health.

Engine-specific provisioning, additional source adapters, connector health integration, richer composition semantics, and data-space exchange remain roadmap work.

Data product contract

apiVersion: data.devantler.tech/v1alpha1
kind: DataProduct
metadata:
  name: harbour-observations
spec:
  id: https://data-products.example.com/products/harbour
  name: Harbour observations
  description: Queryable temperature and salinity observations.
  version: v1.0.0
  owner:
    name: data-platform-team
  outputs:
    - name: observations
      protocol: OpenAPI
      url: https://data-products.example.com/products/harbour/api/observations
      contractUrl: https://data-products.example.com/products/harbour/openapi.json
      mediaType: application/json
  ui:
    title: Explore harbour observations
    url: https://data-products.example.com/products/harbour/ui

Another product composes this output without copying its data:

spec:
  inputs:
    - name: harbour
      productRef:
        name: harbour-observations
        output: observations

The custom resource is control-plane metadata. Product data and credentials do not belong in the Kubernetes API. Credentials remain in Secrets consumed directly by provisioner or connector workloads.

Decentralized UI contract

The spec.ui.url page belongs to the data product, not the registry. A compatible catalogue may render it in a sandboxed iframe or link to it directly. The reference registry:

• loads only absolute HTTPS URLs supplied by the product descriptor;
• uses sandbox="allow-forms allow-scripts" without allow-same-origin;
• passes no bearer token, Secret, or Kubernetes identity;
• never imports product JavaScript into the catalogue document.

This keeps each product portable across the reference registry and third-party catalogue implementations. Cross-window capabilities and authentication are deliberately deferred until they have a versioned, least-privilege protocol.

Install

Install the chart with the registry UI explicitly enabled and an existing Gateway API listener. Replace <version> and <verified-image-digest> with the release's chart version and the controller image digest verified against the trusted publisher:

helm upgrade --install data-product-controller \
  oci://ghcr.io/devantler-tech/charts/data-product-controller \
  --version '<version>' \
  --namespace data-product-system \
  --create-namespace \
  --set-string image.digest='sha256:<verified-image-digest>' \
  --set registryUI.enabled=true \
  --set route.enabled=true \
  --set route.host=data-products.example.com

An empty image.tag selects the packaged chart's appVersion without its optional v prefix. An explicit tag overrides that default; image.digest takes precedence over either tag for all workloads. Pin a verified digest for production. The optional HTTP source workload requires a digest even outside production. When reusing saved Helm values, replace any old image override explicitly.

Updating the CRD is a manual step. Helm installs the chart's crds/ directory on first install only and never updates or removes it on helm upgrade, so a release that changes the DataProduct schema does not reach clusters that already have the chart. Apply the new CRD before upgrading:

helm show crds oci://ghcr.io/devantler-tech/charts/data-product-controller --version <version> \
  | kubectl apply -f -

v1alpha1 is deliberately small and expected to change, so plan for this on schema-changing releases.

The UI remains off when registryUI.enabled is omitted. For repository development:

go test ./...
go test -tags=browser ./internal/browser
go build ./...
sh scripts/chart.test.sh
sh scripts/release.test.sh
helm lint charts/data-product-controller

Regenerate deep-copy code, CRDs, and RBAC after editing API types or markers, then distribute the generated CRD to the chart and release artifact:

go run sigs.k8s.io/controller-tools/cmd/controller-gen@v0.21.0 \
  object rbac:roleName=manager-role crd paths=./... \
  output:crd:artifacts:config=config/crd/bases \
  output:rbac:artifacts:config=config/rbac
cp config/crd/bases/data.devantler.tech_dataproducts.yaml \
  charts/data-product-controller/crds/data.devantler.tech_dataproducts.yaml
cp config/crd/bases/data.devantler.tech_dataproducts.yaml deploy/data.devantler.tech_dataproducts.yaml

Design

ADR 0001 records why the Kubernetes resource stays a small control-plane profile and how products remain portable. ADR 0002 defines delegated source ownership and observation. ADR 0003 defines the reference connector's data-plane and credential boundaries.

The vocabulary is informed by the Open Data Mesh Data Product Descriptor Specification, W3C DCAT 3, OpenAPI, AsyncAPI, and the Eclipse Dataspace Protocol. This release does not claim full conformance with those standards.