TeraSky-OSS · vrabbi · Aug 16, 2026 · Aug 16, 2026 · Aug 16, 2026 · Aug 16, 2026
@@ -53,6 +53,28 @@ Each `ConversionWebhookServer` replica is symmetric and self-sufficient — ther
 - **Readiness** gates on both informer cache sync *and* a completed first reconcile pass over every currently-existing config, closing the classic "added to Service endpoints before the registry is populated" gap.
 - A registry miss (a `ConversionReview` for an XRD this replica has no compiled plan for) fails closed with a clear `503`, rather than guessing.
 
+## One cluster, one install
+
+Every operator replica and every `ConversionWebhookServer` replica is
+self-sufficient **inside a single Kubernetes cluster**. There is no leader
+election on the webhook-server path, no shared conversion state, and no
+network hop to another cluster on a `ConversionReview`. That is a design
+constraint, not a current gap.
+
+**Cross-cluster webhook failover is out of scope.** A webhook-server in
+cluster A must not serve conversions for a resource whose apiserver lives
+in cluster B. Each cluster runs its own fully independent install
+(chart + `ConversionWebhookServer` + configs). Fleet consistency is a
+**CI problem**, not an operator feature: run the same config through
+[`convctl test --live`](cli.md#pre-upgrade-checks-testing-against-everything-that-already-exists)
+and [`convctl diff --live`](cli.md#convctl-diff) against every kubecontext
+before merge — see [Fleet CI](gitops/fleet-ci.md).
+
+Asking the operator to fail over conversion state across clusters would
+need a redesign (shared registry, cross-cluster identity, a different
+availability model). Until that exists, do not point `spec.conversion`
+at a Service in another cluster.
+
 ## Safety by construction: the hazards this design closes
 
 | Scenario | Mitigation |

@@ -71,8 +71,10 @@ convctl test --crd crd.yaml --config crdconversionconfig.yaml --samples ./sample
 | `-c, --config` | Path to an `XRDConversionConfig` or `CRDConversionConfig` YAML file. **Required.** |
 | `-s, --samples` | Path to a directory of sample objects — one file per sample (or multi-doc YAML). Mutually exclusive with `--live`; exactly one of the two is required. |
 | `--live` | Fetch samples from a live cluster instead — see [Pre-upgrade checks](#pre-upgrade-checks-testing-against-everything-that-already-exists) below. |
-| `--kubeconfig` | Path to a kubeconfig file. Only used with `--live`. Falls back to `$KUBECONFIG`, then `~/.kube/config`, exactly like `kubectl`. |
-| `--context` | Kubeconfig context to use. Only used with `--live`. Falls back to the kubeconfig's `current-context`. |
+| `--kubeconfig` | Path to a kubeconfig file. Only used with `--live`. Falls back to `$KUBECONFIG`, then `~/.kube/config`, exactly like `kubectl`. Mutually exclusive with `--kubeconfig-dir`. |
+| `--context` | Kubeconfig context to use. Only used with `--live`. Falls back to the kubeconfig's `current-context`. Mutually exclusive with `--contexts`. |
+| `--contexts` | Repeatable / comma-separated kubeconfig context names. Runs `--live` once per context and aggregates the report. A single name keeps the one-cluster report shape. |
+| `--kubeconfig-dir` | Directory of kubeconfig files. `--live` runs against each file (that file's `current-context`, or each `--contexts` name). README / hidden files are skipped. |
 | `-o, --output` | `table` (default), `json`, or `junit` — see [Output formats](#output-formats) below. |
 | `--output-file` | Write the full report to this file instead of stdout. A short pass/loss/fail/error summary is still printed to stdout either way, so a CI log isn't empty on success. |
 | `--strict` | Escalate warnings (e.g. a rule that's never exercised by any sample) to failures. |
@@ -452,6 +454,12 @@ convctl test --crd crd.yaml --config new-crd-config.yaml --live \
 
 This is the tool to run before applying a new or changed `XRDConversionConfig`/`CRDConversionConfig`: does it hold up against every object that already exists in the cluster, not just your fixtures? `--kubeconfig`/`--context` resolve exactly like `kubectl` does. The invoking identity only needs `get`/`list` on the target resource type — no write access, and nothing related to this operator's own CRDs or webhook server.
 
+To run the same pair of checks (`diff --live` + `test --live`) against every
+cluster in a fleet before merge, see [Fleet CI](gitops/fleet-ci.md).
+`convctl test --xrd xrd.yaml --config proposed.yaml --live --contexts east,west -o junit`
+produces one JUnit document with a `<testsuite>` per cluster (a cluster
+that cannot be reached is an `<error>` suite, not a silent skip).
+
 ## Shell completion
 
 `convctl completion [bash|zsh|fish|powershell]` (built into Cobra) prints a completion script for your shell:
@@ -466,7 +474,7 @@ Once installed, flags complete as follows:
 
 - `--xrd` / `--crd` / `--config` / `--sample` on offline commands complete YAML files; `--samples` completes directories.
 - `migrate-storage --xrd` / `--crd` complete **cluster resource names** (XRDs and CRDs listed from the current kubeconfig context), not files. `--namespace` completes live namespaces the same way.
-- `--context` on `test`, `diff`, and `migrate-storage` completes kubeconfig context names. `--kubeconfig` already on the command line is honored, so `convctl migrate-storage --kubeconfig ./other --context <tab>` lists contexts from that file.
+- `--context` on `test`, `diff`, and `migrate-storage` completes kubeconfig context names. `--contexts` on `test` uses the same list. `--kubeconfig` already on the command line is honored, so `convctl migrate-storage --kubeconfig ./other --context <tab>` lists contexts from that file.
 - `--output` and `test --fail-on` complete their allowed values.
 
 Cluster lookups during completion time out after two seconds and fall back to no suggestions if the apiserver is unreachable, so a hung cluster cannot freeze tab-complete.
@@ -20,6 +20,10 @@ isolation, these show a complete, runnable config you can copy and adapt.
 | [`crossplane-xr-multiversion/`](https://github.com/terasky-oss/declarative-conversion-operator/tree/main/examples/crossplane-xr-multiversion) | Staged Crossplane XR lifecycle: one-version XRD + ConfigMap Composition, add a spoke, promote the hub (new Composition + retarget `compositionRef`), add `v3`, promote `v3` as the standard, deprecate `v1` (including `convctl migrate-storage` and dropping the version block). GitOps alternative: [`gitops/`](https://github.com/terasky-oss/declarative-conversion-operator/tree/main/examples/crossplane-xr-multiversion/gitops) + [`convctl generate kyverno`](../cli.md#convctl-generate-kyverno) (`--gitops-engine simulate\|flux\|argo`). | [`FieldRename`](../strategies/field-rename.md) — see the [lifecycle walkthrough](xr-lifecycle.md) |
 | [`native-crd/`](https://github.com/terasky-oss/declarative-conversion-operator/tree/main/examples/native-crd) | The same model against a plain Kubernetes CRD, with no Crossplane anywhere. | [`FieldRename`](../strategies/field-rename.md), [`Delete`](../strategies/delete.md) |
 
+Operator install via Flux or Argo (apply order and `driftPolicy`) is
+[`examples/gitops/`](https://github.com/terasky-oss/declarative-conversion-operator/tree/main/examples/gitops)
+— see [GitOps operator sync](../gitops/operator-sync.md).
+
 For a single fixture that exercises *every* built-in strategy at once, see the
 [kitchen-sink walkthrough](kitchen-sink.md).
 

@@ -68,10 +68,20 @@ Yes — `convctl test --live` fetches every existing instance of the target type
 
 ### Does this integrate with CI?
 
-`convctl test` supports `--output json` for scripting and `--output junit` for CI systems with JUnit test-result reporting (GitHub Actions, GitLab, Jenkins), plus `--output-file` to write the report to a specific path. Exit codes are deliberately distinct for "a test failed" (`1`) versus "the tool was used wrong" (`2`), so CI can tell the two apart.
+`convctl test` supports `--output json` for scripting and `--output junit` for CI systems with JUnit test-result reporting (GitHub Actions, GitLab, Jenkins), plus `--output-file` to write the report to a specific path. Exit codes are deliberately distinct for "a test failed" (`1`) versus "the tool was used wrong" (`2`), so CI can tell the two apart. For a fleet, `--live --contexts` / `--kubeconfig-dir` emit one aggregated JUnit document — see [Fleet CI](gitops/fleet-ci.md).
 
 ## Operations and scaling
 
+### Can one webhook-server serve conversions for resources in another cluster?
+
+No. Each cluster runs its own operator install. Cross-cluster webhook
+failover (cluster A's pods converting objects for cluster B's apiserver)
+is out of scope — the runtime has no shared state and no leader election
+to make that safe. Use the same config YAML in each cluster and
+`convctl test --live` / `diff --live` per kubecontext to keep them
+consistent ([Fleet CI](gitops/fleet-ci.md)). See
+[Architecture: One cluster, one install](architecture.md#one-cluster-one-install).
+
 ### Can I run more than one `ConversionWebhookServer`?
 
 Yes — it's designed for this. Each instance is cluster-scoped, deployable, and independently scalable (Deployment, Service, cert-manager Certificate, optional HPA/PodDisruptionBudget). The chart installs exactly one, marked `default: true`; you can create more for tenancy or scale-out and assign specific configs to them via `webhookServerRef`. Every replica of every instance is symmetric and self-sufficient — no leader election, no shared state — so horizontal scaling is just adding pods.

@@ -0,0 +1,69 @@
+# Reference GitHub Actions workflow — copy into a platform repo.
+# This file is not wired into this operator's own CI.
+#
+# Each matrix.context is one cluster. fail-fast is false so one red
+# cluster does not hide the rest. Upload the JUnit / diff JSON as
+# artifacts for a test reporter or PR review.
+name: convctl fleet
+
+on:
+  pull_request:
+    paths:
+      - "**/xrdconversionconfig.yaml"
+      - "**/crdconversionconfig.yaml"
+      - ".github/workflows/convctl-fleet.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  fleet:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        context: [prod-us, prod-eu]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install convctl
+        run: |
+          # Pin to a released convctl, or build from this repo:
+          # go install github.com/terasky-oss/declarative-conversion-operator/cmd/convctl@latest
+          echo "install convctl and place it on PATH" >&2
+          exit 1
+
+      - name: Write kubeconfig
+        env:
+          KUBECONFIG_B64: ${{ secrets.FLEET_KUBECONFIG }}
+        run: |
+          mkdir -p "${HOME}/.kube"
+          printf '%s' "${KUBECONFIG_B64}" | base64 -d > "${HOME}/.kube/config"
+          chmod 600 "${HOME}/.kube/config"
+
+      - name: convctl diff --live
+        # Exit 1 is a coverage/claim delta (review artifact). Exit 2 is a
+        # usage/cluster error and must fail the job.
+        run: |
+          mkdir -p fleet-out
+          set +e
+          convctl diff --config path/to/xrdconversionconfig.yaml --live \
+            --context "${{ matrix.context }}" -o json \
+            > "fleet-out/${{ matrix.context }}.diff.json"
+          rc=$?
+          if [ "${rc}" -eq 2 ]; then exit 2; fi
+          exit 0
+
+      - name: convctl test --live
+        if: ${{ always() }}
+        run: |
+          convctl test --config path/to/xrdconversionconfig.yaml \
+            --xrd path/to/xrd.yaml --live \
+            --context "${{ matrix.context }}" \
+            --output junit --output-file "fleet-out/${{ matrix.context }}.junit.xml"
+
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: convctl-${{ matrix.context }}
+          path: fleet-out/
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+# Fleet gate: convctl diff --live + test --live against every cluster.
+# Copy into a platform repo or run from a checkout of this operator.
+#
+# Required:
+#   CONVCTL_CONFIG   path to XRDConversionConfig / CRDConversionConfig
+#   CONVCTL_XRD or CONVCTL_CRD
+# Cluster list (exactly one):
+#   CONTEXTS         space-separated kubeconfig context names
+#   KUBECONFIGS      space-separated kubeconfig file paths
+#
+# Optional:
+#   CONVCTL          convctl binary (default: convctl on PATH)
+#   KUBECONFIG       kubeconfig used with CONTEXTS (kubectl default if unset)
+#   OUT_DIR          where per-cluster JUnit/JSON land (default: ./convctl-fleet-out)
+#   SKIP_DIFF        set to 1 to run only test --live
+#   FAIL_ON_DIFF     set to 1 to fail the gate on a coverage/claim delta (diff exit 1).
+#                    Default: treat exit 1 as a review artifact; fail only on exit 2.
+set -euo pipefail
+
+CONVCTL="${CONVCTL:-convctl}"
+OUT_DIR="${OUT_DIR:-./convctl-fleet-out}"
+SKIP_DIFF="${SKIP_DIFF:-0}"
+FAIL_ON_DIFF="${FAIL_ON_DIFF:-0}"
+
+die() { echo "error: $*" >&2; exit 2; }
+
+[[ -n "${CONVCTL_CONFIG:-}" ]] || die "CONVCTL_CONFIG is required"
+if [[ -n "${CONVCTL_XRD:-}" && -n "${CONVCTL_CRD:-}" ]]; then
+  die "set CONVCTL_XRD or CONVCTL_CRD, not both"
+fi
+if [[ -z "${CONVCTL_XRD:-}" && -z "${CONVCTL_CRD:-}" ]]; then
+  die "CONVCTL_XRD or CONVCTL_CRD is required"
+fi
+if [[ -n "${CONTEXTS:-}" && -n "${KUBECONFIGS:-}" ]]; then
+  die "set CONTEXTS or KUBECONFIGS, not both"
+fi
+if [[ -z "${CONTEXTS:-}" && -z "${KUBECONFIGS:-}" ]]; then
+  die "set CONTEXTS (context names) or KUBECONFIGS (kubeconfig paths)"
+fi
+command -v "${CONVCTL}" >/dev/null 2>&1 || die "${CONVCTL} not found on PATH"
+
+schema_flags=()
+if [[ -n "${CONVCTL_XRD:-}" ]]; then
+  schema_flags+=(--xrd "${CONVCTL_XRD}")
+else
+  schema_flags+=(--crd "${CONVCTL_CRD}")
+fi
+
+mkdir -p "${OUT_DIR}"
+failed=0
+
+run_one() {
+  local label="$1" kubeconfig="$2" context="$3"
+  local safe kube_flags=()
+  safe="$(printf '%s' "${label}" | tr -c 'A-Za-z0-9._-' '_')"
+  [[ -n "${kubeconfig}" ]] && kube_flags+=(--kubeconfig "${kubeconfig}")
+  [[ -n "${context}" ]] && kube_flags+=(--context "${context}")
+
+  echo "=== ${label} ==="
+  if [[ "${SKIP_DIFF}" != 1 ]]; then
+    local diff_rc=0
+    "${CONVCTL}" diff --config "${CONVCTL_CONFIG}" --live \
+      "${kube_flags[@]}" -o json \
+      > "${OUT_DIR}/${safe}.diff.json" || diff_rc=$?
+    if [[ "${diff_rc}" -eq 2 ]]; then
+      echo "diff --live usage/cluster error on ${label}" >&2
+      failed=1
+    elif [[ "${diff_rc}" -eq 1 && "${FAIL_ON_DIFF}" == 1 ]]; then
+      echo "diff --live reported a delta on ${label} (FAIL_ON_DIFF=1)" >&2
+      failed=1
+    elif [[ "${diff_rc}" -eq 1 ]]; then
+      echo "diff --live reported a delta on ${label} (review ${OUT_DIR}/${safe}.diff.json)" >&2
+    fi
+  fi
+  if ! "${CONVCTL}" test --config "${CONVCTL_CONFIG}" --live \
+    "${schema_flags[@]}" "${kube_flags[@]}" \
+    --output junit --output-file "${OUT_DIR}/${safe}.junit.xml" \
+    --quiet; then
+    echo "test --live failed on ${label}" >&2
+    failed=1
+  fi
+}
+
+if [[ -n "${CONTEXTS:-}" ]]; then
+  for ctx in ${CONTEXTS}; do
+    run_one "${ctx}" "${KUBECONFIG:-}" "${ctx}"
+  done
+else
+  for kc in ${KUBECONFIGS}; do
+    run_one "$(basename "${kc}")" "${kc}" ""
+  done
+fi
+
+if [[ "${failed}" -ne 0 ]]; then
+  echo "fleet gate failed; reports in ${OUT_DIR}" >&2
+  exit 1
+fi
+echo "fleet gate passed; reports in ${OUT_DIR}"
@@ -0,0 +1,105 @@
+# Fleet CI: `convctl test` and `diff` across kubecontexts
+
+Each cluster runs its own operator install
+([Architecture: One cluster, one install](../architecture.md#one-cluster-one-install)).
+The operator does not sync conversion state between clusters. Before you
+merge a config change, run the same two read-only checks against **every**
+cluster that will apply that YAML:
+
+1. `convctl diff --live` — what would applying this file change on that
+   cluster (coverage, rule claims, lossiness).
+2. `convctl test --live` — does the proposed mapping still hold up against
+   every live object of the target type.
+
+Neither command writes to the cluster. The invoking identity only needs
+`get`/`list` on the target XRD/CRD and its instances.
+
+## Built-in: `convctl test --live --contexts`
+
+Once you have more than one context in a single kubeconfig:
+
+```console
+convctl test --xrd xrd.yaml --config proposed.yaml --live \
+  --contexts kind-fleet-a,kind-fleet-b \
+  --output junit --output-file fleet.junit.xml
+```
+
+`--kubeconfig-dir ./clusters/` is the same idea when each cluster has its
+own kubeconfig file. One context or one file keeps the existing
+single-cluster report. A connection error on one cluster is recorded as a
+failed suite; the others still run.
+
+`convctl diff` stays one cluster per invocation (`--context`). The
+[shell loop](#shell-loop) still wraps both commands when you want
+`diff --live` in the same gate.
+
+## Shell loop
+
+[`convctl-fleet.sh`](convctl-fleet.sh) is a copy-pasteable wrapper. It
+walks `CONTEXTS` (space-separated kubeconfig context names), writes one
+JUnit file per cluster, and exits non-zero if any cluster failed.
+
+```console
+# Two kind clusters sharing ~/.kube/config:
+export CONTEXTS="kind-fleet-a kind-fleet-b"
+export CONVCTL_XRD=examples/field-rename/xrd.yaml
+export CONVCTL_CONFIG=examples/field-rename/xrdconversionconfig.yaml
+./docs/gitops/convctl-fleet.sh
+```
+
+`KUBECONFIG` / `--kubeconfig` resolve the same way `kubectl` does. To use
+one kubeconfig file per cluster instead of contexts, set `KUBECONFIGS` to
+a list of paths (the script uses each file's `current-context`).
+
+## GitHub Actions matrix
+
+[`convctl-fleet.gha.yml`](convctl-fleet.gha.yml) is a reference workflow,
+not a job this repository runs. Copy it into your platform repo and
+replace the `context` matrix with your fleet. Each matrix leg is one
+cluster; `actions/upload-artifact` collects the JUnit files so a
+test-reporter can show a per-cluster breakdown.
+
+```yaml
+strategy:
+  fail-fast: false
+  matrix:
+    context: [prod-us, prod-eu, staging]
+```
+
+`fail-fast: false` is required: a red cluster must not hide the others.
+
+## What "pass" means
+
+| Command | Exit 0 | Exit 1 | Exit 2 |
+|---|---|---|---|
+| `convctl diff --live` | Cluster config and the file are equivalent | Any coverage/claim/lossy delta | Usage or cluster error |
+| `convctl test --live` | Every path passed, or every loss was already `acknowledgeLossy` | Unacknowledged loss or conversion error | Usage or cluster error |
+
+A fleet gate should fail the PR if **any** cluster's `test --live`
+returns 1 or 2, or if `diff --live` returns 2 (usage / cannot reach the
+cluster). A `diff` exit 1 is a coverage/claim delta — the change you are
+about to roll out — so the reference script and workflow treat it as a
+review artifact by default. Set `FAIL_ON_DIFF=1` when you want that
+delta to fail the gate.
+
+## Two-cluster check
+
+Verify the loop against two kind clusters that share one kubeconfig:
+
+```console
+kind create cluster --name fleet-a
+kind create cluster --name fleet-b
+# install the operator + apply the field-rename XRD on both, then:
+CONTEXTS="kind-fleet-a kind-fleet-b" \
+  CONVCTL_XRD=examples/field-rename/xrd.yaml \
+  CONVCTL_CONFIG=examples/field-rename/xrdconversionconfig.yaml \
+  ./docs/gitops/convctl-fleet.sh
+```
+
+## Related
+
+- [CLI: pre-upgrade checks](../cli.md#pre-upgrade-checks-testing-against-everything-that-already-exists)
+- [CLI: `convctl diff`](../cli.md#convctl-diff)
+- [Upgrade runbook](../operations/upgrade-runbook.md)
+- XRD lifecycle GitOps demo (Flux/Argo + in-cluster `convctl test --live`):
+  [`examples/crossplane-xr-multiversion/gitops/`](https://github.com/terasky-oss/declarative-conversion-operator/tree/main/examples/crossplane-xr-multiversion/gitops)