A Kubernetes pin above the Talos ceiling passes every PR check and is caught only by a failed prod deploy

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

ksail.prod.yaml pins kubernetesVersion: v1.36.4 alongside Talos version: v1.13.9. Renovate
opened #3534 to move that pin to v1.37.0, which exceeds what Talos 1.13.9 supports.

That PR passed 27/27 checks and reported mergeStateStatus: CLEAN — including
🧪 Validate Talos Machine Config. It was enqueued at 18:16:26Z and evicted only when the
merge_group run 33666191785
reached 🚀 Deploy to Prod and Talos rejected the config with InvalidArgument:

✗ 7 changes failed to apply:
* kubelet image is not valid: version of Kubernetes 1.37.0 is too new to be used with Talos 1.13.9
* kube-apiserver image is not valid: ... too new to be used with Talos 1.13.9
* kube-controller-manager image is not valid: ... too new to be used with Talos 1.13.9
* kube-scheduler image is not valid: ... too new to be used with Talos 1.13.9

Talos refused the whole config, so nothing was partially applied.

Problem, and who it affects

The first and only thing that catches an out-of-range Kubernetes pin is a live production deploy.

.github/renovate.json already names this hazard in two places — "a Kubernetes minor/major can exceed
the pinned Talos version's supported Kubernetes ceiling … which would break prod bring-up" — and
correctly disables automerge for it. But disabling automerge only removes the automatic path. The
PR still presents to every human and every agent lane as green, CLEAN, and ready, because the prod
pin is never validated before merge. The renovate config says as much: "CI never validates this prod
pin".

Two concrete costs:

1. Every merge attempt is a failed production deploy, not a red check. Re-queuing costs a real
   apply against the cluster.
2. The standard merge preflight is actively misleading here. Any reviewer or agent applying
   "non-draft + CLEAN + green checks + green review ⇒ merge" reaches the wrong answer. The only
   contrary signal is buried in merge_group history.

Hypothesis

Comparing the pinned kubernetesVersion against the pinned Talos version's supported Kubernetes range
at PR time turns this from a failed prod deploy into an ordinary red check with an actionable
message — moving the guard off production and onto CI, at effectively zero runtime cost.

Smallest useful change

A validation step that reads both pins out of ksail.prod.yaml and fails when kubernetesVersion
falls outside the supported range for the pinned Talos version, naming both versions and the required
lockstep Talos upgrade. Talos publishes its supported Kubernetes range per release, and the
apply-time error message shows the check it already performs — so the pairing is derivable rather
than a table to hand-maintain.

The check belongs beside the existing 🧪 Validate Talos Machine Config job so it runs on any PR
touching that file.

Acceptance criteria

• A PR raising kubernetesVersion above the pinned Talos version's ceiling fails a required
  check at PR time, before the merge queue.
• The failure message names the current Talos pin, the requested Kubernetes version, and that a
  lockstep Talos upgrade is required — per the fail-with-the-fix convention.
• An in-range bump still passes (negative control — the check must not simply always fail).
• RED/GREEN proof: the check fails against chore(deps): update dependency kubernetes/kubernetes to v1.37.0 #3534's v1.37.0 + Talos v1.13.9 pairing and passes
  against the current v1.36.4 + v1.13.9 pairing.
• The supported range is derived, not hard-coded, or its staleness is itself guarded.

Size

Small–medium. One validation script plus its test and CI wiring, in the shape of the existing
scripts/ + scripts/tests/ validators.

Note on scope

This does not propose automerging the pin — that stays off for the reasons the renovate config
gives. It only moves the detection of an out-of-range bump from production to CI.

Blocks nothing; #3534 stays parked on its Talos blocker independently of this.