Skip to content

Talos is pinned to v1.13.9, capping Kubernetes at v1.36 and making the v1.37.0 bump permanently unmergeable #3626

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

The Kubernetes pin cannot advance past v1.36.x while Talos stays on the v1.13 line, and the dependency PR that would advance it is now permanently unmergeable.

ksail.prod.yaml on main (read 2026-09-06):

kubernetesVersion: v1.36.4
talos:
  version: v1.13.9
  iso: 125127

#3534 (chore(deps): update dependency kubernetes/kubernetes to v1.37.0) changes exactly one line — kubernetesVersion: v1.36.4v1.37.0 — and the compatibility validator added in #3583 rejects it:

ksail.prod.yaml: Kubernetes v1.37.0 is not verified compatible with Talos v1.13.9:
version of Kubernetes 1.37.0 is too new to be used with Talos 1.13.9.
Choose compatible pins; complete any Talos upgrade separately before raising Kubernetes.

This matches the constraint ksail.prod.yaml already documents in its own comments: "The Talos v1.13 line supports Kubernetes 1.31-1.36, so bump this in lockstep with the Talos pin below."

Why this is costing something now

#3534's own PR checks are green, and it reports mergeStateStatus: CLEAN. Its head was built 2026-09-02T17:57Z; the validator landed 2026-09-04T19:35Z (#3583). The PR's checks therefore predate the guard and cannot see it — the guard runs unconditionally in the merge group, so the PR passes every pre-merge signal and is then evicted.

It has been enqueued and evicted twice on 2026-09-06 alone — 10:00:50Z and 13:54:32Z (the second was mine, before I had pulled the merge-group history). Each attempt costs a full merge-group CI cycle. Renovate keeps the PR open, and nothing on its surface indicates it cannot merge, so any dispatch reading the pentad alone will re-enqueue it again.

This is not a defect in the guard or in the talos: path filter — ksail.prod.yaml is correctly listed in that filter, and the guard is doing exactly what #3583 built it to do. The stale-checks interaction is a consequence of the PR predating the guard.

Expected behaviour

Talos moves to the v1.14 line first, and the Kubernetes pin then advances one minor at a time behind it, per the sequencing ksail.prod.yaml already documents ("Upgrade Kubernetes one minor at a time, and never in the same deploy as a Talos upgrade").

siderolabs/talos v1.14.0 went GA on 2026-09-03 (resolved from proxy.golang.org, a package registry — no repository was inspected), so the upstream side is available.

Scope and sequencing

This is deliberately two deploys, not one, and the ordering is load-bearing:

  1. Talos v1.13.9v1.14.x. This also needs a matching Hetzner ISO id: iso: 125127 is pinned alongside the version, and the file's comment records that the pin must match the ISO and that Hetzner removes old ISOs (the v1.11.2 ISO was withdrawn on 2026-03-18). A Talos ISO that Hetzner still publishes must be identified as part of this step. Note Talos node upgrades can exceed the 10-minute drain timeout, evicting the bump from the merge queue #3388 — Talos node upgrades can exceed the 10-minute drain timeout and get evicted from the queue — is a known hazard on this path.
  2. Kubernetes v1.36.4v1.37.0, one minor, in a separate deploy, once step 1 has landed and the validator accepts the pair.

Acceptance criteria

  • talos.version is on the v1.14 line with an iso id that Hetzner currently publishes, and the cluster is healthy after the upgrade
  • go run ./scripts/validate-talos-kubernetes-compatibility ksail.prod.yaml accepts the resulting pin pair
  • Kubernetes advances to v1.37.0 in a separate deploy, with the validator green in the merge group rather than only on the PR
  • chore(deps): update dependency kubernetes/kubernetes to v1.37.0 #3534 either merges or is closed as superseded by the sequenced upgrade

Size

Medium, and mostly risk rather than diff — both steps are small edits to ksail.prod.yaml, with the exposure concentrated in the live node upgrade in step 1.

Notes

Until step 1 lands, #3534 should not be re-enqueued; it is parked on this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions