Skip to content

chore(talos): prod is pinned to a Talos line that has reached its Kubernetes ceiling #3599

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

Renovate has offered chore(deps): update dependency kubernetes/kubernetes to v1.37.0 (#3534) since
2026-09-02. It is green (16 success / 6 skipped / 0 failing), CLEAN, and has sat three days with
auto-merge never armed. It cannot be merged: it moves kubernetesVersion in ksail.prod.yaml to
v1.37.0 while leaving the Talos pin at version: v1.13.9.

The Talos v1.13 support matrix lists supported Kubernetes as 1.36, 1.35, 1.34, 1.33, 1.32, 1.31
1.37 is not in it. ksail.prod.yaml's own comment states the same constraint and requires the two to
move in lockstep. So prod's Kubernetes pin is currently held one minor behind by the OS pin beneath
it, and every future Kubernetes offer will stall the same way until the OS line moves.

Problem and who it affects

The cluster is pinned to an OS line that has reached its Kubernetes ceiling. This affects the operator
directly: Kubernetes updates — including any that carry security fixes — cannot land at all while the
pin is where it is, and the stall is silent. Nothing fails; a green PR simply never merges, which is
the hardest state to notice, and #3534 demonstrates it has already gone unnoticed for three days.

Expected behaviour

The Talos pin is on a line whose support matrix includes the Kubernetes version the repository intends
to run, so ordinary Kubernetes bumps are mergeable rather than structurally blocked.

The upgrade path is clean, and the sequencing matters

Talos v1.14 lists supported Kubernetes as 1.37, 1.36, 1.35, 1.34, 1.33. That range includes
the version prod runs today (1.36.4), which is what makes this safe to do in the order the
repository already mandates:

Upgrade Kubernetes one minor at a time, and never in the same deploy as a Talos OS upgrade. A single
'ksail cluster update' carrying both rolls one fleet-wide upgrade with two independent failure
modes, so land the OS bump first, let every node report the new Talos version, and only then move
this pin.

So the OS bump lands alone, with kubernetesVersion untouched at v1.36.4. Only once every node
reports the new Talos version does #3534 become mergeable, as its own separate deploy.

Smallest useful change

Move the Talos pin in ksail.prod.yaml from v1.13.9 to the v1.14 line, in a deploy that changes
nothing else. Note the pin is two coupled fields, not one — the file states the ISO must match the
version (iso: 125127, currently the v1.12.4 image), so the corresponding Hetzner ISO id for the
chosen v1.14 patch has to be resolved as part of this work rather than assumed.

Two known operational constraints already recorded in this repository apply to the rolling upgrade and
should not be rediscovered during it: the 30m per-node eviction budget exists because Longhorn pins a
zero-disruption PDB while an instance-manager still serves attached volumes, and #3388 records that
Talos node upgrades can exceed the drain timeout and get the bump evicted from the merge queue.

Success signal

Every node reports the v1.14 Talos version, prod is still on Kubernetes v1.36.4 and healthy, and
#3534 then merges on its own without further changes. Guardrail: no workload disruption beyond the
expected rolling drain.

Acceptance criteria

  • ksail.prod.yaml pins a Talos v1.14 patch with a matching ISO id, and kubernetesVersion is
    unchanged in that same change.
  • The rolling upgrade completes with every node reporting the new Talos version.
  • chore(deps): update dependency kubernetes/kubernetes to v1.37.0 #3534 is subsequently mergeable and merges as a separate deploy.
  • The support-matrix constraint comment in ksail.prod.yaml is updated to state the v1.14 range, so
    the next bump is checked against the line actually pinned.

Blocks #3534. Rough size: M — the change is small, the rolling upgrade and its drain behaviour are the
work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions