Skip to content

fix: upgrades keep the VIP; etcd restarts one member at a time - #46

Merged
marcos-mendez merged 6 commits into
19.xfrom
feat/vip-etcd-rolling-upgrades
Oct 8, 2026
Merged

marcos-mendez merged 6 commits into
19.xfrom
feat/vip-etcd-rolling-upgrades

Conversation

@marcos-mendez

Copy link
Copy Markdown
Collaborator

The packaging half of Keel-Linux/keel#87, for the maintainer's requirement of 2026-10-10: an apt full-upgrade on a node whose VIP is active must neither drop the VIP nor trigger a failover. The design, and the measurements, are in keel#87.

What an upgrade did before this PR

  • keel-overlay-vip 0.1.1. dh_installsystemd --no-enable --no-start generated a preinst that runs deb-systemd-invoke stop keel-vip-check.timer keel-vip.service on every upgrade, and a postinst that never starts either unit again. On the holder, ExecStopPost dropped the VIP. On the replica, the controller was stopped as well, so nothing claimed the VIP. The VIP was down until a reboot or keel spec apply.
  • keel-overlay-etcd 0.2.0. No upgrade action of its own. etcd-server's postinst restarts etcd.service on every upgrade, and nothing coordinates that restart across members. With apt running on two members of three at once, the cluster loses its majority while both restart.

Changes

  • keel-overlay-vip 0.2.0
    • Built with --no-stop-on-upgrade, so there is no preinst stop.
    • The postinst runs deb-systemd-invoke try-restart keel-vip.service on an upgrade, and on a file trigger for /usr/lib/python3/dist-packages/keel, so an upgrade of keel restarts the helper and it runs the new code. A unit that was stopped is never started.
    • RestartSec=2.
    • Depends on keel (>= 0.21.0). That version keeps the address across the restart: the kernel bounds it with a lifetime, ExecStopPost keeps it while the unit restarts, and the next controller renews the same lease.
  • keel-overlay-etcd 0.3.0
    • Ships /usr/lib/systemd/system/etcd.service.d/keel-overlay-etcd.conf:
      • ExecStop=+keel mesh etcd gate stop waits until every other voter is healthy and no other member is restarting, then takes a restart lock held on an etcd lease.
      • ExecStartPost=-+keel mesh etcd gate started releases the lock once this member is back in the majority.
    • The stop is never refused for good: the gate gives up after 300 s, and at shutdown it does not wait.
    • A node with no etcd cluster passes the gate at once, and so does a member that does not answer (a crash).
    • The postinst and postrm run daemon-reload.
    • Depends on keel (>= 0.21.0).
  • tests/overlay-install.bats: real upgrades to builds of the same packages with a bumped version (with_version), with fab's policy-rc.d moved aside for those tests. The cases:
    • a running controller is try-restarted, with a new invocation, and its timer runs;
    • a stopped controller stays stopped;
    • an upgrade of keel restarts the controller through the trigger;
    • etcd restarts through the gate in well under the gate's wait.
  • The install job builds keel at the head of keel#87. Pin it to keel#87's merge commit once that PR is merged. keel#87 should merge first.

Test plan

  • build / trixie: both packages build and pass lintian (clean locally).
  • install / trixie: overlay-install.bats, including the four new upgrade tests.
  • After keel#87 merges, re-pin KEEL_COMMIT to its merge commit.

navigator added 3 commits October 8, 2026 06:38
0.1.1's dh_installsystemd --no-start made the preinst stop
keel-vip.service and its check on every upgrade, and nothing started
them again: the holder dropped the VIP and the replica had no
controller, until a reboot. Built with --no-stop-on-upgrade now, the
postinst try-restarts keel-vip.service where it runs, on an upgrade of
the package and, through a file trigger on keel's Python files, of keel.
keel 0.21.0 keeps the address over that restart. RestartSec=2.
A drop-in of etcd.service runs keel mesh etcd gate stop as its ExecStop
and keel mesh etcd gate started as its ExecStartPost: etcd-server's
restart on upgrade waits until every other member is healthy and none
restarts, so apt on two members at once never loses the majority.
tests/overlay-install.bats upgrades to version-bumped builds: a running
VIP controller restarted, a stopped one left stopped, keel's upgrade
restarting it through the trigger, etcd restarting through the gate.
The install job builds keel at Keel-Linux/keel#87.
@marcos-mendez
marcos-mendez merged commit 27beb65 into 19.x Oct 8, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant