Repository navigation
fix: upgrades keep the VIP; etcd restarts one member at a time - #46
Merged
Merged
Conversation
added 3 commits
October 8, 2026 06:38
0.1.1's dh_installsystemd --no-start made the preinst stop keel-vip.service and its check on every upgrade, and nothing started them again: the holder dropped the VIP and the replica had no controller, until a reboot. Built with --no-stop-on-upgrade now, the postinst try-restarts keel-vip.service where it runs, on an upgrade of the package and, through a file trigger on keel's Python files, of keel. keel 0.21.0 keeps the address over that restart. RestartSec=2.
A drop-in of etcd.service runs keel mesh etcd gate stop as its ExecStop and keel mesh etcd gate started as its ExecStartPost: etcd-server's restart on upgrade waits until every other member is healthy and none restarts, so apt on two members at once never loses the majority. tests/overlay-install.bats upgrades to version-bumped builds: a running VIP controller restarted, a stopped one left stopped, keel's upgrade restarting it through the trigger, etcd restarting through the gate. The install job builds keel at Keel-Linux/keel#87.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The packaging half of Keel-Linux/keel#87, for the maintainer's requirement of 2026-10-10: an
apt full-upgradeon a node whose VIP is active must neither drop the VIP nor trigger a failover. The design, and the measurements, are in keel#87.What an upgrade did before this PR
dh_installsystemd --no-enable --no-startgenerated a preinst that runsdeb-systemd-invoke stop keel-vip-check.timer keel-vip.serviceon every upgrade, and a postinst that never starts either unit again. On the holder, ExecStopPost dropped the VIP. On the replica, the controller was stopped as well, so nothing claimed the VIP. The VIP was down until a reboot orkeel spec apply.etcd.serviceon every upgrade, and nothing coordinates that restart across members. With apt running on two members of three at once, the cluster loses its majority while both restart.Changes
--no-stop-on-upgrade, so there is no preinst stop.deb-systemd-invoke try-restart keel-vip.serviceon an upgrade, and on a file trigger for/usr/lib/python3/dist-packages/keel, so an upgrade of keel restarts the helper and it runs the new code. A unit that was stopped is never started.RestartSec=2./usr/lib/systemd/system/etcd.service.d/keel-overlay-etcd.conf:ExecStop=+keel mesh etcd gate stopwaits until every other voter is healthy and no other member is restarting, then takes a restart lock held on an etcd lease.ExecStartPost=-+keel mesh etcd gate startedreleases the lock once this member is back in the majority.daemon-reload.tests/overlay-install.bats: real upgrades to builds of the same packages with a bumped version (with_version), with fab's policy-rc.d moved aside for those tests. The cases:Test plan
build / trixie: both packages build and pass lintian (clean locally).install / trixie: overlay-install.bats, including the four new upgrade tests.KEEL_COMMITto its merge commit.