EAI-7893 Stop persisting kube-proxy's live iptables rules - #294
Conversation
The "Save iptables" handler runs iptables-save, a whole-machine dump, and nothing flushed it, so it fired at the very end of the play with RKE2 already up. The file netfilter-persistent replays at boot was therefore a snapshot of kube-proxy's and Cilium's live state. That breaks in two directions. The replay runs about 23 seconds before RKE2, and kube-proxy's KUBE-FORWARD rule needs an nfacct object that only exists once kube-proxy has created it, so the restore fails with "RULE_APPEND failed (No such file or directory)" and the unit stays failed. Where it does not fail it is worse: it replays a stale copy of another controller's state before that controller starts. One int-test node had 110 KB of KUBE-SVC-*/KUBE-SEP-* chains naming pod IPs from two months earlier. Flushing handlers after prepare_node fixes both. Every notifier has run by then, the port rules and the REJECT removals, and the ruleset is still only ours. New nodes are enough for the timing, but already-bloomed ones keep the bad file: the handler only fires when a port rule changes, which on a prepared node it does not. So validate_node now discards a rules.v4 that contains KUBE, CILIUM or ts- chains and lets the flushed handler write a correct one. Detection is on those chain names rather than size or age, so a hand-written file that predates RKE2 is left alone. Verified against six nodes: the one file never overwritten by the handler is kept, and all five snapshots taken from running clusters are matched, including one whose unit has not failed yet because its snapshot predates kube-proxy's KUBE-FORWARD chain.
Deleting the file and leaving the handler to write the replacement has a gap: the handler only runs where flush_handlers is reached, so a targeted run such as --tags iptables would delete the file and never write anything back, leaving the node with no persisted rules at all. Writing in place means the file is never absent on any code path. Also leave the file alone when it holds ufw chains. ufw does not read rules.v4, it programs its own chains from /etc/ufw/*.rules, so a file containing both is one where two tools already disagree about who owns the ruleset, and bloom is not the place to settle that. The check now returns 0 rewrite, 1 leave alone, 2 ufw owns it, 3 no file yet, with the no-file arm explicit because grep on a missing path also exits 2 and would otherwise report a fresh node as a ufw node. Neither path can tighten a firewall. iptables-save records whatever policy the node already has, and it runs in validate_node before RKE2 starts, so what it captures is still the operator's own ruleset. Verified against six nodes and five synthetic files: a hand-written rules.v4 with :INPUT DROP is left alone, a ufw file is left alone, a missing file reports as missing, and the five snapshots taken from running clusters are rewritten.
Kaytoo VM verificationFresh Kaytoo VM Branch binary built from Detection matrix
Case 3 is the arm that needs the explicit no-file check to exist: Case 1 in detailThe file's md5 does change across a case-1 run, but not from anything in this Post-fix file restores
One negative I could not reproduce on this VM, stated plainly: the polluted Not coveredA full |
1 participant
Related:
Jira: https://amd.atlassian.net/browse/EAI-7893
Problem
Bloom's
Save iptableshandler runsiptables-save > /etc/iptables/rules.v4.Handlers fire at the end of the play, by which point RKE2 is up and kube-proxy
and Cilium have programmed the live ruleset.
iptables-saveis a whole-machinedump, so bloom writes another controller's runtime state into the file
netfilter-persistentreplays at boot.Two failures follow:
KUBE-FORWARDrule needs annfacctaccounting object that only exists oncekube-proxy has created it, so the restore fails with
RULE_APPEND failed (No such file or directory)andnetfilter-persistentis left permanently failed. This is what the fleet smoke test reports.
snapshot of cluster state. On int-test-003 the file had grown to 110 KB of
KUBE-SVC-*/KUBE-SEP-*chains naming pod IPs from two months earlier.Re-running bloom does not repair an affected node. The handler only fires when
a port rule actually changes, which on an already-prepared node it does not.
Fix
Two parts:
prepare_node, so the snapshot is takenwhile the ruleset is still only bloom's own. Every notifier has run by that
point: the port rules in
prepare_nodeand the REJECT removals invalidate_node.validate_node, detect arules.v4that already holds anothercontroller's chains and replace it in place with a clean
iptables-save.This repairs nodes bloomed by an earlier version.
Detection keys on
KUBE/CILIUM/ts-chains rather than on file size or age,so a hand-written
rules.v4predating RKE2 survives untouched. Filescontaining
ufw-chains are skipped entirely and only logged: ufw does notread
rules.v4, it programs its own chains from/etc/ufw/*.rules, so such afile is one where two tools already disagree about ownership, and bloom is not
the right place to settle that.
Safety for non-OCI users
The replacement is written in place rather than deleted and left to the
handler, because a targeted run such as
--tags iptableswould otherwisedelete the file and never reach
flush_handlersto write the replacement. Thefile is never absent, on any code path.
No option here can lock an operator out of their own machine.
iptables-restoreapplies policy lines before rules, and the capture recordswhatever policies the node already has, so this can only preserve the current
firewall posture, never tighten it. It runs in
validate_node, before RKE2starts, so the live ruleset it captures is still the operator's own.
Verification
All four detection arms exercised on a fresh Kaytoo VM (Ubuntu 24.04, OCI
image with its own 4489-byte
rules.v4, ufw not installed), running thebranch binary via
bloom cli --export.