Skip to content

fix: inspect, confirm and monit on a Keel Core (keel#60, keel#61) - #64

Merged
marcos-mendez merged 2 commits into
mainfrom
fix/inspect-on-a-keel-core
Oct 2, 2026
Merged

marcos-mendez merged 2 commits into
mainfrom
fix/inspect-on-a-keel-core

Conversation

@marcos-mendez

@marcos-mendez marcos-mendez commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

The keel bugs in the maintainer's screenshots of the Core image (2026-10-02), plus keel#60 and item 1 of keel#61. keel 0.15.1.

Root causes

Screen Symptom Root cause Fix
047, 059 appliance: not inferred, header "unknown appliance" probe_appliance accepted only the turnkey- prefix; a Keel image writes keel-core-19.0-trixie-amd64 keel- read like turnkey-
048, 059, 045 inspect exits 13 on a fresh Core instance.fqdn was in REQUIRED, whose rule is "a hook prompts when unset"; no hook asks for it, apply skips /etc/hosts without it, and no appliance has one out of the box dropped from REQUIRED (still reported as not inferred)
058, 061 hub.api_key: skip with a Keel Cloud key saved probe_hub looked only for a TurnKey Hub registration the presence of /etc/keel/secrets/cloud_api_key gives {file: <secrets-dir>/cloud_api_key}, never read
034, 057 an eth1 the container does not have every image ships an allow-hotplug eth1 placeholder stanza, and inspect reported every stanza on the live root, a hotplug-only interface absent from /sys/class/net is left out with a report line; auto ones and offline roots unchanged
034, 058 updates_at_first_boot: force after Skip the first boot left no trace, so inspect used the update posture (the cron-apt install action every image ships) reads /var/lib/inithooks/sec-updates, which inithooks' 95secupdates now writes (Keel-Linux/inithooks#33, keel16)
040 "one not confirmed in time has been reverted" after another session confirmed with no marker, confirm assumed a revert each change's outcome is kept in /var/lib/keel/network/last.json (0600); already confirmed exits 0 and says so, reverted is called reverted
069, 073, 096 monit summary always fails keel never set monit's interface keel.conf sets set httpd unixsocket /run/monit.sock uid root gid root permission 0600 + allow localhost: root-only socket, no network listener (decision 0021). monit 5.34.3 supports it; checked with monit -t -v
keel#60 emitted spec loses the monitor channels inspect never read /etc/keel/monitor.json channels read back, token and URL files by reference; report names channels only; diff now also withholds observed channel values
keel#61 (1) comment-only online_api_credentials.yaml read as registered any non-blank text counted a file of comments only counts as absent

keel#61 item 2 (monit's first cycle) is fixed in keel-core (a monit drop-in ordered after Core's units); its PR links here.

Tests

Tests written first, then the code. 2308 pass, 8 skip; coverage 99% (same 12 lines missed as on main, all new code covered). The real-monit tests ran against monit 5.34.3 via KEEL_MONIT. New fixture tests/fixtures/inspect/keelcore is a fresh Core after first boot.

Real test

LXC container from debian-13-keel-core_19.0-3+step7b-20261001_amd64.tar.zst on the test VM, with keel 0.15.1 and the inithooks, confconsole and keel-core packages of their PRs (built before the renumbering to keel16 and keel11) installed before the first boot. The review fixes (record last, forget on arm) are covered by unit tests:

  • inspect: core 19.0 (trixie, amd64), exit 0, no eth1, updates_at_first_boot: skip, hub.api_key: {file: ...cloud_api_key}, monitor.notify read back;

  • emitted spec (with the overlay states inspect cannot infer added) re-applied: 0 changes;

  • overlay confirmed from lxc-attach, then from the console: "already confirmed ... it stays", exit 0;

  • monit summary works; /run/monit.sock is srw------- root root; no TCP listener;

  • container destroyed afterwards.

  • CI green

  • Review

Closes #60. Refs #61.

The bugs of the maintainer's screenshots of the Core image (2026-10-02):

- inspect reads keel-<name>-<version>-<codename>-<arch> in
  /etc/turnkey_version, as a Keel image writes it;
- instance.fqdn is no longer required: no hook asks for it, so inspect
  exited 13 on every new appliance;
- hub.api_key references the Keel Cloud key's file when the first boot
  stored one, found by its presence alone;
- an allow-hotplug interface the live machine does not have is left
  out (the image's eth1 placeholder);
- security.updates_at_first_boot is read from the line 95secupdates
  now leaves before the update posture, which said force after Skip;
- keel network confirm with nothing waiting says how the last change
  ended, from /var/lib/keel/network/last.json: already confirmed (exit
  0) or reverted;
- keel.conf sets monit's interface to a root-only Unix socket, so monit
  summary and status work, with no network listener.

keel#60: inspect reads the monitor's channels back from monitor.json,
tokens and URL files by reference; diff withholds observed channel
values too. keel#61 item 1: a comment-only CrowdSec
online_api_credentials.yaml counts as absent.

Closes #60
Review of #64. The outcome of a confirmation or a revert is recorded
after the marker is cleared and the timers disarmed, and a record that
cannot be written (a full disk) is reported as a note instead of
crashing: before, it could leave the marker and let the timer revert a
change the operator had confirmed. Arming a change removes the previous
record, so an older confirmation never answers for a change whose end
recorded nothing; a record that cannot be removed refuses the change
before anything moves.

The keelcore fixture's cloud_api_key is committed: a global *_key ignore
rule had left it out, and the repository's .gitignore now excepts it.
inithooks' record ships in 2.3.6+keel16, not keel15.
@marcos-mendez
marcos-mendez merged commit 1030245 into main Oct 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

inspect cannot read back monitor alert channels, so a re-emitted spec loses them

1 participant