Add a supervisor-readiness audit to the hourly balena fleet job

[vpetersson-bot]

Why

The hourly fleet job already unpins, HUPs and bumps supervisors, but it reports only a pinned-release histogram. There is currently no way to answer "how much of the fleet is on a supervisor new enough to update itself?" without ad-hoc API queries.

That number has become load-bearing. balena supervisor v19 (2026-07-20) is the first release whose core-next service retrieves a queued host OS update on its own. Every older supervisor depends on balenaCloud pushing the update over Cloudlink via the device-actions service (https://actions.<deviceUrlsBase>/v2/<uuid>/resinhup, which this script already calls) — and device-actions is a balenaCloud-only component with no self-hosted equivalent.

The consequence: a device still below supervisor 19 cannot be host-updated at all once it is pointed at a self-hosted backend, and there is no way to recover it remotely afterwards. The uplift can therefore only be driven while balenaCloud is still in the picture, which makes "what fraction of the fleet is already there, and how fast is it moving?" the number that paces any such plan.

This PR adds the measurement. It does not change what the job does to devices.

What

A read-only --audit phase that reports, per fleet:

• balenaOS version histogram
• supervisor version histogram
• count and share of devices already at supervisor major >= 19

plus a fleet-wide rollup line at the end. It is wired into the hourly workflow on both the dry-run and apply branches, so the burndown shows up every run.

Sample output (synthetic data, real shape):

# screenly_ose/anthias-pi4: devices=3520 online=2546 self-update-capable=140 (4.0%)
  by balenaOS version:
       196 x 6.1.7+rev4
       193 x 6.1.1+rev4
       ...
    ... and 8 more balenaOS version(s)
  by supervisor version:
      1800 x 16.5.0
       900 x 17.0.9
       600 x 15.4.0
       140 x 19.1.4
        60 x 18.0.3
        20 x unknown

== supervisor >= 19 readiness: 140/3520 (4.0%) | online=2546 ==

Two deliberate choices worth reviewing

The audit lists every device, not just online ones. list_fleet_devices() filters is_online eq true because a resinhup needs a reachable device. The audit uses a separate list_fleet_inventory() with no such filter: the offline tail is exactly the population an uplift has to wait for, so excluding it would flatter the burndown and make the plan look shorter than it is.

The audit never selects uuid. Actions logs on this repo are world-readable. The existing phases gate per-device detail behind --verbose; the audit has no per-device output mode at all, so it does not fetch identifiers in the first place — the privacy property holds by construction rather than by discipline. There is a test asserting the $select stays uuid-free.

Risk

Additive and off by default. --audit issues only GETs and never mutates, regardless of --apply. Existing phase behaviour is untouched. An audit listing failure is counted and logged but does not stop the maintenance phases from running.

Extra API load is one paged device listing per fleet per hour (~10-15 requests).

Testing

• uv run pytest tests/test_balena_unpin_audit.py — 16 new tests pass (threshold behaviour incl. missing/blank supervisor versions, histogram bucketing and tail collapse, totals accumulation, empty fleet, and the uuid-free $select assertion)
• uv run ruff check . — clean
• uv run ruff format --check . — clean (250 files)
• uv run mypy . — clean for the changed files; the only findings project-wide are two pre-existing import-not-found for pygit2 / python_on_whales in tools/image_builder/__main__.py, which are unrelated and absent from the venv used here
• CLI smoke: --help renders, missing-token path still exits 2, audit output verified against a stubbed API

Not exercised against the live balena API — the audit path has no token here, so the first real numbers will come from the next scheduled run (or a workflow_dispatch with dry-run left ticked, which now also prints the burndown).

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud