lint-format's 3-minute timeout + apt.llvm.org install silently skips the entire build and test graph

[BitHighlander]

What happened

Run 32071353150 (PR #470, bb68a5dde):

static-analysis           success
check-submodules          success
secret-scan               success
lint-format               CANCELLED   (21:29:20 -> 21:32:36, 3m16s)
build-arm-firmware        SKIPPED
build-emulator            SKIPPED
unit-tests                SKIPPED
python-dylib-tests        SKIPPED
python-integration-tests  SKIPPED

lint-format did not fail the formatting check — it never reached it. Step conclusions:

Checkout                              success
Install clang-format-20 (pinned)      CANCELLED
Check code formatting                 skipped

Root cause

lint-format has timeout-minutes: 3, and its install step performs three network operations against a third party before any linting happens:

wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
echo "deb http://apt.llvm.org/noble/ llvm-toolchain-noble-20 main" | sudo tee /etc/apt/sources.list.d/llvm.list
sudo apt-get update -qq && sudo apt-get install -y -qq clang-format-20

When apt.llvm.org is slow, the install alone exceeds the 3-minute job budget. The job is cancelled mid-step and every downstream job is skipped, because they all declare needs: [lint-format, static-analysis, check-submodules, secret-scan].

Why this matters more than a flake

The entire build and test signal for a security release is one slow third-party apt mirror away from silently disappearing. The run surfaces success on the jobs that did complete, so at a glance it reads as healthy while ARM cross-compile, unit tests, dylib tests and integration tests all produced nothing.

This is the same failure family that already cost this release once: secret-scan was failing and silently skipping the whole downstream graph (fixed earlier via .gitleaks.toml). A gate stage that skips rather than fails is indistinguishable from a gate stage that passed, unless someone reads the job list.

There is a supply-chain dimension too: the pipeline fetches and trusts a GPG key over the network on every run (apt-key add, itself deprecated), then installs from that repo. For a firmware security release the toolchain should not be assembled from a third-party mirror at build time.

Suggested fix, in preference order

1. Stop fetching the toolchain at CI time. The pinned builder image kktech/firmware@sha256:7438e539… is already used for the ARM build; run clang-format from a pinned image or a cached apt package instead of apt.llvm.org. This removes both the flake and the supply-chain surface.
2. If the external install stays, raise timeout-minutes well above the observed install time and cache the package (actions/cache on the .deb), so a slow mirror costs latency rather than signal.
3. Independently: make a skipped gate loud. Add a final job with if: always() that fails when any required job's result is skipped or cancelled. Today those states are silent, and that silence has now hidden two different problems on this release.

Found while validating #470.