Skip to content

dev certs: the parametric minting tool, with a real handshake behind it - #722

Open
zah wants to merge 3 commits into
devfrom
dev-certs/minting-tooling
Open

zah wants to merge 3 commits into
devfrom
dev-certs/minting-tooling

Conversation

@zah

@zah zah commented Oct 2, 2026

Copy link
Copy Markdown
Member

environment-domains-and-dev-certificates.md §4 puts the development-certificate
minting tooling in this repo, parametric over root, domains and validity,
because the scheme is shared by metacraft-labs, agent-harbor and blocksense and
each runs its own root CA under it. The tool therefore names no org — everything
comes from a JSON spec, which is the consumer repo's lib/dev-certificates.nix
evaluated:

nix eval --json --file lib/dev-certificates.nix > spec.json
mint-dev-certificates init-root --spec spec.json --key-out ca.key --cert-out certs/root-ca.crt
mint-dev-certificates mint --spec spec.json --root-key ca.key --root-cert certs/root-ca.crt \
    --key-out-dir keys --cert-out-dir certs
mint-dev-certificates report --spec spec.json --cert-dir certs

What it does, and the one thing it refuses

  • init-root generates the root key (umask before keygen, not chmod after)
    and a self-signed root with CA:true,pathlen:0. It refuses to overwrite
    either output: re-minting a root invalidates every issued leaf and every
    machine's trust store at the same instant, so that is never an overwrite.
  • mint issues one leaf per spec row, verifies each one against the root it
    was just signed by and that it carries every requested SAN, then writes
    fullchain.pem (leaf first, then root) and a mode-600 key.
  • report is §4.1: remaining validity for every leaf, exit non-zero
    below the threshold, and a never-minted leaf counts as a failure, not a skip.

The refusal that matters is the reserved-TLD guard. A private root that can
issue for codetracer.com is a private root that can impersonate production to
everyone who trusts it — and every workstation in the org will trust this one.
The spec is hand-edited, so the guard sits next to the key rather than in review.
It covers .local explicitly, because §2 rules mDNS out by name and a guard that
quietly accepted it would route around that.

It deliberately does not seal anything. Sealing needs the consumer repo's
agenix recipient set, which is an evaluation of that flake; reaching for it
here would make this package depend on a tree it cannot see. The caller seals.

The test is a handshake, not a file-shape assertion

§7: "certificates being installed is not evidence that TLS works." So
scripts/tests/dev-certificates-minting.sh mints a real root and real leaves
with real openssl and completes a real verified handshake against
openssl s_server — for a wildcard name and for the apex — with three negative
controls so the positive one means something:

  • a name the leaf does not carry must be rejected on a verification error,
    not on a refused connection;
  • a client with no trust anchor must refuse the same leaf;
  • a refused spec must leave no root key behind.

35 assertions, 35 passing, by hand and inside the Nix sandbox, at
d741a43. It runs in CI as the flake check dev-certificates-minting, and is
runnable by hand with bash scripts/tests/dev-certificates-minting.sh (bash,
openssl, jq).

Two of those controls earned their place immediately, which is the honest
argument for writing them:

  • The reserved-TLD guard shipped rejecting every legitimate name.
    local name="$1" bare="$name" expands all of local's words before
    performing any of its assignments, so bare was empty and everything fell
    through to the refusal. A positive-only test would have called that a pass.
  • The handshake's own readiness probe consumed -naccept 1's single slot,
    so the client met a closed port — and the negative control then passed for
    exactly that wrong reason. It now asserts on the verification message.

Checks run locally at d741a43

  • nix build .#checks.x86_64-linux.dev-certificates-minting — 35/35.
  • nix build .#packages.x86_64-linux.mint-dev-certificates — builds; the
    wrapper is ShellCheck-clean via writeShellApplication.
  • shellcheck -S warning on both scripts — clean.
  • prek run --files <the five touched files> in .#pre-commit — all hooks pass.

aarch64-darwin was not available here; the script handles the GNU/BSD stat and
date splits explicitly, and the check runs under nixpkgs' bash rather than the
system one, but the darwin leg is unverified.

Follow-on

The consumer side lands in metacraft-labs/infra: the declared domain set
(lib/dev-certificates.nix, PR #1677), the agenix-sealed leaf keys, and the
NixOS / nix-darwin module that installs §5's /etc/mcl-dev-certs layout plus
the system trust store.

`environment-domains-and-dev-certificates.md` §4 puts the development-certificate
minting tooling in this repo, "parametric over root, domains and validity",
because the scheme is shared by metacraft-labs, agent-harbor and blocksense and
each of them runs its own root CA under it. So the tool names no org: everything
comes from a JSON spec, which is the consumer repo's `lib/dev-certificates.nix`
evaluated.

Three subcommands. `init-root` generates the root key and self-signed root, and
refuses to overwrite either — re-minting a root invalidates every issued leaf
and every machine's trust store at the same instant, so that is never an
overwrite. `mint` issues one leaf per spec row, keys for the caller to seal and
`fullchain.pem` to commit in the clear. `report` is §4.1: it prints remaining
validity for every leaf and EXITS NON-ZERO below the threshold, counting a
never-minted leaf as a failure rather than a skip.

The guard worth having is the reserved-TLD refusal. A private root that can
issue for `codetracer.com` is a private root that can impersonate production to
everyone who trusts it, and every workstation in the org will trust this one.
The spec is hand-edited, so the refusal sits next to the key rather than in
review, and it covers `.local` explicitly — §2 rules mDNS out by name, and a
guard that accepted it would route around that.

§7 says certificates being installed is not evidence that TLS works, so the test
is not a file-shape assertion: it mints a real root and real leaves and
completes a real verified handshake against `openssl s_server`, both for a
wildcard name and for the apex. Three negative controls make that mean
something — a name the leaf does not carry must be REJECTED on a verification
error (not on a refused connection), and a client with no trust anchor must
refuse the leaf. 35 assertions, green by hand and inside the Nix sandbox at this
commit.

Two of those controls earned their place immediately. The reserved-TLD guard
shipped rejecting every legitimate name, because `local name="$1" bare="$name"`
expands all of `local`'s words before performing any of its assignments, so
`bare` was empty and every name fell through to the refusal. And the handshake's
own readiness probe consumed `-naccept 1`'s single slot, so the client met a
closed port — at which point the negative control passed for exactly that wrong
reason. It now asserts on the verification message.
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for your Pull Request!

Below you will find a summary of the cachix status of each package, for each supported platform.

package x86_64-linux x86_64-darwin aarch64-darwin
attic-migrate-flake ✅ Cached at <0> 🚫 not supported ❌ build failed
attic-migrate-flake-idempotent ✅ Cached at <0> 🚫 not supported ❌ build failed
attic-push-flake-outputs-action ✅ Cached at <0> 🚫 not supported ❌ build failed
aztec ✅ Cached at <0> 🚫 not supported ❌ build failed
cachix ✅ Cached at <0> 🚫 not supported ❌ build failed
cachix-deploy-metrics ✅ Cached at <0> 🚫 not supported ❌ build failed
ci-build-batching ✅ Cached at <0> 🚫 not supported ❌ build failed
ci-image ✅ Cached at <0> 🚫 not supported 🚫 not supported
consumer-flake-cachix-inventory ✅ Cached at <0> 🚫 not supported ❌ build failed
consumer-flake-cachix-inventory-tool ✅ Cached at <0> 🚫 not supported ❌ build failed
consumer-flake-no-cachix-residual ✅ Cached at <0> 🚫 not supported ❌ build failed
consumer-flake-no-cachix-residual-tool ✅ Cached at <0> 🚫 not supported ❌ build failed
dcd ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-attic-nginx-logs-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-attic-push-substitute-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-break-glass-runbook-sections ✅ Cached at <0> 🚫 not supported ❌ build failed
deployment-cache-backend-policy ✅ Cached at <0> 🚫 not supported ❌ build failed
deployment-cache-corruption-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-cachix-fallback-simulation ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-current-flow-inventory ✅ Cached at <0> 🚫 not supported ❌ build failed
deployment-darwin-activation-integration ✅ Cached at <0> 🚫 not supported ❌ build failed
deployment-dashboard-query-fixtures ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-direct-ssh-attic-restore-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-direct-ssh-rollback-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-direct-ssh-success-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-e2e-rehearsal-skill-doc ✅ Cached at <0> 🚫 not supported ❌ build failed
deployment-event-metrics ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-event-schema-examples ✅ Cached at <0> 🚫 not supported ❌ build failed
deployment-incus-rehearsal-image ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-incus-rehearsal-script-static ❌ build failed 🚫 not supported 🚫 not supported
deployment-metrics-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-no-default-cachix-deploy-call ❌ build failed 🚫 not supported 🚫 not supported
deployment-parallel-cache-push-local ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-production-cutover-simulation-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-pull-agent-darwin-integration 🚫 not supported 🚫 not supported ❌ build failed
deployment-pull-agent-latest-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-pull-agent-lock-contention-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-pull-agent-locks-migration-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-pull-agent-rejects-invalid-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-pull-agent-selfheal-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-pull-agent-static ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-reconciler-lock-contention-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-reconciler-skill-doc ❌ build failed 🚫 not supported ❌ build failed
deployment-reconciler-supersession-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-reconciler-timer-retry-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-reconciler-timer-static ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-scheduled-canary-local-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-skill-doc-command-references ❌ build failed 🚫 not supported ❌ build failed
deployment-summary-artifact ✅ Cached at <0> 🚫 not supported ❌ build failed
desktop-vms-golden ✅ Cached at <0> 🚫 not supported 🚫 not supported
dev-certificates-minting ❌ build failed 🚫 not supported ❌ build failed
disko ✅ Cached at <0> 🚫 not supported 🚫 not supported
dmd ✅ Cached at <0> 🚫 not supported 🚫 not supported
dscanner ✅ Cached at <0> 🚫 not supported 🚫 not supported
dub ✅ Cached at <0> 🚫 not supported ❌ build failed
encrypted-s3-artifact ✅ Cached at <0> 🚫 not supported ❌ build failed
erigon ✅ Cached at <0> 🚫 not supported 🚫 not supported
fleet-alertmanager-config ✅ Cached at <0> 🚫 not supported ❌ build failed
fleet-alertmanager-config-deadman ✅ Cached at <0> 🚫 not supported ❌ build failed
folder-size-metrics ✅ Cached at <0> 🚫 not supported 🚫 not supported
foundry ✅ Cached at <0> 🚫 not supported ❌ build failed
garm ✅ Cached at <0> 🚫 not supported ❌ build failed
garm-fleet-alert-rules ✅ Cached at <0> 🚫 not supported ❌ build failed
garm-fleet-external-checks ✅ Cached at <0> 🚫 not supported ❌ build failed
garm-provider-aws ✅ Cached at <0> 🚫 not supported ❌ build failed
garm-provider-vmharness ✅ Cached at <0> 🚫 not supported ✅ Cached at <0>
geth ✅ Cached at <0> 🚫 not supported ❌ build failed
git-hooks-reprobuild-handoff ✅ Cached at <0> 🚫 not supported ❌ build failed
github-actions-fit-alert-rules ✅ Cached at <0> 🚫 not supported ❌ build failed
github-actions-fit-exporter ✅ Cached at <0> 🚫 not supported ❌ build failed
ldc ✅ Cached at <0> 🚫 not supported 🚫 not supported
lido-withdrawals-automation ✅ Cached at <0> 🚫 not supported ❌ build failed
linux-vm-cloud-init-image ✅ Cached at <0> 🚫 not supported 🚫 not supported
mcl-devops ✅ Cached at <0> 🚫 not supported ✅ Cached at <0>
mcl-devops-no-stale-references ❌ build failed 🚫 not supported ❌ build failed
mcl-devops-single-bin-mcl-provider ❌ build failed 🚫 not supported ❌ build failed
mev-boost ✅ Cached at <0> 🚫 not supported 🚫 not supported
mint-dev-certificates ✅ Cached at <0> 🚫 not supported ❌ build failed
netbird-with-agenix-login-script ✅ Cached at <0> 🚫 not supported 🚫 not supported
nethermind ✅ Cached at <0> 🚫 not supported 🚫 not supported
nimbus ✅ Cached at <0> 🚫 not supported 🚫 not supported
nix ✅ Cached at <0> <1> 🚫 not supported
nix-eval-jobs ✅ Cached at <0> <1> 🚫 not supported
nix-fast-build ✅ Cached at <0> 🚫 not supported ✅ Cached at <0>
nixos-anywhere ✅ Cached at <0> 🚫 not supported 🚫 not supported
nixos-rebuild-ng ✅ Cached at <0> 🚫 not supported ❌ build failed
pyroscope ✅ Cached at <0> 🚫 not supported ❌ build failed
random-alerts ✅ Cached at <0> 🚫 not supported ❌ build failed
reusable-flake-checks-artifact-fallback ✅ Cached at <0> 🚫 not supported ❌ build failed
reusable-flake-checks-attic-credentials ✅ Cached at <0> 🚫 not supported ❌ build failed
reusable-flake-checks-mcl-ref ✅ Cached at <0> 🚫 not supported ❌ build failed
reusable-flake-checks-s3-mirror-compat ✅ Cached at <0> 🚫 not supported ❌ build failed
reusable-lint-hook-discovery ❌ build failed 🚫 not supported ❌ build failed
reusable-terraform-drift-workflow ✅ Cached at <0> 🚫 not supported ❌ build failed
reusable-terraform-policy-advisories ❌ build failed 🚫 not supported ❌ build failed
reusable-terraform-source-identity ❌ build failed 🚫 not supported ❌ build failed
runner-label-tool ✅ Cached at <0> 🚫 not supported ❌ build failed
rustToolchain ✅ Cached at <0> 🚫 not supported ❌ build failed
seal-alerting-secrets ✅ Cached at <0> 🚫 not supported ❌ build failed
secret-integration ✅ Cached at <0> 🚫 not supported 🚫 not supported
serve-d ✅ Cached at <0> 🚫 not supported 🚫 not supported
setup-nix-hosted-disk-reclamation ❌ build failed 🚫 not supported ❌ build failed
setup-nix-substituter-preflight ✅ Cached at <0> 🚫 not supported ❌ build failed
setup-nix-transfer-resilience ❌ build failed 🚫 not supported ❌ build failed
t_aws_burst_runners ❌ build failed 🚫 not supported 🚫 not supported
t_aws_spot_runners ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_central_garm_recovery ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_ci_runs_on_capability ✅ Cached at <0> 🚫 not supported ❌ build failed
t_cross_host_publish_substitute ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_fit_exporter_repos_env_roundtrip ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_fleet_alert_routing ✅ Cached at <0> 🚫 not supported ❌ build failed
t_fleet_alerting ✅ Cached at <0> 🚫 not supported ❌ build failed
t_garm_api_watchdog ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_busy_runner_not_reaped ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_capability_placement ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_central_multi_host ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_credentials_no_race ❌ build failed 🚫 not supported 🚫 not supported
t_garm_deleting_capacity ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_incus_runner_host ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_incus_storage_pool_source ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_instance_lifecycle ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_job_cache_self_heal ❌ build failed 🚫 not supported 🚫 not supported
t_garm_macos_pools_supported ✅ Cached at <0> 🚫 not supported ❌ build failed
t_garm_macos_runner_install_wrapper ✅ Cached at <0> 🚫 not supported ❌ build failed
t_garm_multi_provider ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_nixos_service_boots ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_pools_labels ❌ build failed 🚫 not supported 🚫 not supported
t_garm_provider_remote ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_provider_remote_old_daemon ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_provider_vmharness_backend ✅ Cached at <0> 🚫 not supported ❌ build failed
t_garm_provider_vmharness_protocol ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_provider_vmharness_windows_toolchain ✅ Cached at <0> 🚫 not supported ❌ build failed
t_garm_reconcile ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_remote_incus_capabilities ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_remote_libvirt_settings ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_stale_scaleset_job_reaped ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_webhook_delivery ❌ build failed 🚫 not supported 🚫 not supported
t_pools_cutover_complete ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_private_repo_hybrid_fallback ✅ Cached at <0> 🚫 not supported ❌ build failed
t_public_repo_free_ci ✅ Cached at <0> 🚫 not supported ❌ build failed
t_repro_binary_cache_systemd_healthz ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_repro_cache_client_render ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_repro_deploy_agent_https_converges ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_repro_lease_reaper_eval ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_runner_fit_monitoring ✅ Cached at <0> 🚫 not supported ❌ build failed
t_runner_label_taxonomy ✅ Cached at <0> 🚫 not supported ❌ build failed
t_runner_mode_manager ✅ Cached at <0> 🚫 not supported ❌ build failed
t_runner_mode_switch ✅ Cached at <0> 🚫 not supported ❌ build failed
t_s3_artifact_store_acl_and_creds ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_vmharness_create_fails_on_dead_guest ✅ Cached at <0> 🚫 not supported ❌ build failed
t_vmharness_image_is_honoured ✅ Cached at <0> 🚫 not supported ❌ build failed
t_vmharness_serve_inventory_textfile ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_vmharness_serve_linux_deploy ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_vmharness_serve_win_libvirt ✅ Cached at <0> 🚫 not supported 🚫 not supported
terraform ✅ Cached at <0> 🚫 not supported 🚫 not supported
terraform-ci-matrix ✅ Cached at <0> 🚫 not supported ❌ build failed
terranix ✅ Cached at <0> 🚫 not supported 🚫 not supported
test-linux-vm-cloud-init ✅ Cached at <0> 🚫 not supported 🚫 not supported
test_darwin_pull_agent_already_current_is_transaction_free 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_already_current_recovers_lifecycle_before_convergence 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_assertion_gate 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_entrypoint_survives_generation_change 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_launchd_contract 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_lock_contention_is_non_destructive 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_recreates_missing_lock_dir ✅ Cached at <0> 🚫 not supported ❌ build failed
test_darwin_pull_agent_rejects_invalid_durable_state 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_rejects_untrusted_and_wrong_target_manifests 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_same_id_sequence_high_water 🚫 not supported 🚫 not supported ❌ build failed
vm-harness ✅ Cached at <0> 🚫 not supported ❌ build failed
vmharness-serve-darwin-posture ✅ Cached at <0> 🚫 not supported ❌ build failed
web3signer ✅ Cached at <0> 🚫 not supported 🚫 not supported
yaml-automation-runner ✅ Cached at <0> 🚫 not supported 🚫 not supported

zah added 2 commits October 2, 2026 09:05
`prek run --all-files` is red on `dev` for two files neither of which this
branch touches, so it is red on every PR opened against it. Red lint is not
pre-existing noise to merge past — it is the thing to fix first — so it is fixed
here, in its own commit.

`terraform/github/README.md` had a closing code fence with prose on the same
line:

    ```
    repositories = map (r: r // (…)) inventory.repositories;
    ``` The `mergeQueues` output reports each queued

Prettier's own fix for that is to escalate the fence to FOUR backticks, which is
why this is worth a careful look rather than an `--all-files --fix`: a 4-backtick
fence would have swallowed the next seventy lines of prose — through the
`protectedRepos` paragraph and the second code block — into one code block. The
defect is the fence, so the fence is what is repaired: it closes on its own line
and the sentence starts a new paragraph. Prettier then wants no change at all.

`upstream-patches/garm-busy-runner-reap/fix.patch` ended with a blank line.
end-of-file-fixer removes it; the last content line keeps its newline, so the
patch is unchanged as a patch.

`prek run --all-files` is clean at this commit.
`__darwinAllowLocalNetworking = true` on the check derivation. macOS's Nix sandbox
denies even loopback unless a derivation asks for it, so the four handshake
subtests would have failed on an aarch64-darwin builder while passing on Linux —
the exact asymmetry that makes a green Linux leg misleading. The attribute is
ignored on other platforms, so it is set unconditionally.

This is preemptive and the darwin leg stays UNVERIFIED: no darwin builder was
available, and the aarch64-darwin CI batch that would have exercised it died with
"the self-hosted runner lost communication with the server" before reporting
anything. Both the check and this commit message say so rather than implying
coverage.

Also: the EXIT trap now kills the TLS server. `stop_srv` runs on the happy path,
but a failed assertion between `serve` and it would leave an `openssl s_server`
holding a port on a shared self-hosted runner for as long as the machine stays up.

35/35 by hand and in the Nix sandbox at this commit.
@zah

zah commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

On the five failing checks: four are pre-existing on dev, one is a runner flake

None are caused by this branch, and I established that by measurement rather than
by reading the diff. Recorded in full as
metacraft-specs/issues/2026-10-02-devops-modules-dev-has-four-failing-checks.md.

Check Verdict How
t_garm_pools_labels pre-existing drv path byte-identical to origin/dev (m6jhrg8f…-vm-test-run-t_garm_pools_labels.drv) — and it is the same path the job reported building. Same inputs cannot give a different result
t_garm_webhook_delivery pre-existing drv path byte-identical to origin/dev
mcl-devops-no-stale-references pre-existing, and a false positive built the dev variant: same message, same line — checks/pre-commit.nix:50, a comment about a flake-parts config.mcl, which is a different mcl. The gate's own output labels it [tool named in prose]
deployment-no-default-cachix-deploy-call pre-existing built the dev variant: same AssertionError: workflow must default non-nix-runner to the ephemeral Linux class
ci / batch 3/6 | aarch64-darwin infrastructure the only annotation is "The self-hosted runner lost communication with the server". No test failed; the log is gone

The last two of those four have different drv paths from dev, because both
hash the whole source tree — adding any file changes the hash. That difference is
not evidence of causation, which is why I compared the messages instead. "The drv
differs" is the easy wrong conclusion there.

One real fix came out of chasing it

376eab2 adds __darwinAllowLocalNetworking = true to the check. macOS's Nix
sandbox denies even loopback unless a derivation asks, so the four handshake
subtests would have failed on an aarch64-darwin builder while passing on Linux —
the exact asymmetry that makes a green Linux leg misleading. The darwin leg is
still unverified: no darwin builder was available here, and the batch that
would have exercised it is the one whose runner died.

It also makes the EXIT trap kill the TLS server, so a failed assertion cannot leave
an openssl s_server holding a port on a shared self-hosted runner.

35/35 by hand and in the Nix sandbox at 376eab2.

What this means for merging

This PR cannot reach an all-green CI while dev is red, so merging it is a
judgement about the four known failures rather than about this branch. The issue
above is the record to check them against.

@zah

zah commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

Update: the failure count moved to 7, and one of the new ones I can only partly attribute

Re-running changed the darwin picture, so correcting my own table rather than
leaving it to be read as current:

Check Verdict
ci / batch 1/6 | aarch64-darwin runner drop — annotation is only "The self-hosted runner lost communication with the server"
ci / batch 3/6 | aarch64-darwin runner drop — same annotation
ci / batch 5/6 | aarch64-darwin real build failure: error: Cannot build '…-garm-provider-vmharness-0.1.0.drv'
the three x86_64-linux ones unchanged, pre-existing as established above

On batch 5/6, what I can and cannot show. garm-provider-vmharness is built by
packages/garm-provider-vmharness/, which this branch does not touch; my
packages/default.nix edit only adds an attribute. Evaluating that package for
aarch64-darwin gives a byte-identical derivation on this branch and on
origin/dev (vhpsqz8gdmfnh55ml3915s9h6mmwb80f-…).

But I am not claiming that settles it, because the drv named in the CI log is
6fvdqnbvps55r65gx7vdckhzq98xwks6-… — a different path from the attribute I
evaluated, so the batch reaches that package by some route I have not identified. The
clean control would be a recent dev CI run to compare against, and there isn't
one: dev's most recent run of any kind is 2026-09-21. Worth knowing on its own
— it means "does this also fail on dev?" currently has no cheap answer in this
repo.

So: two runner drops, three established pre-existing, and one darwin package failure
that is very likely pre-existing but which I have not proved to the same standard as
the others.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant