[ML] Harden post-checkout Vault reads against slow checkouts (retry, logging, token refresh)

[edsavage]

Summary

Makes the post-checkout hook's Vault reads resilient to slow Buildkite checkouts, which were red-failing otherwise-green macOS test jobs.

Background

The macOS/Orka agents have recently been seeing abysmal GitHub transfer rates (as low as ~100 KiB/s) caused by a MacStadium firewall issue (packet discards; tracked by Platform Eng Productivity). On ml-cpp PR builds this pushed the macOS-aarch64 checkout from ~2.4 min (Oct 9) to 11–20 min (Oct 13).

The agent's environment hook mints VAULT_TOKEN once, at job start. When the clone takes longer than the token's TTL, the token has expired by the time post-checkout runs, so its secret reads fail with permission denied / invalid token. For the test steps the only read is BUILDKITE_ANALYTICS_TOKEN; leaving it empty makes the test-collector plugin's pre-exit hook exit 1 — failing a job in which all tests passed. (The same expiry silently disables sccache on build steps.)

This is intermittent, tracking the firewall degradation — which is why only macOS, and only on slow-clone days, was affected.

Changes

Built up over a few commits, all scoped to .buildkite/hooks/post-checkout:

1. Retry each Vault read a few times (handles genuine transient blips).
2. Log the real Vault error on each failed attempt instead of swallowing it with 2>/dev/null — this is what surfaced the invalid token root cause.
3. vault_reauth: when a read fails with invalid token (the slow-checkout/expiry symptom), mint a fresh VAULT_TOKEN using the same OIDC/JWT flow as the agent (elastic/vault-buildkite-plugin) and retry. Best-effort — never fails the job if re-auth is unavailable.

All Vault reads in the hook route through the shared helper, so analytics and sccache/other reads all benefit.

Testing

• Helper logic unit-tested locally: re-auth fires exactly once on invalid token, the read then succeeds, stdout stays clean, and the best-effort fallback never fails when buildkite-agent/vault are unavailable.
• CI: build passes on all platforms. The fast-clone path is confirmed non-regressive; the re-auth recovery path engages only on a genuinely slow clone (intermittent, firewall-driven).

The real root-cause fix (firewall upgrade) is infra-owned; this change makes our builds resilient in the meantime and to any future slow-checkout episode.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[Copilot]

🟡 Changes recommended

Command substitution prevents the refreshed Vault token from propagating to the hook’s parent shell.

1 open finding

• Token refresh is lost inside command substitution subshell · New

What changed in this PR

Hardens Buildkite post-checkout Vault access against expired tokens and transient failures.

Changes:

• Adds retry/error logging for Vault reads.
• Adds OIDC-based Vault reauthentication.
• Routes secret reads through a shared helper.

File	Description
.buildkite/​hooks/​post-checkout	Adds resilient Vault authentication and reads.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[edsavage]

Good catch — fixed in cac6d57. The helper now takes an output-variable name and assigns via printf -v, and is called in the parent shell rather than inside $(...). A VAULT_TOKEN refreshed by vault_reauth now propagates to the hook, so multi-secret steps re-authenticate at most once (subsequent reads reuse the fresh token) and later hook commands see the refreshed token.