CI red on main: 1.85 (MSRV) job fails reproducibly — code passes locally, ARC runner logs not persisted

[27Bslash6]

Summary

CI has been red on main for 6+ days (since the last pushes on 2026-04-26). The failure is isolated to the 1.85 (MSRV floor) matrix job; stable, beta, and wasm32 compilation check all pass. I re-ran the failed job today (2026-06-06) — it failed again, so this is reproducible, not a flake.

Job	Result
1.85	❌ failure (reproduced on re-run)
stable	✅
beta	✅
wasm32 compilation check	✅

Run: https://github.com/cachekit-io/cachekit-rs/actions/runs/26684715852

The code is NOT the cause

The 1.85 job runs cargo test --features "cachekitio,redis,encryption,l1,macros". I reproduced the exact command locally on rustc 1.85.1 (same toolchain CI installs) — everything is green:

• cargo +1.85 test --features "…" → all suites ok, 0 failed
• cargo +1.85 test --no-run --features "…" → compiles clean
• cargo +1.85 test --doc --features "…" → doctests pass

The committed Cargo.lock compiles fine on 1.85 — this is not a transitive-dependency MSRV bump (my first hypothesis, now ruled out).

The real blocker: the runner isn't persisting logs

I cannot read why CI failed because the cachekit-lean (self-hosted ARC) runner is not persisting job logs:

• GET /actions/jobs/{id}/logs → BlobNotFound (immediately after completion, both the 6-day-old run and today's fresh re-run)
• gh run view --json jobs → the 1.85 job's steps[] / failed-steps array is empty

So the GitHub UI/API has no step breakdown and no log blob for this job. That's an observability defect on the ARC runners independent of the test failure, and it's what makes this undiagnosable remotely.

Impact

The Release workflow installs stable only, so 0.3.0 published despite red CI — the breakage is fully masked. A real regression that only trips the MSRV floor would ship unnoticed.

Hypothesis (unverified — needs runner-side logs)

Code passes locally on 1.85 + only the 1.85 job fails + no step logs ⇒ the failure is environmental on the runner pod, before/around test execution rather than in the tests. Most likely the toolchain-install step (rustup toolchain install 1.85 --profile minimal --component rustfmt,clippy) or a pod-level resource/network issue. Note the 1.85 job installs rustfmt+clippy it never uses (both run steps are gated if: matrix.rust != '1.85').

Recommended next steps (ranked)

1. Make CI logs survive — the ARC blob-upload is broken; until it's fixed nothing here is diagnosable. Quick mitigation: tee the test output to a file and actions/upload-artifact it, so failures are inspectable even when blob upload fails.
2. Re-run with logs captured, read the actual error.
3. Cheap candidate fix to test in the same pass: drop --component rustfmt,clippy from the 1.85 toolchain install (it's unused on that job) — removes a download/failure surface and speeds the job.
4. Consider gating Release/publish on CI green (or at least alerting on red main), so a red MSRV job can't ship silently again.

---

Filed from the 2026-06-06 fleet review. Local repro: cargo 1.85.1 (d73d2caf9) / rustc 1.85.1 (4eb161250).

[27Bslash6]

Root cause found + fixed (diagnosed live on the ARC runner)

Got cluster (k9s) access and captured the 1.85 job's pod logs directly, since GitHub never persists them.

It's OOM, not the code. Sequence:

• Install Rust toolchain succeeds; failure is in the cargo test step, which runs ~10 min then the runner is SIGKILLed (exit 137) mid-step — the step records no conclusion, and the killed runner never finalizes its log upload (→ the BlobNotFound we kept hitting).
• cargo reads the node's 32-thread CPU count, not the pod's cgroup quota, so it spawns ~32 parallel rustc in the 6Gi cachekit-lean runner. Memory peak (codegen/link) exceeds 6Gi → thrash → OOM-kill at ~10:00 (reproduced twice: 10:03 and 10:02).
• Passes locally because my box has 64GB and no cap.

Why 1.85 specifically: stable/beta run clippy first (warms artifacts → incremental, lower-peak test build); 1.85 skips fmt/clippy and does the full cold 32-way compile in the test step → tips over the ceiling.

Fix (proven on a live run): cap CARGO_BUILD_JOBS=4 → 1.85 goes from killed at ~10:00 to green in 75s (whole matrix green). PR #27.

Secondary issues this surfaced (worth separate follow-up):

1. ARC runners don't persist job logs — /actions/jobs/{id}/logs → BlobNotFound, empty steps[]. Any future failure on these runners is undiagnosable from GitHub. (My recommendation feat: intent-based cache presets (CacheKit::io, ::minimal, ::production, ::secure) #1 — tee + upload-artifact — still stands.)
2. Release runs on stable only, so 0.3.0 shipped despite red CI. Consider gating release on CI green.
3. Same cargo-vs-cgroup parallelism footgun likely affects cachekit-core's CI on these runners.