feat(proxies): cargo/crates pull-through cache, and fix the Go module proxy

[luthermonson]

The Go module proxy has never served a request

It binds to the CNI bridge gateway (10.88.0.1:8082), but that address does not exist when the daemon starts. The bridge is created lazily by CNI on the first job container, and both cleanup() and cleanStaleBridge() delete it outright (pkg/networking/network_linux.go). So Listen fails with EADDRNOTAVAIL on every boot, the error is logged as a warning and swallowed, and GOPROXY is never injected into any container.

Start() runs MkdirAll before the failing Listen, which is exactly why cache/gomod exists on every node, empty, at 4 KB. That empty directory was the clue.

Three further defects would have kept it useless even with the bind fixed:

1. EnvVars() advertised the bound address rather than the configured one — meaningless inside a container after any fallback.
2. GOPROXY=…,direct is not fail-open: the comma separator only falls through on 404/410, so a wedged proxy hard-failed builds. Now |direct.
3. cleanup was dead code (if !cleanup { cleanup = true }) — the cache was wiped on every shutdown regardless of configuration.

Changes

• pkg/proxies/listen.go (new) — Listen() with a wildcard fallback, so the gateway address becomes reachable as soon as CNI brings the bridge up. A genuine error (port in use) is still returned rather than masked.
• pkg/proxies/go — advertise the configured address; |direct so any proxy error falls through to upstream; honour cleanup = false.
• pkg/proxies/cargo (new) — pull-through cache for the sparse index, .crate tarballs and rustup dist. The index is revalidated on a TTL with conditional GETs (If-None-Match / If-Modified-Since); tarballs are immutable and cached permanently. config.json is rewritten so dl points at the proxy while api still reaches the real registry, so cargo publish/search keep working. Concurrent misses are collapsed by a per-path mutex.
• pkg/runtime — support bind-mounting proxy-generated files into job containers.

Why a mounted config file rather than environment variables

Cargo ignores CARGO_SOURCE_* and CARGO_REGISTRIES_* for source replacement. This was verified empirically against a dead endpoint rather than assumed: with the env vars set, cargo reported "Updating crates.io index" and went straight to the real registry; the same settings in a config file produced "Updating ephemerd index" and the expected connection error.

So the proxy generates a .cargo/config.toml and mounts it read-only at the container root, relying on Cargo's ancestor-directory config search. That applies to any checkout path, needs no knowledge of the workdir or the image's CARGO_HOME (left writable), and a repository's own .cargo/config.toml still wins. No workflow changes required. rustup takes RUSTUP_DIST_SERVER, where an env var does work.

Had this been built on the env-var assumption, the proxy would have been silently bypassed — the same class of failure as the Go proxy above.

Fail-open

Three stages: proxy not started → nothing injected, jobs go direct; upstream down with a cached copy → serve stale with a warning; upstream down with nothing cached → 307 to the origin. Genuine 404s pass through.

Config

[cargo_proxy]: enabled (false), port (8083), upstream (https://index.crates.io), rustup_upstream (https://static.rust-lang.org), index_ttl (10m), cleanup (false).

cleanup defaults false, deliberately unlike [module_proxy] — wiping a pull-through cache on every restart is the bug diagnosed above. Registered in managedCaches() as LiveSafe: true; the generated container config lives outside the cache root so cache clear cargo cannot pull a mounted file out from under a running job.

Testing

go build ./... and go vet ./... clean; go test ./... passes apart from a pre-existing pkg/runner failure (fresh worktree lacks the gitignored runner archive). ~45 tests against an httptest fake upstream, no network access.

Not yet run on a node — opened as a draft. The Go-proxy fix in particular is worth confirming live, since the symptom (an empty 4 KB directory) is easy to mistake for "working but unused".

Follow-ups found, not fixed

• Manager.GatewayIP() hardcodes 10.88.0.1 when Subnet is empty, while Config.subnet() → pickSubnet() may auto-select a different 10.x/16 on conflict. On such a host the two disagree.
• The GatewayPorts ACCEPT rules land in EPHEMERD-FORWARD, jumped from FORWARD, but container→gateway traffic hits INPUT — so those rules are currently no-ops. Harmless today (reachability comes from the default INPUT policy), but the carve-out is not doing what it appears to.

sccache

docs/arch/sccache-evaluation.md. Recommends building it, sequenced after the disk-pressure GC, with a hard size cap wired into that GC from day one. Local-disk backend, partitioned per repo, and scoped to cargo build rather than docker build so it stays disjoint from the BuildKit layer cache instead of storing the same output twice.