feat(hydro_lang, hydro_deploy): replay captured rustc commands for final test builds, skipping cargo

[shadaj]

Continuing the per-test CI cost investigation: with warm caches, per-phase
profiling showed the final cargo rustc of each generated example dominating at
~1.35s of a ~1.8s test (serial, 32-core). Decomposition of that 1.35s:
~320ms fixed cargo overhead (a no-op cargo rustc with nothing dirty),
~180ms rustc frontend, ~240ms codegen, ~430ms link (already on rust-lld),
plus ~30ms per-job build-dir population and cargo's cross-process target-dir
locking (a likely contention source under nextest parallelism in CI that serial
profiling doesn't even see). sccache can't help: the example links a dylib, so
the compile is never cacheable.

Since every final build against a given prebuild runs an identical rustc
command modulo a few per-test values, capture it once and replay it directly:

• New hydro_concurrent_cargo::final_rustc module (shared so both hydro_lang
  and hydro_deploy use it): the first final build after a prebuild runs cargo
  with -v, and the printed rustc invocation is parsed (shell-tokenizer
  handling cargo's unix/windows quoting), validated, and stored as a template
  with placeholders for the per-test values: crate name, source path, out dir,
  and -C metadata/-C extra-filename (unique per test so multiple dlopen'ed
  cdylibs don't collide). Per-job target dir paths are rewritten to the shared
  target dir (their deps/.fingerprint/build entries are symlinks into it
  anyway). Templates live next to the prebuild lock file and are keyed to the
  prebuild stamp (features_hash:timestamp), so any prebuild rerun invalidates
  them — staleness tracking is exactly as strong as the prebuild's own.
• Replay invokes rustc directly (no cargo): substitutes the per-test values,
  parses rustc's --json=artifacts stderr for the linked artifact, and
  surfaces rendered diagnostics. Any anomaly (parse failure, missing files,
  compile error) deletes the template and falls back to the unchanged cargo
  path, which remains the source of truth for command construction and error
  reporting.
• Replay still takes the per-job lock via new
  hydro_concurrent_cargo::lock_job_dir (serializing concurrent builds of an
  identical flow) but skips job-dir population entirely; no cargo also means
  no cargo target-dir lock contention. New PrebuildGuard::read_stamp() /
  lock_path() expose the stamp for template keying.
• hydro_lang::compile_trybuild_example (sim + maelstrom tests): replay is
  used for all non-fuzz builds; fuzz mode keeps the cargo path (different
  crate layout + linker flags).
• hydro_deploy build_crate_memoized (localhost deploy tests): replay is
  used for local dylib example builds with default profile/rustflags — the
  same conditions under which deploy_graph enables dynamic linking; the
  replayed artifact keeps shared_library_path pointing at the shared
  debug/deps.

Measured (32-core local, serial, warm): sim test mean 1814ms -> 1391ms (-23%);
final compile phase 1347ms (final_build) -> 954ms (final_rustc), and
populate_job_dir (~30ms) disappears from the steady state. The cargo capture
build itself is unchanged in cost and happens once per prebuild per feature
set. Full hydro_lang suite (188 tests incl. sim_fuzzer) and hydro_test
localhost deploy suite (18 tests) pass, with all steady-state builds confirmed
replaying; clippy and fmt clean.

Scripts updated: profile_test_phases.sh documents the new final_rustc phase;
bench_trybuild.sh removes captured templates when touching examples so its
warm phase still exercises (and measures) the cargo capture path.

Considered and rejected for now: deriving the rustc command from the prebuild's
lib invocation (the example target differs in crate-type/emit/externs/link
args, so it would mean reimplementing cargo's argv construction) and
constructing the command from first principles (fragile against profile
settings like debug = "line-tables-only" that trybuild copies into the
generated workspace). Remaining per-test costs, in case of follow-up: ~430ms
link + ~240ms codegen (biggest lever: -Zlink-native-libraries=no saved
~175ms but is nightly-only), ~180ms flow_build, ~65ms sim_codegen, ~26ms
cargo_metadata.

[cloudflare-workers-and-pages]

Deploying hydro with    Cloudflare Pages

Latest commit:	95a6fa4
Status:	✅  Deploy successful!
Preview URL:	https://1fd50ff9.hydroflow.pages.dev
Branch Preview URL:	https://sandbox-d501e509-750f-4480-b.hydroflow.pages.dev

View logs

[shadaj]

This had very little impact on CI times, and introduces a lot of brittle code, so closing for now.