Skip to content

build: hold one machine-wide lock per dev Cargo build - #6440

Open
Hmbown wants to merge 2 commits into
mainfrom
build/machine-build-lock
Open

Hmbown wants to merge 2 commits into
mainfrom
build/machine-build-lock

Conversation

@Hmbown

@Hmbown Hmbown commented Sep 23, 2026

Copy link
Copy Markdown
Owner

No-Issue: local multi-agent build coordination for 0.10.1 dogfooding; no public issue.

Several agents share this checkout on a memory-constrained Mac, and
Cargo's own lock is per target directory, so builds into different
target dirs ran concurrently and needed manual coordination.

scripts/dev-cargo.sh and scripts/dev-test.sh (through the shared
codewhale_dev_cache_exec_cargo) now run Cargo under an exclusive flock
on /build.lock. A waiting build prints the holder's pid,
directory and command. Cargo runs as a child of the lock holder, never
via exec, so a daemon it starts (an sccache server) cannot inherit the
descriptor and pin the lock. Nested runs skip it through
CODEWHALE_BUILD_LOCK_HELD; CODEWHALE_BUILD_LOCK=0 opts out. Advisory
only; documented in docs/BUILD_PERFORMANCE.md.

Evidence: two concurrent holders serialized (the second waited 2s,
named the first, then ran and propagated exit status 7); a nested call
skipped the lock; codewhale_dev_cache_exec_cargo --version ran under it
and recorded the holder. scripts/dev-cache.test.sh: all 22 checks
passed.

🤖 Generated with Claude Code

Several agents share this checkout on a memory-constrained Mac, and
Cargo's own lock is per target directory, so builds into different
target dirs ran concurrently and needed manual coordination.

scripts/dev-cargo.sh and scripts/dev-test.sh (through the shared
codewhale_dev_cache_exec_cargo) now run Cargo under an exclusive flock
on <cache root>/build.lock. A waiting build prints the holder's pid,
directory and command. Cargo runs as a child of the lock holder, never
via exec, so a daemon it starts (an sccache server) cannot inherit the
descriptor and pin the lock. Nested runs skip it through
CODEWHALE_BUILD_LOCK_HELD; CODEWHALE_BUILD_LOCK=0 opts out. Advisory
only; documented in docs/BUILD_PERFORMANCE.md.

Evidence: two concurrent holders serialized (the second waited 2s,
named the first, then ran and propagated exit status 7); a nested call
skipped the lock; codewhale_dev_cache_exec_cargo --version ran under it
and recorded the holder. scripts/dev-cache.test.sh: all 22 checks
passed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 23, 2026 15:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

The self-hosted macOS runner is also the developer machine, and its
trusted Test job runs Cargo directly, so it still built concurrently
with local agents (observed: `cargo run -p codewhale-tui -- eval` from
the runner beside local work). The Test job now holds the same machine
lock from before the first test build to the end of the job, on the
self-hosted runner only and only when the runner's `.env` sets
CODEWHALE_BUILD_LOCK_FILE; otherwise it logs that it builds unlocked.
The holder is a background process released by an always() step, and
the runner kills orphans at job end, so a cancelled or crashed job
cannot pin the lock. The job exports CODEWHALE_BUILD_LOCK_HELD so
scripts inside it do not wait on their own lock. dev-cargo.sh accepts
the same CODEWHALE_BUILD_LOCK_FILE override.

Evidence: the step script extracted from ci.yml was run locally against
a contended lock: it waited for a running holder (3 s), then a second
build waited while it held the lock and ran once the hold file was
removed; with the variable unset it exits 0 without locking. YAML parses;
dev-cache.sh passes `sh -n`. Hosted acceptance needs a self-hosted run
after the runner's .env is set and the runner restarted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants