fix(build): stop two builds fighting over target/<profile>/deps/liboxidex.* - #649
Merged
Conversation
#639 stopped tests/ffi_c_integration.rs's nested `cargo build --lib` from building the DEFAULT feature graph over target/debug/deps/liboxidex.rlib, which is what made `cargo test --all-features` fail with E0432 on both parsers::magika_detector doctests. It forwarded the outer feature graph and recorded that the nested build "resolves to the unit cargo has already compiled, so it writes nothing at all". Measured on `cargo test --all-features -v`, that last part is false. A nested cargo inherits no profile either: it ran under `dev` while `cargo test` builds the lib under `test`, and those are different units here because [profile.test] sets opt-level = 2, codegen-units = 4 against [profile.dev]'s 0 and 16 -- both read off the rustc invocations, which also confirm the lib unit carries no -C extra-filename. So 44s into the FFI test the nested build logged "Compiling oxidex" / "Finished `dev` profile", and target/debug/deps/liboxidex.rlib went from 119,540,352 bytes to 99,273,872 -- the same library, recompiled unoptimised, underneath the doctests about to link it. Benign only because the features now match. A test cannot learn the profile NAME cargo invoked it under, so a nested build cannot be made to match one. Not building is the fix that holds: cargo builds the lib as a dependency of every integration-test target, emitting all three crate types in one rustc call, so the artifacts already exist. Derive the directory from env::current_exe() and link there. It has to be target/<profile>/deps, not the profile root -- cargo uplifts a lib target only when it is a requested target of the invocation, and under `cargo test` it is a dependency. Verified with `cargo clean -p oxidex && cargo test --all-features --no-run`: afterwards deps/ holds all three artifacts and target/debug/ holds none. The nested build was what uplifted them, which is the only reason the old `-L target/debug` ever resolved. Deriving from current_exe() also handles CARGO_TARGET_DIR and --target <triple>, which the hard-coded target/debug never did, and makes the release path link the cdylib that invocation actually built instead of a nested debug one. forwarded_features_cover_every_declared_feature goes with it: it guarded the feature forwarding of a nested build that no longer exists. Verified: - `cargo test --all-features`: exit 0, 0 failed, doctests 221 passed. The SHA-256 of target/debug/deps/liboxidex.rlib is identical before and after the run -- nothing overwrites it now. - FFI suite 45.44s -> 0.69s; the 44.75s was the nested rebuild. - `cargo test --release --all-features --features tag-comparison-binary --test ffi_c_integration`: exit 0, 41 C assertions pass. - `cargo fmt --check` clean; `cargo clippy --all-features --test ffi_c_integration -- -D warnings` exit 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Any --release invocation whose unit graph also contains test targets compiles
src/lib.rs twice:
* once panic=unwind, because cargo forces unwind on test targets and their
dependencies -- the rlib the test binaries link;
* once panic=abort, for the `oxidex`, `tag-comparison` and `jpeg-tag-matrix`
bins. Cargo builds the package's bins alongside its tests, and a bin is a
final artifact, so it reads `panic` from [profile.release].
Measured with `cargo test --release --all-features --features
tag-comparison-binary --test ffi_c_integration --no-run -v`: two
`--crate-type lib` rustc invocations, identical but for `-C panic=abort` on
one, neither carrying -C extra-filename, and all three bins built with
`-C panic=abort`. Because `[lib] crate-type = ["lib", "staticlib", "cdylib"]`
makes cargo drop the filename hash, both units write
target/release/deps/liboxidex.{rlib,a,dylib} -- 4 "output filename collision"
warnings (rust-lang/cargo#6313) and then a race. When abort wins, dependents
fail with "requires panic strategy `abort` ... incompatible with ... `unwind`"
on unchanged source. This is the hazard [profile.bench] already documents,
with [profile.release] as the abort source.
The [profile.bench] cure -- pin panic = "unwind" -- does not transfer, because
[profile.release]'s panic = 'abort' is what shipped binaries want. Scoping the
override to the invocations that need it is the equivalent: the bins those
recipes build are throwaway test-run artifacts. `just build-bin-release`,
`cargo build --release`, the tag-comparison bin recipes and the Release Build
CI job carry no prefix, so released binaries still abort.
With the prefix the same command reports 0 collision warnings and a single lib
unit, and all three bins build unwind.
CI is unaffected either way: it runs no --release test step (`cargo nextest run
--all-features` and `cargo test --all-features` are both test-profile), which
is also why this never showed up as a red build -- only as an intermittent
local `just test`.
Not fixed here: the root enabler is the dropped -C extra-filename, and the only
complete cure is moving the C ABI into its own crate so `oxidex` can go back to
crate-type = ["lib"]. That touches cbindgen, the committed header, the python
bindings and the packaging assets, and CI cannot validate it, so it wants its
own change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two builds were fighting over
target/<profile>/deps/liboxidex.*. Both are thesame bug class:
[lib] crate-type = ["lib", "staticlib", "cdylib"]makes cargodrop
-C extra-filename, so every configuration of the lib shares one set offilenames and whichever writer finishes last wins.
1. The C FFI test's nested build (finishes #639)
#639 stopped
tests/ffi_c_integration.rs's nestedcargo build --libfrombuilding the default feature graph over the rlib the doctests link — the
E0432 on both
parsers::magika_detectordoctests. It forwarded the featuregraph and recorded that the nested build now "writes nothing at all".
Measured on
cargo test --all-features -v, that part is false. A nested cargoinherits no profile either: it ran under
devwhilecargo testbuilds thelib under
test, which are different units here ([profile.test]opt-level = 2, codegen-units = 4vs[profile.dev]'s0/16, read off therustc invocations). 44s into the FFI test it logged
Compiling oxidex/Finished `dev` profile, andtarget/debug/deps/liboxidex.rlibwent119,540,352 → 99,273,872 bytes underneath the doctests.
A test can't learn the profile name cargo invoked it under, so a nested build
can't be made to match one. It doesn't need to build at all: cargo builds the
lib as a dependency of every integration-test target, all three crate types in
one rustc call. The test now derives its link directory from
env::current_exe()and links what's already there.It has to be
target/<profile>/deps, not the profile root — cargo uplifts a libtarget only when it's a requested target. After
cargo clean -p oxidex && cargo test --all-features --no-run,deps/holds allthree artifacts and
target/debug/holds none; the nested build was doing thatuplift, which is the only reason the old
-L target/debugresolved.2.
cargo test --releaseraces two lib unitsAny
--releaseinvocation whose graph also contains test targets compilessrc/lib.rstwice: unwind (cargo forces it on test targets and their deps) andabort (the three bins are final artifacts reading
panicfrom[profile.release]). Measured withcargo test --release --all-features --features tag-comparison-binary --test ffi_c_integration --no-run -v:two
--crate-type libinvocations identical but for-C panic=abort, neithercarrying
-C extra-filename, 4output filename collisionwarnings(rust-lang/cargo#6313), then a race. When abort wins, dependents fail with
"requires panic strategy
abort… incompatible with …unwind" on unchangedsource. This is what the
[profile.bench]comment already documents, with[profile.release]as the abort source.That cure — pin
panic = "unwind"— doesn't transfer:[profile.release]'spanic = 'abort'is what shipped binaries want. So the override is scoped tothe invocations that need it (
unwindin the justfile). The bins those recipesbuild are throwaway test-run artifacts;
just build-bin-release,cargo build --release, the tag-comparison bin recipes and the Release Build CIjob carry no prefix, so released binaries still abort. With the prefix the
same command reports 0 warnings and a single lib unit.
CI is unaffected either way — it runs no
--releasetest step, which is whythis never showed as a red build, only as an intermittent local
just test.Not fixed here
The root enabler is the dropped
-C extra-filename. The only complete cure ismoving the C ABI into its own crate so
oxidexcan return tocrate-type = ["lib"]; that touches cbindgen, the committed header, the pythonbindings and the packaging assets, and CI can't validate it. Wants its own change.
Verification
cargo test --all-features(post-merge with origin/main): exit 0, 0 failures,doctests 221 passed / 0 failed. On a warm re-run the SHA-256 of
target/debug/deps/liboxidex.rlibis identical before and after — nothingoverwrites it now.
cargo test --release --all-features --features tag-comparison-binary --test ffi_c_integration:exit 0, 41 C assertions pass, now linking the release cdylib that invocation
actually built.
cargo fmt --checkclean;cargo clippy --all-features --test ffi_c_integration -- -D warningsexit 0.just --dry-run testshows the prefix expanded; shipping recipes verified unprefixed.🤖 Generated with Claude Code