Force test executables to rerun past delta-build caching - #1213
Merged
Conversation
Delta builds skip actually running a test executable when nothing changed since its last build, reusing the previous run's cached result. That's the right call for an ordinary rebuild, but it quietly defeats :unity ↳ :shuffle_tests: shuffled test-case order is decided at runtime inside the compiled executable itself, not at compile or link time, so an unchanged executable that never runs again also never produces a new order. Test execution is now forced whenever shuffling is enabled, so the feature actually does something past the very first build. The same override is available on demand via a new --force-test-rerun flag, for anyone who wants fresh results regardless of caching.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Delta builds skip actually running a test executable when nothing changed since its last build, reusing the previous run's cached result. That's correct for an ordinary rebuild, but it quietly defeats
:unity↳:shuffle_tests: shuffled test-case order is decided at runtime inside the compiled executable itself, not at compile or link time, so an unchanged executable that never runs again also never produces a new order.:unity↳:shuffle_testsis enabled, so the feature actually does something on any build past the very first.--force-test-rerunbuild flag, for anyone who wants fresh results regardless of caching, independent of shuffling.Test plan
spec/units/test_build_executor_spec.rb(#stage_execute) covers both trigger paths, plus the existing unchanged/cached-result behavior.spec/units/bin/cli_helper_spec.rb(#process_force_test_rerun) covers the flag's no-op/raise/pass conditions, mirroring#process_testcase_filters's existing validation shape.spec/system/delta_builds_spec.rbnew scenarios confirm both the CLI flag and:shuffle_testscause a test executable to actually rerun on an otherwise fully-cached rebuild (no recompiling/relinking, butRunningstill occurs), and the full file's pre-existing scenarios still pass with no regressions.