fix(reproduce): scale default dataloader workers to CPU count, not OS - #181
Open
skywalker-lt wants to merge 1 commit into
Open
fix(reproduce): scale default dataloader workers to CPU count, not OS#181skywalker-lt wants to merge 1 commit into
skywalker-lt wants to merge 1 commit into
Conversation
…st OS eff7e3e set --workers to 0 whenever the host is Windows. That disables the worker pool outright: build_dataloader computes nw = min(cpu_count // nd, workers), so 0 stays 0 and every batch is decoded and augmented inline on the training process. Mosaic/resize no longer overlap the GPU step, so Windows users silently get materially slower epochs with no warning -- the only hint is a help string. The stated reason was multiprocessing deadlocks, but the usual cause on Windows is a missing entry-point guard for spawn, and all three reproduce scripts already have 'if __name__ == "__main__":'. Replace the OS branch with default_workers(), which scales to the machine: half the visible cores, capped at 16, floored at 1. Worker cost tracks cores and memory rather than platform, so this also drops the 'import platform' the branch introduced; hard-coded per-OS constants go stale as machines change. cores 1 2 4 8 16 32 64 workers 1 1 2 4 8 16 16 A >=32-core Linux host still gets 16, matching the previous default, so existing reproduction runs are unaffected. Windows now gets a real pool sized to the laptop instead of 0. --workers 0 remains available for any box that genuinely stalls, and the help text says so.
skywalker-lt
marked this pull request as draft
July 28, 2026 15:08
skywalker-lt
marked this pull request as ready for review
July 28, 2026 15:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
PR #174 made
--workersdefault to0whenever the host is Windows with the intention to avoid I/O deadlocks. However,build_dataloadercomputesnw = min(os.cpu_count() // nd, workers), so0stays0and every batch is decoded and augmented inline on the training process. The augmentation no longer overlap the GPU step, hence Windows users get materially slower epochs even with powerful CPUs.🛠️ Fix
Replace the OS-branch with
default_workers(), sized to the machine:Now worker cost tracks core count and memory rather than the operating system.
📁 file changed
scripts/reproduce/_reproduce_common.py