Skip to content

Stop the macOS binary printing a traceback on every diff - #201

Merged
ESultanik merged 2 commits into
masterfrom
tqdm-multiprocessing-lock
Sep 16, 2026
Merged

ESultanik merged 2 commits into
masterfrom
tqdm-multiprocessing-lock

Conversation

@ESultanik

Copy link
Copy Markdown
Collaborator

Every diff through the macOS binary built from master writes a traceback to stderr. It is visible in the
Build binary artifacts dry run on master (run 35127349018),
immediately after the correct diff output:

ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'

The smoke test passed anyway, because it only checks the exit status.

Root cause

Not numpy. graphtage/__main__.py:357 constructs the first progress bar, and tqdm builds a
multiprocessing.RLock for its default write lock:

__main__.py:357   printer.tqdm(...)
progress.py:60    tqdm(*args, **kwargs)
tqdm/std.py:770   cls._lock = TqdmDefaultWriteLock()
tqdm/std.py:119   cls.mp_lock = RLock()
resource_tracker.py:216   self.ensure_running()

Registering that lock's semaphore starts multiprocessing.resource_tracker, which re-executes sys.executable
with the interpreter's own flags. Under PyInstaller sys.executable is the Graphtage binary, so the helper re-runs
Graphtage with -B -S -I. Locally that surfaces as graphtage: error: unrecognized arguments: -B -S -I; on the CI
runner the re-execution fails earlier, importing numpy in isolated mode.

Two things make this new and macOS-only:

  • macOS spawns where Linux forks, so only macOS re-executes at all. The Linux binary is clean.
  • graphtage/__init__.py now imports batch_distance, which imports numpy at module scope, so the re-executed
    process had numpy on its import path for the first time. v0.4.0's binary does not show this.

The diff itself was never wrong. All the noise came from the helper.

Fix

Graphtage never uses multiprocessing, so the lock guards nothing. graphtage/progress.py installs a threading lock
at import, which keeps tqdm from building the multiprocessing one.

Measured on a rebuilt macOS binary

Invocation stderr before stderr after exit
--no-status --format yaml (the smoke test's 2-key diff) 1948 bytes 0 1
--no-status --no-color on a 45-key diff (reaches the numpy kernel) 1948 bytes 0 1

The rendered diff is byte-identical to the source install before and after, and identical under both
GRAPHTAGE_BATCH_BACKEND settings. --version and -dumpversion were already clean.

Testing

test_progress.py gains test_drawing_a_progress_bar_starts_no_helper_process, which asserts the resource tracker
never starts. Its four existing siblings cannot catch this: from a source checkout the helper is a real interpreter
and exits quietly, so stderr stays empty either way. Verified the new test fails without the fix.

The second commit closes the gap that let this ship undetected: the binary smoke test now fails if --no-status
writes anything to stderr, which is the contract test_no_status_suppresses_progress already holds the source
install to.

Full suite: 273 passed, 1770 subtests. Ruff and actionlint clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F2sHz5c5TvMs9tFn2HhwaC

ESultanik and others added 2 commits September 16, 2026 13:58
Every diff through the macOS binary printed a traceback to stderr:

    ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'

tqdm builds a multiprocessing.RLock for its default write lock the first time a
progress bar is constructed, at __main__.py:357. Registering that lock's
semaphore starts multiprocessing.resource_tracker, which re-executes
sys.executable with the interpreter's own flags. Under PyInstaller
sys.executable is the Graphtage binary, so the helper re-ran Graphtage with
`-B -S -I`, and that re-execution failed importing numpy in isolated mode.

Only macOS was affected, because it spawns rather than forks. Only 0.5.0 showed
it, because graphtage/__init__.py now imports batch_distance, which imports
numpy at module scope, so the re-executed process had numpy on its import path
for the first time. The diff itself was always correct; the noise came entirely
from the helper.

Graphtage never uses multiprocessing, so the lock guards nothing. Installing a
threading lock at import keeps tqdm from building the multiprocessing one.

Measured on a rebuilt macOS binary, for both the smoke test's 2-key diff and a
45-key diff large enough to reach the numpy kernel: stderr goes from 1948 bytes
to 0, exit statuses are unchanged, and the rendered diff stays byte-identical to
the source install.

The existing tests in test_progress.py cannot catch this, because from a source
checkout the helper is a real interpreter and exits quietly. The new test
asserts the tracker never starts. Verified it fails without the fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2sHz5c5TvMs9tFn2HhwaC
The smoke test checked only the exit status, so the macOS binary shipped a
traceback on every diff while the step stayed green: the diff was correct and
the traceback came from a helper process.

`--no-status` promises an empty stderr, and test_progress.py already asserts
that for a source checkout. Hold the frozen binary to the same contract, where
re-executing sys.executable is a live hazard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2sHz5c5TvMs9tFn2HhwaC
@ESultanik
ESultanik merged commit 291ca75 into master Sep 16, 2026
14 checks passed
@ESultanik
ESultanik deleted the tqdm-multiprocessing-lock branch September 16, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant