Colab-ready GPU bootstrap and validation for ArtP and DArtP

[jsalsman]

Summary

This PR adds a reproducible, end-to-end GPU bootstrap and smoke-test workflow for PathBench’s ArtP and DArtP evaluators, including support for fresh Google Colab T4 runtimes.

The new tools/test_gpu_predictors.py helper:

• optionally installs the required Ubuntu/Debian build packages;
• builds and installs the pinned espeak-ng revision (2ea41210);
• verifies the espeak-ng executable, shared library, runtime behavior, and installation marker;
• creates and reuses an isolated Python virtual environment;
• installs pinned CUDA-enabled PyTorch and torchaudio packages;
• verifies that PyTorch can access the NVIDIA GPU;
• installs PathBench, pytest, KenLM, and the remaining test dependencies;
• runs focused ArtP and DArtP smoke tests;
• uses a headless Matplotlib backend in notebook environments;
• optionally retrieves the English KenLM model from the existing Zenodo archive using HTTP byte ranges, avoiding a download of the complete 35 GB archive;
• validates the downloaded model using its compressed and expanded sizes, ZIP CRC-32, SHA-256, and KenLM;
• reuses existing verified native dependencies, Python packages, and model files on subsequent runs;
• retains the previous warning-and-skip behavior when the optional language-model download is not requested.

The helper makes no host changes by default. Native package installation and the espeak-ng build require the explicit --install-system-dependencies option.

NVIDIA driver handling is kept separate and safe:

• Google Colab and other provider-managed environments use their existing host driver;
• driver replacement is refused in Colab, containers, WSL, virtual machines, and other GPU-passthrough environments;
• an experimental bare-metal Ubuntu workflow provides a non-destructive preflight;
• actual driver installation requires a separate confirmation flag, uses Ubuntu’s recommended signed package, and stops with a reboot-required message.

The dependency configuration uses the published phonemizer-fork==3.3.2 release rather than an inaccessible Git repository.

Documentation and network-free tests cover:

• native setup opt-in behavior;
• Ubuntu/Debian platform and privilege detection;
• package installation and failure propagation;
• pinned espeak-ng installation and reuse;
• safe NVIDIA driver preflight and managed-environment refusal;
• CUDA/PyTorch setup;
• range-based ZIP extraction;
• archive metadata and response validation;
• interrupted requests and retries;
• disk-space checks;
• CRC-32 and SHA-256 failures;
• model validation and cache reuse.

An opt-in live test verifies that the expected English model remains available at the pinned location in the Zenodo archive.

Google Colab validation

Proof

https://colab.research.google.com/drive/1byutWFQaM67evO53oMblQMzPCjMNMTln

The notebook uses a single input cell on a fresh T4 runtime to:

1. install uv and Python 3.12;
2. clone PathBench and select the tested revision;
3. invoke the helper with --install-system-dependencies;
4. install the native Ubuntu prerequisites;
5. build and validate the pinned espeak-ng;
6. create the CUDA-enabled Python environment;
7. download and verify the English KenLM model;
8. verify access to the Tesla T4;
9. run both ArtP and DArtP smoke tests; and
10. report the final repository, Python, PyTorch, CUDA, GPU, espeak-ng, and model configuration.

Validated configuration:

• Python 3.12
• T4 GPU
• PyTorch 2.6.0+cu124
• PyTorch CUDA runtime 12.4
• pinned espeak-ng commit: 2ea41210
• wiki_en_token.arpa.bin size: 14,600,342,241 bytes

Expected final smoke-test result:

• ArtP: passed
• DArtP: passed
• focused GPU suite: 2 passed

This upstream PR consolidates the work developed through PRs #1–#7 in the fork.

[jsalsman]

$ python3.12 tools/test_gpu_predictors.py --help
usage: test_gpu_predictors.py [-h] [--python PYTHON]
                              [--pytorch-version PYTORCH_VERSION]
                              [--cuda-version CUDA_VERSION] [--venv VENV]
                              [--download-language-model]
                              [--language-model-url LANGUAGE_MODEL_URL]
                              [--language-model-sha256 LANGUAGE_MODEL_SHA256]
                              [--language-model-cache LANGUAGE_MODEL_CACHE]
                              [--force-language-model-download]

Create a CUDA-enabled PathBench virtual environment, verify GPU access, and
run the ArtP and DArtP smoke tests.

options:
  -h, --help            show this help message and exit
  --python PYTHON       Python 3.10-3.12 executable (default: PYTHON or
                        python3)
  --pytorch-version PYTORCH_VERSION
                        matching PyTorch and torchaudio version (default:
                        2.6.0)
  --cuda-version CUDA_VERSION
                        CUDA wheel version, such as 12.4 (default: 12.4)
  --venv VENV           virtual-environment destination (default:
                        tools/gpu_venv)
  --download-language-model
                        download and install the English DArtP model (large;
                        opt in)
  --language-model-url LANGUAGE_MODEL_URL
                        model/archive URL (PATHBENCH_LANGUAGE_MODEL_URL)
  --language-model-sha256 LANGUAGE_MODEL_SHA256
                        SHA-256 for a custom URL
                        (PATHBENCH_LANGUAGE_MODEL_SHA256)
  --language-model-cache LANGUAGE_MODEL_CACHE
                        download cache (PATHBENCH_LANGUAGE_MODEL_CACHE;
                        default: ~/.cache/pathbench)
  --force-language-model-download
                        discard a cached artifact and download it again

[jsalsman]

P.S. The credential-free packaging changes in this PR are separate from—but supportive of—the GPU/Colab bootstrap work:

• pyproject.toml replaces the Git-hosted phonemizer-fork and pyctcdecode dependencies with pinned PyPI releases. This allows fresh installations, including those created by the GPU helper, to resolve dependencies without GitHub credentials or direct VCS access.

• .github/workflows/tests.yml adds a dependency-resolution check that performs a clean pip dry run and fails if any runtime dependency still resolves to a Git or GitHub URL. It is intended as a regression check for the new credential-free installation guarantee; it does not add GPU CI.

• tests/test_string_clean.py verifies that the published phonemizer-fork package retains the import API used by pathbench/string_clean.py. It also contains an opt-in test for the language-specific IPA output produced with the pinned espeak-ng revision.

• docs/installation.rst and the relevant README.md section update the installation documentation to reflect that Python dependencies now come from package-index releases, while espeak-ng remains a separately pinned system dependency.

These files were therefore added together as one packaging compatibility change. They are not required to implement the GPU helper itself, but they ensure that the helper—and ordinary PathBench installations—can install without relying on GitHub-hosted Python dependencies. If maintainers would prefer to keep this PR narrowly focused on GPU bootstrap, I’m happy to split these packaging changes into a separate PR.

[karkirowle]

Thanks for your PR.

I briefly looked at this and I will need more time to look at it. However, I do agree that having a Colab would be nice for
I think the two things that these PR tries to solve is the fork pull that breaks the PyPi publication, and including a Colab. It would be nicer to separate these.

For the Colab link, I don’t see any immediate demonstrations that a result is reproduced with the unit tests?
For the phonemizer fork, I think the reason why I included it is a caching error in phonemizer, not the espeak-ng error, which depends on the espeak-ng version. That caching error might not be triggered by a Colab-level testing.