Warm predictor: torch.compile + GPU-resident export - #122
Merged
Conversation
Two speed additions to the persistent LesionSegmenter service, both of which only make sense for a warm/persistent process: torch.compile (gated by LESIONSEG_COMPILE, default off): compiles the patch-level network once at startup with max-autotune-no-cudagraphs. The sliding-window patch is a constant [128,224,224], so one compile serves every patch of every case. Measured 1.18x on inference, accuracy-safe (voxel agreement 0.99996, lesion Dice 0.999 vs eager). The ~20s-3min compile is paid once at startup, behind the health gate -- while it runs, lesionseg_predict.py falls back to the cold path so the site keeps working. Env-gated so it can be dropped without a redeploy if a node shows the compile-latency variance seen on some shared GB10 nodes; the warm predictor's main win (cold-start elimination) is independent of it. GPU-resident export (default for softmax-argmax models): predict_logits returns logits on the CPU, so convert_logits moves the ~1.5 GB, 43-channel volume CPU->GPU for the (GPU-patched) resample and back. This runs the same resample, spacings and argmax GPU-resident and returns only the uint8 label map. Verified 0 differing voxels vs convert_logits across a size-stratified sample incl. anisotropic cases; 3.6-25x on the export stage. Region-based models fall back to convert_logits. Verified the service starts, compiles, and listens; export logic matches PR BodyMaps#121 (shipped) which was validated bit-exact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two speed additions to the persistent LesionSegmenter warm service (
lesionseg_warm_server.py), both of which only make sense for a warm/persistent process.torch.compile (env-gated,
LESIONSEG_COMPILE, default off)Compiles the patch-level network once at startup (
max-autotune-no-cudagraphs). The sliding-window patch is a constant[128,224,224], so one compile serves every patch of every case.lesionseg_predict.pyuses the cold path, so the site keeps working.GPU-resident export (default for softmax-argmax models)
predict_logitsreturns logits on the CPU, soconvert_logitsmoves the ~1.5 GB, 43-channel volume CPU→GPU for the (GPU-patched) resample and back. This runs the same resample, spacings, and argmax GPU-resident and returns only the uint8 label map.convert_logits(0 differing voxels across a size-stratified sample incl. anisotropic cases); 3.6–25× on the export stage. Same logic as the already-merged LesionSegmenter: GPU-resident export (no CPU round trip) #121.convert_logits.Deploy notes (not automatic)
Going live also requires, on the server: starting the warm service, and setting
LESIONSEG_WARM_URLin.envsolesionseg_predict.pyroutes to it (with automatic cold-path fallback if it is down/busy/mismatched). No frontend/DB change.Testing
Verified the service starts, compiles (17.7s on an idle node), and listens; the export logic is identical to #121 which was validated bit-exact end-to-end.