Skip to content

Warm predictor: torch.compile + GPU-resident export - #122

Merged
aperson30 merged 1 commit into
BodyMaps:mainfrom
aperson30:perf/warm-compile-gpuexport
Aug 4, 2026
Merged

Warm predictor: torch.compile + GPU-resident export#122
aperson30 merged 1 commit into
BodyMaps:mainfrom
aperson30:perf/warm-compile-gpuexport

Conversation

@aperson30

@aperson30 aperson30 commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Two speed additions to the persistent LesionSegmenter warm service (lesionseg_warm_server.py), both of which only make sense for a warm/persistent process.

torch.compile (env-gated, LESIONSEG_COMPILE, default off)

Compiles the patch-level network once at startup (max-autotune-no-cudagraphs). The sliding-window patch is a constant [128,224,224], so one compile serves every patch of every case.

  • 1.18× on inference, accuracy-safe (voxel agreement 0.99996, lesion Dice 0.999 vs eager — measured over 20 runs on an idle node with zero latency stalls).
  • The one-time compile (~20s–3min) is paid at startup behind the health gate: while it runs, the health check is unreachable and lesionseg_predict.py uses the cold path, so the site keeps working.
  • Env-gated so it can be dropped without a redeploy if a node shows compile-latency variance (seen on some shared GB10 nodes). The warm predictor’s main win — cold-start elimination — is independent of compile.

GPU-resident export (default for softmax-argmax models)

predict_logits returns logits on the CPU, so convert_logits moves the ~1.5 GB, 43-channel volume CPU→GPU for the (GPU-patched) resample and back. This runs the same resample, spacings, and argmax GPU-resident and returns only the uint8 label map.

Deploy notes (not automatic)

Going live also requires, on the server: starting the warm service, and setting LESIONSEG_WARM_URL in .env so lesionseg_predict.py routes to it (with automatic cold-path fallback if it is down/busy/mismatched). No frontend/DB change.

Testing

Verified the service starts, compiles (17.7s on an idle node), and listens; the export logic is identical to #121 which was validated bit-exact end-to-end.

Two speed additions to the persistent LesionSegmenter service, both of which only
make sense for a warm/persistent process:

torch.compile (gated by LESIONSEG_COMPILE, default off): compiles the patch-level
network once at startup with max-autotune-no-cudagraphs. The sliding-window patch is
a constant [128,224,224], so one compile serves every patch of every case. Measured
1.18x on inference, accuracy-safe (voxel agreement 0.99996, lesion Dice 0.999 vs
eager). The ~20s-3min compile is paid once at startup, behind the health gate -- while
it runs, lesionseg_predict.py falls back to the cold path so the site keeps working.
Env-gated so it can be dropped without a redeploy if a node shows the compile-latency
variance seen on some shared GB10 nodes; the warm predictor's main win (cold-start
elimination) is independent of it.

GPU-resident export (default for softmax-argmax models): predict_logits returns logits
on the CPU, so convert_logits moves the ~1.5 GB, 43-channel volume CPU->GPU for the
(GPU-patched) resample and back. This runs the same resample, spacings and argmax
GPU-resident and returns only the uint8 label map. Verified 0 differing voxels vs
convert_logits across a size-stratified sample incl. anisotropic cases; 3.6-25x on the
export stage. Region-based models fall back to convert_logits.

Verified the service starts, compiles, and listens; export logic matches PR BodyMaps#121
(shipped) which was validated bit-exact.
@aperson30
aperson30 merged commit 94ce02a into BodyMaps:main Aug 4, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant