[amrit110]

Problem

After warmup, call shapes that warmup didn't cover compiled 1–7 new graphs inside the request that hit them. The timings_ms breakdowns from 0.14.0 identified these shapes:

• tables of about 20k rows or more, which hand the compiled blocks differently strided inputs;
• regression predicts of 1,024+ rows;
• variants produced by overlapping calls on one worker.

A compile costs 5–25 s on an A100 and about 4× that on prod's A40s. While it runs it holds the device, so other models' calls wait too. On prod, a 5,000 × 500 TabICL fit compiled 3 graphs on every call (25.8 s, then 25.7 s on the repeat).

Fix

After warmup, a call that no compiled graph fits runs eagerly instead of compiling one. This uses torch.compiler.set_stance("eager_on_recompile"), which is process-wide and called once per worker at the end of warmup (tabicl_compile.stop_compiling_new_graphs). TABCTX_COMPILE_AFTER_WARMUP=1 restores the old behaviour.

Measured on the devbox (A100, serving config)

Call	This PR	Before (compile in request)
First 20,000 × 100 fit	3.0 s, 0 graphs	12.5 s, 3 graphs
First 1,000-row regression predict	0.14 s	1.8 s, 1 graph
10,000 × 100 fit (warmed family)	1.08 s	1.07 s
20,000 × 100 / 40,000 × 100 / 5,000 × 500 after their compile	2.9 / 6.6 / 3.0 s	2.3 / 5.2 / 2.2 s

Eager costs about 30% on the families warmup misses, compared with a graph that was already compiled. The alternative is a compile stall several times larger, and per worker. I tried widening warmup to cover those families (20,000 × 4 tables, 1,100-row predicts); it didn't cover them, because the stride family depends on the column count too, so I reverted it.

Also: prod_bench.py --log-calls saves every call with its timings_ms.

Tests: unit tests for the stance and the setting. pytest tests/unit: 567 passed. A devbox mixed-load run with worker processes is in progress, and I'll add its numbers here.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 53.33333% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.21%. Comparing base (2c4f3e7) to head (2799c58).

Files with missing lines	Patch %	Lines
src/tabctx/backends/tabicl_compile.py	35.29%	11 Missing ⚠️
src/tabctx/serve/workers.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #67      +/-   ##
==========================================
- Coverage   80.42%   80.21%   -0.22%     
==========================================
  Files          45       45              
  Lines        3454     3483      +29     
==========================================
+ Hits         2778     2794      +16     
- Misses        676      689      +13     

Flag	Coverage Δ
integration	29.74% <10.00%> (-0.17%)	⬇️
unit	76.65% <53.33%> (-0.19%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/tabctx/serve/factory.py	93.14% <100.00%> (+0.22%)	⬆️
src/tabctx/serve/workers.py	70.31% <0.00%> (-0.41%)	⬇️
src/tabctx/backends/tabicl_compile.py	24.19% <35.29%> (+4.19%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amrit110]

Devbox, 2×A100, prod-like Ray Serve with a worker process per GPU (benchmarks/gke/serve_bench.sh)

Not one of the run's 606 calls compiled a graph (n_graphs_compiled). The two earlier runs, before this PR, compiled 2–7 graphs on fits and predicts during mixed load.

	Before (two runs)	This PR
TabICL 5,000 × 500 fit, first / repeat	22.1 / 22.4 s	4.6 / 4.6 s
Mixed load: TabICL fit p95 / max	14.1 / 18.0 s, and 6.8 / 19.2 s	4.9 / 7.9 s
Mixed load: TabICL predict max	3.4 s and 8.9 s	1.9 s
Mixed load: TabPFN fit p95 / max	11.1 / 16.1 s, and 7.1 / 12.1 s	6.4 / 7.0 s

Review follow-ups:

• The stance now flips in serve/factory.finish_warmup once every backend in the process has warmed up, and only where the compiled blocks are installed. This fixes later GPUs in in-process multi-GPU mode.
• The stance test checks behaviour.
• The torch-free tests run in CI.