Repository navigation
Fixed 10–30 s stalls on fit and predict across table sizes (recompiles?) #47
Description
Found in the prod A40 sweep on 2026-10-20 (benchmarks/sweep_admission.py, direct to tabctx 0.12.0, 4× A40). These costs don't scale with the table, so they hurt small tables most.
| Symptom | Evidence |
|---|---|
| TabPFN fit, about 30 s fixed cost | tabpfn-3.5 10×100: 30.3 s; 10×250: 30.3 s; 10×500: 0.65 s. At 10×20,000, the first shape on the CSV upload path, fit jumps from 5.6 s (15,000, JSON) to 37.8 s |
| TabICL predict, about 24 s fixed cost | tabicl-v2 at 500 features: 0.49 s at 2,000 rows, then 24.0 / 26.2 / 23.8 / 25.1 / 23.9 / 25.6 / 24.4 / 24.2 s from 3,500 to 45,000 rows. Server-side time from the Serve log (predict 200 23961.9ms). Fit times are normal |
| TabICL predict at 100,000×10 | 10.4 s, against about 0.2 s up to 65,000 |
| Dynamo recompile limit | Worker stderr: torch._dynamo hit config.recompile_limit (8) at the start of the sweep, last reason: expected type of 'k' to be a tensor type, but found NoneType in tabicl forward. After the limit the frame runs uncompiled |
Ruled out: host-memory spill (0 spilled contexts), the gateway (direct calls), client-side time.
Hypotheses to test
- A recompile on each new shape (
dynamic=False), or a guard failure onk=Noneversus a tensor, rebuilding the compiled block. That fits a flat cost that doesn't scale. - TabPFN: the first call per (task, preprocessing) path compiles or warms up something not covered by
warmup(), and the CSV path takes a different dtype or layout (float64 from the CSV parser versus float32 from JSON?) that misses the warm cache. - TabICL predict above a context size takes a different path (offload, or a fallback after the recompile limit).
Plan
- Reproduce on the GKE devbox with
TORCH_LOGS=recompiles,graph_breaksand per-stage timings (GPU profiling harness: per-stage time and peak-memory breakdown (GKE first, then A40) #31): the same shape sequence, and a repeated predict on one context. - Fix the cause: dynamic shapes or bucketed shapes, a guard-stable signature, warmup over the paths actually served, and the same dtype on both input paths.
- Accept: no fixed stall above 1 s at any size in the sweep's shapes, for both models; a recompile count that stays bounded over a long mixed run; and no regression in small-table p50.
Activity
[-]P0 latency: fixed 10–30 s stalls on fit and predict (recompiles?) across table sizes[/-][+]Fixed 10–30 s stalls on fit and predict across table sizes (recompiles?)[/+]fix(predict): size predict chunks with a measured predict cost model (#…
fix(compile): apply dynamo settings per call -- serving threads saw a…
fix(compile): apply dynamo settings per call -- serving threads saw a…
fix(compile): never compile TabICL graphs inside a request after warm…
release: 0.14.1 (#68)
amrit110 commentedon Oct 21, 2026
Closing: the stalls inside tabctx are fixed and verified on prod. The remaining stalls are outside tabctx, tracked in aieng-proxy-gateway#472.
timings_ms, per-request timing breakdowns, and showed that TabICL still compiled graphs inside requests. A 5,000 × 500 fit compiled 3 graphs on every call, taking 25.8 s.On prod with 0.14.1 (
benchmarks/baselines/prod-bench-0.14.1.json):On the devbox (2×A100, worker processes), none of the run's 606 calls compiled a graph.
The remaining 45–230 s stalls are all in the gateway-to-cluster hop, on large request bodies only. Measured with the gateway's new
Server-Timing(aieng-proxy-gateway#475).