Runtime JIT compilation contaminates published TTFT and TPOT measurements

[simone-chen]

Describe the bug

InferenceX can publish TTFT and TPOT measurements while workers are still compiling kernels. These rows mix cold-start compilation with steady-state inference, but the DB dump does not mark them as contaminated.

The latest stable dump still shows the problem:

• Release: db-dump/2026-08-31
• Published: 2026-08-31
• 83,561 benchmark rows; 8,342 latest single-turn operating points
• 357 strong-signal TTFT-tail candidates and 17 strong-signal TPOT-tail candidates
• 193 TTFT and 1 TPOT candidates recur enough to be high-priority checks
• The candidate set is unchanged from db-dump/2026-08-24

Tail statistics alone do not prove JIT. The MiniMax M3 example below is
log-confirmed; the DeepSeek-R1 example is a persistent regression correlated
with warmup/cache changes.

Reproduction: log-confirmed MiniMax M3

Workflow: run 30933463657, attempt 2

Artifact: multinode_server_logs_minimaxm3_8k1k_fp4_dynamo-vllm_prefill-tp4-pp1-dcp1-pcp1-ep1-dpfalse-nw1_decode-tp4-pp1-dcp1-pcp1-ep1-dpfalse-nw6_disagg-true_spec-mtp_conc12x24x48_b200-dgxc_07

Configuration:

• DB config ID 2202; benchmark IDs 438518/438520/438519
• MiniMax M3 FP4, B200, Dynamo-vLLM, MTP/EAGLE3
• ISL 8192, OSL 1024
• One TP4 prefill worker and six TP4 decode workers
• C12 process: 18:38:17–18:40:37 UTC
• Unmeasured probe: 18:38:32.17–18:38:42.79 UTC
• Main C12 request burst begins at 18:38:42.81 UTC
• The server reports enable_jit_warmup=True

Search the worker logs for JIT compilation during inference. There are 50
warnings: 37 Triton and 13 CuTeDSL. Thirty-four occur after the main C12
request burst begins.

All log-confirmed JIT sites

Worker	Role / host	Total	During main C12	Compiled targets
prefill_w0	prefill / gpu-2-0	2	0	eagle_prepare_next_token_padded_kernel, FlashAttentionForwardSm100
decode_w0	decode / gpu-2-0	8	8	all eight targets below
decode_w1	decode / gpu-2-6	8	0	all eight targets compile during the unmeasured probe
decode_w2	decode / gpu-2-6	8	8	all eight targets below
decode_w3	decode / gpu-2-9	8	8	all eight targets below
decode_w4	decode / gpu-2-9	8	2	six before and two during main C12
decode_w5	decode / gpu-2-4	8	8	all eight targets below

Compiler	Target	Warning count
Triton	_gqa_sparse_decode_kernel	6
Triton	_pack_topk_ids_weights_kernel	6
Triton	eagle_prepare_next_token_padded_kernel	7
Triton	eagle_step_slot_mapping_metadata_kernel	6
Triton	rejection_greedy_sample_kernel	6
Triton	eagle_prepare_inputs_padded_kernel	6
CuTeDSL	FlashAttentionForwardSm100	7
CuTeDSL	FlashAttentionForwardCombine	6

Correlating frontend routing/completion logs with each worker's JIT timestamps
identifies all ten first-batch requests whose assigned decode worker compiled
before their first token. The two requests assigned to already-warmed
decode_w1 are shown as controls.

Request	Worker	TTFT	Avg ITL	Assessment
d201ac38-61c1-4e7c-bf6d-341e8a354d4d	decode_w0	6,896.00 ms	4.73 ms	JIT overlap
07159dbd-c125-4a6f-a0fd-2804c29cd77d	decode_w3	6,893.38 ms	7.44 ms	JIT overlap
fba4e068-fb99-4d7f-b9c5-870797482580	decode_w5	6,458.97 ms	4.33 ms	JIT overlap
32b926ae-671e-477f-9c47-a93e768689a9	decode_w2	11,895.40 ms	27.53 ms	JIT overlap
6b003fe1-25f4-4a80-ac7c-7cbbc049cca6	decode_w4	5,876.69 ms	5.24 ms	JIT overlap
3d6b088b-e48d-41fd-a597-f4cd19becd62	decode_w0	7,998.77 ms	4.17 ms	JIT overlap
2ba6e7fa-b2d1-4e2d-88eb-7fb1dfe214c1	decode_w3	7,993.33 ms	3.41 ms	JIT overlap
e8c4cf15-f3ff-4cb6-88da-d498c5a02a76	decode_w5	7,441.74 ms	3.53 ms	JIT overlap
093f21d7-c835-4918-a69b-9af4ad68aa62	decode_w2	10,812.43 ms	28.18 ms	JIT overlap
10d773a7-0625-42ee-a015-f6edab51ec04	decode_w4	6,553.20 ms	3.26 ms	JIT overlap
6e34c4b3-8d71-4cc3-aba4-32157afe08e5	decode_w1	1,294.30 ms	6.75 ms	No measured JIT
7c42eb3a-163c-40dc-94a2-36e37356d777	decode_w1	2,701.49 ms	5.12 ms	No measured JIT

The confirmed victim is the C12 benchmark record and its TTFT, TPOT, ITL, and
E2EL distributions.

C12 metric	Median	Mean	P99
TTFT	383.38 ms	1,091.96 ms	10,290.33 ms
TPOT	3.28 ms	8.41 ms	48.36 ms
ITL	105.48 ms	268.49 ms	1,808.31 ms
E2EL	3,367.95 ms	8,692.52 ms	41,801.65 ms

C24 and C48 also have TPOT tails, but no later JIT warning appears. They should
not be attributed to JIT without trace evidence.

Reproduction: persistent DeepSeek-R1 TTFT victim

Configuration: DSR1 FP4, B200, SGLang, ISL 1024, OSL 8192, TP4, PP1,
attention DP1, MoE EP1, MoE TP4.

Concurrency	TTFT median	TTFT mean	TTFT P99	Flagged history
32	722 ms	7.88 s	75.26 s	26/31 runs
64	631 ms	7.29 s	69.23 s	27/32 runs
128	912 ms	9.18 s	86.71 s	29/33 runs

The older server artifacts have expired, so this case is probable rather than
log-confirmed. The regression starts immediately after:

• dd7e3850 removed explicit DSR1 JIT warmup.
• 15b7c00b removed the persistent FlashInfer cache.
• 17d5e204 changed the model to the V2 name; the launch script still matched only the old name, reducing 1k/8k measured requests from CONC * 20 to CONC * 10.

Before the changes, C32 TTFT was 464 ms mean / 1.11 s P99. It became 3.77 s /
68.92 s on 2025-10-07 and remains 7.88 s / 75.26 s in the latest selected row.

Expected behavior

• Warmup covers every prefill/decode worker and every measured shape/config.
• Measurement starts only after all workers complete warmup.
• A runtime-JIT warning after the measurement boundary invalidates or clearly
  flags the affected benchmark row.
• The dump exports JIT events or a contamination flag before Actions artifacts
  expire.
• Dashboard means exclude or visibly mark contaminated samples.

Acceptance criteria

• Rerunning MiniMax M3 config 2202 produces no runtime-JIT warning after the measurement boundary.
• Rerunning the DSR1 configuration removes the recurring C32/C64/C128 TTFT cold tail.
• CI fails or marks a benchmark invalid when Triton, CuTeDSL, FlashInfer, or another monitored compiler runs during measurement.
• DB exports retain the event type, target, worker, timestamp, and affected benchmark ID.
• C24/C48 MiniMax M3 TPOT tails are traced separately instead of being assumed to be JIT.