From 1296ac66fbddb0bad6080ab8ad169ba14fb606c3 Mon Sep 17 00:00:00 2001 From: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> Date: Thu, 6 Aug 2026 09:57:29 -0700 Subject: [PATCH] [nvbugs/6567554][fix] Fix layer-wise benchmark GEN phase for DeepSeek-V4 The GEN phase aborted in attn_metadata.prepare() with a C++ assertion "Request ID not found in IndexMapper". Three independent defects in the layer-wise benchmark harness contributed: 1. create_kv_cache_manager budgeted max_tokens from round_up(max_seq_len, tokens_per_block), which under-counts what KVCacheManagerV2 reserves per request: max_blocks_per_seq leaves room for one extra decode token and pads to a multiple of 4 blocks for the copy_block_offsets kernel, and the derived quota is divided by max_util_for_resume. For the reported shape that is 13 blocks/request against 9 budgeted, so add_dummy_requests could not place the whole batch. Its failure path releases every already-registered request and returns None, leaving all request IDs unregistered, which surfaced two layers later as the IndexMapper assertion. Mirror the manager's arithmetic instead. 2. run.py defaulted --moe-backend-for-prefill to the literal "CUTLASS", so --moe-backend AUTO resolved a backend for decode but never for prefill. On an FP8_BLOCK_SCALES checkpoint CUTLASS dispatches to a Hopper-only kernel and prefill aborted on SM100. Default prefill to AUTO when decode is AUTO. 3. create_run_pack sized all_rank_num_tokens by world_size, but the field holds one entry per attention-DP rank. With attention DP disabled the MoE non-DP path requires a single-element list, so any world_size > 1 run tripped an assertion in calculate_num_chunks. Use mapping.dp_size. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> --- examples/layer_wise_benchmarks/README.md | 2 +- examples/layer_wise_benchmarks/run.py | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/examples/layer_wise_benchmarks/README.md b/examples/layer_wise_benchmarks/README.md index 9fecec99a595..0f2bc1afd962 100644 --- a/examples/layer_wise_benchmarks/README.md +++ b/examples/layer_wise_benchmarks/README.md @@ -394,7 +394,7 @@ Limitations: 1. Error `fp8 blockscale gemm only support Hopper` on Blackwell. - The default MoE backend "CUTLASS" does not support FP8 weights. Please choose the same MoE backend as your end-to-end config. A typical solution is to add the `--moe-backend DEEPGEMM` (or `TRTLLM`, `CUTEDSL`) and `--moe-backend-for-prefill DEEPGEMM` options. + The default MoE backend "CUTLASS" does not support FP8 weights. Please choose the same MoE backend as your end-to-end config. A typical solution is to add the `--moe-backend DEEPGEMM` (or `TRTLLM`, `CUTEDSL`) and `--moe-backend-for-prefill DEEPGEMM` options. Passing `--moe-backend AUTO` alone also works: prefill then defaults to `AUTO` too. 2. Error `huggingface_hub.errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2/resolve/main/config.json`. diff --git a/examples/layer_wise_benchmarks/run.py b/examples/layer_wise_benchmarks/run.py index eb712bd66164..fd0202f69555 100644 --- a/examples/layer_wise_benchmarks/run.py +++ b/examples/layer_wise_benchmarks/run.py @@ -164,7 +164,9 @@ def comma_separated_floats(s): if args.max_num_tokens is None: args.max_num_tokens = args.max_batch_size * max(args.seq_len_q_list) if args.moe_backend_for_prefill is None: - args.moe_backend_for_prefill = "CUTLASS" + # Let prefill resolve like decode when decode is AUTO; the "CUTLASS" default cannot + # serve every checkpoint (see README limitations). + args.moe_backend_for_prefill = "AUTO" if args.moe_backend == "AUTO" else "CUTLASS" if args.use_low_precision_moe_combine is None: args.use_low_precision_moe_combine = False if args.enable_autotuner is None: