diff --git a/docs/source/blogs/tech_blog/blog01_Pushing_Latency_Boundaries_Optimizing_DeepSeek-R1_Performance_on_NVIDIA_B200_GPUs.md b/docs/source/blogs/tech_blog/blog01_Pushing_Latency_Boundaries_Optimizing_DeepSeek-R1_Performance_on_NVIDIA_B200_GPUs.md index b5e3e6558a60..d87a05467141 100644 --- a/docs/source/blogs/tech_blog/blog01_Pushing_Latency_Boundaries_Optimizing_DeepSeek-R1_Performance_on_NVIDIA_B200_GPUs.md +++ b/docs/source/blogs/tech_blog/blog01_Pushing_Latency_Boundaries_Optimizing_DeepSeek-R1_Performance_on_NVIDIA_B200_GPUs.md @@ -125,7 +125,7 @@ The modules in the diagram are: | Baseline: CUDA Graph + EP8TP8 | 67 | [modeling_deepseekv3.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/models/modeling_deepseekv3.py) | | Multi Stream to overlap shared expert with sparse experts | 73 | [modeling_deepseekv3.py#L506](https://github.com/NVIDIA/TensorRT-LLM/blob/14bfb5e0d6e81aec3306a1324cf074566646f886/tensorrt_llm/_torch/models/modeling_deepseekv3.py#L506) | | Optimize MLA Kernel | 80 | [PR #3763](https://github.com/NVIDIA/TensorRT-LLM/pull/3763) | -| Optimize TopK Kernels | 84 | • [RoutingKernelTopK.cuh](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/RoutingKernelTopK.cuh)
• [noAuxTcKernels.cu](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/kernels/noAuxTcKernels.cu) | +| Optimize TopK Kernels | 84 | • [RoutingKernelTopK.cuh](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingKernelTopK.cuh)
• [noAuxTcKernels.cu](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/kernels/noAuxTcKernels.cu) | | Optimize Fuse_A_GEMM | 89 | [attention.py#L345](https://github.com/NVIDIA/TensorRT-LLM/blob/d6b741ddfe7f8a80718c10d49773c42abc0a254f/tensorrt_llm/_torch/modules/attention.py#L345) | | MTP3_Vanilla | 154 | evolve to MTP3_Autoregressive | | Evolve to MTP3_Autoregressive + Optimize Router GEMM | 164 | [modeling_deepseekv3.py#L304](https://github.com/NVIDIA/TensorRT-LLM/blob/d6b741ddfe7f8a80718c10d49773c42abc0a254f/tensorrt_llm/_torch/models/modeling_deepseekv3.py#L304) | diff --git a/docs/source/blogs/tech_blog/blog04_Scaling_Expert_Parallelism_in_TensorRT-LLM.md b/docs/source/blogs/tech_blog/blog04_Scaling_Expert_Parallelism_in_TensorRT-LLM.md index 68937fd1acc0..f77c2d10bd83 100644 --- a/docs/source/blogs/tech_blog/blog04_Scaling_Expert_Parallelism_in_TensorRT-LLM.md +++ b/docs/source/blogs/tech_blog/blog04_Scaling_Expert_Parallelism_in_TensorRT-LLM.md @@ -536,7 +536,7 @@ export EXPERT_STATISTIC_PATH=./expert_statistic export EXPERT_STATISTIC_ITER_RANGE=100-200 ``` -Prepare a dataset following the [benchmarking documentation](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/performance/perf-benchmarking.md#preparing-a-dataset) and save it as `./dataset.json`. +Prepare a dataset following the [benchmarking documentation](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/developer-guide/perf-benchmarking.md#preparing-a-dataset) and save it as `./dataset.json`. Run 32-way expert parallelism inference on the prepared dataset. Please refer to the [LLM API MGMN example](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llm-api/llm_mgmn_trtllm_bench.sh) for details on running `trtllm-bench` on Slurm. diff --git a/docs/source/blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md b/docs/source/blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md index c9c257a6fb8c..abd3e352fcdd 100644 --- a/docs/source/blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md +++ b/docs/source/blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md @@ -128,7 +128,7 @@ For more information on how to use Dynamo with TensorRT LLM, please refer to [th ### Triton Inference Server -The third approach to do disaggregated LLM inference with TensorRT LLM utilizes the Triton Inference Server. With this approach a Triton ensemble model is employed, comprising a preprocessor, an orchestrator implemented as [a Python business logic scripting (BLS) backend](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/bls.html), and a post-processor. The orchestrator is responsible for routing client requests to context and generation instances, managing the flow of prompt tokens, and handling the return of generated tokens. This approach is illustrated in Figure 5. The Triton Inference Server approach relies on the Triton TensorRT LLM backend and the Executor API, which is supported only for the TensorRT backend. For more information on how to use this approach, please refer to [this documentation](https://github.com/NVIDIA/TensorRT-LLM/tree/main/triton_backend/all_models/disaggregated_serving#running-disaggregated-serving-with-triton-tensorrt-llm-backend). +The third approach to do disaggregated LLM inference with TensorRT LLM utilizes the Triton Inference Server. With this approach a Triton ensemble model is employed, comprising a preprocessor, an orchestrator implemented as [a Python business logic scripting (BLS) backend](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/bls.html), and a post-processor. The orchestrator is responsible for routing client requests to context and generation instances, managing the flow of prompt tokens, and handling the return of generated tokens. This approach is illustrated in Figure 5. The Triton Inference Server approach relies on the Triton TensorRT LLM backend and the Executor API, which is supported only for the TensorRT backend. For more information on how to use this approach, please refer to [this documentation](https://github.com/NVIDIA/TensorRT-LLM/tree/v1.2.0/triton_backend/all_models/disaggregated_serving#running-disaggregated-serving-with-triton-tensorrt-llm-backend).
diff --git a/docs/source/blogs/tech_blog/blog08_Scaling_Expert_Parallelism_in_TensorRT-LLM_part2.md b/docs/source/blogs/tech_blog/blog08_Scaling_Expert_Parallelism_in_TensorRT-LLM_part2.md index ec4745b24abd..a444b90b6aee 100644 --- a/docs/source/blogs/tech_blog/blog08_Scaling_Expert_Parallelism_in_TensorRT-LLM_part2.md +++ b/docs/source/blogs/tech_blog/blog08_Scaling_Expert_Parallelism_in_TensorRT-LLM_part2.md @@ -249,7 +249,7 @@ To address the increased host overhead when scaling parallelism in the system, w TensorRT LLM is designed to be composed of both C++ and Python code, so that C++ can handle the most performance-sensitive parts while Python handles higher-level logic. As we try to put more logic into Python to make the program easier to read and debug, there are still frequent conversations through binding interfaces between C++ and Python. Besides, since most of the logic is implemented in Python, there are several layers of implementation that communicate with each other through inter-process communication overhead. Frequent binding calls and serialization/deserialization introduced by inter-process communication slow down the core library. -To improve program efficiency, we used environment variables introduced in the [performance analysis guidance](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/performance/perf-analysis.md) to measure and profile CPU overhead, and improved performance by reducing and reusing different binding calls as much as possible, and delaying Python object deserialization to avoid duplicated serialization and reduce message size when doing inter-process communication. This optimization was added in [PR 5224](https://github.com/NVIDIA/TensorRT-LLM/pull/5224). We have also reduced Python garbage collection (GC) impacts in [PR 5141](https://github.com/NVIDIA/TensorRT-LLM/pull/5141). +To improve program efficiency, we used environment variables introduced in the [performance analysis guidance](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/developer-guide/perf-analysis.md) to measure and profile CPU overhead, and improved performance by reducing and reusing different binding calls as much as possible, and delaying Python object deserialization to avoid duplicated serialization and reduce message size when doing inter-process communication. This optimization was added in [PR 5224](https://github.com/NVIDIA/TensorRT-LLM/pull/5224). We have also reduced Python garbage collection (GC) impacts in [PR 5141](https://github.com/NVIDIA/TensorRT-LLM/pull/5141). To enable powerful NVTX markers for easier analysis of host overheads, TensorRT LLM provides several useful environment variables: diff --git a/docs/source/blogs/tech_blog/blog09_Deploying_GPT_OSS_on_TRTLLM.md b/docs/source/blogs/tech_blog/blog09_Deploying_GPT_OSS_on_TRTLLM.md index a027a59761de..ba898230665b 100644 --- a/docs/source/blogs/tech_blog/blog09_Deploying_GPT_OSS_on_TRTLLM.md +++ b/docs/source/blogs/tech_blog/blog09_Deploying_GPT_OSS_on_TRTLLM.md @@ -357,7 +357,7 @@ moe_config: ## Troubleshooting Tips -- If you encounter CUDA out-of-memory errors, try reducing `--max_batch_size`, `--max_num_tokens`, or `--kv_cache_free_gpu_memory_fraction`. See the [doc](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md) for the explanation of these parameters. +- If you encounter CUDA out-of-memory errors, try reducing `--max_batch_size`, `--max_num_tokens`, or `--kv_cache_free_gpu_memory_fraction`. See the [doc](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/legacy/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md) for the explanation of these parameters. - Add `print_iter_log: true` to extra LLM API options YAML file to inspect the per-iteration log. - Check GPU utilization with `nvidia-smi` while the server is running to inspect GPU status and memory usage. - If the container fails to start, verify that the NVIDIA Container Toolkit is properly installed diff --git a/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md b/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md index 398dea4054a1..7afbc9b005f9 100644 --- a/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md +++ b/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md @@ -295,7 +295,7 @@ For the full property catalog (every `*_group`, `*_rank`, the flattened `seq_mes ## NVL72 Scaling Results -All performance numbers in this section use the same measured workload and publish one distributed recipe: [`examples/visual_gen/configs/wan22_t2v_bf16_gb200_nvl72.yml`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/visual_gen/configs/wan22_t2v_bf16_gb200_nvl72.yml). +All performance numbers in this section use the same measured workload and publish one distributed recipe: [`examples/visual_gen/configs/wan22_t2v_bf16_gb200_nvl72.yaml`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/visual_gen/configs/wan22_t2v_bf16_gb200_nvl72.yaml). | Item | Value | | :--- | :--- | @@ -348,7 +348,7 @@ compilation_config: ### Cosmos3-Super -The same runtime scales a very different model unchanged. Cosmos3-Super is a 64B Mixture-of-Transformers with 64 attention heads, generating a longer 189-frame clip — so each denoising step is an even larger dense prefill than Wan. Only the workload and the per-width recipe change; the CFG, Ulysses, Attention2D, and parallel-VAE knobs are identical ([`examples/visual_gen/configs/cosmos3_t2v_bf16_gb200_nvl72.yml`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/visual_gen/configs/cosmos3_t2v_bf16_gb200_nvl72.yml)) +The same runtime scales a very different model unchanged. Cosmos3-Super is a 64B Mixture-of-Transformers with 64 attention heads, generating a longer 189-frame clip — so each denoising step is an even larger dense prefill than Wan. Only the workload and the per-width recipe change; the CFG, Ulysses, Attention2D, and parallel-VAE knobs are identical ([`examples/visual_gen/configs/cosmos3_t2v_bf16_gb200_nvl72.yaml`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/visual_gen/configs/cosmos3_t2v_bf16_gb200_nvl72.yaml)) | Item | Value | | :--- | :--- | @@ -386,7 +386,7 @@ The same `--visual_gen_args` YAML drives `trtllm-serve` for online serving. Dist export CONTAINER_IMAGE=/path/to/tensorrt-llm.sqsh export PROJECT_ROOT=/path/to/TensorRT-LLM export MODEL=Wan-AI/Wan2.2-T2V-A14B-Diffusers -export SERVER_CONFIG=examples/visual_gen/configs/wan22_t2v_bf16_gb200_nvl72.yml +export SERVER_CONFIG=examples/visual_gen/configs/wan22_t2v_bf16_gb200_nvl72.yaml # Fill in and with your SLURM account and partition sbatch -A -p -N 18 --ntasks-per-node=4 --ntasks=72 examples/visual_gen/serve/benchmark_visual_gen_mgmn_distributed.sh diff --git a/docs/source/developer-guide/kv-cache-compression-development.md b/docs/source/developer-guide/kv-cache-compression-development.md index eae5f288549d..b99efd2727fb 100644 --- a/docs/source/developer-guide/kv-cache-compression-development.md +++ b/docs/source/developer-guide/kv-cache-compression-development.md @@ -201,8 +201,8 @@ defines its compressed format and implements the relevant APIs below. | [`KvCacheCompressionConfig`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/llmapi/llm_args.py#L3783) and [`create_kv_cache_compression_manager()`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/pyexecutor/_util.py#L2830) | Method selection and supported-combination admission | Pages, kernels, or request mappings | | [`KVCacheCompressionManager`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/pyexecutor/resource_manager.py#L2775) | Algorithm cadence, request state, decisions, format metadata, and algorithm launches | KVCM allocation policy or Attention runtime state | | [`NativeColdPageCodec`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/batch_manager/kv_cache_compression/nativeColdPageCodec.h#L61) | KVCM-layout resolution, provider routing, fallback routing, and Python/native lifetime bridge | Format-specific quantization policy | -| [`KVCacheManagerV2`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py#L789) | Pages, Slots, pools, mappings, migration streams, events, publication, release, rollback, and cold storage | Algorithm scores or quantization decisions | -| [`AttentionBackend`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/attention_backend/interface.py#L1001) | Consumption of the published active GPU representation | Cold storage and migration | +| [`KVCacheManagerV2`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py#L936) | Pages, Slots, pools, mappings, migration streams, events, publication, release, rollback, and cold storage | Algorithm scores or quantization decisions | +| [`AttentionBackend`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/attention/backends/interface.py#L1008) | Consumption of the published active GPU representation | Cold storage and migration | Before physical mutation, a method may reject, defer, or perform a legal no-op. After it submits work or moves bytes, it must follow the framework's completion diff --git a/docs/source/legacy/advanced/gpt-attention.md b/docs/source/legacy/advanced/gpt-attention.md index d5b4d29718bf..67518854438d 100644 --- a/docs/source/legacy/advanced/gpt-attention.md +++ b/docs/source/legacy/advanced/gpt-attention.md @@ -21,7 +21,7 @@ future***. In TensorRT-LLM, the GPT attention operator supports two different types of QKV inputs: Padded and packed (i.e. non padded) inputs. The mode is determined by the global configuration parameter `remove_input_padding` defined -in [`tensorrt_llm.plugin`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/plugin/plugin.py). +in [`tensorrt_llm.plugin`](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tensorrt_llm/plugin/plugin.py). When padding is enabled (that is, `remove_input_padding` is `False`), the sequences that are shorter than the `max_sequence_length` are padded to that maximum @@ -43,7 +43,7 @@ context and generation phases in auto-regressive models like GPT. ### Context Phase If the `context_fmha_type` is set to `disabled` (refer to -[`tensorrt_llm.plugin`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/plugin/plugin.py)), +[`tensorrt_llm.plugin`](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tensorrt_llm/plugin/plugin.py)), the implementation maps to a sequence of GPU kernels that will store the intermediate `Q*K^T` tensor in memory before calling the softmax operator. It is the slowest method and the memory footprint is significant (quadratically diff --git a/docs/source/legacy/advanced/speculative-decoding.md b/docs/source/legacy/advanced/speculative-decoding.md index 2faf885f8d10..0cb781a0cf73 100644 --- a/docs/source/legacy/advanced/speculative-decoding.md +++ b/docs/source/legacy/advanced/speculative-decoding.md @@ -60,16 +60,14 @@ These tokens are then forwarded to the Target model for verification. Upon verification, the Target model may return up to `K+1` tokens. Subsequently, the prompt, now updated with the accepted tokens, is sent back to the Draft model to initiate the generation of new draft tokens. This iterative process continues until a predefined stop conditions are met. -An example orchestration script is available in the Triton backend repository’s -[draft-target-model client example](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/client/python/draft_target_model_client.py). -We provide two styles of running Draft-Target-Model now: using TensorRT-LLM-BLS in Triton Inference Server, or using TensorRT-LLM directly. Detailed steps of running can be found in [examples/draft_target_model/README.md](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/draft_target_model/README.md) and the code can be found in [examples/ngram/run_dtm_ngram.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/ngram/run_dtm_ngram.py). +We provide two styles of running Draft-Target-Model now: using TensorRT-LLM-BLS in Triton Inference Server, or using TensorRT-LLM directly. Detailed steps of running can be found in [examples/draft_target_model/README.md](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/draft_target_model/README.md) and the code can be found in [examples/ngram/run_dtm_ngram.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/ngram/run_dtm_ngram.py). ## NGram The NGram speculative decoding directly copies from the input prompt and previous generated output as draft tokens while generating the later output. It works like Draft-Target-Model but involves only one Target LLM model without further fine-tuning. The NGram profit from the scenarios which have high n-gram overlap between input prompt and output, such as summarization, document QA, multi-turn chat, code editing, etc. -See document in [examples/ngram/README.md](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/ngram/README.md) and the code can be found in [examples/ngram/run_dtm_ngram.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/ngram/run_dtm_ngram.py). +See document in [examples/ngram/README.md](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/ngram/README.md) and the code can be found in [examples/ngram/run_dtm_ngram.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/ngram/run_dtm_ngram.py). ## Medusa @@ -149,7 +147,7 @@ So, only `9` candidates are specified. ### Using Medusa with TensorRT-LLM -For guidance on constructing and executing Medusa with the Python runtime, consult the [Medusa README](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/medusa/README.md). When utilizing the Inflight Fused Batching (IFB) with the C++ API, it is necessary to define the `medusa_choices` explicitly within the model configuration. For detailed instructions, refer to the [model configuration in TensorRT-LLM backend](https://github.com/triton-inference-server/tensorrtllm_backend?tab=readme-ov-file#modify-the-model-configuration) for more details. +For guidance on constructing and executing Medusa with the Python runtime, consult the [Medusa README](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/medusa/README.md). When utilizing the Inflight Fused Batching (IFB) with the C++ API, it is necessary to define the `medusa_choices` explicitly within the model configuration. For detailed instructions, refer to the [model configuration in TensorRT-LLM backend](https://github.com/triton-inference-server/tensorrtllm_backend?tab=readme-ov-file#modify-the-model-configuration) for more details. #### Limitations @@ -163,20 +161,20 @@ However, similar to any new model, you can follow the same approach to define yo The ReDrafter approach enhances the single-model Medusa method by predicting and verifying tokens using the same model. However, unlike Medusa, it predicts draft tokens using a recurrent predictor, where each draft token depends on the previous one. This method also allows the use of beam search to identify more prominent draft tokens. For more details, please read [the ReDrafter paper](https://arxiv.org/html/2403.09919v1). -TensorRT-LLM implements the ReDrafter model such that logits prediction, beam search, and draft token acceptance are performed inside the TensorRT engine. This contrasts with standard model inference, which only predicts logits and performs decoding outside the engine. Since the engine predicts explicit draft tokens instead of implicit tokens decoded from logits, we categorize this speculative decoding method as `explicit_draft_tokens`. Please, visit the [ReDrafter README](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/redrafter/README.md) for information about building and running the model. ReDrafter supports both Inflight Fused Batching runtime and Python static batching runtime. +TensorRT-LLM implements the ReDrafter model such that logits prediction, beam search, and draft token acceptance are performed inside the TensorRT engine. This contrasts with standard model inference, which only predicts logits and performs decoding outside the engine. Since the engine predicts explicit draft tokens instead of implicit tokens decoded from logits, we categorize this speculative decoding method as `explicit_draft_tokens`. Please, visit the [ReDrafter README](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/redrafter/README.md) for information about building and running the model. ReDrafter supports both Inflight Fused Batching runtime and Python static batching runtime. ## EAGLE The EAGLE approach enhances the single-model Medusa method by predicting and verifying tokens using the same model. Similarly to ReDrafter, it predicts draft tokens using a recurrent predictor where each draft token depends on the previous one. However, unlike ReDrafter, it uses a single-layer transformer model to predict draft tokens from previous hidden states and decoded tokens. In the EAGLE-1 decoding tree needs to be known during the decoding. In the EAGLE-2 this tree is asssembled during the execution by searching for the most probable hypothesis along the beam. -Similarly to ReDrafter, TensorRT-LLM implements the EAGLE model such that logits prediction, draft tokens acceptance and draft token generation are performed inside of the TensorRT engine(EAGLE-1 and EAGLE-2 are both supported). Please, visit the [EAGLE README](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/eagle/README.md) for information about building and running the model. +Similarly to ReDrafter, TensorRT-LLM implements the EAGLE model such that logits prediction, draft tokens acceptance and draft token generation are performed inside of the TensorRT engine(EAGLE-1 and EAGLE-2 are both supported). Please, visit the [EAGLE README](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/eagle/README.md) for information about building and running the model. > **EAGLE3 note.** If the EAGLE3 draft head config omits `draft_vocab_size`, TensorRT-LLM assumes it matches `vocab_size` and emits a warning. Set `draft_vocab_size` explicitly if the draft head uses a different vocabulary. ### Disaggregated Serving -[Disaggregated Serving](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disaggregated-service.md) with EAGLE3 using the two model approach is supported in the Pytorch backend. Please refer to the following [Dynamo example](https://github.com/ai-dynamo/dynamo/blob/main/examples/tensorrt_llm/llama4_plus_eagle.md) on how to run EAGLE3 with Disaggregated Serving for Llama 4 Maverick. +[Disaggregated Serving](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/disagg-serving.md) with EAGLE3 using the two model approach is supported in the Pytorch backend. Please refer to the following [Dynamo example](https://github.com/ai-dynamo/dynamo/blob/main/docs/fern/pages/developer-guide/additional-resources/tensorrt-llm-details/llama4-plus-eagle.md) on how to run EAGLE3 with Disaggregated Serving for Llama 4 Maverick. ## Lookahead Decoding -Lookahead decoding algorithm operates through two parallel computation branches within the same model: a lookahead branch that generates n-grams using a fixed-sized 2D window, and a verification branch that validates promising n-gram candidates. This approach eliminates the necessity for additional model training or fine-tuning and can be enabled for any autoregressive model. Refer to the [Lookahead decoding README](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/lookahead/README.md) for information about building and running the model. +Lookahead decoding algorithm operates through two parallel computation branches within the same model: a lookahead branch that generates n-grams using a fixed-sized 2D window, and a verification branch that validates promising n-gram candidates. This approach eliminates the necessity for additional model training or fine-tuning and can be enabled for any autoregressive model. Refer to the [Lookahead decoding README](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/lookahead/README.md) for information about building and running the model. diff --git a/docs/source/legacy/architecture/workflow.md b/docs/source/legacy/architecture/workflow.md index 64460ee2a215..ec7be9aff7f6 100644 --- a/docs/source/legacy/architecture/workflow.md +++ b/docs/source/legacy/architecture/workflow.md @@ -57,7 +57,7 @@ class LLaMAForCausalLM (DecoderModelForCausalLM): Then, in the convert_checkpoint.py script in the -[`examples/models/core/llama/`](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/models/core/llama/) directory of the GitHub repo, +[`examples/models/core/llama/`](https://github.com/NVIDIA/TensorRT-LLM/tree/v1.2.0/examples/models/core/llama/) directory of the GitHub repo, the logic can be greatly simplified. Even if the model definition code of TensorRT-LLM LLaMA class is changed due to some reason, the `from_hugging_face` API will keep the same, thus the existing workflow using this interface will not be affected. @@ -110,7 +110,7 @@ TensorRT-LLM relies on NVIDIA Modelopt toolkit to support some of the quantizati In TensorRT-LLM 0.8 version: * For Modelopt-supported quantization algorithms, a standalone script, - [example/quantization/quantize.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/quantization/quantize.py) + [example/quantization/quantize.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/quantization/quantize.py) can export TensorRT-LLM checkpoints, and the trtllm-build command needs to be executed to build the checkpoints to engines. * For the non-Modelopt quantization algorithms, users need to use the per-model convert_checkpoint.py scripts to export TensorRT-LLM checkpoints. diff --git a/docs/source/legacy/performance/performance-tuning-guide/benchmarking-default-performance.md b/docs/source/legacy/performance/performance-tuning-guide/benchmarking-default-performance.md index 7277b0afa5b0..3d8ae5c668a9 100644 --- a/docs/source/legacy/performance/performance-tuning-guide/benchmarking-default-performance.md +++ b/docs/source/legacy/performance/performance-tuning-guide/benchmarking-default-performance.md @@ -73,10 +73,10 @@ if __name__ == '__main__': TensorRT-LLM also has a command line interface for building and saving engines. This workflow consists of two steps -1. Convert model checkpoint (HuggingFace, Nemo) to TensorRT-LLM checkpoint via `convert_checkpoint.py`. Each supported model has a `convert_checkpoint.py` associated it with it and can be found in the examples folder. For example, the `convert_checkpoint.py` script for Llama models can be found [here](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/models/core/llama/convert_checkpoint.py) +1. Convert model checkpoint (HuggingFace, Nemo) to TensorRT-LLM checkpoint via `convert_checkpoint.py`. Each supported model has a `convert_checkpoint.py` associated it with it and can be found in the examples folder. For example, the `convert_checkpoint.py` script for Llama models can be found [here](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/convert_checkpoint.py) 2. Build engine by passing TensorRT-LLM checkpoint to `trtllm-build` command. The `trtllm-build` command is installed automatically when the `tensorrt_llm` package is installed. -The README in the examples folder for supported models walks through building engines using this flow for a wide variety of situations. The examples folder for Llama models can be found at [https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/models/core/llama](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/models/core/llama). +The README in the examples folder for supported models walks through building engines using this flow for a wide variety of situations. The examples folder for Llama models can be found at [https://github.com/NVIDIA/TensorRT-LLM/tree/v1.2.0/examples/models/core/llama](https://github.com/NVIDIA/TensorRT-LLM/tree/v1.2.0/examples/models/core/llama). ## Benchmarking with `trtllm-bench` diff --git a/docs/source/legacy/performance/performance-tuning-guide/fp8-quantization.md b/docs/source/legacy/performance/performance-tuning-guide/fp8-quantization.md index 3a40ed1046b6..1e64a10dd357 100644 --- a/docs/source/legacy/performance/performance-tuning-guide/fp8-quantization.md +++ b/docs/source/legacy/performance/performance-tuning-guide/fp8-quantization.md @@ -56,7 +56,7 @@ if __name__ == '__main__': main() ``` -For an example of how to build an fp8 engine using the [TensorRT-LLM CLI workflow](./benchmarking-default-performance.md#building-and-saving-engines-via-cli) flow see [TensorRT-LLM LLaMA examples](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/models/core/llama). In short you first run [`examples/quantization/quantize.py`](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/quantization) to quantize and convert the model checkpoint to TensorRT-LLM format and then use `trtllm-build`. +For an example of how to build an fp8 engine using the [TensorRT-LLM CLI workflow](./benchmarking-default-performance.md#building-and-saving-engines-via-cli) flow see [TensorRT-LLM LLaMA examples](https://github.com/NVIDIA/TensorRT-LLM/tree/v1.2.0/examples/models/core/llama). In short you first run [`examples/quantization/quantize.py`](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/quantization) to quantize and convert the model checkpoint to TensorRT-LLM format and then use `trtllm-build`. > ***Note: While quantization aims to preserve model accuracy this is not guaranteed and it is extremely important you check that the quality of outputs remains sufficient after quantization.*** diff --git a/docs/source/torch/scheduler.md b/docs/source/torch/scheduler.md index ce6d74e76ddf..6d90d7ed99a6 100644 --- a/docs/source/torch/scheduler.md +++ b/docs/source/torch/scheduler.md @@ -26,7 +26,7 @@ The inputs to `SimpleScheduler` include `active_requests` and `inflight_request_ To customize the scheduler or batching mechanism, implement your own `CapacityScheduler` and `MicroBatchScheduler` by inheriting their respective classes. If two-step scheduling is unnecessary, inherit `RequestScheduler` and implement `schedule_request` directly. -An example of a `CapacityScheduler` implementation is the `GuaranteedNoEvictScheduler` class, found in [scheduler.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/pyexecutor/scheduler.py). +An example of a `CapacityScheduler` implementation is the `GuaranteedNoEvictScheduler` class, found in [scheduler.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/pyexecutor/scheduler/scheduler.py). This class was used before the C++ binding of `CapacityScheduler` and initially employed a Python-based scheduler. It inherits `CapacityScheduler` and implements its own `schedule_request` method. This method processes all `active_requests` and tries to schedule more requests that can fit in the KV cache. diff --git a/examples/auto_deploy/README.md b/examples/auto_deploy/README.md index 343ad8c87f35..eb59f73c687a 100644 --- a/examples/auto_deploy/README.md +++ b/examples/auto_deploy/README.md @@ -113,7 +113,7 @@ Currently `AutoQuantize` supports only `effective_bits` as the performance const #### 1. Quantize a model with ModelOpt -Refer to [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer/blob/main/examples/llm_autodeploy/README.md) for generating quantized model checkpoint. +Refer to [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer/blob/0.39.0/examples/llm_autodeploy/README.md) for generating quantized model checkpoint. #### 2. Deploy the quantized model with AutoDeploy diff --git a/examples/kv_cache_compression/README.md b/examples/kv_cache_compression/README.md index 910909287524..769e3454b77b 100644 --- a/examples/kv_cache_compression/README.md +++ b/examples/kv_cache_compression/README.md @@ -16,6 +16,6 @@ cache compression method can be enabled for each LLM instance. | TriAttention | Periodically evict lower-importance generation tokens and compact the GPU KV cache | [TriAttention](triattention.md) | For a feature overview and configuration guidance, see -[KV Cache Compression](https://nvidia.github.io/TensorRT-LLM/features/kv-cache-compression.html). +[KV Cache Compression](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/kv-cache-compression.md). To implement another compression method, see the -[KV Cache Compression Development Guide](https://nvidia.github.io/TensorRT-LLM/developer-guide/kv-cache-compression-development.html). +[KV Cache Compression Development Guide](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/developer-guide/kv-cache-compression-development.md). diff --git a/examples/kv_cache_compression/nvfp4_cold_page.md b/examples/kv_cache_compression/nvfp4_cold_page.md index 676b47895e12..7c57ed749843 100644 --- a/examples/kv_cache_compression/nvfp4_cold_page.md +++ b/examples/kv_cache_compression/nvfp4_cold_page.md @@ -11,7 +11,7 @@ optimization: the active GPU cache and Attention kernels continue to use the model's normal runtime KV type. For an overview of all available compression methods, see -[KV Cache Compression](https://nvidia.github.io/TensorRT-LLM/features/kv-cache-compression.html). +[KV Cache Compression](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/kv-cache-compression.md). ## Motivation @@ -76,7 +76,7 @@ required for Pages to cross a compression boundary. On Linux 6.11 through 6.13, mixed models that need both NVFP4 Attention lifecycles and lossless SSM/GDN fallback lifecycles are not supported. See the -[KV Cache Compression Development Guide](https://nvidia.github.io/TensorRT-LLM/developer-guide/kv-cache-compression-development.html) +[KV Cache Compression Development Guide](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/developer-guide/kv-cache-compression-development.md) for the current storage-path limitation. ### Tested Models diff --git a/examples/kv_cache_compression/triattention.md b/examples/kv_cache_compression/triattention.md index 5a675bb27f5c..b372296b0d14 100644 --- a/examples/kv_cache_compression/triattention.md +++ b/examples/kv_cache_compression/triattention.md @@ -3,7 +3,7 @@ This document describes enabling TriAttention KV-cache compression in TensorRT-LLM. For an overview of all available compression methods, see -[KV Cache Compression](https://nvidia.github.io/TensorRT-LLM/features/kv-cache-compression.html). +[KV Cache Compression](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/kv-cache-compression.md). TriAttention is a training-free, decode-time KV-cache eviction method for long-context LLM inference. During generation it periodically scores the cached tokens by a trigonometric importance measure derived from offline per-head query statistics (calibration), keeps the most important `budget` tokens, and physically compacts the cache — reducing KV-cache memory so more sequences fit on a GPU at once. diff --git a/examples/models/core/deepseek_v3/README.md b/examples/models/core/deepseek_v3/README.md index b15f78d3d9ae..9bd0aa89a1dc 100644 --- a/examples/models/core/deepseek_v3/README.md +++ b/examples/models/core/deepseek_v3/README.md @@ -472,7 +472,7 @@ settings for your specific use case. ### Dynamo NVIDIA Dynamo is a high-throughput low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments. -Dynamo supports TensorRT LLM as one of its inference engine. For details on how to use TensorRT LLM with Dynamo please refer to [LLM Deployment Examples using TensorRT-LLM](https://github.com/ai-dynamo/dynamo/blob/main/examples/tensorrt_llm/README.md) +Dynamo supports TensorRT LLM as one of its inference engine. For details on how to use TensorRT LLM with Dynamo please refer to [LLM Deployment Examples using TensorRT-LLM](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/trtllm/README.md) ### tensorrtllm_backend for triton inference server (Prototype) To serve the model using [tensorrtllm_backend](https://github.com/triton-inference-server/tensorrtllm_backend.git), make sure the version is v0.19+ in which the pytorch path is added as a prototype feature. diff --git a/examples/models/core/qwen/README.md b/examples/models/core/qwen/README.md index 6c7654357a18..de3bb97f676a 100644 --- a/examples/models/core/qwen/README.md +++ b/examples/models/core/qwen/README.md @@ -289,7 +289,7 @@ For further details, please refer to [speculative-decoding.md](../../../../docs/ ### Dynamo NVIDIA Dynamo is a high-throughput low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments. -Dynamo supports TensorRT LLM as one of its inference engine. For details on how to use TensorRT LLM with Dynamo please refer to [LLM Deployment Examples using TensorRT-LLM](https://github.com/ai-dynamo/dynamo/blob/main/examples/tensorrt_llm/README.md) +Dynamo supports TensorRT LLM as one of its inference engine. For details on how to use TensorRT LLM with Dynamo please refer to [LLM Deployment Examples using TensorRT-LLM](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/trtllm/README.md) ## Qwen3-Next