NVIDIA · nv-guomingz · Sep 11, 2026 · Sep 10, 2026
@@ -367,7 +367,7 @@ Address these points in the PR description:
 
 ### API Stability Tests
 
-Some APIs are protected by the [API stability testsuite](tests/api_stability). If your PR breaks a protected API, the stability tests will fail with `API stability validation failed`. In this case, request review from the API code owners.
+Some APIs are protected by the [API stability testsuite](../../../tests/unittest/api_stability). If your PR breaks a protected API, the stability tests will fail with `API stability validation failed`. In this case, request review from the API code owners.
 
 ## Quantified Impact of Common Mistakes
 

@@ -6,31 +6,25 @@ This document explains how to build and run the C++ tests, and the included [res
 
 The unit tests can be launched via the Pytest script in [test_unit_tests.py](../../tests/integration/defs/cpp/test_unit_tests.py). These do not require engines to be built. The Pytest script will also build TRT-LLM.
 
-The Pytest scripts in [test_e2e.py](../../tests/integration/defs/cpp/test_e2e.py) and [test_multi_gpu.py](../../tests/integration/defs/cpp/test_multi_gpu.py) build TRT-LLM, build engines, and generate expected outputs and execute the end-to-end C++ tests all in one go.
-`test_e2e.py` and `test_multi_gpu.py` contain single and multi-device tests, respectively.
+The Pytest script in [test_multi_gpu.py](../../tests/integration/defs/cpp/test_multi_gpu.py) builds TRT-LLM, builds engines, and generates expected outputs and executes the end-to-end multi-device C++ tests all in one go.
 
 To get an overview of the tests and their parameterization, call:
 
 ```bash
 pytest tests/integration/defs/cpp/test_unit_tests.py --collect-only
-pytest tests/integration/defs/cpp/test_e2e.py --collect-only
 pytest tests/integration/defs/cpp/test_multi_gpu.py --collect-only
 ```
 
 All tests take the number of the CUDA architecture of the GPU you wish to use as a parameter e.g. 90 for Hopper.
 
-It is possible to choose unit tests or a single model for end-to-end tests.
+It is possible to choose individual unit tests or multi-device tests.
 Example calls could look like this:
 
 ```bash
 export LLM_MODELS_ROOT="/path/to/model_cache"
 
 pytest tests/integration/defs/cpp/test_unit_tests.py::test_unit_tests[runtime-90]
 
-pytest tests/integration/defs/cpp/test_e2e.py::test_model[llama-90]
-
-pytest tests/integration/defs/cpp/test_e2e.py::test_benchmarks[gpt-90]
-
 pytest tests/integration/defs/cpp/test_multi_gpu.py::TestDisagg::test_symmetric_executor[gpt-mpi_kvcache-90]
 ```
 

@@ -10,7 +10,7 @@ Currently TRT-LLM supports `disaggregated-service`, where the context and genera
 
 For more information on disaggregated service in LLM inference, one can refer to papers such as [DistServe](https://arxiv.org/abs/2401.09670), [SplitWise](https://arxiv.org/abs/2311.18677).
 
-An [architectural and performance overview](../../../docs/source/blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md), as well as [usage examples](../../../examples/disaggregated/README.md), are provided.
+An [architectural and performance overview](../../blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md), as well as [usage examples](../../../../examples/disaggregated/README.md), are provided.
 
 ## Environment Variables
 

@@ -195,7 +195,7 @@ loader = ModelWeightsLoader(external_checkpoint_dir, llava_dict)
 loader.generate_tllm_weights(trtllm_model)
 ```
 Users need to specify the different part from the default `tllm_to_externel_key_dict`. The loader still have support across different precisions.
-The support for LLaVA and Exaone is in `LLaMAForCausalLM.from_hugging_face()` of [model.py](../../../tensorrt_llm/models/llama/model.py), and can also be taken as examples.
+The support for LLaVA and Exaone is in `LLaMAForCausalLM.from_hugging_face()` of [model.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tensorrt_llm/models/llama/model.py), and can also be taken as examples.
 
 ### Models with customized weight layout
 For models with different weight layout, users can write the conversion loop explicitly and do customized operations.
@@ -229,7 +229,7 @@ for tllm_key, _ in tqdm(trtllm_model.named_parameters()):
 loader.fill(tllm_weights)
 ```
 This will apply `preprocess` after `load_tensor()` and before `postprocess`, and demonstrates how to convert the loaded shard into default HF layout. The loader still have support for precisions quantized from FP16/BF16 (e.g. INT8-wo/INT4-wo), the other precisions may require special operations, and can be addressed inside the `preprocess` function.
-The support for Qwen-1 is in `QWenForCausalLM.from_hugging_face()` of [model.py](../../../tensorrt_llm/models/qwen/model.py), and can also be taken as example.
+The support for Qwen-1 is in `QWenForCausalLM.from_hugging_face()` of [model.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tensorrt_llm/models/qwen/model.py), and can also be taken as example.
 
 ### Fully customized
 If the model weights loader cannot satisfy the requirements, users can write the conversion loop totally on their own.

@@ -11,12 +11,12 @@ Splitting your model weights between multiple GPUs requires them to communicate
 
 In pipeline parallelism, the model is split into sets of contiguous layers and each GPU houses one of these sets. In this setup, the only required communication is for each GPU to send the outputs of its set to the GPU with the next set.
 
-![Pipeline Parallel Visualization](../../media/Pipeline_Parallel_Vis.svg)
+![Pipeline Parallel Visualization](../../../media/Pipeline_Parallel_Vis.svg)
 
 
  On the other hand, tensor parallelism takes each layer of the model and splits it between the GPUs. This means that every GPU houses a portion of every layer. However since each layer needs the full outputs of the previous layer as an input, each GPU has to perform the heavier All-Reduce communication operation to share its results with all other GPUs before it can begin processing the next layer. While this seems disadvantageous, because each GPU only holds partial layers, it also performs smaller matrix multiplications, allowing it to compute its outputs quicker.
 
- ![Tensor Parallel Visualization](../../media/Tensor_Parallelism_Vis.svg)
+ ![Tensor Parallel Visualization](../../../media/Tensor_Parallelism_Vis.svg)
 
 
  Ultimately deciding the best strategy comes down to whether the extra overhead from the All-Reduce operation overshadows the gains from the smaller matrix multiplications. If the interconnects between the GPUs are sufficiently fast, the gains from the reduced computation burden per layer can outweigh the additional communication cost. Consequently, a general rule of thumb is that if your GPUs have fast connections between them like NVLink then tensor parallel is likely a good choice. However if the communication will go over slow connections (across nodes for example) pipeline parallel is likely better. Overall we provide the following guidelines:

@@ -10,26 +10,26 @@ One of TensorRT-LLM's key features is its inflight batching scheduler and runtim
 
 This section visualizes how TensorRT-LLM schedules requests based on max-batch size and max-num tokens. The example starts out with a newly initialized engine as well as a few unscheduled requests that have come in. For the sake of this example, toy values are set to `max batch size = 4` and `max num tokens = 12`. Each square block represents a token, and its color represents which request it belongs to.
 
-![TRT-LLM Scheduler Visualization 1](../../media/TRTLLM_Scheduler_Vis_1.svg)
+![TRT-LLM Scheduler Visualization 1](../../../media/TRTLLM_Scheduler_Vis_1.svg)
 
 
 Now the scheduler takes the first two requests, Request 1 and Request 2, and schedules them to execute the context phase. However, it cannot schedule any more requests because the prompts of the first two requests had 5 tokens each, leaving a budget of 2 tokens due to the max num tokens limit. Since all remaining requests have more than 2 prompt tokens none of them can be scheduled (context chunking can help in this situation, see the paged context attention section below). The tokens are marked with a "C" on them to represent that they are prompt tokens that were processed in the context phase.
 
 > Note: The tokens for different requests are shown on different rows simply for visualization purposes and are not representative of actual memory layouts
 
-![TRT-LLM Scheduler Visualization 2](../../media/TRTLLM_Scheduler_Vis_2.svg)
+![TRT-LLM Scheduler Visualization 2](../../../media/TRTLLM_Scheduler_Vis_2.svg)
 
 Now the engine runs an iteration of execution, completing the context phases for both of the scheduled requests. After it is done, the kv-cache of the prompts for both requests have been created and the first token has been generated. Tokens that were generated are marked with "G(n)" - for example a token marked "G1" represents that it is the first token generated for its request.
 
 TRT-LLM prioritizes scheduling requests in generation phase first so the two generated tokens are queued to be processed in the next iteration. Now, since the two previously scheduled requests have entered generation phase and only take up two tokens out of the max num token budget of 12, the scheduler is able to schedule two additional requests, Request 3 and Request 4. It cannot schedule the last request, Request 5, even though there is space for it in the max num tokens budget because of the max batch size limit of 4.
 
-![TRT-LLM Scheduler Visualization 3](../../media/TRTLLM_Scheduler_Vis_3.svg)
+![TRT-LLM Scheduler Visualization 3](../../../media/TRTLLM_Scheduler_Vis_3.svg)
 
 After the next iteration of execution, the second tokens for Requests 1 and 2 have been generated, and the first tokens for Request 3 and 4 have been generated. Lets say that G2 that was generated for Request 1 is the stop token, signifying that Request 1 is completed. In this case the scheduler would evict Request 1 before performing another execution iteration and prepare to return it to the user. This eviction puts the state of the engine below the max batch size limit and allows Request 5 to be scheduled.
 
 Another thing to note is that G1 that was generated for Request 2 has been added to the kv-cache for request 2, representing how kv-cache for a request grows as more and more tokens are generated.
 
-![TRT-LLM Scheduler Visualization 4](../../media/TRTLLM_Scheduler_Vis_4.svg)
+![TRT-LLM Scheduler Visualization 4](../../../media/TRTLLM_Scheduler_Vis_4.svg)
 
 Overall, the max batch size and max num tokens limits play a key part in deciding when requests are actually executed, and tuning them can have significant impacts on throughput numbers as well as how the engine balances previously scheduled requests in generation phase with context phase on new requests
 
@@ -101,7 +101,7 @@ Previously we recommended enabling paged context attention even though in our ca
 
 The [visualization](#understanding-the-trt-llm-scheduler) of the TensorRT-LLM scheduler showed that initially Request 3 couldn't be scheduled because it would put the scheduler over the max-num tokens limit. However with context chunking, this is no longer the case, and the first chunk of Request 3 would be able to be scheduled.
 
-![TRT-LLM Scheduler Visualization Chunked Context 1](../../media/TRTLLM_Scheduler_Vis_Chunked_Context_1.svg)
+![TRT-LLM Scheduler Visualization Chunked Context 1](../../../media/TRTLLM_Scheduler_Vis_Chunked_Context_1.svg)
 
 This is extremely beneficial for several reasons. Firstly it eliminates the possibility of requests with large prompts relative to max num tokens being unable to be scheduled due to other requests that are already in-flight. In production workloads, this can help improve worst case TTFT numbers. Secondly it allows for setting smaller values of max num tokens since you no longer need max num tokens to be at least as large as the longest prompt you want to support. For long-context cases this is extremely important, because setting extremely large values of max-num tokens takes away from memory available to be used as kv-cache. Given that in the worst case scenario chunked context has minimal impact on performance but can significantly benefit it in many scenarios, it's recommended that you always enable it.
 

@@ -18,7 +18,7 @@ Solution: try running build script with `--clean`, or try running `rm -r build c
 
 ## Debug on Unit Tests
 
-Here is an example to print the values of the MLP output tensor in a unit test ([full example](../../../../tests/unittest/others/test_debugging_api.py)).
+Here is an example to print the values of the MLP output tensor in a unit test ([full example](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tests/unittest/others/test_debugging_api.py)).
 
 1. Register the intermediate tensors as the network outputs with `register_network_output` API.
 

@@ -51,8 +51,8 @@ There are also interfaces for warming up `PyTorchModelEngine`, especially when u
 ## Customize KV Cache Manager
 
 To customize `KVCacheManager`, implement all the necessary interfaces.
-Then, integrate it into the `PyExecutor`. For the PyTorch backend, the relevant code is in [pytorch_model_registry.py](../../../tensorrt_llm/_torch/pyexecutor/backend_registries/pytorch_model_registry.py).
-In the `create_pytorch_model_based_executor` function, the `KVCacheManager` is instantiated as follows:
+Then, integrate it into the `PyExecutor`. For the PyTorch backend, the relevant code is in [_util.py](../../../tensorrt_llm/_torch/pyexecutor/_util.py).
+In `KvCacheCreator._create_kv_cache_manager`, the `KVCacheManager` is instantiated along these lines:
 
 ```python
     kv_cache_manager = KVCacheManager(

@@ -176,7 +176,7 @@ For expert users, `build_and_run_ad.py` provides advanced configuration capabili
 
 #### CLI Arguments with Dot Notation
 
-The script supports flexible CLI argument parsing using dot notation to modify nested configurations dynamically. You can target any field in both the [`ExperimentConfig`](./build_and_run_ad.py) and nested [`AutoDeployConfig`](../../tensorrt_llm/_torch/auto_deploy/llm_args.py)/[`LlmArgs`](../../tensorrt_llm/_torch/auto_deploy/llm_args.) objects:
+The script supports flexible CLI argument parsing using dot notation to modify nested configurations dynamically. You can target any field in both the [`ExperimentConfig`](./build_and_run_ad.py) and nested [`AutoDeployConfig`](../../tensorrt_llm/_torch/auto_deploy/llm_args.py)/[`LlmArgs`](../../tensorrt_llm/_torch/auto_deploy/llm_args.py) objects:
 
 ```bash
 # Configure model parameters

@@ -90,7 +90,7 @@ The output will be like:
 For more information, you can refer to [examples/llm-api](../../../llm-api).
 
 ## TRT flow
-The next section describes how to convert the weights from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. We will use llama's [convert_checkpoint.py](../../core/llama/convert_checkpoint.py) for the HyperCLOVAX model and then build the model with `trtllm-build`.
+The next section describes how to convert the weights from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. We will use llama's [convert_checkpoint.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/convert_checkpoint.py) for the HyperCLOVAX model and then build the model with `trtllm-build`.
 
 ### Convert checkpoint and build TensorRT engine(s)
 
@@ -230,7 +230,7 @@ trtllm-build \
 ```
 
 ### Run Engine
-Test your engine with the [run.py](../../../run.py) script:
+Test your engine with the [run.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/run.py) script:
 
 ```bash
 python3 ../../../run.py \
@@ -254,5 +254,5 @@ python ../../../summarize.py \
     --engine_dir trt_engines/$MODEL_NAME/fp16/1-gpu
 ```
 
-The TensorRT LLM HyperCLOVAX implementation is based on the LLaMA model. The implementation can be found in [llama/model.py](../../../../tensorrt_llm/models/llama/model.py).
-For more examples, see [`examples/models/core/llama/README.md`](../../core/llama/README.md)
+The TensorRT LLM HyperCLOVAX implementation is based on the LLaMA model. The implementation can be found in [llama/model.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tensorrt_llm/models/llama/model.py).
+For more examples, see [`examples/models/core/llama/README.md`](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/README.md)
@@ -394,7 +394,7 @@ Once all servers are running, you can send requests to `http://localhost:8000/v1
 
 ## TRT flow
 
-The next section describes how to convert weights from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. We will use LLaMA's [convert_checkpoint.py](../llama/convert_checkpoint.py) for EXAONE models and then build the model with `trtllm-build`.
+The next section describes how to convert weights from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. We will use LLaMA's [convert_checkpoint.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/convert_checkpoint.py) for EXAONE models and then build the model with `trtllm-build`.
 
 ### Convert checkpoint and build TensorRT engine(s)
 
@@ -534,7 +534,7 @@ trtllm-build \
 
 
 ### Run Engine
-Test your engine with the [run.py](../../../run.py) script:
+Test your engine with the [run.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/run.py) script:
 
 ```bash
 python3 ../../../run.py \
@@ -558,7 +558,7 @@ python ../../../summarize.py \
     --engine_dir trt_engines/exaone/fp16/1-gpu
 ```
 
-For more examples regarding EXAONE-3.0 & EXAONE-Deep's TRT flow, see [`examples/models/core/llama/README.md`](../llama/README.md)
+For more examples regarding EXAONE-3.0 & EXAONE-Deep's TRT flow, see [`examples/models/core/llama/README.md`](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/README.md)
 
 
 

@@ -324,7 +324,7 @@ def buildStageConfigs(stageName, platform, testlist, testCount, gpuCount, nodeCo
 
 When adding a test, either increment `testCount` on an existing entry or add a new `buildStageConfigs` block. Stages are grouped by node count (2 Nodes, 3 Nodes, 4 Nodes, etc.).
 
-For the full step-by-step guide including how to derive test-db filenames and GPU/node counts from disaggregated config YAMLs, see [`tests/scripts/perf-sanity/README.md`](../../tests/scripts/perf-sanity/README.md) ("Step-by-Step: Adding or Re-enabling Disaggregated Perf Sanity Tests").
+For the full step-by-step guide including how to derive test-db filenames and GPU/node counts from disaggregated config YAMLs, see [`tests/integration/defs/perf/README_test_perf_sanity.md`](../../../tests/integration/defs/perf/README_test_perf_sanity.md) ("Step-by-Step: Adding or Re-enabling Disaggregated Perf Sanity Tests").
 
 ## Post-Processing and Triage
 

@@ -38,6 +38,6 @@ Future work includes the following aspects:
 
 - Provide information for combined performance optimization with backends. Scaffolding can provide some information which is helpful for LLM inference engine. For a example, Controller may aware the prefix relation between generation requests, that would be helpful for kvcache reuse.
 
-- Develop auxiliary components to support generic requirements. Now we have developed some interesting auxiliary components. For an [example](examples/scaffolding/token_budget_majority_vote.py), we developed `GenerationTokenCounter` as a task collection decorator so that Controller could get the output tokens count for itself and its Sub-Controller. There are still many such works waiting for us to do.
+- Develop auxiliary components to support generic requirements. Now we have developed some interesting auxiliary components. For an [example](../../examples/scaffolding/token_budget_majority_vote.py), we developed `GenerationTokenCounter` as a task collection decorator so that Controller could get the output tokens count for itself and its Sub-Controller. There are still many such works waiting for us to do.
 
 You can see more specific work in this [link](https://github.com/NVIDIA/TensorRT-LLM/issues/3706#issuecomment-2820015957).