From f2c25a8d97762227288a88609e9483c28775e0cf Mon Sep 17 00:00:00 2001 From: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Date: Thu, 10 Sep 2026 17:01:50 +0800 Subject: [PATCH] [https://nvbugs/6731971][doc] Fix broken relative paths flagged by test_relative_path_validity Correct link depths, repoint moved targets, and pin legacy-only files to v1.2.0. Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> --- .../skills/trtllm-code-contribution/SKILL.md | 2 +- cpp/tests/README.md | 10 ++-------- .../legacy/advanced/disaggregated-service.md | 2 +- .../legacy/architecture/model-weights-loader.md | 4 ++-- .../deciding-model-sharding-strategy.md | 4 ++-- .../tuning-max-batch-size-and-max-num-tokens.md | 10 +++++----- docs/source/legacy/reference/troubleshooting.md | 2 +- docs/source/torch/kv_cache_manager.md | 4 ++-- examples/auto_deploy/README.md | 2 +- examples/models/contrib/hyperclovax/README.md | 8 ++++---- examples/models/core/exaone/README.md | 6 +++--- jenkins/scripts/perf/README.md | 2 +- tensorrt_llm/scaffolding/README.md | 2 +- tests/integration/defs/accuracy/README.md | 8 ++++---- .../defs/perf/README_release_test.md | 17 ++++++----------- 15 files changed, 36 insertions(+), 47 deletions(-) diff --git a/.claude/skills/trtllm-code-contribution/SKILL.md b/.claude/skills/trtllm-code-contribution/SKILL.md index 86b6d9c5f966..49032565e5fa 100644 --- a/.claude/skills/trtllm-code-contribution/SKILL.md +++ b/.claude/skills/trtllm-code-contribution/SKILL.md @@ -367,7 +367,7 @@ Address these points in the PR description: ### API Stability Tests -Some APIs are protected by the [API stability testsuite](tests/api_stability). If your PR breaks a protected API, the stability tests will fail with `API stability validation failed`. In this case, request review from the API code owners. +Some APIs are protected by the [API stability testsuite](../../../tests/unittest/api_stability). If your PR breaks a protected API, the stability tests will fail with `API stability validation failed`. In this case, request review from the API code owners. ## Quantified Impact of Common Mistakes diff --git a/cpp/tests/README.md b/cpp/tests/README.md index 8ce1c354afe0..950d9a96f348 100644 --- a/cpp/tests/README.md +++ b/cpp/tests/README.md @@ -6,20 +6,18 @@ This document explains how to build and run the C++ tests, and the included [res The unit tests can be launched via the Pytest script in [test_unit_tests.py](../../tests/integration/defs/cpp/test_unit_tests.py). These do not require engines to be built. The Pytest script will also build TRT-LLM. -The Pytest scripts in [test_e2e.py](../../tests/integration/defs/cpp/test_e2e.py) and [test_multi_gpu.py](../../tests/integration/defs/cpp/test_multi_gpu.py) build TRT-LLM, build engines, and generate expected outputs and execute the end-to-end C++ tests all in one go. -`test_e2e.py` and `test_multi_gpu.py` contain single and multi-device tests, respectively. +The Pytest script in [test_multi_gpu.py](../../tests/integration/defs/cpp/test_multi_gpu.py) builds TRT-LLM, builds engines, and generates expected outputs and executes the end-to-end multi-device C++ tests all in one go. To get an overview of the tests and their parameterization, call: ```bash pytest tests/integration/defs/cpp/test_unit_tests.py --collect-only -pytest tests/integration/defs/cpp/test_e2e.py --collect-only pytest tests/integration/defs/cpp/test_multi_gpu.py --collect-only ``` All tests take the number of the CUDA architecture of the GPU you wish to use as a parameter e.g. 90 for Hopper. -It is possible to choose unit tests or a single model for end-to-end tests. +It is possible to choose individual unit tests or multi-device tests. Example calls could look like this: ```bash @@ -27,10 +25,6 @@ export LLM_MODELS_ROOT="/path/to/model_cache" pytest tests/integration/defs/cpp/test_unit_tests.py::test_unit_tests[runtime-90] -pytest tests/integration/defs/cpp/test_e2e.py::test_model[llama-90] - -pytest tests/integration/defs/cpp/test_e2e.py::test_benchmarks[gpt-90] - pytest tests/integration/defs/cpp/test_multi_gpu.py::TestDisagg::test_symmetric_executor[gpt-mpi_kvcache-90] ``` diff --git a/docs/source/legacy/advanced/disaggregated-service.md b/docs/source/legacy/advanced/disaggregated-service.md index 4e23ad5179a8..f43b9292dc27 100644 --- a/docs/source/legacy/advanced/disaggregated-service.md +++ b/docs/source/legacy/advanced/disaggregated-service.md @@ -10,7 +10,7 @@ Currently TRT-LLM supports `disaggregated-service`, where the context and genera For more information on disaggregated service in LLM inference, one can refer to papers such as [DistServe](https://arxiv.org/abs/2401.09670), [SplitWise](https://arxiv.org/abs/2311.18677). -An [architectural and performance overview](../../../docs/source/blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md), as well as [usage examples](../../../examples/disaggregated/README.md), are provided. +An [architectural and performance overview](../../blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md), as well as [usage examples](../../../../examples/disaggregated/README.md), are provided. ## Environment Variables diff --git a/docs/source/legacy/architecture/model-weights-loader.md b/docs/source/legacy/architecture/model-weights-loader.md index d41191364f1e..1a3c9d51b729 100644 --- a/docs/source/legacy/architecture/model-weights-loader.md +++ b/docs/source/legacy/architecture/model-weights-loader.md @@ -195,7 +195,7 @@ loader = ModelWeightsLoader(external_checkpoint_dir, llava_dict) loader.generate_tllm_weights(trtllm_model) ``` Users need to specify the different part from the default `tllm_to_externel_key_dict`. The loader still have support across different precisions. -The support for LLaVA and Exaone is in `LLaMAForCausalLM.from_hugging_face()` of [model.py](../../../tensorrt_llm/models/llama/model.py), and can also be taken as examples. +The support for LLaVA and Exaone is in `LLaMAForCausalLM.from_hugging_face()` of [model.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tensorrt_llm/models/llama/model.py), and can also be taken as examples. ### Models with customized weight layout For models with different weight layout, users can write the conversion loop explicitly and do customized operations. @@ -229,7 +229,7 @@ for tllm_key, _ in tqdm(trtllm_model.named_parameters()): loader.fill(tllm_weights) ``` This will apply `preprocess` after `load_tensor()` and before `postprocess`, and demonstrates how to convert the loaded shard into default HF layout. The loader still have support for precisions quantized from FP16/BF16 (e.g. INT8-wo/INT4-wo), the other precisions may require special operations, and can be addressed inside the `preprocess` function. -The support for Qwen-1 is in `QWenForCausalLM.from_hugging_face()` of [model.py](../../../tensorrt_llm/models/qwen/model.py), and can also be taken as example. +The support for Qwen-1 is in `QWenForCausalLM.from_hugging_face()` of [model.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tensorrt_llm/models/qwen/model.py), and can also be taken as example. ### Fully customized If the model weights loader cannot satisfy the requirements, users can write the conversion loop totally on their own. diff --git a/docs/source/legacy/performance/performance-tuning-guide/deciding-model-sharding-strategy.md b/docs/source/legacy/performance/performance-tuning-guide/deciding-model-sharding-strategy.md index 2970d9e7e444..ca47c07e3601 100644 --- a/docs/source/legacy/performance/performance-tuning-guide/deciding-model-sharding-strategy.md +++ b/docs/source/legacy/performance/performance-tuning-guide/deciding-model-sharding-strategy.md @@ -11,12 +11,12 @@ Splitting your model weights between multiple GPUs requires them to communicate In pipeline parallelism, the model is split into sets of contiguous layers and each GPU houses one of these sets. In this setup, the only required communication is for each GPU to send the outputs of its set to the GPU with the next set. -![Pipeline Parallel Visualization](../../media/Pipeline_Parallel_Vis.svg) +![Pipeline Parallel Visualization](../../../media/Pipeline_Parallel_Vis.svg) On the other hand, tensor parallelism takes each layer of the model and splits it between the GPUs. This means that every GPU houses a portion of every layer. However since each layer needs the full outputs of the previous layer as an input, each GPU has to perform the heavier All-Reduce communication operation to share its results with all other GPUs before it can begin processing the next layer. While this seems disadvantageous, because each GPU only holds partial layers, it also performs smaller matrix multiplications, allowing it to compute its outputs quicker. - ![Tensor Parallel Visualization](../../media/Tensor_Parallelism_Vis.svg) + ![Tensor Parallel Visualization](../../../media/Tensor_Parallelism_Vis.svg) Ultimately deciding the best strategy comes down to whether the extra overhead from the All-Reduce operation overshadows the gains from the smaller matrix multiplications. If the interconnects between the GPUs are sufficiently fast, the gains from the reduced computation burden per layer can outweigh the additional communication cost. Consequently, a general rule of thumb is that if your GPUs have fast connections between them like NVLink then tensor parallel is likely a good choice. However if the communication will go over slow connections (across nodes for example) pipeline parallel is likely better. Overall we provide the following guidelines: diff --git a/docs/source/legacy/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md b/docs/source/legacy/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md index ed09e344b717..e1bbc8bfef02 100644 --- a/docs/source/legacy/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md +++ b/docs/source/legacy/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md @@ -10,26 +10,26 @@ One of TensorRT-LLM's key features is its inflight batching scheduler and runtim This section visualizes how TensorRT-LLM schedules requests based on max-batch size and max-num tokens. The example starts out with a newly initialized engine as well as a few unscheduled requests that have come in. For the sake of this example, toy values are set to `max batch size = 4` and `max num tokens = 12`. Each square block represents a token, and its color represents which request it belongs to. -![TRT-LLM Scheduler Visualization 1](../../media/TRTLLM_Scheduler_Vis_1.svg) +![TRT-LLM Scheduler Visualization 1](../../../media/TRTLLM_Scheduler_Vis_1.svg) Now the scheduler takes the first two requests, Request 1 and Request 2, and schedules them to execute the context phase. However, it cannot schedule any more requests because the prompts of the first two requests had 5 tokens each, leaving a budget of 2 tokens due to the max num tokens limit. Since all remaining requests have more than 2 prompt tokens none of them can be scheduled (context chunking can help in this situation, see the paged context attention section below). The tokens are marked with a "C" on them to represent that they are prompt tokens that were processed in the context phase. > Note: The tokens for different requests are shown on different rows simply for visualization purposes and are not representative of actual memory layouts -![TRT-LLM Scheduler Visualization 2](../../media/TRTLLM_Scheduler_Vis_2.svg) +![TRT-LLM Scheduler Visualization 2](../../../media/TRTLLM_Scheduler_Vis_2.svg) Now the engine runs an iteration of execution, completing the context phases for both of the scheduled requests. After it is done, the kv-cache of the prompts for both requests have been created and the first token has been generated. Tokens that were generated are marked with "G(n)" - for example a token marked "G1" represents that it is the first token generated for its request. TRT-LLM prioritizes scheduling requests in generation phase first so the two generated tokens are queued to be processed in the next iteration. Now, since the two previously scheduled requests have entered generation phase and only take up two tokens out of the max num token budget of 12, the scheduler is able to schedule two additional requests, Request 3 and Request 4. It cannot schedule the last request, Request 5, even though there is space for it in the max num tokens budget because of the max batch size limit of 4. -![TRT-LLM Scheduler Visualization 3](../../media/TRTLLM_Scheduler_Vis_3.svg) +![TRT-LLM Scheduler Visualization 3](../../../media/TRTLLM_Scheduler_Vis_3.svg) After the next iteration of execution, the second tokens for Requests 1 and 2 have been generated, and the first tokens for Request 3 and 4 have been generated. Lets say that G2 that was generated for Request 1 is the stop token, signifying that Request 1 is completed. In this case the scheduler would evict Request 1 before performing another execution iteration and prepare to return it to the user. This eviction puts the state of the engine below the max batch size limit and allows Request 5 to be scheduled. Another thing to note is that G1 that was generated for Request 2 has been added to the kv-cache for request 2, representing how kv-cache for a request grows as more and more tokens are generated. -![TRT-LLM Scheduler Visualization 4](../../media/TRTLLM_Scheduler_Vis_4.svg) +![TRT-LLM Scheduler Visualization 4](../../../media/TRTLLM_Scheduler_Vis_4.svg) Overall, the max batch size and max num tokens limits play a key part in deciding when requests are actually executed, and tuning them can have significant impacts on throughput numbers as well as how the engine balances previously scheduled requests in generation phase with context phase on new requests @@ -101,7 +101,7 @@ Previously we recommended enabling paged context attention even though in our ca The [visualization](#understanding-the-trt-llm-scheduler) of the TensorRT-LLM scheduler showed that initially Request 3 couldn't be scheduled because it would put the scheduler over the max-num tokens limit. However with context chunking, this is no longer the case, and the first chunk of Request 3 would be able to be scheduled. -![TRT-LLM Scheduler Visualization Chunked Context 1](../../media/TRTLLM_Scheduler_Vis_Chunked_Context_1.svg) +![TRT-LLM Scheduler Visualization Chunked Context 1](../../../media/TRTLLM_Scheduler_Vis_Chunked_Context_1.svg) This is extremely beneficial for several reasons. Firstly it eliminates the possibility of requests with large prompts relative to max num tokens being unable to be scheduled due to other requests that are already in-flight. In production workloads, this can help improve worst case TTFT numbers. Secondly it allows for setting smaller values of max num tokens since you no longer need max num tokens to be at least as large as the longest prompt you want to support. For long-context cases this is extremely important, because setting extremely large values of max-num tokens takes away from memory available to be used as kv-cache. Given that in the worst case scenario chunked context has minimal impact on performance but can significantly benefit it in many scenarios, it's recommended that you always enable it. diff --git a/docs/source/legacy/reference/troubleshooting.md b/docs/source/legacy/reference/troubleshooting.md index fac5c3d18c01..c840d3e87b8d 100644 --- a/docs/source/legacy/reference/troubleshooting.md +++ b/docs/source/legacy/reference/troubleshooting.md @@ -18,7 +18,7 @@ Solution: try running build script with `--clean`, or try running `rm -r build c ## Debug on Unit Tests -Here is an example to print the values of the MLP output tensor in a unit test ([full example](../../../../tests/unittest/others/test_debugging_api.py)). +Here is an example to print the values of the MLP output tensor in a unit test ([full example](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tests/unittest/others/test_debugging_api.py)). 1. Register the intermediate tensors as the network outputs with `register_network_output` API. diff --git a/docs/source/torch/kv_cache_manager.md b/docs/source/torch/kv_cache_manager.md index 57d02c8ab59f..a95499e0e389 100644 --- a/docs/source/torch/kv_cache_manager.md +++ b/docs/source/torch/kv_cache_manager.md @@ -51,8 +51,8 @@ There are also interfaces for warming up `PyTorchModelEngine`, especially when u ## Customize KV Cache Manager To customize `KVCacheManager`, implement all the necessary interfaces. -Then, integrate it into the `PyExecutor`. For the PyTorch backend, the relevant code is in [pytorch_model_registry.py](../../../tensorrt_llm/_torch/pyexecutor/backend_registries/pytorch_model_registry.py). -In the `create_pytorch_model_based_executor` function, the `KVCacheManager` is instantiated as follows: +Then, integrate it into the `PyExecutor`. For the PyTorch backend, the relevant code is in [_util.py](../../../tensorrt_llm/_torch/pyexecutor/_util.py). +In `KvCacheCreator._create_kv_cache_manager`, the `KVCacheManager` is instantiated along these lines: ```python kv_cache_manager = KVCacheManager( diff --git a/examples/auto_deploy/README.md b/examples/auto_deploy/README.md index eb59f73c687a..2ab1478d3ed1 100644 --- a/examples/auto_deploy/README.md +++ b/examples/auto_deploy/README.md @@ -176,7 +176,7 @@ For expert users, `build_and_run_ad.py` provides advanced configuration capabili #### CLI Arguments with Dot Notation -The script supports flexible CLI argument parsing using dot notation to modify nested configurations dynamically. You can target any field in both the [`ExperimentConfig`](./build_and_run_ad.py) and nested [`AutoDeployConfig`](../../tensorrt_llm/_torch/auto_deploy/llm_args.py)/[`LlmArgs`](../../tensorrt_llm/_torch/auto_deploy/llm_args.) objects: +The script supports flexible CLI argument parsing using dot notation to modify nested configurations dynamically. You can target any field in both the [`ExperimentConfig`](./build_and_run_ad.py) and nested [`AutoDeployConfig`](../../tensorrt_llm/_torch/auto_deploy/llm_args.py)/[`LlmArgs`](../../tensorrt_llm/_torch/auto_deploy/llm_args.py) objects: ```bash # Configure model parameters diff --git a/examples/models/contrib/hyperclovax/README.md b/examples/models/contrib/hyperclovax/README.md index e880dd745c76..b52b04600a7a 100644 --- a/examples/models/contrib/hyperclovax/README.md +++ b/examples/models/contrib/hyperclovax/README.md @@ -90,7 +90,7 @@ The output will be like: For more information, you can refer to [examples/llm-api](../../../llm-api). ## TRT flow -The next section describes how to convert the weights from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. We will use llama's [convert_checkpoint.py](../../core/llama/convert_checkpoint.py) for the HyperCLOVAX model and then build the model with `trtllm-build`. +The next section describes how to convert the weights from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. We will use llama's [convert_checkpoint.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/convert_checkpoint.py) for the HyperCLOVAX model and then build the model with `trtllm-build`. ### Convert checkpoint and build TensorRT engine(s) @@ -230,7 +230,7 @@ trtllm-build \ ``` ### Run Engine -Test your engine with the [run.py](../../../run.py) script: +Test your engine with the [run.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/run.py) script: ```bash python3 ../../../run.py \ @@ -254,5 +254,5 @@ python ../../../summarize.py \ --engine_dir trt_engines/$MODEL_NAME/fp16/1-gpu ``` -The TensorRT LLM HyperCLOVAX implementation is based on the LLaMA model. The implementation can be found in [llama/model.py](../../../../tensorrt_llm/models/llama/model.py). -For more examples, see [`examples/models/core/llama/README.md`](../../core/llama/README.md) +The TensorRT LLM HyperCLOVAX implementation is based on the LLaMA model. The implementation can be found in [llama/model.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/tensorrt_llm/models/llama/model.py). +For more examples, see [`examples/models/core/llama/README.md`](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/README.md) diff --git a/examples/models/core/exaone/README.md b/examples/models/core/exaone/README.md index 1c6ddad4c067..33601c88631f 100644 --- a/examples/models/core/exaone/README.md +++ b/examples/models/core/exaone/README.md @@ -394,7 +394,7 @@ Once all servers are running, you can send requests to `http://localhost:8000/v1 ## TRT flow -The next section describes how to convert weights from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. We will use LLaMA's [convert_checkpoint.py](../llama/convert_checkpoint.py) for EXAONE models and then build the model with `trtllm-build`. +The next section describes how to convert weights from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. We will use LLaMA's [convert_checkpoint.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/convert_checkpoint.py) for EXAONE models and then build the model with `trtllm-build`. ### Convert checkpoint and build TensorRT engine(s) @@ -534,7 +534,7 @@ trtllm-build \ ### Run Engine -Test your engine with the [run.py](../../../run.py) script: +Test your engine with the [run.py](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/run.py) script: ```bash python3 ../../../run.py \ @@ -558,7 +558,7 @@ python ../../../summarize.py \ --engine_dir trt_engines/exaone/fp16/1-gpu ``` -For more examples regarding EXAONE-3.0 & EXAONE-Deep's TRT flow, see [`examples/models/core/llama/README.md`](../llama/README.md) +For more examples regarding EXAONE-3.0 & EXAONE-Deep's TRT flow, see [`examples/models/core/llama/README.md`](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/models/core/llama/README.md) diff --git a/jenkins/scripts/perf/README.md b/jenkins/scripts/perf/README.md index 8c2f89679615..7456882bfe78 100644 --- a/jenkins/scripts/perf/README.md +++ b/jenkins/scripts/perf/README.md @@ -324,7 +324,7 @@ def buildStageConfigs(stageName, platform, testlist, testCount, gpuCount, nodeCo When adding a test, either increment `testCount` on an existing entry or add a new `buildStageConfigs` block. Stages are grouped by node count (2 Nodes, 3 Nodes, 4 Nodes, etc.). -For the full step-by-step guide including how to derive test-db filenames and GPU/node counts from disaggregated config YAMLs, see [`tests/scripts/perf-sanity/README.md`](../../tests/scripts/perf-sanity/README.md) ("Step-by-Step: Adding or Re-enabling Disaggregated Perf Sanity Tests"). +For the full step-by-step guide including how to derive test-db filenames and GPU/node counts from disaggregated config YAMLs, see [`tests/integration/defs/perf/README_test_perf_sanity.md`](../../../tests/integration/defs/perf/README_test_perf_sanity.md) ("Step-by-Step: Adding or Re-enabling Disaggregated Perf Sanity Tests"). ## Post-Processing and Triage diff --git a/tensorrt_llm/scaffolding/README.md b/tensorrt_llm/scaffolding/README.md index ff6adc0672e7..7f887abd2efd 100644 --- a/tensorrt_llm/scaffolding/README.md +++ b/tensorrt_llm/scaffolding/README.md @@ -38,6 +38,6 @@ Future work includes the following aspects: - Provide information for combined performance optimization with backends. Scaffolding can provide some information which is helpful for LLM inference engine. For a example, Controller may aware the prefix relation between generation requests, that would be helpful for kvcache reuse. -- Develop auxiliary components to support generic requirements. Now we have developed some interesting auxiliary components. For an [example](examples/scaffolding/token_budget_majority_vote.py), we developed `GenerationTokenCounter` as a task collection decorator so that Controller could get the output tokens count for itself and its Sub-Controller. There are still many such works waiting for us to do. +- Develop auxiliary components to support generic requirements. Now we have developed some interesting auxiliary components. For an [example](../../examples/scaffolding/token_budget_majority_vote.py), we developed `GenerationTokenCounter` as a task collection decorator so that Controller could get the output tokens count for itself and its Sub-Controller. There are still many such works waiting for us to do. You can see more specific work in this [link](https://github.com/NVIDIA/TensorRT-LLM/issues/3706#issuecomment-2820015957). diff --git a/tests/integration/defs/accuracy/README.md b/tests/integration/defs/accuracy/README.md index 063e1944bcc2..ab890aaf6284 100644 --- a/tests/integration/defs/accuracy/README.md +++ b/tests/integration/defs/accuracy/README.md @@ -30,9 +30,9 @@ The following tasks are currently supported: \* Rouge is an informal evaluation metric for code completion. -For LLM API supported tasks, the core evaluation logics (i.e., evaluators) are implemented in the [`tensorrt_llm.evaluate`](../../../../tensorrt_llm/evaluate) module. They are also shared with the CLI tool [`trtllm-eval`](../../../../examples/trtllm-eval). CLI flow tasks typically require calling a standalone script like [`summarize.py`](../../../../examples/summarize.py). +For LLM API supported tasks, the core evaluation logics (i.e., evaluators) are implemented in the [`tensorrt_llm.evaluate`](../../../../tensorrt_llm/evaluate) module. They are also shared with the CLI tool [`trtllm-eval`](../../../../examples/trtllm-eval). CLI flow tasks typically require calling a standalone script like [`summarize.py`](https://github.com/NVIDIA/TensorRT-LLM/blob/v1.2.0/examples/summarize.py). -New accuracy tests are strongly recommended to be added to this test suite, in particular, in the LLM API style (i.e., [test_llm_api.py](./test_llm_api.py) and [test_llm_api_pytorch.py](./test_llm_api_pytorch.py)). There are some legacy accuracy tests outside this test suite (e.g., the tests in [examples](../examples) folder), but they are not recommended anymore. +New accuracy tests are strongly recommended to be added to this test suite, in particular, in the LLM API style (i.e., [test_llm_api_pytorch.py](./test_llm_api_pytorch.py)). There are some legacy accuracy tests outside this test suite (e.g., the tests in [examples](../examples) folder), but they are not recommended anymore. ## Background: Why This Test Suite? @@ -138,7 +138,7 @@ If all the evaluated accuracies are equal to or higher than the corresponding th ### Add New Test Cases with Existing Tasks -We suggest supporting the model with LLM API, and then add tests to [test_llm_api.py](./test_llm_api.py) or [test_llm_api_pytorch.py](./test_llm_api_pytorch.py). Typically, a test class is responsible for a model (corresponding to a unique Hugging Face model ID); it contains several test methods for different features (e.g., quantizations, parallelisms). For example, in [test_llm_api_pytorch.py](./test_llm_api_pytorch.py) the model [`meta-llama/Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) has the test class defined as: +We suggest supporting the model with LLM API, and then add tests to [test_llm_api_pytorch.py](./test_llm_api_pytorch.py). Typically, a test class is responsible for a model (corresponding to a unique Hugging Face model ID); it contains several test methods for different features (e.g., quantizations, parallelisms). For example, in [test_llm_api_pytorch.py](./test_llm_api_pytorch.py) the model [`meta-llama/Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) has the test class defined as: ```python class TestLlama3_1_8BInstruct(LlmapiAccuracyTestHarness): @@ -267,7 +267,7 @@ The remaining fields of the task class should be straightforward: Please see `AccuracyTask.evaluate` in [accuracy_core.py](./accuracy_core.py) for how `EVALUATOR_CLS` and `EVALUATOR_KWARGS` are used. -The new task class is all set. Use it in [test_llm_api.py](./test_llm_api.py) or [test_llm_api_pytorch.py](./test_llm_api_pytorch.py). +The new task class is all set. Use it in [test_llm_api_pytorch.py](./test_llm_api_pytorch.py). ## Hypothesis Testing Methodology diff --git a/tests/integration/defs/perf/README_release_test.md b/tests/integration/defs/perf/README_release_test.md index 35c5d34a28b9..13aa38c5a6fc 100644 --- a/tests/integration/defs/perf/README_release_test.md +++ b/tests/integration/defs/perf/README_release_test.md @@ -109,17 +109,12 @@ if self._config.backend == "pytorch": ### 3.1 Full Test Cycles -1. **llm_perf_full.yml** - Release performance test - - [test_lists/qa/llm_perf_full.yml](../../test_lists/qa/llm_perf_full.yml) -2. **llm_perf_cluster.yml** - Cluster performance test(for Blackwell) - - [test_lists/qa/llm_perf_cluster.yml](../../test_lists/qa/llm_perf_cluster.yml) -3. **llm_perf_nim.yml** - NIM performance test - - [test_lists/qa/llm_perf_nim.yml](../../test_lists/qa/llm_perf_nim.yml) - -### 3.2 Sanity Test Cycles - -- **llm_perf_sanity.yml** - Release performance sanity test - - [test_lists/qa/llm_perf_sanity.yml](../../test_lists/qa/llm_perf_sanity.yml) +1. **llm_perf_core.yml** - Release performance test; the former `llm_perf_full.yml`, `llm_perf_cluster.yml` and `llm_perf_sanity.yml` were merged into it + - [test_lists/qa/llm_perf_core.yml](../../test_lists/qa/llm_perf_core.yml) +2. **llm_perf_disagg.yml** - Disaggregated serving performance test + - [test_lists/qa/llm_perf_disagg.yml](../../test_lists/qa/llm_perf_disagg.yml) +3. **llm_perf_multinode.yml** - Multi-node performance test + - [test_lists/qa/llm_perf_multinode.yml](../../test_lists/qa/llm_perf_multinode.yml) ## 4. Test Configuration Description