diff --git a/tests/README.md b/tests/README.md index 582942b68629..11e15c18ac78 100644 --- a/tests/README.md +++ b/tests/README.md @@ -55,7 +55,7 @@ pip install -r requirements-dev.txt cd tests/integration/defs # example 1: run a case -pytest "accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_auto_dtype" +pytest "accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_dummy_load_format" # example 2: run a test list pytest --rootdir . --test-list= diff --git a/tests/integration/defs/accuracy/README.md b/tests/integration/defs/accuracy/README.md index ab890aaf6284..33a3b9f06ba1 100644 --- a/tests/integration/defs/accuracy/README.md +++ b/tests/integration/defs/accuracy/README.md @@ -8,7 +8,6 @@ In addition, most tests are based on the offline API -- [LLM API](https://nvidia This test suite is organized as following: * [accuracy_core.py](./accuracy_core.py) provides the test harness, including hypothesis testing logics, evaluation task configurations, and common utilities. -* [test_cli_flow.py](./test_cli_flow.py) contains the tests with CLI workflow, i.e., checkpoint conversion, engine building and evaluation. * [test_llm_api_pytorch.py](./test_llm_api_pytorch.py) contains the tests with LLM API and PyTorch backend. * [references](./references) registers the reference accuracies for each task, each model and each specification (e.g., data type, quantization). * [scripts](./scripts) provides some utility scripts that may help setup accuracy tests. @@ -110,19 +109,17 @@ The accuracy references are registered in the YAML files in [references](./refer * Model level: Each model is indexed by its unique Hugging Face model ID in each YAML file. * Accuracy specification level: Each accuracy specification is some feature combination that has justifiable accuracy difference from the default accuracy. -For example, in [references/mmlu.yaml](./references/mmlu.yaml) the model [`meta-llama/Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) has accuracy references as following: +For example, in [references/mmlu.yaml](./references/mmlu.yaml) the model [`mistralai/Ministral-8B-Instruct-2410`](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) has accuracy references as following: ```yaml -meta-llama/Llama-3.1-8B-Instruct: - - accuracy: 68.17 - - quant_algo: FP8 - accuracy: 67.93 +mistralai/Ministral-8B-Instruct-2410: + - accuracy: 66.35 - quant_algo: FP8 kv_cache_quant_algo: FP8 - accuracy: 67.87 + accuracy: 65.96 ``` -The first item is the default accuracy specification (i.e., using original Hugging Face model data type and no quantization), and the reference accuracy is 68.17. The second item is an accuracy specification with FP8 GEMM quantization, with a slightly lower reference accuracy 67.93. The third item is a specification with FP8 GEMM and KV cache quantization, with a further slightly lower reference accuracy 67.87. +The first item is the default accuracy specification (i.e., using original Hugging Face model data type and no quantization), and the reference accuracy is 66.35. The second item is an accuracy specification with FP8 GEMM and KV cache quantization, with a slightly lower reference accuracy 65.96. Model data type and quantization decide the precision in model computation, so accuracy differences can be *justified* if different data types or quantizations are used. Hence, they are the most typical components in accuracy specifications. Please see other categories of accuracy specifications documented in `AccuracyTask.get_hypothesis_testing_params` in [accuracy_core.py](./accuracy_core.py). Note that we exclude most inference features such as parallelism, because theoretically they should not affect model accuracy. Think from the opposite perspective, if enabling tensor parallelism results in statistically significant accuracy loss, we might need to check whether some accuracy bugs exist. @@ -138,14 +135,14 @@ If all the evaluated accuracies are equal to or higher than the corresponding th ### Add New Test Cases with Existing Tasks -We suggest supporting the model with LLM API, and then add tests to [test_llm_api_pytorch.py](./test_llm_api_pytorch.py). Typically, a test class is responsible for a model (corresponding to a unique Hugging Face model ID); it contains several test methods for different features (e.g., quantizations, parallelisms). For example, in [test_llm_api_pytorch.py](./test_llm_api_pytorch.py) the model [`meta-llama/Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) has the test class defined as: +We suggest supporting the model with LLM API, and then add tests to [test_llm_api_pytorch.py](./test_llm_api_pytorch.py). Typically, a test class is responsible for a model (corresponding to a unique Hugging Face model ID); it contains several test methods for different features (e.g., quantizations, parallelisms). For example, in [test_llm_api_pytorch.py](./test_llm_api_pytorch.py) the model [`mistralai/Ministral-8B-Instruct-2410`](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) has the test class defined as: ```python -class TestLlama3_1_8BInstruct(LlmapiAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct" - MODEL_PATH = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" +class TestMinistral8BInstruct(LlmapiAccuracyTestHarness): + MODEL_NAME = "mistralai/Ministral-8B-Instruct-2410" + MODEL_PATH = f"{llm_models_root()}/Ministral-8B-Instruct-2410" - def test_bfloat16(self, ...): + def test_auto_dtype(self, ...): # create an LLM instance with tested features enabled, optionally with pytest parameters llm = LLM(self.MODEL_PATH, ...) # use a context manager to explicitly deconstruct the LLM instance upon exiting @@ -167,7 +164,7 @@ The last step is registering the accuracy reference. If the new test case shares Otherwise, run the new test case without reference by prepending `TRTLLM_ACCURACY_NO_REFERENCE=1`. For example, ```bash -TRTLLM_ACCURACY_NO_REFERENCE=1 pytest -vs "test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=TRTLLM-torch_compile]" +TRTLLM_ACCURACY_NO_REFERENCE=1 pytest -vs "test_llm_api_pytorch.py::TestMinistral8BInstruct::test_auto_dtype" ``` The results would look like: @@ -183,8 +180,6 @@ We can clearly see the evaluated accuracies from the test logs. If the accuracie The new test case is all set. See [tests/README.md](../../../README.md) for how to register the new case to the CI or QA list. -If the model supports CLI flow only, please follow other cases in [test_cli_flow.py](./test_cli_flow.py). - ### Add New Tasks We recommend reading [Hypothesis Testing Methodology](#hypothesis-testing-methodology) before introducing a new evaluation task. diff --git a/tests/integration/defs/accuracy/accuracy_core.py b/tests/integration/defs/accuracy/accuracy_core.py index 0d8ed4ca0e1f..c017a22e53e1 100644 --- a/tests/integration/defs/accuracy/accuracy_core.py +++ b/tests/integration/defs/accuracy/accuracy_core.py @@ -13,12 +13,10 @@ # See the License for the specific language governing permissions and # limitations under the License. import gc -import json import math import os -import tempfile from dataclasses import dataclass, field -from typing import Dict, List, Optional, Union +from typing import List, Optional, Union import pytest import scipy @@ -30,15 +28,11 @@ from tensorrt_llm._torch.auto_deploy import LLM as AutoDeployLLM from tensorrt_llm.evaluate.audio_asr import AudioASREvaluator from tensorrt_llm.llmapi import SamplingParams -from tensorrt_llm.llmapi.llm_args import DecodingBaseConfig, TorchLlmArgs +from tensorrt_llm.llmapi.llm_args import DecodingBaseConfig from tensorrt_llm.logger import logger -from tensorrt_llm.models.modeling_utils import QuantConfig -from tensorrt_llm.quantization import QuantAlgo from tensorrt_llm.sampling_params import LogitsProcessor -from ..common import venv_check_call, venv_mpi_check_call from ..conftest import llm_models_root -from ..trt_test_alternative import check_call, exists from .video_mme import VideoMME as VideoMMEEvaluator @@ -176,7 +170,7 @@ def assert_acceptance_length(test_key: str, al_value: float) -> None: Args: test_key: Key in acceptance_length.yaml identifying the test variant, - e.g. ``"TestLlama3_1_8BInstruct::test_dflash"``. + e.g. ``"TestGPTOSS::test_dflash"``. al_value: Observed mean acceptance length to check. Population: @@ -662,418 +656,6 @@ class LongBenchV1(AccuracyTask): apply_chat_template=True) -class CliFlowAccuracyTestHarness: - # Model - MODEL_NAME = None - MODEL_PATH = None - MODEL_FORMAT = "HF" - EXAMPLE_FOLDER = None - - @pytest.fixture(autouse=True, scope="class") - @classmethod - def setup_class(cls, request): - cls.llm_venv = request.getfixturevalue("llm_venv") - cls.llm_root = request.getfixturevalue("llm_root") - - @pytest.fixture(autouse=True, scope="function") - def setup_method(self): - with tempfile.TemporaryDirectory( - prefix=self.MODEL_NAME.replace("/", "-"), - dir=self.llm_venv.get_working_directory()) as workspace: - self.ckpt_dir = f"{workspace}/cmodels" - self.engine_dir = f"{workspace}/engines" - yield - - def install_requirements(self): - requirements = f"{self.llm_root}/examples/{self.EXAMPLE_FOLDER}/requirements.txt" - if exists(requirements): - self.llm_venv.run_cmd( - ["-m", "pip", "install", "-r", requirements], - env={ - "CMAKE_POLICY_VERSION_MINIMUM": - "3.5" # https://github.com/google/sentencepiece/issues/1111 - }) - - def initialize_case(self, - tasks: Optional[List[AccuracyTask]] = None, - dtype: str = 'auto', - quant_algo: Optional[str] = None, - kv_cache_quant_algo: Optional[str] = None, - spec_dec_algo: Optional[str] = None, - extra_acc_spec: Optional[str] = None, - tp_size: int = 1, - pp_size: int = 1, - cp_size: int = 1, - extra_convert_args: Optional[list] = None, - extra_build_args: Optional[list] = None, - extra_summarize_args: Optional[list] = None, - extra_mmlu_args: Optional[list] = None, - extra_eval_long_context_args: Optional[list] = None, - env: Optional[Dict[str, str]] = None): - self.tasks = [CnnDailymail(self.MODEL_NAME)] if tasks is None else tasks - self.dtype = dtype - self.quant_algo = quant_algo - self.kv_cache_quant_algo = kv_cache_quant_algo - self.spec_dec_algo = spec_dec_algo - self.extra_acc_spec = extra_acc_spec - self.tp_size = tp_size - self.pp_size = pp_size - self.cp_size = cp_size - self.extra_convert_args = extra_convert_args - self.extra_build_args = extra_build_args - self.extra_summarize_args = extra_summarize_args - self.extra_mmlu_args = extra_mmlu_args - self.extra_eval_long_context_args = extra_eval_long_context_args - self.env = env - - def convert(self): - logger.info("Converting model to TensorRT LLM checkpoint...") - - is_prequantized = False - for quant_config_file in [ - "hf_quant_config.json", "quant_config.json", - "quantize_config.json" - ]: - if exists(f"{self.MODEL_PATH}/{quant_config_file}"): - is_prequantized = True - break - if not is_prequantized and exists(f"{self.MODEL_PATH}/config.json"): - with open(f"{self.MODEL_PATH}/config.json") as f: - hf_config = json.load(f) - if "quantization_config" in hf_config: - is_prequantized = True - - quant_config = QuantConfig(quant_algo=self.quant_algo, - kv_cache_quant_algo=self.kv_cache_quant_algo) - if not is_prequantized and quant_config._requires_modelopt_quantization: - script = f"{self.llm_root}/examples/quantization/quantize.py" - else: - script = f"{self.llm_root}/examples/{self.EXAMPLE_FOLDER}/convert_checkpoint.py" - - convert_cmd = [ - script, - f"--output_dir={self.ckpt_dir}", - f"--dtype={self.dtype}", - ] - - if "nemotron_nas" in self.EXAMPLE_FOLDER: - convert_cmd.append("--trust_remote_code") - - if self.MODEL_FORMAT == "NEMO": - convert_cmd.append(f"--nemo_ckpt_path={self.MODEL_PATH}") - else: - convert_cmd.append(f"--model_dir={self.MODEL_PATH}") - - if self.tp_size > 1: - convert_cmd.append(f"--tp_size={self.tp_size}") - if self.pp_size > 1: - convert_cmd.append(f"--pp_size={self.pp_size}") - if self.cp_size > 1: - convert_cmd.append(f"--cp_size={self.cp_size}") - - if not is_prequantized and quant_config._requires_modelopt_quantization: - if self.quant_algo == QuantAlgo.MIXED_PRECISION: - assert self.extra_convert_args is not None - assert any( - x.startswith("--autoq_format") - for x in self.extra_convert_args) - else: - convert_cmd.append( - f"--qformat={quant_config._get_modelopt_qformat()}") - if (kv_cache_dtype := - quant_config._get_modelopt_kv_cache_dtype()) is not None: - convert_cmd.append(f"--kv_cache_dtype={kv_cache_dtype}") - else: - if self.quant_algo == QuantAlgo.NVFP4: - convert_cmd.append("--use_nvfp4") - elif self.quant_algo == QuantAlgo.FP8: - if self.EXAMPLE_FOLDER != "models/core/gpt": # --use_fp8 flag is not needed for gpt. - convert_cmd.append("--use_fp8") - elif self.quant_algo == QuantAlgo.FP8_PER_CHANNEL_PER_TOKEN: - convert_cmd.append("--use_fp8_rowwise") - elif quant_config._use_plugin_sq: - convert_cmd.append("--smoothquant=0.5") - if "PER_TOKEN" in self.quant_algo: - convert_cmd.append("--per_token") - if "PER_CHANNEL" in self.quant_algo: - convert_cmd.append("--per_channel") - elif self.quant_algo == QuantAlgo.W8A16: - convert_cmd.extend( - ["--use_weight_only", "--weight_only_precision=int8"]) - elif self.quant_algo == QuantAlgo.W4A16: - convert_cmd.extend( - ["--use_weight_only", "--weight_only_precision=int4"]) - elif self.quant_algo == QuantAlgo.W8A16_GPTQ: - convert_cmd.extend([ - "--use_weight_only", "--weight_only_precision=int8_gptq", - "--per_group", "--group_size=64" - ]) - elif self.quant_algo == QuantAlgo.W4A16_GPTQ: - convert_cmd.extend([ - "--use_weight_only", "--weight_only_precision=int4_gptq", - "--per_group" - ]) - - if self.kv_cache_quant_algo == QuantAlgo.INT8: - convert_cmd.append("--int8_kv_cache") - elif self.kv_cache_quant_algo == QuantAlgo.FP8: - if self.EXAMPLE_FOLDER != "models/core/gpt": # --fp8_kv_cache flag is not needed for gpt. - convert_cmd.append("--fp8_kv_cache") - - if quant_config._requires_calibration: - convert_cmd.append( - f"--calib_dataset={llm_models_root()}/datasets/cnn_dailymail") - - if self.extra_convert_args: - convert_cmd.extend(self.extra_convert_args) - - venv_check_call(self.llm_venv, convert_cmd) - - def build(self): - logger.info("Building engines...") - max_batch_size = max(task.MAX_BATCH_SIZE for task in self.tasks) - max_input_len = max(task.MAX_INPUT_LEN for task in self.tasks) - max_seq_len = max(task.MAX_INPUT_LEN + task.MAX_OUTPUT_LEN - for task in self.tasks) - build_cmd = [ - "trtllm-build", - f"--checkpoint_dir={self.ckpt_dir}", - f"--output_dir={self.engine_dir}", - f"--max_batch_size={max_batch_size}", - f"--max_input_len={max_input_len}", - f"--max_seq_len={max_seq_len}", - f"--workers={self.tp_size * self.pp_size * self.cp_size}", - ] - if self.extra_build_args: - build_cmd.extend(self.extra_build_args) - check_call(" ".join(build_cmd), shell=True, env=self.llm_venv._new_env) - - def summarize(self, task: AccuracyTask): - logger.info("Running summarize...") - summarize_cmd = [ - f"{self.llm_root}/examples/summarize.py", - f"--engine_dir={self.engine_dir}", - f"--hf_model_dir={self.MODEL_PATH}", - f"--max_input_length={task.MAX_INPUT_LEN}", - f"--output_len={task.MAX_OUTPUT_LEN}", - f"--dataset_dir={task.DATASET_DIR}", - f"--rouge_dir={task.ROUGE_DIR}", "--test_trt_llm", - "--random_seed=0", "--check_accuracy" - ] - if self.MODEL_FORMAT == "NEMO": - summarize_cmd.extend([ - f"--vocab_file={self.ckpt_dir}/tokenizer.model", - "--no_add_special_tokens" - ]) - - hypothesis_testing_params = task.get_hypothesis_testing_params( - dtype=self.dtype, - quant_algo=self.quant_algo, - kv_cache_quant_algo=self.kv_cache_quant_algo, - spec_dec_algo=self.spec_dec_algo, - extra_acc_spec=self.extra_acc_spec) - logger.info( - f"Hypothesis testing report:\n{hypothesis_testing_params.report()}") - num_samples = hypothesis_testing_params.num_samples - threshold = hypothesis_testing_params.threshold - - if num_samples < task.MAX_BATCH_SIZE: - max_ite = 1 - batch_size = num_samples - else: - max_ite = math.ceil(num_samples / task.MAX_BATCH_SIZE) - batch_size = task.MAX_BATCH_SIZE - summarize_cmd.extend([ - f"--batch_size={batch_size}", f"--max_ite={max_ite}", - f"--tensorrt_llm_rouge1_threshold={threshold}" - ]) - - if isinstance(task, Humaneval): - summarize_cmd.append("--eval_task=code_completion") - elif isinstance(task, ZeroScrolls): - summarize_cmd.append("--eval_task=summarize_long") - elif isinstance(task, SlimPajama6B): - max_tokens_in_paged_kv_cache = int( - batch_size * (task.MAX_INPUT_LEN + task.MAX_OUTPUT_LEN) * 1.1) - summarize_cmd.extend([ - "--eval_task=eval_context_ppl", - f"--min_input_length={task.MIN_INPUT_LEN}", - f"--max_tokens_in_paged_kv_cache={max_tokens_in_paged_kv_cache}" - ]) - - if task.MAX_INPUT_LEN + task.MAX_OUTPUT_LEN > TorchLlmArgs.model_fields[ - "max_num_tokens"].default: - summarize_cmd.append("--enable_chunked_context") - - if self.extra_summarize_args: - summarize_cmd.extend(self.extra_summarize_args) - - world_size = self.tp_size * self.pp_size * self.cp_size - if world_size == 1: - venv_check_call(self.llm_venv, summarize_cmd, env=self.env) - else: - venv_mpi_check_call( - self.llm_venv, - ["mpirun", "-n", - str(world_size), "--allow-run-as-root"], summarize_cmd) - - def mmlu(self, task: AccuracyTask): - logger.info("Running mmlu...") - hypothesis_testing_params = task.get_hypothesis_testing_params( - dtype=self.dtype, - quant_algo=self.quant_algo, - kv_cache_quant_algo=self.kv_cache_quant_algo, - spec_dec_algo=self.spec_dec_algo, - extra_acc_spec=self.extra_acc_spec) - logger.info( - f"Hypothesis testing report:\n{hypothesis_testing_params.report()}") - num_samples = hypothesis_testing_params.num_samples - threshold = hypothesis_testing_params.threshold - - mmlu_cmd = [ - "trtllm-eval", - f"--model={self.engine_dir}", - f"--tokenizer={self.MODEL_PATH}", - "--backend=tensorrt", - ] - - if self.extra_mmlu_args: - mmlu_cmd.extend(self.extra_mmlu_args) - - mmlu_cmd.extend([ - "mmlu", f"--dataset_path={task.DATASET_DIR}", - f"--num_samples={num_samples}", "--random_seed=0", - "--check_accuracy", f"--accuracy_threshold={threshold}" - ]) - - check_call(" ".join(mmlu_cmd), shell=True, env=self.llm_venv._new_env) - - def eval_long_context(self, task: AccuracyTask): - logger.info("Running construct_synthetic_dataset...") - data_gen_cmd = [ - f"{self.llm_root}/examples/infinitebench/construct_synthetic_dataset.py", - "--test_case=build_passkey", f"--test_level={task.LEVEL}" - ] - venv_check_call(self.llm_venv, data_gen_cmd) - - logger.info("Running eval_long_context...") - eval_cmd = [ - f"{self.llm_root}/examples/eval_long_context.py", "--task=passkey", - f"--engine_dir={self.engine_dir}", - f"--tokenizer_dir={self.MODEL_PATH}", - f"--max_input_length={task.MAX_INPUT_LEN}", - "--enable_chunked_context" - ] - hypothesis_testing_params = task.get_hypothesis_testing_params( - dtype=self.dtype, - quant_algo=self.quant_algo, - kv_cache_quant_algo=self.kv_cache_quant_algo, - spec_dec_algo=self.spec_dec_algo, - extra_acc_spec=self.extra_acc_spec) - logger.info( - f"Hypothesis testing report:\n{hypothesis_testing_params.report()}") - num_samples = hypothesis_testing_params.num_samples - threshold = hypothesis_testing_params.threshold - - batch_size = min(task.MAX_BATCH_SIZE, num_samples) - eval_cmd.extend([ - f"--batch_size={batch_size}", f"--stop_idx={num_samples}", - f"--tensorrt_llm_accuracy_threshold={threshold}" - ]) - - if self.extra_eval_long_context_args: - eval_cmd.extend(self.extra_eval_long_context_args) - - world_size = self.tp_size * self.pp_size * self.cp_size - if world_size == 1: - venv_check_call(self.llm_venv, eval_cmd, env=self.env) - else: - venv_mpi_check_call( - self.llm_venv, - ["mpirun", "-n", - str(world_size), "--allow-run-as-root"], eval_cmd) - - def evaluate(self): - for task in self.tasks: - if isinstance(task, - (CnnDailymail, Humaneval, ZeroScrolls, SlimPajama6B)): - self.summarize(task) - elif isinstance(task, MMLU): - self.mmlu(task) - elif isinstance(task, (PassKeyRetrieval64k, PassKeyRetrieval128k)): - self.eval_long_context(task) - else: - raise ValueError(f"Not registered dataset: {task.DATASET}.") - - def run(self, - tasks: Optional[List[AccuracyTask]] = None, - dtype: str = 'auto', - quant_algo: Optional[str] = None, - kv_cache_quant_algo: Optional[str] = None, - spec_dec_algo: Optional[str] = None, - extra_acc_spec: Optional[str] = None, - tp_size: int = 1, - pp_size: int = 1, - cp_size: int = 1, - extra_convert_args: Optional[list] = None, - extra_build_args: Optional[list] = None, - extra_summarize_args: Optional[list] = None, - extra_eval_long_context_args: Optional[list] = None, - env: Optional[Dict[str, str]] = None, - timeout_manager=None): - """Run all accuracy test phases with timeout management. - - If timeout_manager is provided, each phase will be wrapped to track and deduct remaining timeout. - """ - # Use timeout_manager to manage timeout for each phase - if timeout_manager is not None: - with timeout_manager.timed_operation("install_requirements"): - self.install_requirements() - with timeout_manager.timed_operation("initialize_case"): - self.initialize_case( - tasks=tasks, - dtype=dtype, - quant_algo=quant_algo, - kv_cache_quant_algo=kv_cache_quant_algo, - spec_dec_algo=spec_dec_algo, - extra_acc_spec=extra_acc_spec, - tp_size=tp_size, - pp_size=pp_size, - cp_size=cp_size, - extra_convert_args=extra_convert_args, - extra_build_args=extra_build_args, - extra_summarize_args=extra_summarize_args, - extra_eval_long_context_args=extra_eval_long_context_args, - env=env) - with timeout_manager.timed_operation("convert"): - self.convert() - with timeout_manager.timed_operation("build"): - self.build() - with timeout_manager.timed_operation("evaluate"): - self.evaluate() - else: - # fallback: no timeout management - self.install_requirements() - self.initialize_case( - tasks=tasks, - dtype=dtype, - quant_algo=quant_algo, - kv_cache_quant_algo=kv_cache_quant_algo, - spec_dec_algo=spec_dec_algo, - extra_acc_spec=extra_acc_spec, - tp_size=tp_size, - pp_size=pp_size, - cp_size=cp_size, - extra_convert_args=extra_convert_args, - extra_build_args=extra_build_args, - extra_summarize_args=extra_summarize_args, - extra_eval_long_context_args=extra_eval_long_context_args, - env=env) - self.convert() - self.build() - self.evaluate() - - class LlmapiAccuracyTestHarness: # Model MODEL_NAME = None diff --git a/tests/integration/defs/accuracy/references/acceptance_length.yaml b/tests/integration/defs/accuracy/references/acceptance_length.yaml index f8118fd2d3cf..8d645c427371 100644 --- a/tests/integration/defs/accuracy/references/acceptance_length.yaml +++ b/tests/integration/defs/accuracy/references/acceptance_length.yaml @@ -1,8 +1,5 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -TestLlama3_1_8BInstruct::test_eagle3: - ref_al: 1.987311931289228 - min_al: 1.8879463347247665 TestQwen3_4B::test_eagle3: ref_al: 2.02809899228679 min_al: 1.9266940426724504 @@ -12,27 +9,12 @@ TestDeepSeekV3Lite::test_bfloat16: TestNemotronV3Super::test_nvfp4_4gpus_block_reuse: ref_al: 3.4718918456404553 min_al: 3.2982972533584323 -TestLlama3_1_8BInstruct::test_dflash: - ref_al: 3.0108915792789954 - min_al: 2.8603470003150453 TestGPTOSS::test_dflash: ref_al: 2.925596923467016 min_al: 2.779317077293665 -TestLlama3_1_8BInstruct::test_pard: - ref_al: 3.3641240317850247 - min_al: 3.195917830195773 TestDeepSeekV4ProDSpark::test_gsm8k_dep8_megamoe_deepgemm: ref_al: 4.448784989234082 min_al: 4.226345739772378 -TestLlama3_1_8BInstruct::test_ngram: - ref_al: 1.3434705770970525 - min_al: 1.2762970482421998 -TestLlama3_1_8BInstruct::test_suffix_automaton: - ref_al: 1.449815006993638 - min_al: 1.377324256643956 -TestLlama3_1_8BInstruct::test_draft_target_dynamic_draft_len: - ref_al: 3.243380267550134 - min_al: 3.0812112541726275 TestQwen3_5_4B::test_dflash: ref_al: 3.96417002949639 min_al: 3.7659615280215704 diff --git a/tests/integration/defs/accuracy/references/cnn_dailymail.yaml b/tests/integration/defs/accuracy/references/cnn_dailymail.yaml index c15fab97de1e..0c70741c3bf3 100644 --- a/tests/integration/defs/accuracy/references/cnn_dailymail.yaml +++ b/tests/integration/defs/accuracy/references/cnn_dailymail.yaml @@ -31,50 +31,6 @@ TinyLlama/TinyLlama-1.1B-Chat-v1.0: accuracy: 27.882 - extra_acc_spec: pp_size=4 accuracy: 15.123 -meta-llama/Llama-3.1-8B: - - accuracy: 24.360 - - quant_algo: W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN - accuracy: 25.004 - - quant_algo: NVFP4 - kv_cache_quant_algo: FP8 - accuracy: 25.469 - - quant_algo: FP8 - kv_cache_quant_algo: FP8 - accuracy: 24.359 - - quant_algo: FP8_PER_CHANNEL_PER_TOKEN - accuracy: 24.814 - - quant_algo: FP8_PER_CHANNEL_PER_TOKEN - extra_acc_spec: meta_recipe - accuracy: 24.922 - - quant_algo: MIXED_PRECISION - extra_acc_spec: autoq_format=int4_awq,fp8,w4a8_awq;auto_quantize_bits=5.8 - accuracy: 22.721 -meta-llama/Llama-3.1-8B-Instruct: - - accuracy: 33.640 - - spec_dec_algo: Eagle - accuracy: 33.640 - - spec_dec_algo: Eagle3 - accuracy: 33.640 - - spec_dec_algo: PARD - accuracy: 33.640 - - extra_acc_spec: logprobs=2 - accuracy: 30.522 - - quant_algo: FP8 - accuracy: 33.841 - - quant_algo: FP8 - kv_cache_quant_algo: FP8 - accuracy: 33.757 - - dtype: float16 - spec_dec_algo: Medusa - accuracy: 33.663 - - quant_algo: FP8 - extra_acc_spec: temperature=0.8,top_p=0.95 - accuracy: 28.631 - - extra_acc_spec: beam_width=2 - accuracy: 31.223 - - quant_algo: FP8 - extra_acc_spec: beam_width=2 - accuracy: 31.201 mistralai/Mistral-Small-3.1-24B-Instruct-2503: - accuracy: 29.20 - quant_algo: FP8 diff --git a/tests/integration/defs/accuracy/references/gsm8k.yaml b/tests/integration/defs/accuracy/references/gsm8k.yaml index 2f29dff68fd7..dffc36089764 100644 --- a/tests/integration/defs/accuracy/references/gsm8k.yaml +++ b/tests/integration/defs/accuracy/references/gsm8k.yaml @@ -1,38 +1,5 @@ # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -meta-llama/Llama-3.1-8B-Instruct: - - accuracy: 74.20 - - spec_dec_algo: NGram - accuracy: 74.20 - - spec_dec_algo: SA - accuracy: 74.20 - - spec_dec_algo: Eagle - accuracy: 74.20 - - spec_dec_algo: Eagle3 - accuracy: 74.20 - - spec_dec_algo: Eagle3 - extra_acc_spec: use_sa_spec - accuracy: 74.20 - - spec_dec_algo: PARD - accuracy: 74.20 - - spec_dec_algo: PARD - extra_acc_spec: use_sa_spec - accuracy: 74.20 - - spec_dec_algo: Draft_Target - accuracy: 74.20 - - spec_dec_algo: DFlash - accuracy: 74.20 - - quant_algo: FP8 - accuracy: 74.30 - - quant_algo: FP8 - kv_cache_quant_algo: FP8 - accuracy: 72.85 - - quant_algo: FP8 - kv_cache_quant_algo: NVFP4 - accuracy: 69.75 - - quant_algo: NVFP4 - kv_cache_quant_algo: FP8 - accuracy: 66.03 meta-llama/Llama-4-Maverick-17B-128E-Instruct: # B200 TP4/EP4 EAGLE-3 baseline over the full 1319-sample split. - quant_algo: FP8 diff --git a/tests/integration/defs/accuracy/references/json_mode_eval.yaml b/tests/integration/defs/accuracy/references/json_mode_eval.yaml index 3fda403fb2c8..afb57ed0dbff 100644 --- a/tests/integration/defs/accuracy/references/json_mode_eval.yaml +++ b/tests/integration/defs/accuracy/references/json_mode_eval.yaml @@ -1,11 +1,3 @@ -meta-llama/Llama-3.1-8B-Instruct: - - accuracy: 74.00 - - spec_dec_algo: Eagle - accuracy: 74.00 - - spec_dec_algo: Eagle3 - accuracy: 74.00 - - spec_dec_algo: NGram - accuracy: 74.00 deepseek-ai/DeepSeek-V3-Lite: - accuracy: 77.00 - spec_dec_algo: MTP diff --git a/tests/integration/defs/accuracy/references/longbench_v2.yaml b/tests/integration/defs/accuracy/references/longbench_v2.yaml index 9e511ce85138..357dc405097c 100644 --- a/tests/integration/defs/accuracy/references/longbench_v2.yaml +++ b/tests/integration/defs/accuracy/references/longbench_v2.yaml @@ -7,5 +7,3 @@ DeepSeek-R1-0528: kv_cache_quant_algo: FP8 spec_dec_algo: MTP accuracy: 52.093 -meta-llama/Llama-3.1-8B-Instruct: - - accuracy: 25.80 diff --git a/tests/integration/defs/accuracy/references/mmlu.yaml b/tests/integration/defs/accuracy/references/mmlu.yaml index 3f660f21947e..da6cb0b2fabb 100644 --- a/tests/integration/defs/accuracy/references/mmlu.yaml +++ b/tests/integration/defs/accuracy/references/mmlu.yaml @@ -1,35 +1,3 @@ -meta-llama/Llama-3.1-8B: - - accuracy: 66.06 - - quant_algo: NVFP4 - kv_cache_quant_algo: FP8 - accuracy: 63.16 - - quant_algo: FP8_PER_CHANNEL_PER_TOKEN - accuracy: 65.55 - - quant_algo: MIXED_PRECISION - extra_acc_spec: autoq_format=int4_awq,fp8,w4a8_awq;auto_quantize_bits=5.8 - accuracy: 64.99 -meta-llama/Llama-3.1-8B-Instruct: - - accuracy: 68.17 - - spec_dec_algo: Eagle - accuracy: 68.20 - - spec_dec_algo: Eagle3 - accuracy: 68.20 - - spec_dec_algo: NGram - accuracy: 68.17 - - quant_algo: FP8 - accuracy: 67.93 - - quant_algo: FP8 - extra_acc_spec: temperature=0.8,top_p=0.95 - accuracy: 64.62 - - quant_algo: FP8 - kv_cache_quant_algo: FP8 - accuracy: 67.87 - - quant_algo: FP8 - kv_cache_quant_algo: NVFP4 - accuracy: 66.45 - - quant_algo: NVFP4 - kv_cache_quant_algo: FP8 - accuracy: 65.11 mistralai/Mistral-Small-3.1-24B-Instruct-2503: - accuracy: 81.7 - quant_algo: FP8 diff --git a/tests/integration/defs/accuracy/references/passkey_retrieval_64k.yaml b/tests/integration/defs/accuracy/references/passkey_retrieval_64k.yaml index 9878c4ab84f1..e69de29bb2d1 100644 --- a/tests/integration/defs/accuracy/references/passkey_retrieval_64k.yaml +++ b/tests/integration/defs/accuracy/references/passkey_retrieval_64k.yaml @@ -1,4 +0,0 @@ -meta-llama/Llama-3.1-8B: - - accuracy: 99 - - quant_algo: FP8_PER_CHANNEL_PER_TOKEN - accuracy: 99 diff --git a/tests/integration/defs/accuracy/test_cli_flow.py b/tests/integration/defs/accuracy/test_cli_flow.py deleted file mode 100644 index 39af5bf0aa96..000000000000 --- a/tests/integration/defs/accuracy/test_cli_flow.py +++ /dev/null @@ -1,283 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -import pytest - -from tensorrt_llm.llmapi import EagleDecodingConfig -from tensorrt_llm.quantization import QuantAlgo - -from ..conftest import (get_sm_version, llm_models_root, parametrize_with_ids, - skip_no_nvls, skip_post_blackwell, skip_pre_ada, - skip_pre_hopper) -from .accuracy_core import (MMLU, CliFlowAccuracyTestHarness, CnnDailymail, - Humaneval, PassKeyRetrieval64k, ZeroScrolls) - -# skip trt flow cases on post-Blackwell-Ultra -if get_sm_version() >= 103: - pytest.skip( - "TRT workflow tests are not supported on post Blackwell-Ultra architecture", - allow_module_level=True) - - -class TestStarcoder2_15B(CliFlowAccuracyTestHarness): - MODEL_NAME = "bigcode/starcoder2-15b" - MODEL_PATH = f"{llm_models_root()}/starcoder2-model" - EXAMPLE_FOLDER = "models/core/gpt" - - -class TestGptNext(CliFlowAccuracyTestHarness): - MODEL_NAME = "gpt-next" - MODEL_PATH = f"{llm_models_root()}/gpt-next/megatron_converted_843m_tp1_pp1.nemo" - MODEL_FORMAT = "NEMO" - EXAMPLE_FOLDER = "models/core/gpt" - - def test_auto_dtype(self): - # bfloat16 - self.run(dtype='auto') - - -class TestMinitron4BBase(CliFlowAccuracyTestHarness): - MODEL_NAME = "nvidia/Minitron-4B-Base" - MODEL_PATH = f"{llm_models_root()}/nemotron/Minitron-4B-Base" - EXAMPLE_FOLDER = "models/core/gpt" - - def test_auto_dtype(self): - self.run(tasks=[Humaneval(self.MODEL_NAME)], dtype='auto') - - @skip_pre_ada - def test_fp8(self, mocker): - # Accuracy regression when using large batch size - mocker.patch.object(Humaneval, "MAX_BATCH_SIZE", 1) - self.run(tasks=[Humaneval(self.MODEL_NAME)], - quant_algo=QuantAlgo.FP8, - kv_cache_quant_algo=QuantAlgo.FP8) - - -class TestNemotronMini4BInstruct(CliFlowAccuracyTestHarness): - MODEL_NAME = "nvidia/Nemotron-Mini-4B-Instruct" - MODEL_PATH = f"{llm_models_root()}/nemotron/Nemotron-Mini-4B-Instruct" - EXAMPLE_FOLDER = "models/core/gpt" - - @skip_pre_ada - def test_fp8_prequantized(self, mocker): - mocker.patch.object( - self.__class__, "MODEL_PATH", - f"{llm_models_root()}/nemotron/nemotron-mini-4b-instruct_vfp8-fp8-bf16-export" - ) - self.run(quant_algo=QuantAlgo.FP8, kv_cache_quant_algo=QuantAlgo.FP8) - - -# Long sequence length test: -# Model FP16 7B + 32K tokens in KV cache = 14 * 1024 MB + 32K * 0.5 MB = 30720 MB + scratch memory -@pytest.mark.skip_less_device_memory(40000) -class TestLongAlpaca7B(CliFlowAccuracyTestHarness): - MODEL_NAME = "Yukang/LongAlpaca-7B" - MODEL_PATH = f"{llm_models_root()}/LongAlpaca-7B" - EXAMPLE_FOLDER = "models/core/llama" - - def test_auto_dtype(self): - self.run(tasks=[ZeroScrolls(self.MODEL_NAME)]) - - def test_multiblock_aggressive(self): - # MMHA + aggressive Multi_block_mode (export TRTLLM_ENABLE_MMHA_MULTI_BLOCK_DEBUG=1) - self.run(tasks=[ZeroScrolls(self.MODEL_NAME)], - extra_build_args=["--gemm_plugin=auto"], - env={ - "TRTLLM_ENABLE_MMHA_MULTI_BLOCK_DEBUG": "1", - "TRTLLM_MMHA_BLOCKS_PER_SEQUENCE": "32" - }) - - -class TestMamba130M(CliFlowAccuracyTestHarness): - MODEL_NAME = "state-spaces/mamba-130m-hf" - MODEL_PATH = f"{llm_models_root()}/mamba/mamba-130m-hf" - EXAMPLE_FOLDER = "models/core/mamba" - - def test_auto_dtype(self): - self.run(dtype='auto') - - -class TestVicuna7B(CliFlowAccuracyTestHarness): - MODEL_NAME = "lmsys/vicuna-7b-v1.3" - MODEL_PATH = f"{llm_models_root()}/vicuna-7b-v1.3" - EXAMPLE_FOLDER = "models/core/llama" - EAGLE_MODEL_NAME = "yuhuili/EAGLE-Vicuna-7B-v1.3" - EAGLE_MODEL_PATH = f"{llm_models_root()}/EAGLE-Vicuna-7B-v1.3" - - @skip_post_blackwell - @parametrize_with_ids("cuda_graph,chunked_context,typical_acceptance", - [(False, False, False), (True, False, False), - (True, True, False), (True, False, True)]) - def test_eagle(self, cuda_graph, chunked_context, typical_acceptance, - mocker): - mocker.patch.object(self.__class__, "EXAMPLE_FOLDER", "eagle") - mocker.patch.object(CnnDailymail, "MAX_BATCH_SIZE", 8) - - extra_summarize_args = [ - "--eagle_choices=[[0], [0, 0], [1], [0, 1], [2], [0, 0, 0], [1, 0], [0, 2], [3], [0, 3], [4], [0, 4], [2, 0], [0, 5], [0, 0, 1], [5], [0, 6], [6], [0, 7], [0, 1, 0], [1, 1], [7], [0, 8], [0, 0, 2], [3, 0], [0, 9], [8], [9], [1, 0, 0], [0, 2, 0], [1, 2], [0, 0, 3], [4, 0], [2, 1], [0, 0, 4], [0, 0, 5], [0, 0, 0, 0], [0, 1, 1], [0, 0, 6], [0, 3, 0], [5, 0], [1, 3], [0, 0, 7], [0, 0, 8], [0, 0, 9], [6, 0], [0, 4, 0], [1, 4], [7, 0], [0, 1, 2], [2, 0, 0], [3, 1], [2, 2], [8, 0], [0, 5, 0], [1, 5], [1, 0, 1], [0, 2, 1], [9, 0], [0, 6, 0], [0, 0, 0, 1], [1, 6], [0, 7, 0]]" - ] - if cuda_graph: - extra_summarize_args.append("--cuda_graph_mode") - if chunked_context: - extra_summarize_args.append("--enable_chunked_context") - if typical_acceptance: - extra_summarize_args.extend( - ["--eagle_posterior_threshold=0.09", "--temperature=0.7"]) - - self.run(spec_dec_algo=EagleDecodingConfig. - model_fields["decoding_type"].default, - extra_convert_args=[ - f"--eagle_model_dir={self.EAGLE_MODEL_PATH}", - "--max_draft_len=63", "--num_eagle_layers=4", - "--max_non_leaves_per_layer=10" - ], - extra_build_args=[ - "--speculative_decoding_mode=eagle", "--max_draft_len=63" - ], - extra_summarize_args=extra_summarize_args) - - -class TestTinyLlama1_1BChat(CliFlowAccuracyTestHarness): - MODEL_NAME = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" - MODEL_PATH = f"{llm_models_root()}/llama-models-v2/TinyLlama-1.1B-Chat-v1.0" - EXAMPLE_FOLDER = "models/core/llama" - - def test_auto_dtype(self): - self.run(dtype='auto') - - @skip_post_blackwell - @pytest.mark.parametrize("precision", ["int8", "int4"]) - def test_weight_only(self, precision: str): - quant_algo = QuantAlgo.W8A16 if precision == "int8" else QuantAlgo.W4A16 - self.run(quant_algo=quant_algo) - - @skip_pre_ada - def test_fp8(self): - self.run(quant_algo=QuantAlgo.FP8, kv_cache_quant_algo=QuantAlgo.FP8) - - @pytest.mark.skip_less_device(4) - def test_pp4(self): - # Test num_hidden_layers (22) undivisible by pp_size (4) - self.run(extra_acc_spec="pp_size=4", pp_size=4) - - -class TestLlama3_1_8B(CliFlowAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B" - MODEL_PATH = f"{llm_models_root()}/llama-3.1-model/Meta-Llama-3.1-8B" - EXAMPLE_FOLDER = "models/core/llama" - - def test_auto_dtype(self): - self.run(dtype='auto') - - @skip_pre_ada - def test_fp8(self): - self.run(quant_algo=QuantAlgo.FP8, kv_cache_quant_algo=QuantAlgo.FP8) - - @skip_pre_ada - @skip_post_blackwell - def test_fp8_rowwise(self): - self.run(tasks=[CnnDailymail(self.MODEL_NAME), - MMLU(self.MODEL_NAME)], - quant_algo=QuantAlgo.FP8_PER_CHANNEL_PER_TOKEN) - - @skip_pre_ada - @skip_post_blackwell - def test_fp8_rowwise_meta_recipe(self): - self.run(quant_algo=QuantAlgo.FP8_PER_CHANNEL_PER_TOKEN, - extra_acc_spec="meta_recipe", - extra_convert_args=["--use_meta_fp8_rowwise_recipe"]) - - @pytest.mark.skip_less_device(4) - @pytest.mark.parametrize( - "gemm_allreduce", [False, pytest.param(True, marks=skip_no_nvls)], - ids=["disable_gemm_allreduce_plugin", "enable_gemm_allreduce_plugin"]) - def test_tp4(self, gemm_allreduce: bool): - extra_build_args = None - if gemm_allreduce: - extra_build_args = ["--gemm_allreduce_plugin=bfloat16"] - self.run( - tasks=[PassKeyRetrieval64k(self.MODEL_NAME), - MMLU(self.MODEL_NAME)], - tp_size=4, - extra_build_args=extra_build_args) - - @skip_pre_hopper - @skip_post_blackwell - @pytest.mark.skip_less_device(4) - @pytest.mark.parametrize( - "gemm_allreduce", [False, pytest.param(True, marks=skip_no_nvls)], - ids=["disable_gemm_allreduce_plugin", "enable_gemm_allreduce_plugin"]) - def test_fp8_rowwise_tp4(self, gemm_allreduce: bool): - extra_build_args = None - if gemm_allreduce: - extra_build_args = ["--gemm_allreduce_plugin=bfloat16"] - self.run( - tasks=[PassKeyRetrieval64k(self.MODEL_NAME), - MMLU(self.MODEL_NAME)], - quant_algo=QuantAlgo.FP8_PER_CHANNEL_PER_TOKEN, - tp_size=4, - extra_build_args=extra_build_args) - - -class TestLlama3_1_8BInstruct(CliFlowAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct" - MODEL_PATH = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - EXAMPLE_FOLDER = "models/core/llama" - - def test_auto_dtype(self): - self.run(dtype='auto') - - @skip_pre_hopper - def test_fp8_prequantized(self, mocker): - mocker.patch.object( - self.__class__, "MODEL_PATH", - f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct-FP8") - self.run(quant_algo=QuantAlgo.FP8, kv_cache_quant_algo=QuantAlgo.FP8) - - -class TestGemma2B(CliFlowAccuracyTestHarness): - MODEL_NAME = "google/gemma-2b" - MODEL_PATH = f"{llm_models_root()}/gemma/gemma-2b" - EXAMPLE_FOLDER = "models/core/gemma" - - def test_auto_dtype(self): - self.run(dtype='auto', extra_convert_args=["--ckpt-type=hf"]) - - @pytest.mark.parametrize("precision", ["int8"]) - def test_weight_only(self, precision: str): - quant_algo = QuantAlgo.W8A16 if precision == "int8" else QuantAlgo.W4A16 - self.run(quant_algo=quant_algo, extra_convert_args=["--ckpt-type=hf"]) - - @skip_pre_ada - def test_fp8(self): - self.run(quant_algo=QuantAlgo.FP8, kv_cache_quant_algo=QuantAlgo.FP8) - - -@pytest.mark.skip_less_device_memory(40000) -class TestGemma7B(CliFlowAccuracyTestHarness): - MODEL_NAME = "google/gemma-7b" - MODEL_PATH = f"{llm_models_root()}/gemma/gemma-7b" - EXAMPLE_FOLDER = "models/core/gemma" - - def test_auto_dtype(self): - self.run(dtype='auto', extra_convert_args=["--ckpt-type=hf"]) - - @pytest.mark.parametrize("precision", ["int8"]) - def test_weight_only(self, precision: str): - quant_algo = QuantAlgo.W8A16 if precision == "int8" else QuantAlgo.W4A16 - self.run(quant_algo=quant_algo, extra_convert_args=["--ckpt-type=hf"]) - - @skip_pre_ada - def test_fp8(self): - self.run(quant_algo=QuantAlgo.FP8, kv_cache_quant_algo=QuantAlgo.FP8) diff --git a/tests/integration/defs/accuracy/test_disaggregated_serving.py b/tests/integration/defs/accuracy/test_disaggregated_serving.py index 03bce01d822b..f7f496fc0222 100644 --- a/tests/integration/defs/accuracy/test_disaggregated_serving.py +++ b/tests/integration/defs/accuracy/test_disaggregated_serving.py @@ -42,8 +42,8 @@ from ..conftest import (get_device_count, llm_models_root, parametrize_with_ids, skip_no_hopper, skip_pre_blackwell, skip_pre_hopper) from ..trt_test_alternative import popen -from .accuracy_core import (GSM8K, MMLU, CnnDailymail, - LlmapiAccuracyTestHarness, get_accuracy_task) +from .accuracy_core import (GSM8K, MMLU, LlmapiAccuracyTestHarness, + get_accuracy_task) class Result(GenerationResultBase): @@ -635,410 +635,6 @@ def run_parallel_test(model_name: str, run_accuracy_test(llm, model_name, test_sets) -@pytest.mark.timeout(DEFAULT_TEST_TIMEOUT) -class TestLlama3_1_8BInstruct(LlmapiAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct" - MODEL_PATH = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - # Literal NIXL bypasses the harness's legacy UCX fallback. Omitting the - # runtime then exercises Llama's automatic preference for Python V2. - - @skip_pre_hopper - @pytest.mark.skip_less_device(2) - # overlap scheduler is token-invariant (unit-tested); only block-reuse changes which KV is transferred - # The mismatched pair is kept on purpose: it is the only combination where - # the two servers disagree about which blocks are already resident, so it - # exercises a different transfer path than either symmetric case. - @pytest.mark.parametrize( - "ctx_enable_block_reuse,gen_enable_block_reuse", [(True, True), - (True, False), - (False, False)], - ids=["block_reuse", "ctx_block_reuse_only", "no_block_reuse"]) - def test_auto_dtype(self, ctx_enable_block_reuse, gen_enable_block_reuse): - ctx_server_config = { - "disable_overlap_scheduler": False, - "kv_cache_config": { - "enable_block_reuse": ctx_enable_block_reuse - } - } - ctx_server_config["cache_transceiver_config"] = { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - } - gen_server_config = { - "disable_overlap_scheduler": False, - "kv_cache_config": { - "enable_block_reuse": gen_enable_block_reuse - } - } - gen_server_config["cache_transceiver_config"] = { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - } - disaggregated_server_config = { - "hostname": "localhost", - "backend": "pytorch", - "context_servers": { - "num_instances": 1 - }, - "generation_servers": { - "num_instances": 1 - } - } - with launch_disaggregated_llm(disaggregated_server_config, - ctx_server_config, gen_server_config, - self.MODEL_PATH) as llm: - run_accuracy_test(llm, self.MODEL_NAME, ["MMLU", "GSM8K"]) - - @skip_pre_hopper - @pytest.mark.skip_less_device(2) - def test_beam_search(self): - max_beam_width = 2 - sampling_params = SamplingParams(n=max_beam_width, - best_of=max_beam_width, - use_beam_search=True) - kv_cache_config = { - "free_gpu_memory_fraction": 0.5, - "enable_block_reuse": True, - "enable_partial_reuse": True, - "use_kv_cache_manager_v2": False, - } - cache_transceiver_config = { - "backend": "NIXL", - "transceiver_runtime": "PYTHON", - "max_tokens_in_buffer": 4096, - } - ctx_server_config = { - "disable_overlap_scheduler": True, - "max_beam_width": max_beam_width, - "kv_cache_config": kv_cache_config, - "cache_transceiver_config": cache_transceiver_config, - } - gen_server_config = { - "disable_overlap_scheduler": True, - "max_beam_width": max_beam_width, - "kv_cache_config": kv_cache_config, - "cache_transceiver_config": cache_transceiver_config, - } - disaggregated_server_config = { - "hostname": "localhost", - "backend": "pytorch", - "context_servers": { - "num_instances": 1 - }, - "generation_servers": { - "num_instances": 1 - } - } - with launch_disaggregated_llm(disaggregated_server_config, - ctx_server_config, gen_server_config, - self.MODEL_PATH) as llm: - run_accuracy_test(llm, - self.MODEL_NAME, [CnnDailymail], - extra_acc_spec=f"beam_width={max_beam_width}", - sampling_params=sampling_params) - - @pytest.mark.skip_less_device(2) - def test_ngram(self): - speculative_decoding_config = { - "decoding_type": "NGram", - "max_draft_len": 4, - "max_matching_ngram_size": 4, - "is_keep_all": True, - "is_use_oldest": True, - "is_public_pool": True - } - kv_cache_config = { - "free_gpu_memory_fraction": 0.5, - "enable_block_reuse": False - } - ctx_server_config = { - "disable_overlap_scheduler": True, - "kv_cache_config": kv_cache_config, - "cache_transceiver_config": { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - } - } - gen_server_config = { - "disable_overlap_scheduler": True, - "speculative_config": speculative_decoding_config, - "kv_cache_config": kv_cache_config, - "cache_transceiver_config": { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - } - } - disaggregated_server_config = { - "hostname": "localhost", - "backend": "pytorch", - "context_servers": { - "num_instances": 1 - }, - "generation_servers": { - "num_instances": 1 - } - } - with launch_disaggregated_llm(disaggregated_server_config, - ctx_server_config, gen_server_config, - self.MODEL_PATH) as llm: - run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"]) - - @pytest.mark.skip_less_device(2) - @skip_pre_hopper - @parametrize_with_ids("overlap_scheduler", [True, False]) - @parametrize_with_ids("eagle3_one_model", [True, False]) - def test_eagle3(self, overlap_scheduler, eagle3_one_model): - speculative_decoding_config = { - "decoding_type": "Eagle", - "max_draft_len": 4, - "speculative_model": - f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B", - "eagle3_one_model": eagle3_one_model - } - ctx_server_config = { - "disable_overlap_scheduler": - True, # BS=1 does not need overlap scheduling - "speculative_config": speculative_decoding_config, - "kv_cache_config": { - "free_gpu_memory_fraction": 0.5, - "enable_block_reuse": True # reuse on context requests - }, - "max_num_tokens": 13393 * 2, - "max_batch_size": 1, - "cache_transceiver_config": { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - }, - "cuda_graph_config": None, - } - gen_server_config = { - "disable_overlap_scheduler": not overlap_scheduler, - "speculative_config": speculative_decoding_config, - "kv_cache_config": { - "free_gpu_memory_fraction": 0.5, - "enable_block_reuse": False - }, - "max_num_tokens": 13393 * 2, - "max_batch_size": 16, - "cache_transceiver_config": { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - }, - "cuda_graph_config": None, - } - disaggregated_server_config = { - "hostname": "localhost", - "backend": "pytorch", - "context_servers": { - "num_instances": 1 - }, - "generation_servers": { - "num_instances": 1 - } - } - with launch_disaggregated_llm(disaggregated_server_config, - ctx_server_config, gen_server_config, - self.MODEL_PATH) as llm: - run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"]) - - @pytest.mark.skip_less_device(2) - @skip_pre_hopper - def test_gen_only_spec_dec(self): - speculative_decoding_config = { - "decoding_type": "Eagle", - "max_draft_len": 4, - "speculative_model": - f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B", - "eagle3_one_model": True, - } - ctx_server_config = { - "disable_overlap_scheduler": - True, # BS=1 does not need overlap scheduling - "kv_cache_config": { - "free_gpu_memory_fraction": 0.5, - "enable_block_reuse": True # reuse on context requests - }, - "max_num_tokens": 13393 * 2, - "max_batch_size": 1, - "cache_transceiver_config": { - "backend": "NIXL", - "transceiver_runtime": "PYTHON", - "max_tokens_in_buffer": 4096, - }, - "cuda_graph_config": None, - } - gen_server_config = { - "disable_overlap_scheduler": False, - "speculative_config": speculative_decoding_config, - "kv_cache_config": { - "free_gpu_memory_fraction": 0.5, - "enable_block_reuse": False - }, - "max_num_tokens": 13393 * 2, - "max_batch_size": 16, - "cache_transceiver_config": { - "backend": "NIXL", - "transceiver_runtime": "PYTHON", - "max_tokens_in_buffer": 4096, - }, - "cuda_graph_config": None, - } - disaggregated_server_config = { - "hostname": "localhost", - "backend": "pytorch", - "context_servers": { - "num_instances": 1 - }, - "generation_servers": { - "num_instances": 1 - } - } - with launch_disaggregated_llm(disaggregated_server_config, - ctx_server_config, gen_server_config, - self.MODEL_PATH) as llm: - run_accuracy_test(llm, self.MODEL_NAME, ["GSM8K"]) - - @pytest.mark.skip_less_device(2) - @pytest.mark.skip_less_device_memory(32000) - # grammar backend is disagg-agnostic (runs on gen worker); backend correctness is covered by aggregated tests - @pytest.mark.parametrize("backend", ["xgrammar"]) - def test_guided_decoding(self, backend: str, mocker): - mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"}) - ctx_server_config = { - "disable_overlap_scheduler": True, - "guided_decoding_backend": backend, - "cache_transceiver_config": { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - } - } - gen_server_config = { - "guided_decoding_backend": backend, - "cache_transceiver_config": { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - } - } - disaggregated_server_config = { - "hostname": "localhost", - "backend": "pytorch", - "context_servers": { - "num_instances": 1 - }, - "generation_servers": { - "num_instances": 1 - } - } - with launch_disaggregated_llm(disaggregated_server_config, - ctx_server_config, gen_server_config, - self.MODEL_PATH) as llm: - run_accuracy_test(llm, self.MODEL_NAME, ["JsonModeEval"]) - - @pytest.mark.skip_less_device(2) - @pytest.mark.skip_less_device_memory(48000) - @parametrize_with_ids("eagle3_one_model", [True, False]) - @pytest.mark.parametrize("backend", ["xgrammar", "llguidance"]) - def test_guided_decoding_with_eagle3(self, backend: str, - eagle3_one_model: bool, mocker): - mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"}) - speculative_decoding_config = { - "decoding_type": "Eagle", - "max_draft_len": 3, - "speculative_model": - f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B", - "eagle3_one_model": eagle3_one_model - } - - ctx_server_config = { - "disable_overlap_scheduler": True, - "speculative_config": speculative_decoding_config, - "kv_cache_config": { - "free_gpu_memory_fraction": 0.8, - }, - "guided_decoding_backend": backend, - "cache_transceiver_config": { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - } - } - gen_server_config = { - # Two-model eagle3 does not support overlap scheduler - "disable_overlap_scheduler": not eagle3_one_model, - "speculative_config": speculative_decoding_config, - "kv_cache_config": { - "free_gpu_memory_fraction": 0.8, - }, - "guided_decoding_backend": backend, - "cache_transceiver_config": { - "backend": "NIXL", - "max_tokens_in_buffer": 4096 - } - } - disaggregated_server_config = { - "hostname": "localhost", - "backend": "pytorch", - "context_servers": { - "num_instances": 1 - }, - "generation_servers": { - "num_instances": 1 - } - } - with launch_disaggregated_llm(disaggregated_server_config, - ctx_server_config, gen_server_config, - self.MODEL_PATH) as llm: - run_accuracy_test(llm, self.MODEL_NAME, ["JsonModeEval"]) - - @pytest.mark.parametrize("tp,pp", [(1, 2), (2, 1), (2, 2)], - ids=["tp1pp2", "tp2pp1", "tp2pp2"]) - @pytest.mark.parametrize("testset", ["GSM8K", "MMLU"]) - def test_tp_pp_symmetric(self, tp, pp, testset): - if tp * pp * 2 > get_device_count(): - pytest.skip(f"Not enough devices for tp={tp}*pp={pp} test") - return run_parallel_test(self.MODEL_NAME, - self.MODEL_PATH, - ctx_pp=pp, - ctx_tp=tp, - gen_pp=pp, - gen_tp=tp, - ctx_instances=1, - gen_instances=1, - test_sets=[get_accuracy_task(testset)], - cache_transceiver_backend="NIXL") - - @parametrize_with_ids("ctx_pp", [2, 4]) - @parametrize_with_ids("gen_tp", [1, 2]) - @pytest.mark.parametrize("testset", ["GSM8K", "MMLU"]) - def test_ctx_pp_gen_tp_asymmetric(self, ctx_pp, gen_tp, testset): - if ctx_pp + gen_tp > get_device_count(): - pytest.skip( - f"Not enough devices for ctx_pp={ctx_pp}+gen_tp={gen_tp} test") - return run_parallel_test(self.MODEL_NAME, - self.MODEL_PATH, - ctx_pp=ctx_pp, - ctx_tp=1, - gen_pp=1, - gen_tp=gen_tp, - ctx_instances=1, - gen_instances=1, - test_sets=[get_accuracy_task(testset)], - cache_transceiver_backend="NIXL") - - @pytest.mark.parametrize("testset", ["GSM8K", "MMLU"]) - def test_multi_instance(self, testset): - return run_parallel_test(self.MODEL_NAME, - self.MODEL_PATH, - ctx_pp=1, - ctx_tp=1, - gen_pp=1, - gen_tp=1, - ctx_instances=2, - gen_instances=2, - test_sets=[get_accuracy_task(testset)], - cache_transceiver_backend="NIXL") - - @pytest.mark.timeout(DEFAULT_TEST_TIMEOUT) @skip_pre_hopper class TestDeepSeekV3Lite(LlmapiAccuracyTestHarness): diff --git a/tests/integration/defs/accuracy/test_llm_api_autodeploy.py b/tests/integration/defs/accuracy/test_llm_api_autodeploy.py index 4bc2a7cd8bd1..4fdbf5ffb209 100644 --- a/tests/integration/defs/accuracy/test_llm_api_autodeploy.py +++ b/tests/integration/defs/accuracy/test_llm_api_autodeploy.py @@ -24,12 +24,10 @@ from test_common.llm_data import hf_id_to_local_model_dir from tensorrt_llm._torch.auto_deploy import LLM as AutoDeployLLM -from tensorrt_llm.llmapi import Eagle3DecodingConfig from tensorrt_llm.quantization import QuantAlgo from tensorrt_llm.sampling_params import SamplingParams -from .accuracy_core import (GSM8K, MMLU, MMMU, CnnDailymail, - LlmapiAccuracyTestHarness) +from .accuracy_core import GSM8K, MMLU, MMMU, LlmapiAccuracyTestHarness _AD_CONFIGS_DIR = (Path(get_llm_root()) / 'examples' / 'auto_deploy' / 'model_registry' / 'configs') @@ -193,217 +191,6 @@ def reduced_model_kwargs(num_hidden_layers: int, return {"model_kwargs": overrides} -class TestLlama3_1_8B(LlmapiAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B" - MODEL_PATH = hf_id_to_local_model_dir(MODEL_NAME) - - # Configuration presets for different attention backends - ATTN_BACKEND_CONFIGS = { - "flashinfer": { - "max_batch_size": 512, - "max_seq_len": 8192, - "compile_backend": "torch-cudagraph", - }, - "trtllm": { - "max_batch_size": 512, - "max_seq_len": 8192, - "compile_backend": "torch-cudagraph", - "transforms": { - "fuse_gemms_mixed_children": { - "enabled": True, - }, - "fuse_rope_into_trtllm_attention": { - "enabled": True, - }, - }, - }, - "torch": { - "max_batch_size": 32, - "max_seq_len": 2048, - "compile_backend": "torch-simple", - }, - "triton": { - "max_batch_size": 128, - "max_seq_len": 8192, - "compile_backend": "torch-cudagraph", - }, - } - - def get_default_kwargs(self, - enable_chunked_prefill=False, - attn_backend="flashinfer"): - backend_cfg = self.ATTN_BACKEND_CONFIGS[attn_backend] - - # Filter cuda graph batch sizes to those <= max_batch_size; the LlmArgs - # validator requires cuda_graph_config.max_batch_size <= max_batch_size. - cuda_graph_batch_sizes = [ - size for size in [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] - if size <= backend_cfg["max_batch_size"] - ] - - config = { - "skip_tokenizer_init": False, - "trust_remote_code": True, - "attn_backend": attn_backend, - "max_batch_size": backend_cfg["max_batch_size"], - # 131072 is the max seq len for the model - "max_seq_len": backend_cfg["max_seq_len"], - # max num tokens is derived in the build_config, which is not used by AutoDeploy llmargs. - # Set it explicitly here to 8192 which is the default in build_config. - "max_num_tokens": 8192, - "skip_loading_weights": False, - "kv_cache_config": { - "free_gpu_memory_fraction": 0.7 - }, - "cuda_graph_config": { - "batch_sizes": cuda_graph_batch_sizes, - }, - "transforms": { - "compile_model": { - "backend": backend_cfg["compile_backend"], - }, - "fuse_silu_mul": { - "enabled": True, - }, - }, - } - if enable_chunked_prefill: - config["enable_chunked_prefill"] = True - # NOTE: must be > max(tokens_per_block, max_batch_size) - config["max_num_tokens"] = 512 - return config - - def get_default_sampling_params(self): - eos_id = -1 - beam_width = 1 - return SamplingParams(end_id=eos_id, - pad_id=eos_id, - n=beam_width, - use_beam_search=beam_width > 1) - - @pytest.mark.skip_less_device_memory(32000) - @pytest.mark.parametrize("world_size", [1, 2, 4]) - @pytest.mark.parametrize("enable_chunked_prefill", [False, True]) - @pytest.mark.parametrize( - "attn_backend", - [ - "flashinfer", - "trtllm", - # Torch attention is unpaged. - # Unpaged KV = (batch_size + 1) slots * 2048 tokens * 32 layers * 2 KV * 8 KV heads * 128 dim * 2 bytes. - # For batch_size=32: 8.25 GiB KV + ~15 GiB weights ~= 23.3 GiB. - # If batch size is increased, this parameterization must be gated at a higher memory threshold. - "torch", - "triton", - ], - ) - def test_auto_dtype(self, world_size, enable_chunked_prefill, attn_backend): - kwargs = self.get_default_kwargs(enable_chunked_prefill, attn_backend) - sampling_params = self.get_default_sampling_params() - with AutoDeployLLM(model=self.MODEL_PATH, - tokenizer=self.MODEL_PATH, - world_size=world_size, - **kwargs) as llm: - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm) - if attn_backend != "torch": - task = MMLU(self.MODEL_NAME) - task.evaluate(llm, sampling_params=sampling_params) - - @pytest.mark.skip_less_device_memory(32000) - @pytest.mark.parametrize("world_size", [ - pytest.param(2, marks=pytest.mark.skip_less_device(2)), - pytest.param(4, marks=pytest.mark.skip_less_device(4)), - ]) - def test_attention_dp(self, world_size): - """Test attention data parallelism mode where TP sharding is disabled.""" - kwargs = self.get_default_kwargs(enable_chunked_prefill=True) - # Enable attention DP - this disables TP sharding - kwargs["transforms"]["detect_sharding"] = {"enable_attention_dp": True} - sampling_params = self.get_default_sampling_params() - with AutoDeployLLM(model=self.MODEL_PATH, - tokenizer=self.MODEL_PATH, - world_size=world_size, - **kwargs) as llm: - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm) - task = MMLU(self.MODEL_NAME) - task.evaluate(llm, sampling_params=sampling_params) - - -class TestLlama3_1_8B_Instruct_Eagle3(LlmapiAccuracyTestHarness): - """Accuracy test for Eagle3 one-model speculative decoding with AutoDeploy.""" - - MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct" - MODEL_PATH = hf_id_to_local_model_dir(MODEL_NAME) - EAGLE_MODEL_PATH = hf_id_to_local_model_dir( - "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B") - - def get_default_kwargs(self, attn_backend="flashinfer"): - yaml_paths, _ = _get_registry_yaml_extra(self.MODEL_NAME) - speculative_config = Eagle3DecodingConfig( - max_draft_len=3, - speculative_model=self.EAGLE_MODEL_PATH, - eagle3_one_model=True, - eagle3_layers_to_capture={1, 15, 28}, - ) - # Note: Test crashes with trtllm attn_backend + torch-simple - # See: https://github.com/NVIDIA/TensorRT-LLM/issues/13135 - compile_backend = "torch-cudagraph" if attn_backend == "trtllm" else "torch-simple" - - kwargs = { - "yaml_extra": yaml_paths, - "attn_backend": attn_backend, - "compile_backend": compile_backend, - "skip_tokenizer_init": False, - "trust_remote_code": True, - "max_seq_len": 8192, - "max_num_tokens": 8192, - "enable_iter_perf_stats": True, - "kv_cache_config": { - "free_gpu_memory_fraction": 0.7 - }, - "speculative_config": speculative_config, - # Force the Eagle3 draft to match the target (Llama 3.1 8B is bfloat16). - # Shared KV cache requires matching dtypes between target and draft. - "speculative_model_kwargs": { - "torch_dtype": "bfloat16" - }, - } - kwargs.setdefault("transforms", - {}).setdefault("compile_model", - {})["piecewise_enabled"] = False - - return kwargs - - def get_default_sampling_params(self): - return SamplingParams( - max_tokens=GSM8K.MAX_OUTPUT_LEN, # 256 tokens - truncate_prompt_tokens=GSM8K.MAX_INPUT_LEN, - ) - - def check_acceptance_rate(self, llm, min_acceptance_rate: float): - """Check speculative decoding acceptance rate.""" - _check_acceptance_rate_stats(llm.get_stats(), min_acceptance_rate) - - @skip_pre_hopper - @pytest.mark.skip_less_device_memory(32000) - @pytest.mark.parametrize("attn_backend", ["flashinfer", "trtllm"]) - def test_eagle3_one_model(self, attn_backend): - """Test Eagle3 one-model speculative decoding accuracy on GSM8K.""" - kwargs = self.get_default_kwargs(attn_backend=attn_backend) - - with AutoDeployLLM( - model=self.MODEL_PATH, - tokenizer=self.MODEL_PATH, - **kwargs, - ) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - self.check_acceptance_rate(llm, min_acceptance_rate=0.18) - - class TestNemotronV2(LlmapiAccuracyTestHarness): MODEL_NAME = "nvidia/NVIDIA-Nemotron-Nano-9B-v2" _MODEL_PATH_BASE = f"{llm_models_root()}/NVIDIA-Nemotron-Nano-9B-v2" @@ -1375,22 +1162,11 @@ class TestModelRegistryAccuracy(LlmapiAccuracyTestHarness): """ # Aliases for models that have different names in the registry and the reference accuracy files. MODEL_REFERENCE_ALIASES = { - "nvidia/Llama-3.1-8B-Instruct-FP8": "meta-llama/Llama-3.1-8B-Instruct", - "nvidia/Llama-3.1-8B-Instruct-NVFP4": - "meta-llama/Llama-3.1-8B-Instruct", "nvidia/DeepSeek-R1-0528-NVFP4-v2": "deepseek-ai/DeepSeek-R1-0528", } # Each param: (model_name, config_overrides, tasks). Marks skip when machine lacks GPUs/memory. MODEL_REGISTRY_ACCURACY_PARAMS = [ - pytest.param("meta-llama/Llama-3.1-8B-Instruct", {}, [MMLU, GSM8K], - id="meta-llama_Llama-3.1-8B-Instruct"), - pytest.param("nvidia/Llama-3.1-8B-Instruct-FP8", {}, [MMLU, GSM8K], - marks=skip_pre_ada, - id="nvidia_Llama-3.1-8B-Instruct-FP8"), - pytest.param("nvidia/Llama-3.1-8B-Instruct-NVFP4", {}, [MMLU, GSM8K], - marks=skip_pre_blackwell, - id="nvidia_Llama-3.1-8B-Instruct-NVFP4"), pytest.param("mistralai/Ministral-8B-Instruct-2410", {}, [MMLU, GSM8K], id="mistralai_Ministral-8B-Instruct-2410"), pytest.param( diff --git a/tests/integration/defs/accuracy/test_llm_api_pytorch.py b/tests/integration/defs/accuracy/test_llm_api_pytorch.py index 9bacdaac1e22..4909ecb1b7e6 100644 --- a/tests/integration/defs/accuracy/test_llm_api_pytorch.py +++ b/tests/integration/defs/accuracy/test_llm_api_pytorch.py @@ -33,13 +33,10 @@ # isort: off from tensorrt_llm.llmapi import ( AttentionDpConfig, CudaGraphConfig, DeepSeekSparseAttentionConfig, - DFlashDecodingConfig, DSparkDecodingConfig, DraftTargetDecodingConfig, - Eagle3DecodingConfig, KvCacheConfig, MambaStateConfig, - MiniMaxM3SparseAttentionConfig, MoeConfig, MTPDecodingConfig, - NGramDecodingConfig, PARDDecodingConfig, PrefillCudaGraphBackend, - RocketSparseAttentionConfig, SADecodingConfig, SamplingParams, - SchedulerConfig, SkipSoftmaxAttentionConfig, SAEnhancerConfig, - TorchCompileConfig) + DFlashDecodingConfig, DSparkDecodingConfig, Eagle3DecodingConfig, + KvCacheConfig, MambaStateConfig, MiniMaxM3SparseAttentionConfig, MoeConfig, + MTPDecodingConfig, PrefillCudaGraphBackend, SamplingParams, SchedulerConfig, + SkipSoftmaxAttentionConfig, SAEnhancerConfig, TorchCompileConfig) # isort: on from tensorrt_llm.math_utils import pad_up from tensorrt_llm.quantization import QuantAlgo @@ -222,980 +219,6 @@ def _run_multinode_accuracy(model_path, task.evaluate(llm) -class TestLlama3_1_8B(LlmapiAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B" - MODEL_PATH = f"{llm_models_root()}/llama-3.1-model/Meta-Llama-3.1-8B" - - @pytest.mark.skip_less_device_memory(32000) - def test_auto_dtype(self): - with LLM(self.MODEL_PATH) as llm: - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm) - task = MMLU(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_blackwell - def test_nvfp4(self): - model_path = f"{llm_models_root()}/nvfp4-quantized/Meta-Llama-3.1-8B" - with LLM(model_path) as llm: - assert llm.args.quant_config.quant_algo == QuantAlgo.NVFP4 - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm) - task = MMLU(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_blackwell - @pytest.mark.parametrize("stream_interval", [4, 64], - ids=["stream_interval_4", "stream_interval_64"]) - def test_nvfp4_streaming(self, stream_interval): - # When stream_interval < TLLM_STREAM_INTERVAL_THRESHOLD, hf incremental detokenization is used. - # When stream_interval >= TLLM_STREAM_INTERVAL_THRESHOLD, trtllm implemented incremental detokenization is used. - # The behavior is due to perf considerations, while both paths need to be tested. - with LLM(f"{llm_models_root()}/nvfp4-quantized/Meta-Llama-3.1-8B", - stream_interval=stream_interval) as llm: - assert llm.args.quant_config.quant_algo == QuantAlgo.NVFP4 - assert llm.args.stream_interval == stream_interval - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm, streaming=True) - - -class TestLlama3_1_8BInstruct(LlmapiAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct" - MODEL_PATH = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - @pytest.mark.skip_less_device_memory(32000) - @parametrize_with_ids("attn_backend", ["TRTLLM", "FLASHINFER"]) - # NB: Because greedy sampling is handled via a "fast path", a small non-zero - # temperature is required to also cover the regular (batched) sampling code. - @parametrize_with_ids("use_temperature", [False, True]) - def test_chunked_prefill(self, attn_backend, use_temperature: bool): - with LLM(self.MODEL_PATH, - attn_backend=attn_backend, - enable_chunked_prefill=True, - max_num_tokens=512, - max_stats_len=-1, - enable_iter_perf_stats=True) as llm: - task = MMLU(self.MODEL_NAME) - task.evaluate(llm, - sampling_params=(SamplingParams( - temperature=0.001) if use_temperature else None)) - - # MMLU prepends a fixed 5-shot prefix per subject (~12 blocks at the median), and - # iterates subject by subject, so block reuse should be hit heavily here regardless of - # attention backend. - stats = _latest_kv_cache_stats(llm) - # Uncomment for debugging: - # print(f"[MMLU] backend={attn_backend} " - # f"reused={stats['reusedBlocks']} " - # f"missed={stats['missedBlocks']} " - # f"hit_rate={stats['cacheHitRate']:.4f}") - - # This should be close to 0.8, but keeping it low out of caution for CI. - assert stats["reusedBlocks"] > 0.5 - - @pytest.mark.skip_less_device_memory(32000) - def test_dummy_load_format(self): - llm = LLM(self.MODEL_PATH, load_format="dummy") - with llm: - task = MMLU(self.MODEL_NAME) - task.evaluate(llm, is_integration_test=True) - - @pytest.mark.skip_less_device_memory(32000) - def test_gather_generation_logits_cuda_graph(self): - """RCCA: https://nvbugs/5365525.""" - llm = LLM(self.MODEL_PATH, - gather_generation_logits=True, - cuda_graph_config=CudaGraphConfig()) - with llm: - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm) - - @pytest.mark.parametrize("use_dynamic_tree", [False, True], - ids=["no_dynamic_tree", "dynamic_tree"]) - def test_eagle3_rejection_dynamic_tree_smoke(self, use_dynamic_tree, - mocker): - """Smoke-test one-model Eagle3 rejection sampling with both tree modes.""" - mocker.patch.object(GSM8K, "MAX_OUTPUT_LEN", 128) - - eagle_model_dir = f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B" - spec_config_kwargs = dict( - max_draft_len=4, - speculative_model=eagle_model_dir, - eagle3_one_model=True, - use_rejection_sampling=True, - ) - max_batch_size = 1 - if use_dynamic_tree: - spec_config_kwargs.update( - use_dynamic_tree=True, - dynamic_tree_max_topK=4, - max_total_draft_tokens=16, - ) - - llm = LLM( - self.MODEL_PATH, - tensor_parallel_size=1, - pipeline_parallel_size=1, - attn_backend="TRTLLM", - disable_overlap_scheduler=True, - cuda_graph_config=None, - kv_cache_config=KvCacheConfig(free_gpu_memory_fraction=0.4, - dtype="auto"), - max_seq_len=4096, - max_batch_size=max_batch_size, - speculative_config=Eagle3DecodingConfig(**spec_config_kwargs), - ) - - with llm: - task = GSM8K(self.MODEL_NAME) - sampling_params = SamplingParams(temperature=1.0, - top_p=1.0, - max_tokens=128, - truncate_prompt_tokens=2048) - task.evaluate(llm, - sampling_params=sampling_params, - extra_evaluator_kwargs=dict(apply_chat_template=True), - is_integration_test=True) - - @pytest.mark.skip_less_device_memory(32000) - @parametrize_with_ids("torch_compile", [False, True]) - @parametrize_with_ids("attn_backend", ["TRTLLM", "FLASHINFER"]) - def test_bfloat16(self, attn_backend, torch_compile): - torch_compile_config = _get_default_torch_compile_config(torch_compile) - pytorch_config = dict( - torch_compile_config=torch_compile_config, - cuda_graph_config=CudaGraphConfig(enable_padding=torch_compile, - batch_sizes=[4]), - attn_backend=attn_backend, - disable_overlap_scheduler=torch_compile, - ) - with LLM(self.MODEL_PATH, **pytorch_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - @parametrize_with_ids("torch_compile", [False, True]) - @parametrize_with_ids("attn_backend", ["TRTLLM", "FLASHINFER"]) - @pytest.mark.skip_less_device(4) - @pytest.mark.parametrize("tp_size,pp_size", [(4, 1), (2, 2), (1, 4)], - ids=["tp4", "tp2pp2", "pp4"]) - def test_bfloat16_4gpus(self, tp_size, pp_size, attn_backend, - torch_compile): - torch_compile_config = _get_default_torch_compile_config(torch_compile) - pytorch_config = dict( - torch_compile_config=torch_compile_config, - cuda_graph_config=CudaGraphConfig(enable_padding=torch_compile, - batch_sizes=[4]), - attn_backend=attn_backend, - disable_overlap_scheduler=torch_compile, - ) - with LLM(self.MODEL_PATH, - tensor_parallel_size=tp_size, - pipeline_parallel_size=pp_size, - **pytorch_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_ada - @parametrize_with_ids("torch_compile", [False, True]) - @parametrize_with_ids("attn_backend", ["TRTLLM", "FLASHINFER"]) - @parametrize_with_ids("fp8kv", [False, True]) - def test_fp8(self, fp8kv, attn_backend, torch_compile): - torch_compile_config = _get_default_torch_compile_config(torch_compile) - pytorch_config = dict( - torch_compile_config=torch_compile_config, - cuda_graph_config=CudaGraphConfig(enable_padding=torch_compile, - batch_sizes=[4]), - attn_backend=attn_backend, - disable_overlap_scheduler=torch_compile, - ) - if fp8kv: - pytorch_config["kv_cache_config"] = KvCacheConfig( - dtype="fp8", - free_gpu_memory_fraction= - 0.8, # Prevent cublas/cublasLt handle allocation memory insufficient errors - ) - with LLM( - f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct-FP8", - **pytorch_config) as llm: - assert llm.args.quant_config.quant_algo == QuantAlgo.FP8 - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_ada - @parametrize_with_ids("torch_compile", [False, True]) - @parametrize_with_ids("attn_backend", ["TRTLLM", "FLASHINFER"]) - @parametrize_with_ids("fp8kv", [False, True]) - @pytest.mark.skip_less_device(4) - @pytest.mark.parametrize("tp_size,pp_size", [(4, 1), (2, 2), (1, 4)], - ids=["tp4", "tp2pp2", "pp4"]) - def test_fp8_4gpus(self, tp_size, pp_size, fp8kv, attn_backend, - torch_compile): - if pp_size > 1 and torch_compile: - pytest.skip( - "Pipeline parallel with torch.compile is not supported yet.\n" - "Issue: Unfusing flashinfer_fused_add_rmsnorm causes outputs to be " - "discarded at graph breaks.") - torch_compile_config = _get_default_torch_compile_config(torch_compile) - pytorch_config = dict( - torch_compile_config=torch_compile_config, - cuda_graph_config=CudaGraphConfig(enable_padding=torch_compile, - batch_sizes=[4]), - attn_backend=attn_backend, - disable_overlap_scheduler=torch_compile, - ) - if fp8kv: - pytorch_config["kv_cache_config"] = KvCacheConfig( - dtype="fp8", - free_gpu_memory_fraction= - 0.8, # Prevent cublas/cublasLt handle allocation memory insufficient errors - ) - with LLM( - f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct-FP8", - tensor_parallel_size=tp_size, - pipeline_parallel_size=pp_size, - **pytorch_config) as llm: - assert llm.args.quant_config.quant_algo == QuantAlgo.FP8 - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_hopper - def test_fp8_llm_sampler(self): - model_path = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct-FP8" - with LLM(model_path, max_batch_size=256) as llm: - assert llm.args.quant_config.quant_algo == QuantAlgo.FP8 - - sampling_params = SamplingParams( - temperature=0.8, - top_p=0.95, - ) - - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm, - sampling_params=sampling_params, - extra_acc_spec="temperature=0.8,top_p=0.95") - task = MMLU(self.MODEL_NAME) - task.evaluate(llm, - sampling_params=sampling_params, - extra_acc_spec="temperature=0.8,top_p=0.95") - - @skip_pre_hopper - @parametrize_with_ids("overlap_scheduler", [True, False]) - @parametrize_with_ids("eagle3_one_model", [True, False]) - @parametrize_with_ids("sampler_async_worker", [True, False]) - def test_eagle3(self, overlap_scheduler, eagle3_one_model, - sampler_async_worker): - pytorch_config = dict( - max_batch_size= - 1, # add max_batch_size to avoid error in overlap scheduler - sampler_force_async_worker=sampler_async_worker, - disable_overlap_scheduler=not overlap_scheduler, - cuda_graph_config=CudaGraphConfig(max_batch_size=1, - enable_padding=True), - ) - kv_cache_config = KvCacheConfig( - enable_block_reuse=True, free_gpu_memory_fraction=0.8 - ) # both one-model and two-model supports this feature - - eagle_model_dir = f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - draft_len = 4 - spec_config = Eagle3DecodingConfig(max_draft_len=draft_len, - speculative_model=eagle_model_dir, - eagle3_one_model=eagle3_one_model) - with LLM(model=target_model_dir, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config, - max_stats_len=-1, - enable_iter_perf_stats=True) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - acceptance_length = compute_acceptance_length(llm) - print( - f"[AL] test_eagle acceptance_length = {acceptance_length:.3f}") - assert_acceptance_length( - "TestLlama3_1_8BInstruct::test_eagle3", - acceptance_length, - ) - - @skip_pre_hopper - def test_eagle3_sa(self): - """Accuracy test for EAGLE3 One-Model + Suffix Automaton speculative decoding.""" - pytorch_config = dict( - max_batch_size=1, - disable_overlap_scheduler=False, - cuda_graph_config=CudaGraphConfig(max_batch_size=1, - enable_padding=True), - ) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.8) - - eagle_model_dir = f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - spec_config = Eagle3DecodingConfig(max_draft_len=4, - speculative_model=eagle_model_dir, - eagle3_one_model=True, - sa_config=SAEnhancerConfig()) - - with LLM(model=target_model_dir, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm, extra_acc_spec="use_sa_spec") - - @skip_pre_hopper - def test_eagle3_sa_global_pool(self): - """Accuracy test for EAGLE3 One-Model + Suffix Automaton with global pool enabled.""" - max_batch_size = 32 - pytorch_config = dict( - max_batch_size=max_batch_size, - disable_overlap_scheduler=False, - cuda_graph_config=CudaGraphConfig(max_batch_size=max_batch_size, - enable_padding=True), - ) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.8) - - eagle_model_dir = f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - spec_config = Eagle3DecodingConfig( - max_draft_len=4, - speculative_model=eagle_model_dir, - eagle3_one_model=True, - sa_config=SAEnhancerConfig(enable_global_pool=True)) - - with LLM(model=target_model_dir, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm, extra_acc_spec="use_sa_spec") - - @skip_pre_blackwell - def test_eagle3_sa_dynamic_draft_len(self): - pytorch_config = dict( - max_batch_size=500, - disable_overlap_scheduler=False, - cuda_graph_config=(CudaGraphConfig(max_batch_size=500)), - ) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.8) - - eagle_model_dir = f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - spec_config = Eagle3DecodingConfig( - max_draft_len=4, - speculative_model=eagle_model_dir, - eagle3_one_model=True, - sa_config=SAEnhancerConfig(enable_global_pool=True), - draft_len_schedule={ - 50: 4, - 200: 3, - 350: 2 - }, - ) - - with LLM(model=target_model_dir, - **pytorch_config, - kv_cache_config=kv_cache_config, - enable_chunked_prefill=False, - max_num_tokens=8192, - speculative_config=spec_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm, extra_acc_spec="use_sa_spec") - - @skip_pre_hopper - @parametrize_with_ids("overlap_scheduler", [True, False]) - def test_pard(self, overlap_scheduler): - pytorch_config = dict( - max_batch_size= - 1, # add max_batch_size to avoid error in overlap scheduler - disable_overlap_scheduler=not overlap_scheduler, - cuda_graph_config=CudaGraphConfig(max_batch_size=1, - enable_padding=True), - ) - kv_cache_config = KvCacheConfig( - enable_block_reuse=True, free_gpu_memory_fraction=0.8 - ) # both one-model and two-model supports this feature - - pard_model_dir = f"{llm_models_root()}/PARD-Llama-3.2-1B" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - draft_len = 4 - spec_config = PARDDecodingConfig(max_draft_len=draft_len, - speculative_model=pard_model_dir) - - with LLM(model=target_model_dir, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config, - max_stats_len=-1, - enable_iter_perf_stats=True) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - acceptance_length = compute_acceptance_length(llm) - print(f"[AL] test_pard acceptance_length = {acceptance_length:.3f}") - assert_acceptance_length("TestLlama3_1_8BInstruct::test_pard", - acceptance_length) - - @skip_pre_hopper - def test_pard_sa(self): - """Accuracy test for PARD + Suffix Automaton speculative decoding.""" - pytorch_config = dict( - max_batch_size=1, - disable_overlap_scheduler=False, - cuda_graph_config=CudaGraphConfig(max_batch_size=1, - enable_padding=True), - ) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.8) - - pard_model_dir = f"{llm_models_root()}/PARD-Llama-3.2-1B" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - spec_config = PARDDecodingConfig(max_draft_len=4, - speculative_model=pard_model_dir, - sa_config=SAEnhancerConfig()) - - with LLM(model=target_model_dir, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm, extra_acc_spec="use_sa_spec") - - @skip_pre_hopper - @pytest.mark.skip(reason="PARD accuracy issue with batch size > 1") - def test_pard_sa_global_pool(self): - """Accuracy test for PARD + Suffix Automaton with global pool enabled.""" - max_batch_size = 32 - pytorch_config = dict( - max_batch_size=max_batch_size, - disable_overlap_scheduler=False, - cuda_graph_config=CudaGraphConfig(max_batch_size=max_batch_size, - enable_padding=True), - ) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.8) - - pard_model_dir = f"{llm_models_root()}/PARD-Llama-3.2-1B" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - spec_config = PARDDecodingConfig( - max_draft_len=4, - speculative_model=pard_model_dir, - sa_config=SAEnhancerConfig(enable_global_pool=True)) - - with LLM(model=target_model_dir, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm, extra_acc_spec="use_sa_spec") - - @skip_pre_blackwell - def test_pard_dynamic_draft_len(self): - draft_len_schedule = {50: 4, 200: 3, 350: 2} - max_draft_len = 4 - cuda_graph_config = CudaGraphConfig(max_batch_size=500) - pytorch_config = dict( - disable_overlap_scheduler=False, - cuda_graph_config=cuda_graph_config, - ) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.75) - pard_model_dir = f"{llm_models_root()}/PARD-Llama-3.2-1B" - pard_config = PARDDecodingConfig( - max_draft_len=max_draft_len, - speculative_model=pard_model_dir, - draft_len_schedule=draft_len_schedule, - ) - with LLM(self.MODEL_PATH, - kv_cache_config=kv_cache_config, - enable_chunked_prefill=False, - max_num_tokens=8192, - **pytorch_config, - speculative_config=pard_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_blackwell - def test_pard_sa_dynamic_draft_len(self): - draft_len_schedule = {50: 4, 200: 3, 350: 2} - max_draft_len = 4 - cuda_graph_config = CudaGraphConfig(max_batch_size=500) - pytorch_config = dict( - max_batch_size=500, - disable_overlap_scheduler=False, - cuda_graph_config=cuda_graph_config, - ) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.75) - pard_model_dir = f"{llm_models_root()}/PARD-Llama-3.2-1B" - pard_config = PARDDecodingConfig( - max_draft_len=max_draft_len, - speculative_model=pard_model_dir, - sa_config=SAEnhancerConfig(enable_global_pool=True), - draft_len_schedule=draft_len_schedule, - ) - with LLM(self.MODEL_PATH, - kv_cache_config=kv_cache_config, - enable_chunked_prefill=False, - max_num_tokens=16384, - **pytorch_config, - speculative_config=pard_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm, extra_acc_spec="use_sa_spec") - - def test_dflash(self): - pytorch_config = dict( - max_batch_size=8, - disable_overlap_scheduler=False, - cuda_graph_config=CudaGraphConfig(max_batch_size=8, - enable_padding=True), - ) - kv_cache_config = KvCacheConfig(enable_block_reuse=False, - free_gpu_memory_fraction=0.6) - - dflash_model_dir = f"{llm_models_root()}/LLaMA3.1-8B-Instruct-DFlash-UltraChat" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - spec_config = DFlashDecodingConfig(max_draft_len=4, - speculative_model=dflash_model_dir) - - with LLM(model=target_model_dir, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config, - max_stats_len=-1, - enable_iter_perf_stats=True) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - acceptance_length = compute_acceptance_length(llm) - print( - f"[AL] test_dflash acceptance_length = {acceptance_length:.3f}") - assert_acceptance_length("TestLlama3_1_8BInstruct::test_dflash", - acceptance_length) - - @skip_pre_blackwell - def test_dflash_dynamic_draft_len(self): - # DFlash uses a Qwen3-style draft with q/k_norm and 8K-wide cross-attn - # context, so the per-layer rmsnorm row count scales as - # B * max_ctx * num_kv_heads. Very large batches (e.g. 500) push that - # past the flashinfer rmsnorm kernel's stable range; cap at 200. - draft_len_schedule = {50: 4, 100: 3, 150: 2} - max_draft_len = 4 - pytorch_config = dict( - max_batch_size=200, - disable_overlap_scheduler=False, - cuda_graph_config=CudaGraphConfig(max_batch_size=200, - enable_padding=True), - ) - kv_cache_config = KvCacheConfig(enable_block_reuse=False, - free_gpu_memory_fraction=0.6) - dflash_model_dir = f"{llm_models_root()}/LLaMA3.1-8B-Instruct-DFlash-UltraChat" - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - spec_config = DFlashDecodingConfig( - max_draft_len=max_draft_len, - speculative_model=dflash_model_dir, - draft_len_schedule=draft_len_schedule, - ) - with LLM(model=target_model_dir, - max_seq_len=8192, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_hopper - def test_ngram(self): - max_bs = 16 - - pytorch_config = dict( - disable_overlap_scheduler=True, - cuda_graph_config=CudaGraphConfig( - batch_sizes=[i for i in range(1, max_bs + 1)]), - ) - - kv_cache_config = KvCacheConfig(enable_block_reuse=False, - free_gpu_memory_fraction=0.8) - - spec_config = NGramDecodingConfig( - max_draft_len=4, - max_matching_ngram_size=2, - is_keep_all=True, - is_use_oldest=True, - is_public_pool=True, - ) - - with LLM(model=self.MODEL_PATH, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config, - max_batch_size=max_bs, - max_stats_len=-1, - enable_iter_perf_stats=True) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - acceptance_length = compute_acceptance_length(llm) - print( - f"[AL] test_ngram acceptance_length = {acceptance_length:.3f}") - assert_acceptance_length("TestLlama3_1_8BInstruct::test_ngram", - acceptance_length) - - @skip_pre_hopper - @parametrize_with_ids("enable_global_pool", [False, True]) - def test_suffix_automaton(self, enable_global_pool): - max_bs = 16 - - pytorch_config = dict( - disable_overlap_scheduler=True, - cuda_graph_config=CudaGraphConfig( - batch_sizes=[i for i in range(1, max_bs + 1)]), - ) - - kv_cache_config = KvCacheConfig(enable_block_reuse=False, - free_gpu_memory_fraction=0.8) - - spec_config = SADecodingConfig( - max_draft_len=4, - max_matching_ngram_size=-1, # longest match via suffix automaton - enable_global_pool=enable_global_pool, - ) - - with LLM(model=self.MODEL_PATH, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config, - max_batch_size=max_bs, - max_stats_len=-1, - enable_iter_perf_stats=True) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - acceptance_length = compute_acceptance_length(llm) - print( - f"[AL] test_suffix_automaton enable_global_pool={enable_global_pool} " - f"acceptance_length = {acceptance_length:.3f}") - assert_acceptance_length( - "TestLlama3_1_8BInstruct::test_suffix_automaton", - acceptance_length) - - @skip_pre_blackwell - def test_suffix_automaton_dynamic_draft_len(self): - draft_len_schedule = {50: 4, 200: 3, 350: 2} - max_draft_len = 4 - cuda_graph_config = CudaGraphConfig(max_batch_size=500) - - pytorch_config = dict( - max_batch_size=500, - disable_overlap_scheduler=True, - cuda_graph_config=cuda_graph_config, - ) - - kv_cache_config = KvCacheConfig(enable_block_reuse=False, - free_gpu_memory_fraction=0.8) - spec_config = SADecodingConfig( - max_draft_len=max_draft_len, - max_matching_ngram_size=-1, - enable_global_pool=True, - draft_len_schedule=draft_len_schedule, - ) - - with LLM(model=self.MODEL_PATH, - **pytorch_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config, - enable_chunked_prefill=False, - max_num_tokens=8192) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_hopper - @parametrize_with_ids("spec_dec_algo", ["eagle3", "suffix_automaton"]) - # Builds two LLM() instances back-to-back; the MPI-pool-reuse test layer - # would otherwise hand the second one the first's just-used worker pool. - @pytest.mark.private_mpi_session - def test_one_engine_non_greedy_cuda_graph_matches_eager( - self, spec_dec_algo): - """TRTLLM-14874 regression: capture-only sampling state must not leak into cached graph metadata, non-greedy output must match with and without CUDA graphs.""" - max_bs = 4 - target_model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - if spec_dec_algo == "eagle3": - eagle_model_dir = f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B" - spec_config = Eagle3DecodingConfig( - max_draft_len=4, - speculative_model=eagle_model_dir, - eagle3_one_model=True) - kv_cache_config = KvCacheConfig(enable_block_reuse=True, - free_gpu_memory_fraction=0.8) - else: - spec_config = SADecodingConfig( - max_draft_len=4, - max_matching_ngram_size=-1, # longest match via suffix automaton - ) - kv_cache_config = KvCacheConfig(enable_block_reuse=False, - free_gpu_memory_fraction=0.8) - - def build_llm_kwargs(cuda_graph_config): - return dict(model=target_model_dir, - disable_overlap_scheduler=True, - cuda_graph_config=cuda_graph_config, - kv_cache_config=kv_cache_config, - speculative_config=spec_config, - max_batch_size=max_bs) - - _assert_non_greedy_cuda_graph_matches_eager( - build_llm_kwargs, CudaGraphConfig(max_batch_size=max_bs)) - - @skip_pre_blackwell - def test_draft_target_dynamic_draft_len(self): - draft_len_schedule = {50: 4, 200: 3, 350: 2} - max_draft_len = 4 - cuda_graph_config = CudaGraphConfig(max_batch_size=500) - pytorch_config = dict( - disable_overlap_scheduler=True, - cuda_graph_config=cuda_graph_config, - ) - kv_cache_config = KvCacheConfig( - enable_block_reuse=False, - free_gpu_memory_fraction=0.6, - ) - - spec_config = DraftTargetDecodingConfig( - max_draft_len=max_draft_len, - speculative_model=self.MODEL_PATH, - draft_len_schedule=draft_len_schedule, - ) - - with LLM(model=self.MODEL_PATH, - **pytorch_config, - kv_cache_config=kv_cache_config, - enable_chunked_prefill=False, - max_num_tokens=8192, - speculative_config=spec_config, - max_stats_len=-1, - enable_iter_perf_stats=True) as llm: - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - acceptance_length = compute_acceptance_length(llm) - print(f"[AL] test_draft_target_dynamic_draft_len " - f"acceptance_length = {acceptance_length:.3f}") - assert_acceptance_length( - "TestLlama3_1_8BInstruct::test_draft_target_dynamic_draft_len", - acceptance_length) - - @skip_pre_blackwell - @parametrize_with_ids("torch_compile", [False, True]) - @parametrize_with_ids("attn_backend", ["TRTLLM"]) - @parametrize_with_ids("v2_kv_cache", [True, False]) - def test_nvfp4_kv(self, attn_backend, torch_compile, v2_kv_cache): - torch_compile_config = _get_default_torch_compile_config(torch_compile) - pytorch_config = dict( - torch_compile_config=torch_compile_config, - cuda_graph_config=CudaGraphConfig(enable_padding=torch_compile, - batch_sizes=[4]), - attn_backend=attn_backend, - disable_overlap_scheduler=torch_compile, - ) - pytorch_config["kv_cache_config"] = KvCacheConfig( - dtype="nvfp4", use_kv_cache_manager_v2=v2_kv_cache) - with LLM(f"{llm_models_root()}/Llama-3_1-8B-Instruct_fp8_kv_nvfp4", - **pytorch_config) as llm: - assert llm.args.quant_config.quant_algo == QuantAlgo.FP8 - assert llm.args.quant_config.kv_cache_quant_algo == QuantAlgo.NVFP4 - task = MMLU(self.MODEL_NAME) - task.evaluate(llm) - task = GSM8K(self.MODEL_NAME) - task.evaluate(llm) - - @pytest.mark.parametrize("backend", ["xgrammar", "llguidance"]) - def test_guided_decoding(self, backend: str, mocker): - mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"}) - llm = LLM(self.MODEL_PATH, guided_decoding_backend=backend) - with llm: - task = JsonModeEval(self.MODEL_NAME) - task.evaluate(llm) - - @pytest.mark.timeout(7200) - @pytest.mark.skip_less_device(4) - @pytest.mark.parametrize("backend", ["xgrammar", "llguidance"]) - def test_guided_decoding_4gpus(self, backend: str, mocker): - mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"}) - with LLM(self.MODEL_PATH, - guided_decoding_backend=backend, - tensor_parallel_size=2, - pipeline_parallel_size=2) as llm: - task = JsonModeEval(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_hopper - @parametrize_with_ids("eagle3_one_model", [True, False]) - @pytest.mark.parametrize("backend", ["xgrammar", "llguidance"]) - def test_guided_decoding_with_eagle3(self, backend: str, - eagle3_one_model: bool, mocker): - mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"}) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.8) - cuda_graph_config = CudaGraphConfig(enable_padding=True) - spec_config = Eagle3DecodingConfig( - max_draft_len=3, - speculative_model=f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B", - eagle3_one_model=eagle3_one_model) - llm = LLM( - self.MODEL_PATH, - guided_decoding_backend=backend, - kv_cache_config=kv_cache_config, - cuda_graph_config=cuda_graph_config, - enable_chunked_prefill=True, - max_num_tokens=256, - speculative_config=spec_config, - # Two-model eagle3 does not support overlap scheduler - disable_overlap_scheduler=not eagle3_one_model) - with llm: - task = JsonModeEval(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_hopper - @pytest.mark.parametrize("backend", ["xgrammar", "llguidance"]) - def test_guided_decoding_with_eagle3_low_latency_dispatch( - self, backend: str, mocker): - """Smoke-test enable_low_latency_host_dispatch with Eagle3 + guided decoding. - - Eagle3 spec-dec captures guided-decoder hostfuncs inside the CUDA graph - (via _execute_guided_decoder_if_present in the target forward pass), so - this combination exercises the cudaLaunchHostFunc_v2 / spin-wait path. - """ - mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"}) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.8) - cuda_graph_config = CudaGraphConfig(enable_padding=True) - spec_config = Eagle3DecodingConfig( - max_draft_len=3, - speculative_model=f"{llm_models_root()}/EAGLE3-LLaMA3.1-Instruct-8B", - eagle3_one_model=True) - llm = LLM(self.MODEL_PATH, - guided_decoding_backend=backend, - kv_cache_config=kv_cache_config, - cuda_graph_config=cuda_graph_config, - enable_chunked_prefill=True, - max_num_tokens=256, - speculative_config=spec_config, - enable_low_latency_host_dispatch=True) - with llm: - task = JsonModeEval(self.MODEL_NAME) - task.evaluate(llm) - - @skip_pre_hopper - @pytest.mark.parametrize("backend", ["xgrammar", "llguidance"]) - def test_guided_decoding_with_ngram(self, backend: str, mocker): - mocker.patch.dict(os.environ, {"TRTLLM_XGUIDANCE_LENIENT": "1"}) - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.8) - cuda_graph_config = CudaGraphConfig(enable_padding=True) - spec_config = NGramDecodingConfig(max_draft_len=3, - max_matching_ngram_size=3) - llm = LLM(self.MODEL_PATH, - guided_decoding_backend=backend, - kv_cache_config=kv_cache_config, - cuda_graph_config=cuda_graph_config, - enable_chunked_prefill=True, - max_num_tokens=256, - speculative_config=spec_config, - disable_overlap_scheduler=True) - with llm: - task = JsonModeEval(self.MODEL_NAME) - task.evaluate(llm) - - @parametrize_with_ids("sampler_async_worker", [True, False]) - @parametrize_with_ids("disable_overlap_scheduler", [False, True]) - @parametrize_with_ids( - "enable_cuda_graph,enable_padding", - [ - (False, False), # No CUDA Graph (padding irrelevant) - (True, False), # CUDA Graph without padding - (True, True), # CUDA Graph with padding - ]) - def test_auto_dtype_beam_search(self, enable_cuda_graph, enable_padding, - disable_overlap_scheduler, - sampler_async_worker): - max_beam_width = 2 - sampling_params = SamplingParams(n=max_beam_width, - best_of=max_beam_width, - use_beam_search=True) - - if enable_cuda_graph: - # enable_padding only matters when CUDA Graph is enabled - if enable_padding: - batch_sizes = [ - 1, 8 - ] # Need batch_size != max_batch_size to enable padding - else: - batch_sizes = [1, 2, 4, 8] - cuda_graph_config = CudaGraphConfig(batch_sizes=batch_sizes, - enable_padding=enable_padding) - else: - cuda_graph_config = None - - with LLM( - model=self.MODEL_PATH, - kv_cache_config=KvCacheConfig(free_gpu_memory_fraction=0.5), - max_batch_size=max_beam_width, - max_seq_len=2048, - max_beam_width=max_beam_width, - sampler_force_async_worker=sampler_async_worker, - disable_overlap_scheduler=disable_overlap_scheduler, - cuda_graph_config=cuda_graph_config, - ) as llm: - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm, - sampling_params=sampling_params, - extra_acc_spec="beam_width=2") - - @skip_pre_hopper - @parametrize_with_ids("sampler_async_worker", [True, False]) - @parametrize_with_ids("disable_overlap_scheduler", [False, True]) - @parametrize_with_ids( - "enable_cuda_graph,enable_padding", - [ - (False, False), # No CUDA Graph (padding irrelevant) - (True, False), # CUDA Graph without padding - (True, True), # CUDA Graph with padding - ]) - def test_fp8_beam_search(self, enable_cuda_graph, enable_padding, - disable_overlap_scheduler, sampler_async_worker): - model_path = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct-FP8" - max_beam_width = 2 - sampling_params = SamplingParams(n=max_beam_width, - best_of=max_beam_width, - use_beam_search=True) - if enable_cuda_graph: - # enable_padding only matters when CUDA Graph is enabled - if enable_padding: - batch_sizes = [ - 1, 8 - ] # Need batch_size != max_batch_size to enable padding - else: - batch_sizes = [1, 2, 4, 8] - cuda_graph_config = CudaGraphConfig(batch_sizes=batch_sizes, - enable_padding=enable_padding) - else: - cuda_graph_config = None - - llm = LLM( - model=model_path, - kv_cache_config=KvCacheConfig(free_gpu_memory_fraction=0.5), - max_batch_size=max_beam_width, - max_seq_len=2048, - max_beam_width=max_beam_width, - disable_overlap_scheduler=disable_overlap_scheduler, - sampler_force_async_worker=sampler_async_worker, - cuda_graph_config=cuda_graph_config, - ) - - with llm: - task = CnnDailymail(self.MODEL_NAME) - task.evaluate(llm, - sampling_params=sampling_params, - extra_acc_spec="beam_width=2") - - class TestMinistral8BInstruct(LlmapiAccuracyTestHarness): MODEL_NAME = "mistralai/Ministral-8B-Instruct-2410" MODEL_PATH = f"{llm_models_root()}/Ministral-8B-Instruct-2410" @@ -1380,7 +403,7 @@ def test_bfloat16_2_model_mtp(self): # would otherwise hand the second one the first's just-used worker pool. @pytest.mark.private_mpi_session def test_mtp_non_greedy_cuda_graph_matches_eager(self): - """TRTLLM-14874 regression on the MTP capture path (see test_one_engine_non_greedy_cuda_graph_matches_eager in TestLlama3_1_8BInstruct for the EAGLE3/SA equivalents).""" + """TRTLLM-14874 regression on the MTP capture path.""" kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.3) mtp_config = MTPDecodingConfig(max_draft_len=3, mtp_eagle_one_model=False, @@ -7574,53 +6597,6 @@ def test_nvfp4_4gpus(self): task.evaluate(llm, sampling_params=sampling_params) -@skip_pre_blackwell -class TestLlama3_1_8B_Instruct_RocketKV(LlmapiAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct" - MODEL_PATH = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct/" - - def test_auto_dtype(self): - model_dir = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct/" - if not os.path.exists(model_dir): - pytest.skip(f"Model directory {model_dir} does not exist") - - # Configure model settings - kv_cache_config = KvCacheConfig(enable_block_reuse=False) - - cuda_graph_config = CudaGraphConfig(enable_padding=True, - max_batch_size=64) - - sparse_attention_config = RocketSparseAttentionConfig( - kt_cache_dtype="float8_e5m2", ) - - pytorch_config = dict(cuda_graph_config=cuda_graph_config, - kv_cache_config=kv_cache_config, - sparse_attention_config=sparse_attention_config, - enable_chunked_prefill=False) - - MAX_LEN = 128000 - MAX_NEW_TOKENS = 1024 - - with LLM(model_dir, - max_seq_len=MAX_LEN, - max_num_tokens=128000, - max_batch_size=64, - **pytorch_config) as llm: - task = LongBenchV2(self.MODEL_NAME) - - sampling_params = SamplingParams( - max_tokens=MAX_NEW_TOKENS, - temperature=0.8, - top_p=0.95, - ) - - extra_evaluator_kwargs = dict(max_len=MAX_LEN, - max_output_length=MAX_NEW_TOKENS) - task.evaluate(llm, - sampling_params=sampling_params, - extra_evaluator_kwargs=extra_evaluator_kwargs) - - class TestMistralLarge3_675B(LlmapiAccuracyTestHarness): MODEL_NAME = "mistral/Mistral-Large-3-675B" diff --git a/tests/integration/defs/accuracy/test_llm_api_pytorch_ray.py b/tests/integration/defs/accuracy/test_llm_api_pytorch_ray.py deleted file mode 100644 index 70c71ad74803..000000000000 --- a/tests/integration/defs/accuracy/test_llm_api_pytorch_ray.py +++ /dev/null @@ -1,26 +0,0 @@ -import pytest - -from tensorrt_llm import LLM -from tensorrt_llm.llmapi import KvCacheConfig - -from ..conftest import llm_models_root -from .accuracy_core import MMLU, LlmapiAccuracyTestHarness - -pytestmark = pytest.mark.ray - - -class TestLlama3_1_8BInstruct(LlmapiAccuracyTestHarness): - MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct" - MODEL_PATH = f"{llm_models_root()}/llama-3.1-model/Llama-3.1-8B-Instruct" - - @pytest.mark.skip_less_device(2) - @pytest.mark.skip_less_device_memory(32000) - def test_pp2_ray(self): - kv_cache_config = KvCacheConfig(free_gpu_memory_fraction=0.6) - - with LLM(self.MODEL_PATH, - orchestrator_type="ray", - pipeline_parallel_size=2, - kv_cache_config=kv_cache_config) as llm: - task = MMLU(self.MODEL_NAME) - task.evaluate(llm) diff --git a/tests/integration/test_lists/qa/README.md b/tests/integration/test_lists/qa/README.md index e242584e64f3..194eb2c8db3e 100644 --- a/tests/integration/test_lists/qa/README.md +++ b/tests/integration/test_lists/qa/README.md @@ -90,7 +90,7 @@ cd tests/integration/defs # Run all fp8 functional test pytest --no-header -vs --test-list=../test_lists/qa/llm_function_full.txt -k fp8 # Run a single test case -pytest -vs accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_auto_dtype +pytest -vs accuracy/test_llm_api_pytorch.py::TestMinistral8BInstruct::test_auto_dtype ``` ### Automated Execution diff --git a/tests/integration/test_lists/qa/llm_function_core.txt b/tests/integration/test_lists/qa/llm_function_core.txt index 5c96ee632cf8..b4192b1dfd07 100644 --- a/tests/integration/test_lists/qa/llm_function_core.txt +++ b/tests/integration/test_lists/qa/llm_function_core.txt @@ -22,35 +22,6 @@ accuracy/test_disaggregated_serving.py::TestGPTOSS::test_auto_dtype[True] accuracy/test_disaggregated_serving.py::TestGPTOSS::test_kv_cache_v2_nixl_python[cache_mgr_v1] accuracy/test_disaggregated_serving.py::TestGPTOSS::test_kv_cache_v2_nixl_python[cache_mgr_v2] accuracy/test_disaggregated_serving.py::TestKimiK25::test_nvfp4 -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_auto_dtype[block_reuse] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_auto_dtype[ctx_block_reuse_only] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_auto_dtype[no_block_reuse] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_beam_search -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=1-ctx_pp=2] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=1-ctx_pp=4] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=2-ctx_pp=2] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=2-ctx_pp=4] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=1-ctx_pp=2] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=1-ctx_pp=4] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=2-ctx_pp=2] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=2-ctx_pp=4] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_eagle3[eagle3_one_model=False-overlap_scheduler=False] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_eagle3[eagle3_one_model=True-overlap_scheduler=True] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_gen_only_spec_dec -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding[xgrammar] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[llguidance-eagle3_one_model=False] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[llguidance-eagle3_one_model=True] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[xgrammar-eagle3_one_model=False] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[xgrammar-eagle3_one_model=True] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_multi_instance[GSM8K] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_multi_instance[MMLU] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ngram -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[GSM8K-tp1pp2] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[GSM8K-tp2pp1] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[GSM8K-tp2pp2] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp1pp2] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp2pp1] -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp2pp2] accuracy/test_disaggregated_serving.py::TestNemotron3Super120B::test_auto_dtype[mtp_nextn=0-block_reuse=False-use_py_transceiver=True] accuracy/test_disaggregated_serving.py::TestNemotron3Super120B::test_auto_dtype[mtp_nextn=3-block_reuse=True-use_py_transceiver=True] accuracy/test_disaggregated_serving.py::TestNemotron3Super120B::test_ctx_dp2_gen_tp4 @@ -502,106 +473,6 @@ accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_nvfp4 accuracy/test_llm_api_pytorch.py::TestLagunaXS_2_1::test_bf16_dflash accuracy/test_llm_api_pytorch.py::TestLagunaXS_2_1::test_fp8_dflash accuracy/test_llm_api_pytorch.py::TestLagunaXS_2_1::test_nvfp4_dflash -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4 -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4_streaming[stream_interval_4] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4_streaming[stream_interval_64] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_auto_dtype_beam_search[enable_cuda_graph=False-enable_padding=False-disable_overlap_scheduler=False-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_auto_dtype_beam_search[enable_cuda_graph=False-enable_padding=False-disable_overlap_scheduler=True-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_auto_dtype_beam_search[enable_cuda_graph=True-enable_padding=False-disable_overlap_scheduler=False-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_auto_dtype_beam_search[enable_cuda_graph=True-enable_padding=False-disable_overlap_scheduler=True-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_auto_dtype_beam_search[enable_cuda_graph=True-enable_padding=True-disable_overlap_scheduler=False-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_auto_dtype_beam_search[enable_cuda_graph=True-enable_padding=True-disable_overlap_scheduler=False-sampler_async_worker=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_auto_dtype_beam_search[enable_cuda_graph=True-enable_padding=True-disable_overlap_scheduler=True-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp2pp2-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp2pp2-attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp2pp2-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp2pp2-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_chunked_prefill[use_temperature=False-attn_backend=FLASHINFER] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_chunked_prefill[use_temperature=False-attn_backend=TRTLLM] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_chunked_prefill[use_temperature=True-attn_backend=FLASHINFER] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_chunked_prefill[use_temperature=True-attn_backend=TRTLLM] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_dflash -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_dflash_dynamic_draft_len -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_draft_target_dynamic_draft_len -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_dummy_load_format -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3[sampler_async_worker=False-eagle3_one_model=False-overlap_scheduler=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3[sampler_async_worker=False-eagle3_one_model=True-overlap_scheduler=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3[sampler_async_worker=True-eagle3_one_model=True-overlap_scheduler=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3_rejection_dynamic_tree_smoke[dynamic_tree] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3_rejection_dynamic_tree_smoke[no_dynamic_tree] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3_sa -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3_sa_dynamic_draft_len -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3_sa_global_pool -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=True-attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=True-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_beam_search[enable_cuda_graph=False-enable_padding=False-disable_overlap_scheduler=False-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_beam_search[enable_cuda_graph=False-enable_padding=False-disable_overlap_scheduler=True-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_beam_search[enable_cuda_graph=True-enable_padding=False-disable_overlap_scheduler=False-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_beam_search[enable_cuda_graph=True-enable_padding=False-disable_overlap_scheduler=True-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_beam_search[enable_cuda_graph=True-enable_padding=True-disable_overlap_scheduler=False-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_beam_search[enable_cuda_graph=True-enable_padding=True-disable_overlap_scheduler=False-sampler_async_worker=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_beam_search[enable_cuda_graph=True-enable_padding=True-disable_overlap_scheduler=True-sampler_async_worker=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_llm_sampler -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_gather_generation_logits_cuda_graph -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding[llguidance] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding[xgrammar] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_4gpus[llguidance] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_4gpus[xgrammar] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[llguidance-eagle3_one_model=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[llguidance-eagle3_one_model=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[xgrammar-eagle3_one_model=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[xgrammar-eagle3_one_model=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_ngram[llguidance] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_ngram[xgrammar] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_ngram -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_nvfp4_kv[v2_kv_cache=False-attn_backend=TRTLLM-torch_compile=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_nvfp4_kv[v2_kv_cache=False-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_nvfp4_kv[v2_kv_cache=True-attn_backend=TRTLLM-torch_compile=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_one_engine_non_greedy_cuda_graph_matches_eager[spec_dec_algo=eagle3] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_one_engine_non_greedy_cuda_graph_matches_eager[spec_dec_algo=suffix_automaton] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_pard[overlap_scheduler=False] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_pard[overlap_scheduler=True] -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_pard_dynamic_draft_len -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_pard_sa -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_pard_sa_dynamic_draft_len -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_pard_sa_global_pool -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_suffix_automaton_dynamic_draft_len -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B_Instruct_RocketKV::test_auto_dtype accuracy/test_llm_api_pytorch.py::TestLlama4SpeculativeDecoding::test_llama4_eagle3[dynamic] TIMEOUT (60) accuracy/test_llm_api_pytorch.py::TestLlama4SpeculativeDecoding::test_llama4_eagle3[linear] TIMEOUT (60) accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_auto_dtype[tp_size=8-ep_size=8] TIMEOUT (180) @@ -787,7 +658,6 @@ accuracy/test_llm_api_pytorch_multimodal.py::TestQwen3_5_27B_VL::test_auto_dtype accuracy/test_llm_api_pytorch_multimodal.py::TestQwen3_5_35B_A3B_VL::test_auto_dtype accuracy/test_llm_api_pytorch_multimodal.py::TestQwen3_5_35B_A3B_VL::test_fp8_prequantized accuracy/test_llm_api_pytorch_multimodal.py::TestQwen3_8_Flash_Next_VL::test_nvfp4_1gpu_mtp3_trtllm_ple_offload -accuracy/test_llm_api_pytorch_ray.py::TestLlama3_1_8BInstruct::test_pp2_ray disaggregated/test_aiperf_gate.py::test_all_cancelled_fails disaggregated/test_aiperf_gate.py::test_corrupt_export_fails disaggregated/test_aiperf_gate.py::test_empty_export_fails diff --git a/tests/integration/test_lists/qa/llm_spark_func.yml b/tests/integration/test_lists/qa/llm_spark_func.yml index 0db4b20e4ece..446278687b46 100644 --- a/tests/integration/test_lists/qa/llm_spark_func.yml +++ b/tests/integration/test_lists/qa/llm_spark_func.yml @@ -29,8 +29,6 @@ llm_spark_func: - test_e2e.py::test_ptp_quickstart_advanced[Nemotron-Nano-9B-v2-nvfp4-NVIDIA-Nemotron-Nano-9B-v2-NVFP4] - test_e2e.py::test_ptp_quickstart_advanced_eagle3[GPT-OSS-120B-Eagle3-gpt_oss/gpt-oss-120b-gpt_oss/gpt-oss-120b-Eagle3] - test_e2e.py::test_ptp_quickstart_advanced[Qwen3.6-35B-A3B-nvfp4-Qwen3.6-35B-A3B-NVFP4] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_auto_dtype - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4 - accuracy/test_llm_api_pytorch_multimodal.py::TestGemma4_26B_A4B::test_nvfp4_no_mtp - accuracy/test_llm_api_pytorch_multimodal.py::TestGemma4_26B_A4B::test_nvfp4 - accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_nvfp4[latency_moe_cutlass-torch_compile=False] diff --git a/tests/integration/test_lists/test-db/README.md b/tests/integration/test_lists/test-db/README.md index 74e8a137cceb..051be642868f 100644 --- a/tests/integration/test_lists/test-db/README.md +++ b/tests/integration/test_lists/test-db/README.md @@ -31,8 +31,8 @@ l0_e2e: - '*h100*' linux_distribution_name: ubuntu* tests: - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=TRTLLM-torch_compile=False] + - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[pp4-mtp_nextn=0-fp8kv=False-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False-sampler_async_worker=False] + - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[tp4-mtp_nextn=0-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False-sampler_async_worker=False] ``` ## Generating Test Lists diff --git a/tests/integration/test_lists/test-db/l0_b200.yml b/tests/integration/test_lists/test-db/l0_b200.yml index 669fbeaec14e..1d1a9afc72bb 100644 --- a/tests/integration/test_lists/test-db/l0_b200.yml +++ b/tests/integration/test_lists/test-db/l0_b200.yml @@ -33,12 +33,6 @@ l0_b200: # ------------- PyTorch tests --------------- - unittest/others/test_kv_cache_transceiver.py::test_cpp_nixl_sync_transfer_stress - unittest/others/test_lora_manager.py - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4 - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4_streaming[stream_interval_4] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4_streaming[stream_interval_64] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_nvfp4_kv[v2_kv_cache=False-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_nvfp4_kv[v2_kv_cache=False-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_nvfp4_kv[v2_kv_cache=True-attn_backend=TRTLLM-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestLagunaXS::test_nvfp4 - accuracy/test_llm_api_pytorch.py::TestLagunaXS_2_1::test_bf16_dflash - accuracy/test_llm_api_pytorch.py::TestLagunaXS_2_1::test_nvfp4_dflash @@ -429,7 +423,6 @@ l0_b200: - accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_bf16[tp1-TRTLLM] - accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_bf16_branch_snapshot[enable_branch_snapshot=True] - accuracy/test_llm_api_pytorch.py::TestSeedOss_36B::test_auto_dtype - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B_Instruct_RocketKV::test_auto_dtype - accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_1gpu[v1_kv_cache-True-True-trtllm-auto] - accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_1gpu[v2_kv_cache-True-True-trtllm-auto] - accuracy/test_llm_api_pytorch_multimodal.py::TestGemma4_26B_A4B::test_nvfp4 diff --git a/tests/integration/test_lists/test-db/l0_b300.yml b/tests/integration/test_lists/test-db/l0_b300.yml index a3f29606b8ba..08feca1018fa 100644 --- a/tests/integration/test_lists/test-db/l0_b300.yml +++ b/tests/integration/test_lists/test-db/l0_b300.yml @@ -102,7 +102,6 @@ l0_b300: # DEEPGEMM backend: FP8_BLOCK_SCALES - unittest/_torch/moe/test_moe_module.py::test_configurable_moe_single_gpu[e60_k4_h2048_i1408-seq=1-dtype=torch.bfloat16-backend=DEEPGEMM-quant=FP8_BLOCK_SCALES-routing=Renormalize] # ---- end MoE tests ---- - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_nvfp4 - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4[moe_backend=TRTLLM-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4[moe_backend=CUTLASS-mtp_nextn=2-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_chunked_prefill[quant_dtype=fp8-kv_cache_reuse=True-fp8kv=True-overlap_scheduler=True] # Cover nvbugs 6084445 diff --git a/tests/integration/test_lists/test-db/l0_dgx_b200.yml b/tests/integration/test_lists/test-db/l0_dgx_b200.yml index 59fa1607f02f..381a82ce99de 100644 --- a/tests/integration/test_lists/test-db/l0_dgx_b200.yml +++ b/tests/integration/test_lists/test-db/l0_dgx_b200.yml @@ -130,8 +130,6 @@ l0_dgx_b200: - unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py -m "part3" - unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py -m "part4" - unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py -m "part5" - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - disaggregated/test_disaggregated.py::test_disaggregated_ctxpp2_genpp2[TinyLlama-1.1B-Chat-v1.0] - disaggregated/test_disaggregated.py::test_disaggregated_ctxtp2_genpp2[TinyLlama-1.1B-Chat-v1.0] - disaggregated/test_disaggregated.py::test_disaggregated_ctxpp2_gentp2[TinyLlama-1.1B-Chat-v1.0] @@ -292,9 +290,6 @@ l0_dgx_b200: orchestrator: mpi tests: - accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_bf16[tp2-CUTLASS] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[tp4-mtp_nextn=0-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[tp4-mtp_nextn=2-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[ep4-mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True] @@ -328,7 +323,6 @@ l0_dgx_b200: - accuracy/test_llm_api_pytorch.py::TestQwen3_5_397B_A17B::test_nvfp4[tep4_cutedsl] - accuracy/test_llm_api_pytorch.py::TestQwen3_5_397B_A17B::test_nvfp4[tep4_trtllm] - accuracy/test_llm_api_pytorch.py::TestQwen3_5_397B_A17B::test_nvfp4[adp4_trtllm] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_beam_search - accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_guided_decoding[xgrammar-mtp_nextn=0] - accuracy/test_disaggregated_serving.py::TestQwen3_30B_A3B::test_mixed_ctx_gen_model[ctxpp2gentp2] - accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v1_kv_cache-tp4-cutlass-auto] @@ -347,7 +341,6 @@ l0_dgx_b200: - accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_eagle3_vswa_reuse_4gpus[v2_kv_cache-one_model] - accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_eagle3_vswa_reuse_4gpus[v1_kv_cache-two_model] - unittest/_torch/multi_gpu_modeling -k "deepseek" - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTEDSL-mtp_nextn=2-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTEDSL-mtp_nextn=2-ep4-fp8kv=False-attention_dp=True-cuda_graph=False-overlap_scheduler=False-low_precision_combine=True-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTEDSL-mtp_nextn=2-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=True-torch_compile=False] diff --git a/tests/integration/test_lists/test-db/l0_dgx_b300.yml b/tests/integration/test_lists/test-db/l0_dgx_b300.yml index 7ed967b5398f..5ee651c345d0 100644 --- a/tests/integration/test_lists/test-db/l0_dgx_b300.yml +++ b/tests/integration/test_lists/test-db/l0_dgx_b300.yml @@ -33,11 +33,6 @@ l0_dgx_b300: - unittest/_torch/modeling -k "modeling_llama" - unittest/_torch/modeling -k "modeling_gpt_oss" - unittest/_torch/multi_gpu_modeling -k "deepseek" - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[tp4-mtp_nextn=0-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[tp4-mtp_nextn=2-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[ep4-mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True] diff --git a/tests/integration/test_lists/test-db/l0_dgx_h100.yml b/tests/integration/test_lists/test-db/l0_dgx_h100.yml index 74b8bfc0af49..960bb97b3326 100644 --- a/tests/integration/test_lists/test-db/l0_dgx_h100.yml +++ b/tests/integration/test_lists/test-db/l0_dgx_h100.yml @@ -28,23 +28,14 @@ l0_dgx_h100: - llmapi/test_llm_api_pytorch_bart.py::test_bart_pytorch_continuous_admission_replays_encoder_and_mixed_cuda_graphs[tp2] - llmapi/test_llm_api_pytorch_whisper.py::test_whisper_pytorch_feature_combinations[fp32-kv-v1-graphs-off-greedy-tp2] # ------------- Disaggregated serving tests --------------- - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_eagle3[eagle3_one_model=True-overlap_scheduler=True] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding[xgrammar] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[xgrammar-eagle3_one_model=True] - accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_auto_dtype[False-True] - accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_auto_dtype[True-True] - accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_auto_dtype[False-False] - accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_chunked_prefill - accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_nixl_backend - accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_gen_only_sync - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ngram - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_auto_dtype[block_reuse] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_auto_dtype[ctx_block_reuse_only] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_auto_dtype[no_block_reuse] - unittest/llmapi/apps/test_disagg_serving_perf_metrics.py - disaggregated/test_disaggregated.py::test_disaggregated_cancel_large_context_requests[DeepSeek-V3-Lite-bf16] - # llmapi - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_gen_only_spec_dec # ------------- Model specific tests --------------- - accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_w4a8[tp2_ep1] - condition: @@ -90,13 +81,6 @@ l0_dgx_h100: # spread across four local processes. - unittest/_torch/visual_gen/test_executor_lifecycle_multi_gpu.py::test_sigkill_one_worker_contains_real_multi_gpu_group # ------------- Model specific tests --------------- - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=True-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_4gpus[xgrammar] - accuracy/test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_4gpus_hopper_w4a16 - test_e2e.py::test_ptp_quickstart_advanced_bs1 - test_e2e.py::test_ptp_quickstart_advanced_deepseek_v3_lite_4gpus_adp_balance[DeepSeek-V3-Lite-FP8-DeepSeek-V3-Lite/fp8] @@ -120,16 +104,6 @@ l0_dgx_h100: - disaggregated/test_disaggregated.py::test_disaggregated_python_transceiver_host_offload[TinyLlama-1.1B-Chat-v1.0] - disaggregated/test_disaggregated.py::test_disaggregated_overlap_gen_first[ctx_pp1-TinyLlama-1.1B-Chat-v1.0] - disaggregated/test_disaggregated.py::test_disaggregated_overlap_gen_first[ctx_pp4-TinyLlama-1.1B-Chat-v1.0] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[GSM8K-tp1pp2] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp1pp2] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[GSM8K-tp2pp1] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp2pp1] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=1-ctx_pp=2] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=2-ctx_pp=2] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=1-ctx_pp=2] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=2-ctx_pp=2] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_multi_instance[GSM8K] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_multi_instance[MMLU] - accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_gen_first - accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_gen_first_kv_cache_v1 - accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_gen_first[noadp-mtp0] @@ -294,7 +268,6 @@ l0_dgx_h100: - unittest/_torch/ray_orchestrator/multi_gpu/test_ops.py -m "part4" - unittest/llmapi/test_llm_multi_gpu_pytorch.py -m "gpu2" - unittest/llmapi/test_async_llm.py -m "gpu2" - - accuracy/test_llm_api_pytorch_ray.py::TestLlama3_1_8BInstruct::test_pp2_ray - examples/test_ray.py::test_llm_inference_distributed_ray[tp2] - examples/test_ray.py::test_llm_inference_distributed_ray[pp2] - examples/test_ray.py::test_llm_inference_distributed_ray[tep2] diff --git a/tests/integration/test_lists/test-db/l0_dgx_h200.yml b/tests/integration/test_lists/test-db/l0_dgx_h200.yml index 35a749f4c01c..7e2dc208768a 100644 --- a/tests/integration/test_lists/test-db/l0_dgx_h200.yml +++ b/tests/integration/test_lists/test-db/l0_dgx_h200.yml @@ -31,12 +31,6 @@ l0_dgx_h200: - accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=0-overlap_scheduler=False] - accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=2-overlap_scheduler=True] - accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_auto_dtype[mtp_nextn=2-overlap_scheduler=False] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[GSM8K-tp2pp2] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp2pp2] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=1-ctx_pp=4] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=2-ctx_pp=4] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=1-ctx_pp=4] - - accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=2-ctx_pp=4] - accuracy/test_disaggregated_serving.py::TestGPTOSS::test_auto_dtype[False] - accuracy/test_disaggregated_serving.py::TestGPTOSS::test_auto_dtype[True] - accuracy/test_llm_api_pytorch.py::TestQwen3NextInstruct::test_bf16_4gpu[tep4] @@ -119,22 +113,6 @@ l0_dgx_h200: - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[pp4-mtp_nextn=0-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[pp4-mtp_nextn=0-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[pp4-mtp_nextn=0-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp2pp2-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp2pp2-attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp2pp2-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp2pp2-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=False-attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp2pp2-fp8kv=True-attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_4gpus[llguidance] - test_e2e.py::test_trtllm_bench_mgmn - accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B_Instruct_2507::test_skip_softmax_attention_4gpus[target_sparsity_0.5-fp8kv=False] - accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B_Instruct_2507::test_skip_softmax_attention_4gpus[target_sparsity_0.5-fp8kv=True] diff --git a/tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml b/tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml index b0a6c80b6f14..3f4100e02f7e 100644 --- a/tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml +++ b/tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml @@ -14,12 +14,8 @@ l0_gb200_multi_gpus: stage: pre_merge backend: pytorch tests: - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=True] - unittest/_torch/modules/test_engram.py - unittest/_torch/modules/test_mhc.py - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] # ---- curated pre-merge smoke set: diversify moe_backend across CUTLASS/TRTLLM/CUTEDSL ---- - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-tp4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=TRTLLM-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] @@ -80,8 +76,6 @@ l0_gb200_multi_gpus: - accuracy/test_llm_api_pytorch.py::TestQwen3NextInstruct::test_bf16_4gpu[tep4] - accuracy/test_llm_api_pytorch.py::TestQwen3NextInstruct::test_nvfp4[dep4-trtllm] - accuracy/test_llm_api_pytorch.py::TestQwen3NextInstruct::test_bf16_2gpu_mtp_ar - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-tp2pp2-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-low_precision_combine=False-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=2-tp4-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-low_precision_combine=False-torch_compile=False] @@ -114,9 +108,6 @@ l0_gb200_multi_gpus: - accuracy/test_llm_api_pytorch.py::TestQwen3_235B_A22B::test_nvfp4_4gpus[latency_moe_trtllm_eagle3] TIMEOUT (90) - accuracy/test_llm_api_pytorch.py::TestMistralLarge3_675B::test_nvfp4_4gpus[latency_moe_trtllm] TIMEOUT (90) - accuracy/test_llm_api_pytorch.py::TestMistralLarge3_675B::test_nvfp4_4gpus[latency_moe_trtllm_eagle] TIMEOUT (90) - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=TRTLLM-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] - accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4a16[dp4-auto] # ---- moved to post-merge (MoE CI optimization) ---- diff --git a/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml b/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml index a129bba4ecfd..0658027824da 100644 --- a/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml +++ b/tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml @@ -63,9 +63,6 @@ l0_gb300_multi_gpus: # from a pass, so a vanished checkpoint must be a visible regression). - test_kimi_k3_specdec.py::test_kimi_k3_sa_specdec_logits_parity TIMEOUT (40) - unittest/_torch/multi_gpu_modeling -k "deepseek" - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-tp2pp2-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-low_precision_combine=False-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=2-tp4-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-low_precision_combine=False-torch_compile=False] @@ -81,12 +78,6 @@ l0_gb300_multi_gpus: - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=True-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus_online_eplb[fp8kv=True-moe_backend=TRTLLM] - accuracy/test_llm_api_pytorch.py::TestQwen3_235B_A22B::test_nvfp4_4gpus[latency_moe_trtllm_eagle3] TIMEOUT (90) - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[tp4-mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[ep4-mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-tp4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=True] diff --git a/tests/integration/test_lists/test-db/l0_h100.yml b/tests/integration/test_lists/test-db/l0_h100.yml index a91ef915d8aa..eb898fc3a2b6 100644 --- a/tests/integration/test_lists/test-db/l0_h100.yml +++ b/tests/integration/test_lists/test-db/l0_h100.yml @@ -129,22 +129,6 @@ l0_h100: - unittest/usage/test_e2e_capture.py - accuracy/test_kv_pool_rebalance_accuracy.py::TestKvPoolRebalanceAccuracy::test_rebalance_matches_baseline[no_overlap] - accuracy/test_kv_pool_rebalance_accuracy.py::TestKvPoolRebalanceAccuracy::test_rebalance_matches_baseline[overlap] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_chunked_prefill[use_temperature=False-attn_backend=TRTLLM] TIMEOUT (90) - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_chunked_prefill[use_temperature=True-attn_backend=TRTLLM] TIMEOUT (90) - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_dummy_load_format - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding[xgrammar] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[xgrammar-eagle3_one_model=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3[sampler_async_worker=False-eagle3_one_model=True-overlap_scheduler=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_eagle3[sampler_async_worker=True-eagle3_one_model=True-overlap_scheduler=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_one_engine_non_greedy_cuda_graph_matches_eager[spec_dec_algo=eagle3] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_one_engine_non_greedy_cuda_graph_matches_eager[spec_dec_algo=suffix_automaton] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_pard[overlap_scheduler=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_pard[overlap_scheduler=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_dflash - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales[mtp=disable-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales[mtp=eagle-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales[mtp=vanilla-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True] @@ -372,16 +356,6 @@ l0_h100: - accuracy/test_llm_api_pytorch.py::TestQwen3_8B::test_eagle3[eagle3_one_model=True-enable_chunked_prefill=False-enable_max_concurrency=False-enable_draft_len_schedule=True] - accuracy/test_llm_api_pytorch.py::TestQwen3_8B::test_eagle3[eagle3_one_model=True-enable_chunked_prefill=False-enable_max_concurrency=True-enable_draft_len_schedule=False] - accuracy/test_llm_api_pytorch.py::TestNemotronV3Nano::test_fp8 - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=TRTLLM-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16[attn_backend=TRTLLM-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_chunked_prefill[use_temperature=False-attn_backend=FLASHINFER] TIMEOUT (90) - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_chunked_prefill[use_temperature=True-attn_backend=FLASHINFER] TIMEOUT (90) - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=FLASHINFER-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=True-attn_backend=FLASHINFER-torch_compile=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=0-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=False-enable_chunked_prefill=False-v2_kv_cache=False] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=0-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=False-enable_chunked_prefill=False-v2_kv_cache=True] - accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=0-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=True-enable_chunked_prefill=False-v2_kv_cache=False] @@ -430,10 +404,6 @@ l0_h100: - accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_fp8[latency-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_fp8_block_scales[latency-torch_compile=False] - accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_fp8_block_scales[latency-torch_compile=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding[llguidance] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3[llguidance-eagle3_one_model=True] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_ngram[xgrammar] - - accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_ngram[llguidance] - accuracy/test_llm_api_pytorch_multimodal.py::TestMistralSmall24B::test_auto_dtype[forced_chunked_prefill] - accuracy/test_llm_api_pytorch_multimodal.py::TestQwen3VL::test_auto_dtype[forced_chunked_prefill] - accuracy/test_llm_api_pytorch_multimodal.py::TestNemotron_Nano_12B_V2_VL::test_auto_dtype[forced_chunked_prefill] diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index cc4bc23e48c0..dd6e4ad16c5b 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -1,7 +1,4 @@ accuracy/test_disaggregated_serving.py::TestGLM52NVFP4::test_nvfp4_nixl[cache_mgr_v1] SKIP (https://nvbugs/6619883) -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ngram SKIP (https://nvbugs/6245651) -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[GSM8K-tp2pp1] SKIP (https://nvbugs/6644475) -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp2pp1] SKIP (https://nvbugs/6611817) accuracy/test_disaggregated_serving.py::TestQwen3_30B_A3B::test_mixed_ctx_gen_model[ctxpp2gentp2] SKIP (https://nvbugs/5748664) accuracy/test_dwdp_aggregated.py::TestDwdpAggDeepSeekV3Lite::test_dwdp_agg_accuracy[mode_a_uniform] SKIP (https://nvbugs/6661863) accuracy/test_dwdp_aggregated.py::TestDwdpAggDeepSeekV3Lite::test_dwdp_agg_accuracy[mode_a_uniform_contention_opt] SKIP (https://nvbugs/6644489) @@ -31,9 +28,6 @@ accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-cutl accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-ep4-cutlass-auto] SKIP (https://nvbugs/5596343) accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-tp4-cutlass-auto] SKIP (https://nvbugs/5596343) accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_chunked_prefill[cutlass-auto] SKIP (https://nvbugs/5596343) -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=TRTLLM-torch_compile=False] SKIP (https://nvbugs/5616182) -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=False] SKIP (https://nvbugs/6437412) -accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=True-attn_backend=TRTLLM-torch_compile=False] SKIP (https://nvbugs/6278337) accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_nvfp4[use_msa=True] SKIP (https://nvbugs/6601633) accuracy/test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_4gpus_hopper_w4a16 SKIP (https://nvbugs/6644472) accuracy/test_llm_api_pytorch.py::TestNemotronV3Super::test_nvfp4_8gpus_mtp SKIP (https://nvbugs/6581065) @@ -142,7 +136,6 @@ full:DGX_H100/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat1 full:DGX_H100/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=2-attention_dp=False-cuda_graph=False-overlap_scheduler=False-torch_compile=True-enable_chunked_prefill=False-v2_kv_cache=True] SKIP (https://nvbugs/6633927) full:DGX_H100/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=2-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False-enable_chunked_prefill=False-v2_kv_cache=True] SKIP (https://nvbugs/6633927) full:DGX_H100/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales[mtp=disable-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True] SKIP (https://nvbugs/6700265) -full:GB200/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_auto_dtype[ctx_block_reuse_only] SKIP (https://nvbugs/6525893) full:GB200/accuracy/test_dwdp_disaggregated_serving.py::TestDwdpDeepSeekV3Lite::test_dwdp_accuracy SKIP (https://nvbugs/6276923) full:GB200/accuracy/test_dwdp_disaggregated_serving.py::TestDwdpDeepSeekV3Lite::test_dwdp_accuracy_contention_opt SKIP (https://nvbugs/6276923) full:GB200/accuracy/test_dwdp_disaggregated_serving.py::TestDwdpDeepSeekV3Lite::test_dwdp_accuracy_mode_b_overlap SKIP (https://nvbugs/6276923) @@ -162,9 +155,6 @@ full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4 full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[pp4-mtp_nextn=0-fp8kv=False-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=False-sampler_async_worker=False] SKIP (https://nvbugs/6661948) full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[pp4-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True-sampler_async_worker=False] SKIP (https://nvbugs/6661948) full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-pp4-fp8kv=False-attention_dp=False-cuda_graph=False-overlap_scheduler=False-low_precision_combine=False-torch_compile=True] SKIP (https://nvbugs/6697099) -full:GB300/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=FLASHINFER-torch_compile=False] SKIP (https://nvbugs/6385771) -full:GB300/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[tp4-attn_backend=FLASHINFER-torch_compile=True] SKIP (https://nvbugs/6657571) -full:GB300/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[pp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=False] SKIP (https://nvbugs/6385771) full:GB300/accuracy/test_llm_api_pytorch.py::TestMiniMaxM3::test_mxfp8_piecewise_cuda_graph[use_msa=False] SKIP (https://nvbugs/6714109) full:GB300/accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_fp8_moe_dflash SKIP (https://nvbugs/6316985) full:GB300/accuracy/test_llm_api_pytorch.py::TestQwen3_5_4B::test_bf16 SKIP (https://nvbugs/6487918) @@ -206,23 +196,11 @@ full:H20/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_sc full:H20/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_cuda_graph_padding_4gpus[attention_dp=True-mtp_nextn=0] SKIP (https://nvbugs/6692005) full:H20/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_eagle3_2gpus[triton-one_model-overlap_scheduler] SKIP (https://nvbugs/6373530) full:H20/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_chunked_prefill[triton-auto] SKIP (https://nvbugs/6026676) -full:H20/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[fp8kv=False-attn_backend=TRTLLM-torch_compile=True] SKIP (https://nvbugs/6546909) -full:H20/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=False-attn_backend=TRTLLM-torch_compile=True] SKIP (https://nvbugs/6580087) -full:H20/accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8_4gpus[tp4-fp8kv=True-attn_backend=FLASHINFER-torch_compile=True] SKIP (https://nvbugs/6546909) full:H20/accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[forced_chunked_prefill] SKIP (https://nvbugs/6597570) full:H20/accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[full_budget] SKIP (https://nvbugs/6597570) full:H20/accuracy/test_llm_api_pytorch_multimodal.py::TestNanoV3Omni::test_auto_dtype[bf16] SKIP (https://nvbugs/6618649) full:H20/disaggregated/test_disaggregated.py::test_disaggregated_logprobs_serving[llama-3.1-8b-instruct] SKIP (https://nvbugs/6275959) full:H20/disaggregated/test_disaggregated.py::test_disaggregated_overlap_gen_first[ctx_pp1-TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6692009) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=1-ctx_pp=2] SKIP (https://nvbugs/6649818) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=2-ctx_pp=2] SKIP (https://nvbugs/6649818) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=1-ctx_pp=2] SKIP (https://nvbugs/6649818) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[MMLU-gen_tp=2-ctx_pp=2] SKIP (https://nvbugs/6649818) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding[xgrammar] SKIP (https://nvbugs/6649818) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_multi_instance[GSM8K] SKIP (https://nvbugs/6649818) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_multi_instance[MMLU] SKIP (https://nvbugs/6649818) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[GSM8K-tp1pp2] SKIP (https://nvbugs/6649818) -full:L40S/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_tp_pp_symmetric[MMLU-tp1pp2] SKIP (https://nvbugs/6649818) full:L40S/disaggregated/test_disaggregated.py::test_disaggregated_ctxpp2_genpp2[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6728119) full:L40S/disaggregated/test_disaggregated.py::test_disaggregated_ctxpp2_gentp2[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6728119) full:L40S/disaggregated/test_disaggregated.py::test_disaggregated_ctxtp2_genpp2[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/6728119)