From a74e5072bf846e6e87494856f689c908f36c149e Mon Sep 17 00:00:00 2001 From: ADou Date: Wed, 9 Sep 2026 08:52:10 +0000 Subject: [PATCH] [None][docs] retarget ModelOpt examples llm_ptq paths to hf_ptq MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Model-Optimizer renamed examples/llm_ptq → examples/hf_ptq (llm_ptq 404). Update Qwen/quantization/disaggregated example READMEs: paths, shell quant preset fp8_pc_pt→fp8_per_channel_per_token, drop obsolete --export_fmt hf from huggingface_example.sh invocations, and point the quantization README at github.com/NVIDIA/Model-Optimizer. Signed-off-by: ADou --- examples/disaggregated/README.md | 2 +- examples/models/core/qwen/README.md | 14 +++++++------- examples/quantization/README.md | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/examples/disaggregated/README.md b/examples/disaggregated/README.md index a03b875a0a6b..b6cd6b70abdf 100644 --- a/examples/disaggregated/README.md +++ b/examples/disaggregated/README.md @@ -473,7 +473,7 @@ To enable mixed precision serving, you will need: A quantized checkpoint can be created using `--kv_cache_qformat none`. ```bash -python $MODELOPT_ROOT/examples/llm_ptq/hf_ptq.py \ +python $MODELOPT_ROOT/examples/hf_ptq/hf_ptq.py \ --pyt_ckpt_path=meta-llama/Llama-3.1-8B-Instruct \ --export_path=./weights/Llama-3.1-8B-Instruct-FP8-KV-BF16 \ --sparsity_fmt=dense \ diff --git a/examples/models/core/qwen/README.md b/examples/models/core/qwen/README.md index 6c7654357a18..44794023e186 100644 --- a/examples/models/core/qwen/README.md +++ b/examples/models/core/qwen/README.md @@ -82,12 +82,12 @@ pushd Model-Optimizer pip install -e . # Quantize the Qwen3-235B-A22B model by nvfp4 -# By default, the checkpoint would be stored in `Model-Optimizer/examples/llm_ptq/saved_models_Qwen3-235B-A22B_nvfp4_hf/`. -./examples/llm_ptq/scripts/huggingface_example.sh --model Qwen3-235B-A22B/ --quant nvfp4 --export_fmt hf +# By default, the checkpoint would be stored in `Model-Optimizer/examples/hf_ptq/saved_models_Qwen3-235B-A22B_nvfp4/`. +./examples/hf_ptq/scripts/huggingface_example.sh --model Qwen3-235B-A22B/ --quant nvfp4 -# Quantize the Qwen3-32B model by fp8_pc_pt -# By default, the checkpoint would be stored in `Model-Optimizer/examples/llm_ptq/saved_models_Qwen3-32B_fp8_pc_pt_hf/`. -./examples/llm_ptq/scripts/huggingface_example.sh --model Qwen3-32B/ --quant fp8_pc_pt --export_fmt hf +# Quantize the Qwen3-32B model by fp8_per_channel_per_token +# By default, the checkpoint would be stored in `Model-Optimizer/examples/hf_ptq/saved_models_Qwen3-32B_fp8_per_channel_per_token/`. +./examples/hf_ptq/scripts/huggingface_example.sh --model Qwen3-32B/ --quant fp8_per_channel_per_token popd ``` @@ -98,7 +98,7 @@ To run the benchmark, we suggest using the `trtllm-bench` tool. Please refer to ```bash #!/bin/bash -folder_model=Model-Optimizer/examples/llm_ptq/saved_models_Qwen3-235B-A22B_nvfp4_hf/ +folder_model=Model-Optimizer/examples/hf_ptq/saved_models_Qwen3-235B-A22B_nvfp4/ path_config=config.yml num_gpus=8 ep_size=8 @@ -138,7 +138,7 @@ trtllm-bench --model ${folder_model} --model_path ${folder_model} throughput \ We suggest benchmarking with a real dataset. It will prevent from having improperly distributed tokens in the MoE. Here, we use the `aa_prompt_isl_1k_osl_2k_qwen3_10000samples.txt` dataset. It has 10000 samples with an average input length of 1024 and an average output length of 2048. If you don't have a dataset (this or another) and you want to run the benchmark, you can use the following command to generate a random dataset: ```bash -folder_model=Model-Optimizer/examples/llm_ptq/saved_models_Qwen3-235B-A22B_nvfp4_hf/ +folder_model=Model-Optimizer/examples/hf_ptq/saved_models_Qwen3-235B-A22B_nvfp4/ min_input_len=1024 min_output_len=2048 concurrency=128 diff --git a/examples/quantization/README.md b/examples/quantization/README.md index 23bfa5d40c8e..be8a2712accc 100644 --- a/examples/quantization/README.md +++ b/examples/quantization/README.md @@ -6,8 +6,8 @@ To run quantized models with TensorRT LLM: checkpoints published on the [NVIDIA Hugging Face hub](https://huggingface.co/nvidia)). Quantization settings are detected automatically when the model loads. - To quantize your own model, use the - [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) - Hugging Face export flow (`examples/llm_ptq` in that repository). + [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/Model-Optimizer) + Hugging Face export flow (`examples/hf_ptq` in that repository). See the [quantization feature documentation](https://nvidia.github.io/TensorRT-LLM/features/quantization.html) for supported formats per GPU architecture.