Merge/dev to main 20260807 - #97
Open
zhaoyinglia wants to merge 522 commits into
Open
Conversation
…tensors. NVIDIA#2120" (NVIDIA#2673) * Adds dst.dtype information in copy_ method of quantized tensors. Signed-off-by: Zhiyi Su <dantesuu@gmail.com> * Update transformer_engine/pytorch/tensor/quantized_tensor.py Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Signed-off-by: ZhiyiDanielSu <35579247+zobeideThePlayer@users.noreply.github.com> * Update transformer_engine/pytorch/quantized_tensor.py Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fix reference tensor copy Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Zhiyi Su <dantesuu@gmail.com> Signed-off-by: ZhiyiDanielSu <35579247+zobeideThePlayer@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: Zhiyi Su <dantesuu@gmail.com> Co-authored-by: ZhiyiDanielSu <35579247+zobeideThePlayer@users.noreply.github.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Fuse scale + 0 + cumulative sum for splits to offsets calc Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add unit test and fix bug in kernel for >256 size Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix race Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * check for logical_last_dim > 0 Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * suggestions Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Added new people to CI Signed-off-by: Przemek Tredak <ptredak@nvidia.com> * Removing duplicate Signed-off-by: Przemek Tredak <ptredak@nvidia.com> --------- Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
… parallelism (NVIDIA#2688) * Error out if constructing LayerNormLinear with row tensor parallelism Signed-off-by: Tim Moon <tmoon@nvidia.com> * Disable Userbuffers test for row-TP LayerNormLinear Signed-off-by: Tim Moon <tmoon@nvidia.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Enable cgemm + FP8 tests * Implement CGEMM + MXFP8 * added size check for mxfp8 * added tols for assertions * update tests with recipes * enable tests + is_quantize_recipe_supported Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> --------- Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…kScaling and Float8BlockScaling quantized model init. (NVIDIA#2753) * Updates FusedAdam with FSDP2 and MXFP8 Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * removes xfailing unit test for MXFPr MXFP8 Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * addresses comments related to reset parameters and guard against self.capturable Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * adds e2e unit test Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * adds test to non meta device init Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * attempts to add float8block scaling fsdp hooks Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * adds e2e test for Float8BlockScaling Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * addresses review comments and code cleanup Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * more review comments addressed Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * removes unused block_len param Signed-off-by: Jonathan Mitchell <jomitchell@umb-b300-dp-147.ipp4a1.colossus.nvidia.com> * fixes failing unit test because we still need to xfail nvfp4 dcp Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * lint - replacing todo with note Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1429.ipp1a1.colossus.nvidia.com> --------- Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> Signed-off-by: Jonathan Mitchell <jomitchell@umb-b300-dp-147.ipp4a1.colossus.nvidia.com> Signed-off-by: Jonathan Mitchell <jomitchell@ipp1-1429.ipp1a1.colossus.nvidia.com> Co-authored-by: Jonathan Mitchell <jomitchell@ipp1-1334.ipp1a1.colossus.nvidia.com> Co-authored-by: Jonathan Mitchell <jomitchell@umb-b300-dp-147.ipp4a1.colossus.nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: vthumbe1503 <vthumbe@nvidia.com> Co-authored-by: Jonathan Mitchell <jomitchell@ipp1-1429.ipp1a1.colossus.nvidia.com>
Signed-off-by: Peter St. John <pstjohn@nvidia.com>
* fix for async dcp checkpointing Signed-off-by: Peter St. John <pstjohn@nvidia.com> * Apply suggestions from code review Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Peter St. John <peterc.stjohn@gmail.com> * Update transformer_engine/pytorch/tensor/storage/float8_tensor_storage.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Greptile review feedback: defensive guards for edge cases - Add _quantizer None guard in new_empty dispatch - Replace self.is_cpu with explicit _data/_transpose checks in __reduce_ex__ - Make get_metadata() safe for cleared tensors (both _data and _transpose None) Signed-off-by: Peter St. John <pstjohn@nvidia.com> --------- Signed-off-by: Peter St. John <pstjohn@nvidia.com> Signed-off-by: Peter St. John <peterc.stjohn@gmail.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
…ped Tensor Swizzling (NVIDIA#2669) Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove changes not needed for bf16 Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * keep only pytorch binding for now Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * linting error Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * add fast accumulator support Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * MXFP8 grouped GEMM + tensor-scaled FP8 fixes Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change version to 13.3 Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * fix the test Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Random padding condition shouldnt be done for mxfp8 Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Remove incorrect comment Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * CUBLAS > 13.2 is enough Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * CUBLAS version needed for MXFP8 indeed seems to be 13.3 Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * all changes for grouped gemm Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Accidental line removal added back. Plus need changes ci t trigger Add documentation for scaling factors in common.h Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update cuBLAS version requirement for MXFP8 support Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * grouped gemm: address code review comments - Replace nvte_set/get_grouped_tensor_swizzled_scales with nvte_set_grouped_tensor_param - Add host-side validation: A and B must use same scaling mode (both MXFP8 or both tensor scaling) - Add host-side validation: A and B must both be FP8 or both non-FP8; restrict inputs to FP8/BF16 - Restrict output (C/D) to BF16/FP32; remove FP16 from supported types - Refactor workspace allocation: replace manual offset arithmetic with moving pointer pattern - Use void* + NVTEScalingMode in setup kernel instead of separate float*/char* scale params - Extract use_columnwise(swap_dims) helper to eliminate duplicated MXFP8 columnwise blocks - Split set_fp8_scale_pointers into set_fp8_scale_pointers / set_mxfp8_scale_pointers - Remove scale_inv_ptrs from GroupedOperandSelection; pass workspace pointers directly - Move swizzled-scales validation into validate_grouped_gemm_inputs for fail-fast behavior - Add use_split_accumulator to GroupedMatmulConfig (Hopper only, default false) - Add FP8 test case with per-tensor scales; add BF16/MXFP8 shape-varying test cases Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * address reviee comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * missed merged conflict handling Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor change Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * forgot adding a or Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * resolve merge conflicts Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address minor review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * remove unecessary code Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * one line that broke everything :( Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * unecessary Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * revert caching changes Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * fix minor bug from greptile Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * revert for now Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * address review commentsgp Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Jeremy Berchtold <jberchtold@nvidia.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
…VIDIA#2757) * [Common] Fix linker error for to_string(DType) in distributed tests Make transformer_engine::to_string(DType) inline in common.h so that translation units outside libtransformer_engine.so can resolve it without requiring the symbol to be exported. Regression introduced by 61f9594 which added to_string(DType) calls into TRANSFORMER_ENGINE_TYPE_SWITCH_* macros, causing test object files to reference the symbol that the linker version script hides. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Vladimir Cherepanov <vcherepanov@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Vladimir Cherepanov <vcherepanov@nvidia.com> --------- Signed-off-by: Vladimir Cherepanov <vcherepanov@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…-Cast Fusion Kernel (NVIDIA#2555) * first draft Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * pass numerical unit test Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * format Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * add benchmark script Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * lint and format Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * compile guard Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * warning fix Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * resolve greptile comment Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * minor style fixes Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * fix namespace Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * resolve some comments Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * fix comment Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * attempt to fix compile CI with guard Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * better naming for tests Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * fix deprecate messsage Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * more compile guard Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * new API name Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * fix format all in one Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * try to fix compile CI again Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * AI code review comments Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * to pass oldest compile CI with cuda 12.1 Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * add more guards to nvfp4 Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * make multiply inverse default numerics Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * update numerics of nvfp4 partial cast as well Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * resolve comments Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * add NVTE_BUILD_NUM_PHILOX_ROUNDS after rebase Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> * simplify compile guard messsages Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> --------- Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…Linear` (NVIDIA#2761) * Load multi-param checkpoint from single-param config in GroupedLinear Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Multi-param to single param case Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Multi-param to single param case Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Better varnames Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
…A#2765) * Fix GMM cuBLAS version and SM arch checks Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Update transformer_engine/common/gemm/cublaslt_grouped_gemm.cu Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…DIA#2773) * Pin python 3.13 in vermin check Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Update vermin version for python 3.14 support Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Use sha instead of tag Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* init Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * work finished Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * lint fixes Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * fixes Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: root <pgadzinski@nvidia.com> * removed warning.warn Signed-off-by: root <pgadzinski@nvidia.com> * [PyTorch] Remove dead None-check for num_out_tokens in moe_permute_mask_map_forward num_out_tokens is typed as int in the custom_op signature and can never be None; the check was incorrectly carried over from the class-based upstream version during merge conflict resolution. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> --------- Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> Signed-off-by: root <pgadzinski@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…are detached (NVIDIA#2772) [PyTorch] Change the restore tensor API to ensure tensors are detached from ctx Signed-off-by: Kaining Zhong <kainingz@nvidia.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
…ges it (NVIDIA#2781) Install pytest in onnx L1 test as Pyt container no longer packages it Signed-off-by: Kshitij Janardan Lakhani <klakhani@nvidia.com>
…_descriptors (NVIDIA#2782) * Fix zero-sized groups in update_tma_descriptors Signed-off-by: jberchtold-nvidia <158520091+jberchtold-nvidia@users.noreply.github.com> * Update test_cast_mxfp8_grouped.cu Signed-off-by: jberchtold-nvidia <158520091+jberchtold-nvidia@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: jberchtold-nvidia <158520091+jberchtold-nvidia@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: jberchtold-nvidia <158520091+jberchtold-nvidia@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
NVIDIA#2693) * Enable sm120 support for fused attn if cuDNN is 9.18.1+ Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Force intermediate tensors such as S, Sum_Exp, and Max to be BHS1 shape instead of TH1 for sm120 Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add support for sm120 correct batch, seq dims Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * Add support for sm120 BHS1 style max logit even QKV are THD to avoid incorrect max logit calculation (includes padded tokens in max calculation) Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * Disable fused and flash attn for sm120 filter:kv cache Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * For CP P2P attn, set softmax_lse_in_packed_format to False if sm120+ Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * Assert in TE if T3HD/TH3D layout is used on sm120 before cuDNN F16 sdpa arbitrary kernel call Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * Modify is_ragged_q && cudnn_runtime_version >= 90600 check to also include a check for sm120 Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * nit: Code clean up Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * Disable fused attn for T3HD and TH3D Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * nit: Add missed sm120 guard Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * Modify sm120 condition to be very specific to sm120 and not generalized to sm120+ Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * nit: Fix missing sm120 check in fwd Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * Move the check for sm120 T3HD/TH3D to nvte_get_fused_attn_backend() instead of higher layers in TE stack Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * nit: Check for matching sm120 and not sm120+ Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* code drop Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * code drop Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docs Signed-off-by: root <pgadzinski@nvidia.com> * nvfp4 internals support Signed-off-by: root <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * lint fixes Signed-off-by: root <pgadzinski@nvidia.com> * Update transformer_engine/debug/features/dump_tensors.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Paweł Gadziński <62263673+pggPL@users.noreply.github.com> * fix Signed-off-by: root <pgadzinski@nvidia.com> * Update transformer_engine/debug/features/dump_tensors.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Paweł Gadziński <62263673+pggPL@users.noreply.github.com> * Update transformer_engine/debug/features/dump_tensors.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Paweł Gadziński <62263673+pggPL@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update tests/pytorch/debug/test_log.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Paweł Gadziński <62263673+pggPL@users.noreply.github.com> * Update transformer_engine/debug/features/dump_tensors.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Paweł Gadziński <62263673+pggPL@users.noreply.github.com> * fix Signed-off-by: root <pgadzinski@nvidia.com> * fix Signed-off-by: root <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove dump_quantized_internals support from DumpTensors Drop the dump_quantized_internals config option, the _get_quantized_internals method, and all helper functions for extracting scales/raw data from Float8Tensor, Float8BlockwiseQTensor, MXFP8Tensor, and NVFP4Tensor. Remove corresponding tests: test_dump_tensors_nvfp4_unpacked_codes and NVFP4_DUMP_TENSORS_CONFIG, and scale/data assertions from test_dump_tensors_sanity. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address Greptile review comments - Add dot ('.') to _sanitize_name to handle common PyTorch dotted layer names like 'encoder.layer.0.attention' - Add docstring note about pickle dependency for the 'quantized' key - Add comment explaining weights_only=False in test - Remove redundant local RecipeState import in test_nvfp4_numeric Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * Remove portability suggestion from quantized key docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * Compute rank lazily in _expected_root_dir Avoids relying on stale self.rank when ensure_initialized is called before initialize() has set the rank. Consistent with how nvdlfw_inspect logger resolves rank. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * detach tensors before saving; verify dump filename in test Detach both high_precision and quantized tensors before saving to avoid serializing the autograd graph. For QuantizedTensor this is a zero-copy view (make_like), so no extra GPU allocation. Add filename format assertion to test_dump_tensors_sanity to catch regressions in _sanitize_name or the naming convention. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add empty dump_dict log; assert QuantizedTensor type in test Log a message when no tensors are available to dump so the user has an explicit signal that no file was written. Assert that the quantized key round-trips as a QuantizedTensor to catch regressions in detach() or serialisation path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update transformer_engine/debug/features/dump_tensors.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: Paweł Gadziński <62263673+pggPL@users.noreply.github.com> * Address review: iter subdirs, remove dead rank field, add allclose test and MSE example - Organize dumps into per-iteration subdirectories (iter_000000/) to keep file count manageable per directory. - Remove unused self.rank attribute from TensorLogger. - Add torch.allclose assertion in test to verify serialization correctness. - Add docstring example showing how to load dumps and compute MSE. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> Made-with: Cursor * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: use detach().clone() to avoid shared storage in DumpTensors Using tensor.detach() creates a view sharing the same underlying storage. If any in-place operation modifies the tensor after the dump, the saved data would be silently corrupted. Use .clone() to ensure the dump captures an independent copy of the data. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * test: use torch.equal instead of torch.allclose for serialisation round-trip The saved tensor is an exact bit-for-bit copy (detach().clone()), so torch.equal is the correct check. torch.allclose with its default tolerances could mask a genuine dtype conversion or precision loss introduced by a future change to the serialisation path. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * fix: add tp_size to DumpTensors.inspect_tensor and fix KeyError in call_feature backward compat pop Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> Signed-off-by: root <pgadzinski@nvidia.com> Signed-off-by: Paweł Gadziński <62263673+pggPL@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * change distributed tests infra for fsdp2 Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * verbose flag for reporting Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * add back coments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * another minor fix Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * not needed for this PR Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * address review comments Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * unecessary comments
* add cudnn dln+add Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt> * try fixing cudnn build issue Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt> * guard against cudnn version Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * change itype to wtype for add in rmsnorm_bwd Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt> * remove dead code Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt> * remove dangling todo Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt> --------- Signed-off-by: CarlosGomes98 <carlosmiguel.gomes@live.com.pt> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* add blackwell support filter for 9.7<=cudnn<9.18.1 Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * simplify conditionals Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * fix conditionals again Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * fix conditionals again Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * update the error log Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * remove the python filter and correct the cpp filter Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> --------- Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…quired (NVIDIA#2798) * Disable fused attention for sm120 if determinism is required Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * nit: disable fused attn for sm120 determinism, if training Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> --------- Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
… always and `Max` when `return_max_logit=True` (NVIDIA#2677) * cudnn now returns Stats always and Max only with `return_max_logit=true` Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * fix a typo that caused a bug Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * update doc strings Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix more docs Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * fixes from the feedback Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * update cudnn-frontend to v1.19.1 Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * update the cudnn frontend Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * fix a wrong omission Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Move cuDNN FE to v1.21.0 Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
…2796) * Add warning if using BSHD and max_segments_per_seq > 1 Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update transformer_engine/jax/attention.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: jberchtold-nvidia <158520091+jberchtold-nvidia@users.noreply.github.com> * Update transformer_engine/jax/attention.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Signed-off-by: jberchtold-nvidia <158520091+jberchtold-nvidia@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Kshitij Lakhani <33047503+KshitijLakhani@users.noreply.github.com> Signed-off-by: jberchtold-nvidia <158520091+jberchtold-nvidia@users.noreply.github.com> * Remove warning test Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> --------- Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> Signed-off-by: jberchtold-nvidia <158520091+jberchtold-nvidia@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Kshitij Lakhani <33047503+KshitijLakhani@users.noreply.github.com>
) * Refactor to group_sizes per tensor Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Support first_dims and last_dims instead of a single group_sizes per tensor Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Refactor GMM FFIs to store static attrs as structs Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup C++ v2 FFI Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Fix int64 workspace usage Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Address greptile comments Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Refactor wgrad-specific checks to be generic for GMM in gemm.py Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Refactor XLA FFI struct setup Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Fix edge case in TE v1 GMM Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix issues on Hopper Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Refactor Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address comments Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Lint Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Fixes for Hopper Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * Address review comments Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Grouped quantization test fixes Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> --------- Signed-off-by: Jeremy Berchtold <jberchtold@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Fix convergence table render Signed-off-by: ksivamani <ksivamani@nvidia.com>
…8 CUDA graph capture (NVIDIA#3065) * [PyTorch] Propagate skip_fp8_weight_update in GroupedLinear during FP8 CUDA graph capture GroupedLinear.forward hardcoded None for skip_fp8_weight_update, so the FP8 graph-capture skip tensor was never forwarded during CUDA graph replay. Mirror Linear.forward: when fp8_graph_capturing() is true, read quantization_state.skip_fp8_weight_update_tensor, force is_first_microbatch to False, and thread the tensor into the forward call (the slot _GroupedLinear.forward already unpacks). Fixes NVIDIA#3051 Signed-off-by: LeSingh1 <sshaurya914@gmail.com> * [PyTorch] Add CUDA graph FP8 weight-caching test for GroupedLinear Exercises skip_fp8_weight_update propagation in GroupedLinear during FP8 CUDA graph capture. With fp8_weight_caching enabled, graphed and eager runs only match when is_first_microbatch is threaded into the weight- update skip tensor for every microbatch, which the prior None hardcode prevented. Signed-off-by: LeSingh1 <sshaurya914@gmail.com> --------- Signed-off-by: LeSingh1 <sshaurya914@gmail.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Add NS via cusolvermp to wheel build Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Build dep runtime Signed-off-by: ksivamani <ksivamani@nvidia.com> * Fix Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix Signed-off-by: ksivamani <ksivamani@nvidia.com> * Fix Signed-off-by: ksivamani <ksivamani@nvidia.com> * rm prev cublas req Signed-off-by: ksivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: ksivamani <ksivamani@nvidia.com>
* Add TE op CPU offload opt-out API Signed-off-by: hongbinl <hongbinl@nvidia.com> * Rename op API to activation offloading Signed-off-by: hongbinl <hongbinl@nvidia.com> * Move CPU offload gating to TE op call sites Signed-off-by: hongbinl <hongbinl@nvidia.com> * Preserve grouped linear offload start semantics Signed-off-by: hongbinl <hongbinl@nvidia.com> * Use setter for activation offload policy Signed-off-by: hongbinl <hongbinl@nvidia.com> * Limit activation offload policy helper to marking Signed-off-by: hongbinl <hongbinl@nvidia.com> * Move CPU offload imports to op module scope Signed-off-by: hongbinl <hongbinl@nvidia.com> * Patch activation offload test bound symbols Signed-off-by: hongbinl <hongbinl@nvidia.com> * Refactor base class offloading infrastructure Handle inclusion and exclusion in same function. Check whether CPU offloading is enabled internally. Tweak documentation and style. Signed-off-by: Tim Moon <tmoon@nvidia.com> * Propagate activation offload policy helper Use BasicOperation.mark_for_cpu_offload_if_needed at op call sites and keep explicit offload synchronization checks where needed. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> * Move test into TestFuser suite Signed-off-by: Tim Moon <tmoon@nvidia.com> * Debug failure with grouped linear Signed-off-by: Tim Moon <tmoon@nvidia.com> --------- Signed-off-by: hongbinl <hongbinl@nvidia.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: OpenAI Codex <codex@openai.com>
* guarding max_logits fused attention for cudnn < 9.21.0 Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai> * removing python side check Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai> --------- Signed-off-by: Francesco Bertolotti <francesco.bertolotti@igenius.ai>
…A#3120) Revert "[PyTorch] Add op-level activation offload opt-out API (NVIDIA#3108)" This reverts commit 9b06f26. Signed-off-by: Tim Moon <tmoon@nvidia.com>
…VIDIA#3117) * Refactor grouped MLP into joint fused op Consolidate the experimental grouped MLP forward and backward CuTe DSL fusions into a single joint fused operation. Move grouped-MLP-specific helper logic out of ops/_common.py and update tests to assert the joint forward/backward fusion object. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> * Review suggestions from @greptile-apps Also fix linter warnings. Signed-off-by: Tim Moon <tmoon@nvidia.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Codex <codex@openai.com>
…le names (NVIDIA#3086) Signed-off-by: Teddy Do <tdophung@nvidia.com> Signed-off-by: tdophung <tdophung@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Enable colwise only 2d nvfp4 Signed-off-by: Evgeny <etsykunov@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Enable colwise only for the main 2d kernel Signed-off-by: Evgeny <etsykunov@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve comments Signed-off-by: Evgeny <etsykunov@nvidia.com> * Fix test comment Signed-off-by: Evgeny <etsykunov@nvidia.com> * Enable rectangular shapes in tests Signed-off-by: Evgeny <etsykunov@nvidia.com> --------- Signed-off-by: Evgeny <etsykunov@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Copy grouped MLP tests from TE ops tests Signed-off-by: Tim Moon <tmoon@nvidia.com> * Reduce TE ops test cases Signed-off-by: Tim Moon <tmoon@nvidia.com> * Reduce overparametrized grouped MLP tests Signed-off-by: Tim Moon <tmoon@nvidia.com> * Clean up grouped MLP test leftovers Remove unused imports and helpers left after splitting grouped MLP tests out of the fusible ops suite. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Tim Moon <tmoon@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Review suggestion from @greptile-apps Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Expert Parallelism: common C API + NCCL EP v0.1 backend Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Expert Parallelism: persistent ncclEpHandle cache with allow_handle_mem_reloc gating Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build: NCCL_HOME discovery supports Debian/Ubuntu multiarch lib paths Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * bump NCCL Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Expert Parallelism: require token_dtype in NVTEEpGroupConfig and enforce at dispatch Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Expert Parallelism: document ep_comm lifetime, v0.1 single-GPU scope, static layer registration Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Expert Parallelism: drop version label from initialize scope note Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Expert Parallelism: pointer-keyed LRU handle cache; drop register_layer + NVTEEpHandle struct (NVTE_EP_HANDLE_CACHE_SIZE=-1 disables eviction) Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * bump nccl to latest v0.1 Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * tests/cpp_distributed: drop unused NCCL EP header include path Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: fold nvte_ep_* stubs into ep_api.cpp under #if NVTE_WITH_NCCL_EP Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: dlopen libnccl_ep.so so libtransformer_engine.so loads without it Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: add BUILD_RPATH=NCCL_EP_LIB_DIR for in-tree dev builds Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: polish ep.h docstrings; drop unused NVTE_CHECK_NCCL from logging.h Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: expose zero_copy in NVTEEpGroupConfig; map to NCCL_EP_ZERO_COPY_{ON,OFF} Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * tests/cpp_distributed: exercise zero_copy=ON in EPZeroCopyTest.IdentityAllSymm Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * tests/cpp_distributed: tighten EPZeroCopyTest comments Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/CMakeLists: correct NCCL resolution comment (not bundled with CUDA Toolkit) Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/CMakeLists: shorten NCCL/GIN headers comments Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * setup,common: bundle libnccl_ep.so.0 next to libtransformer_engine.so for wheel install Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * .gitmodules: drop nccl branch pin and align indentation with other submodules Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * setup: gate NCCL EP build on arch >= 90 or native; drop sm_90 fallback Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common,setup,tests: discover nccl.h via find_path/NCCL_INCLUDE_DIR; drop submodule header mirror Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: simplify make_nccl_ep_tensor to take NVTETensor and optional CommWindow Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: move te_dtype_to_nccl_dtype out of EPBackend into anon namespace Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: reword multicast check; drop NVLS framing Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common,tests: replace unicode em-dash and box-drawing chars with ASCII in EP files Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * bump nccl to latest v0.1 Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * nccl commit to 2.31.0a4-1 Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/CMakeLists: point NCCL_EP_INCLUDE_DIR at build/include staged headers Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/CMakeLists: clarify NCCL EP missing-header instructions Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: use int64_t instead of long for handle-cache size env (cpplint runtime/int) Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: fix dangling sizes pointer in make_nccl_ep_tensor (NVTEShape lifetime) Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * common/ep: static-link libnccl_ep.a into libtransformer_engine.so; drop dlopen loader and runtime bundling Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: lock handle_mem_size, guard tensor ndim, forward-declare ncclWindow in public header Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * qa,tests/cpp_distributed: emit per-rank JUnit XML and isolate sub-test failures in L1 CI Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * common/ep: harden NCCL EP init/probe paths and CI build short-circuit Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: drop Sphinx backticks and consolidate EP docstring Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * setup: keep _discover_nccl_home local; revert utils.py move Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * qa/L1_cpp_distributed: build and run each suite independently Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * qa/L1_cpp_distributed: pass --allow-run-as-root --oversubscribe to mpirun in run_test_ep.sh Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * common/ep: check initialized_ under mutex_ in per-step EPBackend ops Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * setup/ep: rename NVTE_BUILD_WITH_NCCL_EP to NVTE_WITH_NCCL_EP, harden NCCL EP build Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * tests/cpp_distributed: anchor NCCL discovery on libnccl, derive include dir from its prefix Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * tests/cpp_distributed: template EP unit tests on token dtype; switch routing to global counter Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * ep: drop misleading rebuild advice in NCCL EP errors and submodule header version log Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * common/ep: drop multicast support check; NCCL EP does not require multicast Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * setup/ep: error out when NVTE_CUDA_ARCHS yields no SM>=90 arch for NCCL EP Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * make core to be RTLD_LAZY Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> --------- Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
) Revert "Expert Parallelism: common C API + NCCL EP backend (NVIDIA#3034)" This reverts commit c3396ee. Signed-off-by: Tim Moon <tmoon@nvidia.com>
* update cublaslt grouped gemm filter Signed-off-by: Xin Yao <xiny@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update nvfp4 filter and tests Signed-off-by: Xin Yao <xiny@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * test correctness Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * better test Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Xin Yao <xiny@nvidia.com> Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Tim Moon <tmoon@nvidia.com> Co-authored-by: Phuong Nguyen <phuonguyen@nvidia.com>
…mel > INT_MAX (NVIDIA#3136) Widen multi_tensor_apply tensor sizes to int64 TensorListMetadataBase::sizes was declared int32 but populated from Tensor::numel(), so a tensor with numel > INT_MAX truncated to a negative size and the consumer kernels then computed out-of-bounds offsets, hitting an illegal memory access at the next sync. Store sizes as int64_t and read them into an int64_t (or the existing index_t on the already-templated Adam kernels) before the n -= chunk_idx * chunk_size subtraction. Widen the chunk_size kernel argument in the non-Adam consumers to int64_t as well so the chunk_idx * chunk_size element offset is computed in 64-bit. Fixes NVIDIA#2918 Signed-off-by: Javier de Jesus <javier.dejesusj9@gmail.com>
Signed-off-by: Kshitij Janardan Lakhani <klakhani@nvidia.com>
Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com>
Signed-off-by: Vladimir Cherepanov <vcherepanov@nvidia.com>
* Expert Parallelism: JAX primitives + VJPs --------- Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…VIDIA#3151) Revert "Add wheel support for Newton-Schulz method via cuSolverMp (NVIDIA#3004)" This reverts commit 20e185c. Signed-off-by: ksivamani <ksivamani@nvidia.com>
…m-mem zero-copy (NVIDIA#3035) * Expert Parallelism: PyTorch wrapper + autograd ops with symm-mem zero-copy Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> --------- Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…NS_PER_RANK (NVIDIA#3150) * nccl with relax num_dispatch_tokens%64!=0 Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Skip EP tests/examples on nodes without NVLink Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> --------- Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…` with `total_recv_tokens_per_rank` placeholder (NVIDIA#3154) * versioning EP C configs Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Rename EP prepare token_counts to recv_tokens_per_expert Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Add total_recv_tokens_per_rank placeholder to nvte_ep_prepare Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Adapt PyTorch EP binding to versioned nvte_ep C config API Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> * Rename EP group config max_num_sms to num_comm_sms Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com> --------- Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…L2 Jax dist (NVIDIA#3159) * Keep the routing map format alive Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> * Fix incorrectly launched multi process EP tests in L2 Jax instead of L2 jax dist Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com> --------- Signed-off-by: Kshitij Lakhani <klakhani@nvidia.com>
* disable 9.23.0/.1 for mxfp8 attention Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Charlene Yang <8636796+cyanguwa@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Resolved conflicts strategy: - Attention files: accept upstream base, will re-apply fork patches in Stage 5 - Module/ops/tensor files: accept upstream (fork changes are te_device_type only) - pytorch/__init__.py: merged (torch_nv + upstream imports) - permutation.py: accept upstream, will re-apply device checks in Stage 5 - Deleted upstream files: removed (fused_attn_f16_max512_seqlen.cu/h, experimental/quantization_nvfp4.py) - Custom recipes: kept fork's quantization_nvfp4.py Next: Stage 4 (build verification) and Stage 5 (re-apply fork patches)
- Remove convert_host_pointers_to_tensor (replaced by copy_data_ptrs_to_device) - Remove get_device_pointer_for_data_and_scales (functionality merged) Both methods removed from: - transformer_engine/plugin/core/backends/vendor/cuda/cuda.py - transformer_engine/plugin/core/backends/vendor/cuda/register_ops.py
Added new pybind APIs to CUDABackend and registry: Memory Management (2): - bulk_allocate: bulk allocate multiple tensors - copy_data_ptrs_to_device: copy host pointers to device Quantization (5): - create_empty_quantized_tensor: create empty quantized tensor - group_dequantize: dequantize grouped tensors - nvfp4_quantize_with_amax: NVFP4 quantization (Ada/Hopper+) - nvfp4_group_quantize_with_amax: NVFP4 group quantization - swizzle_scales_and_pack_ptrs_for_discrete_weights: pointer packing Multi-Tensor Operations (3): - multi_tensor_pad_last_dim: pad last dim for multiple tensors - multi_tensor_swizzle_scales_for_gemm_: in-place scale swizzle - multi_tensor_transpose_to_bhsd: batch transpose to BHSD layout GEMM & Solver (3): - get_grouped_gemm_setup_workspace_size: workspace size calculation - cusolvermp_ctx_create/destroy: cuSolverMP context management Utility (2): - splits_to_offsets_multi: convert splits to offsets - newton_schulz: Newton-Schulz iteration for matrix ops All methods include hardware capability checks where needed.
Modified APIs to match upstream changes: 1. fused_topk_with_score_function_fwd: + Add routing_map_format parameter (default BYTEMAP) 2. fused_topk_with_score_function_bwd: - Remove num_tokens and num_experts parameters + Add routing_map_format parameter (default BYTEMAP) 3. fused_score_for_moe_aux_loss_fwd: + Add routing_map_format parameter (default BYTEMAP) 4. fused_score_for_moe_aux_loss_bwd: - Remove num_tokens and num_experts parameters The routing_map_format parameter enables flexible routing map formats (BYTEMAP vs other layouts) for MOE operations.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please include a brief summary of the changes, relevant motivation and context.
Fixes # (issue)
Type of change
Changes
Please list the changes introduced in this PR:
Checklist: