Skip to content

Add a backwards linear function to be used with the fused mla q up-proj - #3330

Open
chaseblock wants to merge 3 commits into
NVIDIA:mainfrom
chaseblock:lin_bwd
Open

Add a backwards linear function to be used with the fused mla q up-proj#3330
chaseblock wants to merge 3 commits into
NVIDIA:mainfrom
chaseblock:lin_bwd

Conversation

@chaseblock

Copy link
Copy Markdown
Contributor

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Signed-off-by: Chase Block <cblock@nvidia.com>
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Aug 7, 2026
chaseblock added a commit to chaseblock/Megatron-LM that referenced this pull request Aug 7, 2026
Requires TE PR: NVIDIA/TransformerEngine#3330

Signed-off-by: Chase Block <cblock@nvidia.com>
@greptile-apps

greptile-apps Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds a reusable linear-backward entry point for fused operations and exposes it through the fused MLA Q up-projection helper.

  • Constructs LinearBwdArgs for quantized and non-quantized projection backward passes.
  • Supports tensor/sequence parallelism, deferred or fused weight-gradient accumulation, and optional gradient computation.
  • Delegates the fused MLA Q up-projection backward path to the new helper.

Confidence Score: 4/5

The PR should not merge until the outstanding bias-gradient contract is fixed for callers that enable bias.

The new wrapper allows use_bias=True, and _linear_backward computes the corresponding gradient, but the wrapper discards that result and provides no other path for a fused caller to update the bias.

Files Needing Attention: transformer_engine/pytorch/module/linear.py

Important Files Changed

Filename Overview
transformer_engine/pytorch/module/linear.py Adds the shared backward helper, but its optional bias-gradient result remains discarded.
transformer_engine/pytorch/attention/fused_mla_q_uproj.py Adds a thin classmethod that forwards fused MLA projection-backward inputs and keyword options to the shared helper.

Reviews (2): Last reviewed commit: "Remove redundant import in backward_line..." | Re-trigger Greptile

Comment on lines +872 to +873
wgrad, dgrad, _ = _linear_backward(bwd_args)
return dgrad, wgrad

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Bias gradient is discarded

When use_bias=True, _linear_backward computes a bias gradient, but this wrapper discards the third return value and exposes only dgrad and wgrad, preventing the caller from updating the bias parameter.

Knowledge Base Used: PyTorch Fused Modules (transformer_engine/pytorch/module)

Signed-off-by: Chase Block <cblock@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant