Skip to content

[https://nvbugs/6793964][test] Unwaive test_row_linear[2-unbalanced] - #19569

Merged
lori-ren merged 1 commit into
NVIDIA:mainfrom
lori-ren:fix/ipc-mem-handle-reserved
Sep 23, 2026
Merged

lori-ren merged 1 commit into
NVIDIA:mainfrom
lori-ren:fix/ipc-mem-handle-reserved

Conversation

@lori-ren

@lori-ren lori-ren commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Dev Engineer Review

The change removes the DGX H100 waiver and NVBug reference for test_row_linear[2-unbalanced]. No source-code changes are present. The compatibility fix must be available on the target branch before removing the waiver.

QA Engineer Review

tests/integration/test_lists/waives.txt was modified. The test_row_linear[2-unbalanced] waiver was removed. The author reports that both test_row_linear parametrizations pass on 2x H100. Coverage verdict: sufficient.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Removes the DGX H100 skip entry for test_row_linear[2-unbalanced] and its NVBug reference. This test-list-only change does not require a test-db/ or qa/ entry review.

Description

test_row_linear[2-unbalanced] was waived on DGX_H100 after it hit the pytest timeout in
post-merge and was reported as Test terminated unexpectedly. The cause was a
cuda-bindings drift: cudaIpcMemHandle_t stopped exposing .reserved, so IpcMemory
raised on one rank while the other was already waiting in the allgather, deadlocking the
MPI pool until the stage was killed.

#19184 already fixed that on main with a .reserved/getPtr() compat shim in
_ipc_utils.py. This PR removes the waive that outlived it.

Test Coverage

tests/unittest/_torch/multi_gpu/test_linear.py::test_row_linear — both parametrizations
pass on 2x H100. The bug names only [2-unbalanced], but that is an artifact of which case
was in flight when the pool deadlocked, so [2-balanced] was verified as well.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
  • Test cases are provided for new code paths (see test instructions)
  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
  • Any new dependencies have been scanned for license and vulnerabilities

The cudaIpcMemHandle_t compat shim added in NVIDIA#19184 fixed the root cause.

Signed-off-by: Lori Ren <lorir@nvidia.com>
@lori-ren

Copy link
Copy Markdown
Contributor Author

/bot run

@coderabbitai

coderabbitai Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4a1fa787-6e06-442b-a686-6d9115daf5ca

📥 Commits

Reviewing files that changed from the base of the PR and between 134fa24 and 831db7d.

📒 Files selected for processing (1)
  • tests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (1)
  • tests/integration/test_lists/waives.txt

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The pull request removes one DGX H100 skip waiver and its associated NVBug reference for test_row_linear[2-unbalanced].

Changes

Test waiver update

Layer / File(s) Summary
Remove DGX H100 test waiver
tests/integration/test_lists/waives.txt
Removes the waiver for full:DGX_H100/unittest/_torch/multi_gpu/test_linear.py::test_row_linear[2-unbalanced] and NVBug 6793964.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~2 minutes

Change: Other

Suggested reviewers: bowenfu

Merge Risk: ⚪ Minimal · up to 831db

The re-enabled test no longer retains the identified H100 IPC compatibility deadlock trigger. No actionable merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title identifies the NVBugs ticket, test type, and specific waiver removal. It clearly summarizes the main change.
Description check ✅ Passed The description explains the failure cause, the existing fix, the waiver removal, and the test coverage. It includes the required Description and Test Coverage sections. The checklist is partially rep…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #75144 [ run ] triggered by Bot. Commit: 831db7d Link to invocation

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #75144 [ run ] completed with state FAILURE. Commit: 831db7d
/LLM/main/L0_MergeRequest_PR pipeline #61899 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@lori-ren

Copy link
Copy Markdown
Contributor Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #75190 [ run ] triggered by Bot. Commit: 831db7d Link to invocation

@lori-ren
lori-ren enabled auto-merge (squash) September 23, 2026 05:10
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #75190 [ run ] completed with state SUCCESS. Commit: 831db7d
/LLM/main/L0_MergeRequest_PR pipeline #61940 completed with status: 'SUCCESS'

CI Report

Link to invocation

@lori-ren
lori-ren merged commit fa2279b into NVIDIA:main Sep 23, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants