Skip to content

[None][fix] Preserve KV transfer outcomes independently of polling - #19377

Draft
chienchunhung wants to merge 2 commits into
NVIDIA:mainfrom
chienchunhung:dev/dbr-pr3-stable-logical-outcomes
Draft

chienchunhung wants to merge 2 commits into
NVIDIA:mainfrom
chienchunhung:dev/dbr-pr3-stable-logical-outcomes

Conversation

@chienchunhung

@chienchunhung chienchunhung commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Commit each native KV-transfer Attempt's logical outcome when the task/session changes state, rather than when it is first polled. A later cancellation, failure, or physical completion cannot rewrite an earlier Delivered/Failed/Cancelled result. Physical retirement and deadlines are unchanged.

Major changes

  • _LogicalOutcomes: serialize completion against session failure/cancellation; preserve original cause and local/peer cancellation origin.
  • TaskHandle.poll(): observe the committed outcome while keeping report progress separate.
  • Regression tests: cover early/late/fresh observers, queued cancellation, sibling outcomes, original errors, and scatter/cancel ordering.

Verification

  • Test-first commit 7d0430794e, with current-main production code: 14 intended failures / 30 passes. The following fix commit passes the expanded suite.
  • TaskHandle coverage: 27 → 55 cases (+28); TaskHandle/fetch/publish suites: 48 → 76 cases, all passing. These are test counts, not line-coverage percentages.
  • Expanded local checks: 192 passed, 1 skipped, 4 excluded. These ran on macOS with a source-only dependency-stub harness, not native TRT-LLM/NIXL/CUDA. The four executor-fixture exclusions also fail in that harness on baseline.
  • Native CPU CI at this head: 55/55 TaskHandle cases and 76/76 TaskHandle/fetch/publish cases passed on each of x86 and ARM, including all 28 added cases.
  • Full CI #61068 (helper #74244): FAILURE — 61,513 passed / 24 failed / 12,113 skipped. Failures are executor/KV-manager fixture AttributeErrors outside this patch: 20 leaf failures currently waived under NVBUG 6800223 / [None][fix] Initialize KV cache fixtures and unwaive nine cases on all platforms #19382, plus four failing suite wrappers. The focused suites above are green; the full pipeline is not.
  • Ruff lint/format, diff checks, GitHub pre-commit, and separate iterative self-review are clean.

Based on main@e72f70e99d. Size: 208 production + 318 test lines changed.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #74244 [ run ] triggered by Bot. Commit: 8b20e04 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #74244 [ run ] completed with state FAILURE. Commit: 8b20e04
/LLM/main/L0_MergeRequest_PR pipeline #61068 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants