[None][fix] Retire KV transfer ownership after late backend completion

[chienchunhung]

Summary

Allow an ownership-enabled Python KV transfer that became IN_DOUBT to retire safely when its exact retained backend operation later proves DONE. Logical failure/cancellation stays unchanged, and admission quarantine remains closed.

Depends on #19377; merge it first. Both target main and were rebased onto 1a36f4fed; review this PR's isolated change.

Major changes

• Same-handle settlement: recheck retained request/status identity, require positive DONE, and retire the physical claim idempotently.
• Ordered reporting and cleanup: retry IN_DOUBT → FAILED_QUIESCED evidence in worker order; prevent later failure reports overtaking it; release retained storage once.
• Receiver settlement: retain sibling/publication/local-completion gates and permanently reject invalid evidence. FAILED_QUIESCED is physical proof, not delivery success.

Existing opt-in bridge only: matching binaries/configuration, unique request IDs, one immutable selected ADP writer group, and no retry/reroute remain required. No new activation profile, deadline, abort, or restart policy; unproven operations remain retained.

Verification

• Unit-test delta: 36 new cases / 15 test functions cover exact-handle identity, duplicate/concurrent settlement, report ordering/retries, sibling retention and shutdown; 12 cases check unchanged failure/cancellation outcomes.
• Native Linux CI, pre-rebase head 5760bc13b: all 36/36 new cases passed on each x86 and ARM stage (results #61069).
• Current head f372733fe8: full CI #61879 is running.
• Changed-line unit-test coverage: not yet measured in native CI; measure this PR's isolated delta against [None][fix] Preserve KV transfer outcomes independently of polling #19377, not the combined stack.

These CPU unit tests do not establish real NIXL/CUDA or multi-rank qualification.

Rebase is patch-equivalent. Isolated size, excluding #19377: 216 production + 527 test lines changed.

Dev Engineer Review

• Separates committed logical outcomes from physical settlement state.
• Retains IN_DOUBT ownership until the exact backend operation reports DONE.
• Adds FAILED_QUIESCED, identity validation, idempotent writer settlement, ordered retries, and receiver-side settlement checks.
• Keeps failure and cancellation outcomes stable after late reports, sibling events, or session termination.
• Updates TaskHandle construction and removes fallback outcome inference.
• Requires compatibility checks for the new FAILED_QUIESCED wire status.
• Full-CI completion and review-finding counts are unavailable.

QA Engineer Review

• Modified test_peer_fetch.py, test_task_handle.py, and test_transceiver_bounded_polling.py.
• Added test_transfer_late_settlement.py.
• Modified test_transfer_ownership_regressions.py.
• Covers 36 new or updated cases across logical outcome stability, late settlement, ownership quarantine, identity validation, retry ordering, cancellation, failure preservation, shutdown, queue routing, auxiliary transfers, and exactly-once resource release.
• No tests/integration/test_lists/test-db/ or tests/integration/test_lists/qa/ files changed, so no list-entry update is required.
• Coverage verdict: needs follow-up. The tests are CPU-only and do not validate real NIXL, CUDA, or multi-rank behavior. Current full-CI status is unavailable.

Per-File QA Perspective

• tensorrt_llm/_torch/disaggregation/base/backend.py: Verify stable logical outcomes while report state changes.
• tensorrt_llm/_torch/disaggregation/native/handle.py: Verify stable terminal results and the updated typed constructor.
• tensorrt_llm/_torch/disaggregation/native/transfer.py: Verify FAILED_QUIESCED encoding, ownership validation, ordered settlement, shutdown blocking, and unchanged failure or cancellation behavior.
• tests/unittest/disaggregated/test_peer_fetch.py: Covers native receive tasks, admission failures, cancellation metadata, and failure causes. No integration test-list entry applies.
• tests/unittest/disaggregated/test_task_handle.py: Covers ownership, terminal outcomes, late reports, cancellation, sibling events, scatter completion, and resource draining. No integration test-list entry applies.
• tests/unittest/disaggregated/test_transceiver_bounded_polling.py: Covers logical-outcome setup and bounded polling behavior. No integration test-list entry applies.
• tests/unittest/disaggregated/test_transfer_late_settlement.py: Covers late settlement, sender and receiver paths, KV and auxiliary transfers, retries, quarantine, and shutdown. No integration test-list entry applies.
• tests/unittest/disaggregated/test_transfer_ownership_regressions.py: Covers ownership, cancellation, shutdown, publication, and auxiliary settlement regressions. No integration test-list entry applies.

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #74245 [ run ] triggered by Bot. Commit: 5760bc1 Link to invocation

[tensorrt-cicd]

PR_Github #74245 [ run ] completed with state FAILURE. Commit: 5760bc1
/LLM/main/L0_MergeRequest_PR pipeline #61069 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #75126 [ run ] triggered by Bot. Commit: f372733 Link to invocation

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 85903da2-eda8-4756-95e9-a7668cdca635

📥 Commits

Reviewing files that changed from the base of the PR and between 134fa24 and f372733.

📒 Files selected for processing (8)

• tensorrt_llm/_torch/disaggregation/base/backend.py
• tensorrt_llm/_torch/disaggregation/native/handle.py
• tensorrt_llm/_torch/disaggregation/native/transfer.py
• tests/unittest/disaggregated/test_peer_fetch.py
• tests/unittest/disaggregated/test_task_handle.py
• tests/unittest/disaggregated/test_transceiver_bounded_polling.py
• tests/unittest/disaggregated/test_transfer_late_settlement.py
• tests/unittest/disaggregated/test_transfer_ownership_regressions.py

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

Walkthrough

The transfer system now commits task outcomes separately from report progress and physical quiescence. It retains ambiguous transfers until backend settlement, adds FAILED_QUIESCED, preserves report ordering, and expands tests for cancellation, failure, ownership, and late settlement.

Changes

Transfer outcomes and settlement

Layer / File(s)	Summary
Logical outcome arbitration and polling tensorrt_llm/_torch/disaggregation/base/backend.py, tensorrt_llm/_torch/disaggregation/native/handle.py, tensorrt_llm/_torch/disaggregation/native/transfer.py, tests/unittest/disaggregated/test_peer_fetch.py, tests/unittest/disaggregated/test_task_handle.py, tests/unittest/disaggregated/test_transceiver_bounded_polling.py	Tasks and sessions commit logical transfer, failure, or cancellation outcomes. TaskHandle.poll() reads the committed outcome and preserves its cause or peer attribution while report obligations continue to change.
Physical ownership and settlement state tensorrt_llm/_torch/disaggregation/native/transfer.py, tests/unittest/disaggregated/test_transfer_late_settlement.py, tests/unittest/disaggregated/test_transfer_ownership_regressions.py	In-doubt operations remain non-drained until fresh backend completion evidence arrives. FAILED_QUIESCED settlement validates prior evidence, retires writer ownership, and preserves the original logical outcome.
Deferred reporting and worker ordering tensorrt_llm/_torch/disaggregation/native/transfer.py, tests/unittest/disaggregated/test_transfer_late_settlement.py	Workers retain ambiguous KV and auxiliary results, retry reports in order, poll retained backend status, and reject shutdown while settlement reports remain pending.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant SendTaskBase
  participant Worker
  participant Backend
  participant Receiver
  SendTaskBase->>Worker: report IN_DOUBT
  Worker->>Backend: poll retained operation
  Backend-->>Worker: fresh DONE status
  Worker->>Worker: emit FAILED_QUIESCED
  Worker->>Receiver: send settlement result
  Receiver->>Receiver: validate and retire writer ownership

Loading

Suggested reviewers: bowenfu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.34% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 118 functions across 8 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies a fix for retiring KV transfer ownership after late backend completion. It follows the required [None][fix] format and matches the main change.
Description check	✅ Passed	The description clearly explains the problem, solution, implementation details, dependencies, test coverage, and CI status. It does not use the template headings exactly and omits the explicit PR Chec…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}