revert: trtllm routing.priority engine hand-off (#28) - #29
Merged
sopwg612 merged 1 commit intoAug 4, 2026
Merged
Conversation
…ority" This reverts commit 82514b5, restoring handler_base.py and its tests to af77801 exactly. #28 wired routing.priority through to generate_async so per-request priority reached the TRT-LLM waiting queue. We are changing direction: priority will be implemented in the dynamo frontend's router queue only, and no priority value will be forwarded to the engines. Reverting keeps the worker on upstream behaviour (top-level `priority` remains the health-check-only path, pinning 1.0) rather than leaving a plumbed-but-unused engine path around. Nothing was built or deployed from the merge, so no image is affected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqzQrZR2qrvxM6DsxqGciu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #28 (merge commit
82514b574), restoringhandler_base.pyand its tests toaf778016dexactly — verified withgit diff af778016d HEAD -- <file>on both files.Why
#28 fixed the worker to read
routing.priority(it reads a top-level key the Rust frontend never sets) so per-request priority reached TRT-LLM's waiting queue.We're changing direction: priority will live in the dynamo frontend's router queue only, and no priority value will be forwarded to the engines. Reverting leaves the worker on upstream behaviour rather than keeping a plumbed-but-unused engine path around to confuse the next reader.
Blast radius
None. Nothing was built or deployed from the merge — prod decode/prefill still run
dynamo-trtllm-runtime:07-09-62fb5b6f-kvcap-v2-pr15737-{lptol-,}unpinfix, whose base sha predates it.What the revert restores
Note for the frontend-only design
nvext.agent_hints.prioritystill feedspriority_jumpin the router (max(0, priority)seconds of arrival credit under fcfs) — that is frontend-side and unaffected by this revert.strict_prioritylikewise. Only the engine hand-off is removed.Worth knowing before building on it: on prod
gpt-oss-120bthe router queue is effectively always empty —dynamo_frontend_router_queue_pending_requestsmax over 6h is 0, and 24h backpressure is 2056 events all in theno-queueclass, zero instandard. At--router-queue-threshold=3the router dispatches straight through, so a router-side priority queue has nothing to reorder until that threshold drops.🤖 Generated with Claude Code
https://claude.ai/code/session_01QqzQrZR2qrvxM6DsxqGciu