fix(virt-handler): re-register device plugins after kubelet restart - #16
Open
rohan-togetherai wants to merge 2 commits into
Open
fix(virt-handler): re-register device plugins after kubelet restart#16rohan-togetherai wants to merge 2 commits into
rohan-togetherai wants to merge 2 commits into
Conversation
Kubelet restarts exposed two independent failures. PCI and generic health checks only recognized an exact removal of their own socket, so a replaced device-plugin directory or a recreated kubelet socket could leave a plugin attached to a dead kubelet forever. PCI also reused lifecycle channels after teardown, causing its next ListAndWatch stream to deregister immediately and close an already-closed channel. Install the socket-directory watch before Register and hand that watcher to the health loop. Restart on plugin-socket removal, kubelet-socket creation, or device-plugin directory removal or rename, and use fsnotify bit membership so combined operation masks retain their meaning. Derive the registration endpoint from the plugin socket directory, bound Register to the existing five-second connection window, and cancel it when the cycle stops. Recreate PCI lifecycle channels for every Start. Close each cycle's done channel before stopping its server, and construct PCI and generic servers with WaitForHandlers so all old handlers are joined before the next cycle can replace their channels. This closes the cross-cycle race without changing the existing supervision and backoff model. Co-authored-by: Aseef <contact@aseef.dev> Co-authored-by: RITANKAR SAHA <ritankar.saha786@gmail.com>
Drive PCI and generic plugins through the real controlled-device loop against a fake kubelet and verify that both register again and continue advertising devices after a restart. Cover kubelet-socket recreation, device-plugin directory replacement, combined fsnotify masks, and rejection of unrelated events. Exercise the watch-before-register ordering with an in-flight kubelet replacement, prove the production gRPC server waits for handlers while teardown wakes active ListAndWatch streams, and verify hung Register calls are cancelled on shutdown or fail at the deadline.
Broly Security ScanNote Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR. Note ✅ Clean scan Note Re-scan this PR anytime with
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(virt-handler): re-register device plugins after kubelet restart
Problem
After kubelet restarts, PCI and generic device plugins can miss the restart signal and leave extended resources at allocatable 0. PCI restart cycles also reuse closed lifecycle channels, causing immediate deregistration and a duplicate-close panic.
Fix
Watch the device-plugin directory before registration and restart on plugin-socket removal, kubelet-socket creation, or directory removal/rename, including combined fsnotify masks. Reset PCI channels per cycle, join old gRPC handlers before the next cycle, and bound registration by shutdown and a five-second deadline.
Testing
Package tests run PCI and generic plugins through two real supervision cycles against a fake kubelet, cover restart filesystem events and watch ordering, and exercise handler joining and registration cancellation. The full package suite and five focused race-enabled runs pass.