🐛 Drain pre-existing PromoteDisks tasks before RegisterVM specs - #1895
Open
aakashchan wants to merge 1 commit into
Open
🐛 Drain pre-existing PromoteDisks tasks before RegisterVM specs#1895aakashchan wants to merge 1 commit into
aakashchan wants to merge 1 commit into
Conversation
Root-caused several RegisterVM/registervm.go flakes (vmop-4097) to
boot-disk promotion (PromoteDisks_Task) being a real live "disks-only"
Storage vMotion whose duration is governed by datastore/NFC
contention with whatever else is running concurrently. Suite setup
deploys shared VMs (the common Photon VM, and a Windows VM when
Sysprep is enabled) fire-and-forget, without waiting for their own
background disk promotion to finish, so that backlog can still be
draining when RegisterVM specs start their own promotion and blow
past the fixed wait-virtual-machine-condition-update budget.
Changes:
- registervm.go: drain any already-running PromoteDisks tasks once
before the first RegisterVM spec runs (sync.Once-gated, since
Ginkgo has no per-Describe BeforeSuite). The drained/not-drained
result is cached and re-checked on every spec's BeforeEach, so if
the drain never completes within its budget, every RegisterVM spec
is skipped rather than racing its own promotion against known
contention.
- Also wrap the disk-swap sequence's AddDeviceWithProfile call in a
bounded retry: it has been observed failing with a raw storage I/O
error ("msg.disk.policyChangeFailure: 5 (Input/output error)")
under the same kind of datastore contention.
- vcenter/tasks.go: add WaitForNoActiveTask, matching vCenter tasks by
DescriptionId (the same stable identifier vm-operator's own
reconciler checks) rather than a task's Name or localized
Description message, neither of which reliably identifies a task
type. Implemented via each VM's recentTask property resolved
through the property collector -- the same mechanism vm-operator's
reconcile loop uses (pkg/providers/vsphere/vmprovider_vm.go
getRecentTaskInfo) -- in two batched property-collector calls
regardless of inventory size, rather than vCenter's
TaskHistoryCollector API, which didn't reliably surface tasks
already running before the collector was created. Factored the
shared collector-page-fetch plumbing (used by both this and the
existing time-windowed RecentTasks/LookupTask) into
fetchTaskInfoPage.
- wcp.yaml / kind.yaml: add the
default/wait-no-active-promote-disks-tasks interval (10m/10s) this
drain uses.
Fixes: vmop-4097
Contributor
Minimum allowed line rate is |
aakashchan
marked this pull request as ready for review
September 3, 2026 15:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root-caused several RegisterVM/registervm.go flakes (vmop-4097) to boot-disk promotion (PromoteDisks_Task) being a real live "disks-only" Storage vMotion whose duration is governed by datastore/NFC contention with whatever else is running concurrently. Suite setup deploys shared VMs (the common Photon VM, and a Windows VM when Sysprep is enabled) fire-and-forget, without waiting for their own background disk promotion to finish, so that backlog can still be draining when RegisterVM specs start their own promotion and blow past the fixed wait-virtual-machine-condition-update budget.
Changes:
Fixes: vmop-4097
What does this PR do, and why is it needed?
Which issue(s) is/are addressed by this PR? (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #
Are there any special notes for your reviewer:
Please add a release note if necessary: