Skip to content

✨ Add power-state-synced condition for host maintenance mode - #1861

Open
hpannem wants to merge 2 commits into
vmware-tanzu:mainfrom
hpannem:maint-mode-policy-condition
Open

✨ Add power-state-synced condition for host maintenance mode#1861
hpannem wants to merge 2 commits into
vmware-tanzu:mainfrom
hpannem:maint-mode-policy-condition

Conversation

@hpannem

@hpannem hpannem commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

[on top of #1860 ]
What does this PR do, and why is it needed?

Adds a new Reason, InfraInMaintenance, to the existing VirtualMachinePowerStateSynced condition, so tenants/VKS can tell a maintenance-mode-driven power-off apart from a generic NotSynced. When a power-state-change task fails with a NoCompatibleHost fault carrying vCenter's autoevac key (com.vmware.cp.autoevac.HostInMaintenanceMode / ...RestartOnCurrentHostRequired), the condition is set accordingly instead of the generic reason. No new vCenter calls — the signal comes from the failed task's own fault. Gated behind the VMEviction feature flag; behavior is unchanged when it's off.

  • pkg/util/vsphere/task: IsInfraMaintenanceFault detects the fault.
  • pkg/util/vsphere/vm: WrapHardPowerOpFailure wraps it into sentinel ErrInfraMaintenanceFault.
  • pkg/providers/vsphere/vmprovider_vm.go: SetPowerStateSyncedCondition sets the condition from reconcilePowerState's outcome.
  • pkg/providers/vsphere/vmlifecycle/update_status.go: reconcileStatusPowerState now only sets Status.PowerState (condition logic moved above).
  • New VirtualMachineInfraInMaintenanceReason constant in api/v1alpha6.
  • E2E (vmevictionpolicy.go): 3-VM spec — one unpinned VM gets relocated by DRS; two pinned to the same host (via a real vm_host_affinity ComputePolicy, mimicking a VM DRS can't evacuate) get powered off by DRS's own autoevac and surface InfraInMaintenance on VM Operator's retry, then recover once maintenance ends.

Which issue(s) is/are addressed by this PR?

Fixes #

Are there any special notes for your reviewer:

  • E2E labeled experimental, excluded from CI until validated on real hardware.
  • The pinned VMs' ComputePolicy/InfraPolicy have no delete API, so they live until the suite's shared namespace is torn down at the end of the run rather than per-test — flagging in case a temp-namespace refactor is preferred.

Please add a release note if necessary:

VirtualMachine's `VirtualMachinePowerStateSynced` condition now reports reason `InfraInMaintenance` when a VM can't be powered back on because its host is in infrastructure maintenance and can't be evacuated. Gated behind the `supports_infrapolicy_vm_evacuation` capability.

@hpannem hpannem self-assigned this Aug 28, 2026
@hpannem
hpannem requested review from a team and faisalabujabal as code owners August 28, 2026 07:05
@hpannem hpannem added the pro-ai-review-requested Mark the PR as ready for the AI Review bot using a Pro model label Aug 28, 2026
@github-actions github-actions Bot added the size/XXL Denotes a PR that changes 1000+ lines. label Aug 28, 2026
@svc-vmop-bot

Copy link
Copy Markdown

📝 CL Summary:

This changelist introduces two new compute-policy CRDs, AutomaticHostEvacuationPolicy and BestEffortRestartPolicy, to handle workload evacuation and restarts during host maintenance. It refactors the policy evaluation controller into a table-driven registry to seamlessly support these new policy kinds. Additionally, it adds an InfraInMaintenance reason to the VirtualMachinePowerStateSynced condition, allowing consumers like VKS to distinguish maintenance-driven power-offs from generic power-state drift. All new behaviors are safely gated behind the VMEvacuation feature flag.

@svc-vmop-bot svc-vmop-bot added cl-summary-reviewed AI Reviewed PR Summary commit-msg-reviewed AI Reviewed Commit Message labels Aug 28, 2026
Comment thread controllers/vspherepolicy/policyevaluation/policyevaluation_controller.go Outdated
Comment thread external/vsphere-policy/api/v1alpha1/automatichostevacuationpolicy_types.go Outdated
Comment thread external/vsphere-policy/api/v1alpha1/computepolicy_types.go Outdated
Comment thread pkg/providers/vsphere/vcenter/host.go Outdated
Comment thread test/e2e/vmservice/vmservice/hostmaintenancepolicy/hostmaintenancepolicy.go Outdated
Comment thread test/e2e/vmservice/vmservice/hostmaintenancepolicy/hostmaintenancepolicy.go Outdated
Comment thread test/e2e/vmservice/vmservice/hostmaintenancepolicy/hostmaintenancepolicy.go Outdated
Comment thread test/e2e/vmservice/vmservice/hostmaintenancepolicy/hostmaintenancepolicy.go Outdated
@svc-vmop-bot svc-vmop-bot added the ai-reviewed This PR has been reviewed by the AI review bot label Aug 28, 2026
@faisalabujabal faisalabujabal removed the pro-ai-review-requested Mark the PR as ready for the AI Review bot using a Pro model label Aug 28, 2026
@hpannem
hpannem force-pushed the maint-mode-policy-condition branch 5 times, most recently from 0912634 to 6a98778 Compare August 29, 2026 07:08
test fix
issue with condition message update
@hpannem
hpannem force-pushed the maint-mode-policy-condition branch from 08c6cc4 to 282da2f Compare September 7, 2026 22:40
e2e tests
@hpannem
hpannem force-pushed the maint-mode-policy-condition branch from 282da2f to d465d33 Compare September 7, 2026 23:11
@hpannem hpannem added ai-review-requested Mark the PR as ready for the AI Review bot using a standard model pro-ai-review-requested Mark the PR as ready for the AI Review bot using a Pro model and removed ai-reviewed This PR has been reviewed by the AI review bot cl-summary-reviewed AI Reviewed PR Summary commit-msg-reviewed AI Reviewed Commit Message labels Sep 7, 2026
@hpannem
hpannem requested review from aruneshpa and bryanv September 7, 2026 23:16
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Code Coverage

Package Line Rate Health
github.com/vmware-tanzu/vm-operator/controllers/configtarget 74%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/clustercontentlibraryitem 67%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/contentlibraryitem 67%
github.com/vmware-tanzu/vm-operator/controllers/contentlibrary/utils 85%
github.com/vmware-tanzu/vm-operator/controllers/infra/capability/configmap 92%
github.com/vmware-tanzu/vm-operator/controllers/infra/capability/crd 100%
github.com/vmware-tanzu/vm-operator/controllers/infra/configmap 75%
github.com/vmware-tanzu/vm-operator/controllers/infra/node 77%
github.com/vmware-tanzu/vm-operator/controllers/infra/secret 76%
github.com/vmware-tanzu/vm-operator/controllers/infra/validatingwebhookconfiguration 87%
github.com/vmware-tanzu/vm-operator/controllers/infra/workloadnetworkconfig 86%
github.com/vmware-tanzu/vm-operator/controllers/infra/zone 76%
github.com/vmware-tanzu/vm-operator/controllers/storage/storageclass 93%
github.com/vmware-tanzu/vm-operator/controllers/storage/storagepolicy 96%
github.com/vmware-tanzu/vm-operator/controllers/storage/storagepolicyquota 91%
github.com/vmware-tanzu/vm-operator/controllers/storage/volumeattributesclass 93%
github.com/vmware-tanzu/vm-operator/controllers/util/encoding 73%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/storagepolicyusage 96%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/virtualmachine 65%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/volume 87%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine/volumebatch 89%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineclass 73%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineconfigoptions 87%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinegroup 90%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinegrouppublishrequest 88%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineimagecache 89%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinepublishrequest 84%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinereplicaset 67%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice 90%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachineservice/providers 93%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinesetresourcepolicy 81%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinesnapshot 91%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest 72%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest/v1alpha1 72%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest/v1alpha1/conditions 88%
github.com/vmware-tanzu/vm-operator/controllers/virtualmachinewebconsolerequest/v1alpha1/patch 78%
github.com/vmware-tanzu/vm-operator/controllers/vspherepolicy/policyevaluation 87%
github.com/vmware-tanzu/vm-operator/pkg/bitmask 100%
github.com/vmware-tanzu/vm-operator/pkg/builder 89%
github.com/vmware-tanzu/vm-operator/pkg/conditions 90%
github.com/vmware-tanzu/vm-operator/pkg/config 100%
github.com/vmware-tanzu/vm-operator/pkg/config/capabilities 97%
github.com/vmware-tanzu/vm-operator/pkg/config/env 100%
github.com/vmware-tanzu/vm-operator/pkg/context 37%
github.com/vmware-tanzu/vm-operator/pkg/context/generic 100%
github.com/vmware-tanzu/vm-operator/pkg/context/operation 100%
github.com/vmware-tanzu/vm-operator/pkg/crd 77%
github.com/vmware-tanzu/vm-operator/pkg/errors 74%
github.com/vmware-tanzu/vm-operator/pkg/exit 100%
github.com/vmware-tanzu/vm-operator/pkg/log 100%
github.com/vmware-tanzu/vm-operator/pkg/mem 100%
github.com/vmware-tanzu/vm-operator/pkg/patch 78%
github.com/vmware-tanzu/vm-operator/pkg/prober 89%
github.com/vmware-tanzu/vm-operator/pkg/prober/probe 90%
github.com/vmware-tanzu/vm-operator/pkg/prober/worker 77%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere 74%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/clustermodules 73%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/config 88%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/contentlibrary 76%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/credentials 100%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/network 86%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/placement 70%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/session 60%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/storage 44%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/upgrade/virtualmachine 96%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/upgrade/virtualmachine/backfill 96%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/vcenter 86%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/virtualmachine 85%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/virtualmachine/extraconfig 91%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/virtualmachine/networkextraconfig 75%
github.com/vmware-tanzu/vm-operator/pkg/providers/vsphere/vmlifecycle 77%
github.com/vmware-tanzu/vm-operator/pkg/record 84%
github.com/vmware-tanzu/vm-operator/pkg/topology 91%
github.com/vmware-tanzu/vm-operator/pkg/util 79%
github.com/vmware-tanzu/vm-operator/pkg/util/cloudinit 89%
github.com/vmware-tanzu/vm-operator/pkg/util/cloudinit/validate 91%
github.com/vmware-tanzu/vm-operator/pkg/util/image 100%
github.com/vmware-tanzu/vm-operator/pkg/util/kube 91%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/cource 100%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/internal 100%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/networksettings 100%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/proxyaddr 73%
github.com/vmware-tanzu/vm-operator/pkg/util/kube/spq 99%
github.com/vmware-tanzu/vm-operator/pkg/util/linuxprep 97%
github.com/vmware-tanzu/vm-operator/pkg/util/netplan 100%
github.com/vmware-tanzu/vm-operator/pkg/util/nil 100%
github.com/vmware-tanzu/vm-operator/pkg/util/ovfcache 75%
github.com/vmware-tanzu/vm-operator/pkg/util/ovfcache/internal 100%
github.com/vmware-tanzu/vm-operator/pkg/util/paused 100%
github.com/vmware-tanzu/vm-operator/pkg/util/ptr 100%
github.com/vmware-tanzu/vm-operator/pkg/util/resize 98%
github.com/vmware-tanzu/vm-operator/pkg/util/sysprep 98%
github.com/vmware-tanzu/vm-operator/pkg/util/vmopv1 88%
github.com/vmware-tanzu/vm-operator/pkg/util/volumes 100%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/client 66%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/datastore 100%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/fault 100%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/library 95%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/storage 82%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/task 94%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/vm 79%
github.com/vmware-tanzu/vm-operator/pkg/util/vsphere/watcher 85%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig 95%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/anno2extraconfig 100%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/bootoptions 88%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/cdrom 88%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/crypto 92%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/diskpromo 100%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/extensioncompatconstraint 93%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/extraconfig 100%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/networkextraconfig 79%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/policy 97%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/virtualcontroller 93%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/volumes/unmanaged/backfill 98%
github.com/vmware-tanzu/vm-operator/pkg/vmconfig/volumes/unmanaged/register 92%
github.com/vmware-tanzu/vm-operator/pkg/webconsolevalidation 100%
github.com/vmware-tanzu/vm-operator/services/vm-watcher 85%
github.com/vmware-tanzu/vm-operator/webhooks/common 98%
github.com/vmware-tanzu/vm-operator/webhooks/configtarget/validation 87%
github.com/vmware-tanzu/vm-operator/webhooks/persistentvolumeclaim/validation 95%
github.com/vmware-tanzu/vm-operator/webhooks/unifiedstoragequota/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachine/mutation 86%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachine/validation 96%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineclass/mutation 62%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineclass/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineconfigoptions/validation 88%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinegroup/mutation 87%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinegroup/validation 93%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinegrouppublishrequest/mutation 86%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinegrouppublishrequest/validation 88%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineguestoptions/validation 88%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinepublishrequest/validation 90%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinereplicaset/validation 90%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineservice/mutation 67%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachineservice/validation 92%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesetresourcepolicy/validation 89%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesnapshot/mutation 86%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinesnapshot/validation 91%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinewebconsolerequest/v1alpha1/validation 92%
github.com/vmware-tanzu/vm-operator/webhooks/virtualmachinewebconsolerequest/validation 92%
Summary 84% (21858 / 25882)

Minimum allowed line rate is 79%

@svc-vmop-bot svc-vmop-bot added commit-msg-reviewed AI Reviewed Commit Message and removed pro-ai-review-requested Mark the PR as ready for the AI Review bot using a Pro model labels Sep 7, 2026
@svc-vmop-bot svc-vmop-bot added ai-reviewed This PR has been reviewed by the AI review bot and removed ai-review-requested Mark the PR as ready for the AI Review bot using a standard model labels Sep 8, 2026

@aruneshpa aruneshpa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments, but none are absolute blockers.

// Exported so its branches can be unit tested with a synthetic error, since
// vcsim cannot produce a real NoCompatibleHost/autoevac task fault.
func SetPowerStateSyncedCondition(vmCtx pkgctx.VirtualMachineContext, err error) {
if vmCtx.VM.Status.PowerState == vmCtx.VM.Spec.PowerState || errors.Is(err, ErrSetPowerState) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. Why would we set PowerState synced = true if we get ErrSetPowerState?

vmCtx.VM,
vmopv1.VirtualMachinePowerStateSynced,
vmopv1.VirtualMachineInfraInMaintenanceReason,
"VirtualMachine is powered off because the underlying infrastructure is in maintenance")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought: Do you want to keep this generic?
Like: "Unable to reconcile virtual machine power state because the underlying infrastructure is in maintenance"?

}
for _, fm := range fault.FaultMessage {
switch fm.Key {
case FaultMessageKeyHostInMaintenanceMode, FaultMessageKeyRestartOnCurrentHostRequired:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a switch case for a single conditional is weird. Did linter complain against using a simple if statement here?
If not, let's switch to if. Otherwise, ignore.

// power-state-change task when the failure is a NoCompatibleHost fault
// indicating the VM's host is in infrastructure maintenance. Callers use
// errors.Is to detect this condition from any call-stack depth.
var ErrInfraMaintenanceFault = errors.New("vm power op failed due to host infrastructure maintenance")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get rid of "host" in the error message. That's leaking unnecessary detail to the user. Maybe call it:

var ErrInfraMaintenanceFault = errors.New("vm power op failed due to infrastructure in maintenance")


// WrapHardPowerOpFailure builds the error returned by doAndWaitOnHardPowerOp
// for a failed hard power op, wrapping it with ErrInfraMaintenanceFault when
// ti indicates the failure is a NoCompatibleHost fault caused by the VM's

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ti -> the provided task info.

For a second, I thought you meant "it".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-reviewed This PR has been reviewed by the AI review bot commit-msg-reviewed AI Reviewed Commit Message size/XXL Denotes a PR that changes 1000+ lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants