Splunk Operator: Make putting searchheads in manual detention optional during upgrade

[pault28]

Please select the type of request

Enhancement

Tell us more

Describe the request

• Making updates to Search Head Cluster (SHC) on Kubernetes, the Splunk Operator puts a search head in the cluster in manual detention, waits for all searches running on it to complete and then updates/redeploy it. This feature is very good, however it does mean updates can take a significant amount of time to complete in a busy/production cluster. The other side effect is user experience, as they are still sent to the search head in manual detention via the ingress. What would be good is an opt out feature that users can turn on if they do not want this behaviour, effectively allowing for search heads to be taken out of service regardless of ongoing searches happening on it.

Expected behavior

• An opt-out environment such as SH_DISABLE_MANUAL_DETENTION, that when set to true or 1 implies the operator performs the update without putting the search heads in manual detention.

Splunk setup on K8S

• Splunk enterprise 10.0.2 deployed with Splunk Operator 3.0.0

Reproduction/Testing steps

• Making any change that will trigger an update of SHC by the deployer, e.g change the defaults: |-
• Once applied, check the search head clustering and you should see search head n-1 (where n is the number of search heads in the cluster) entering manual detention

K8s environment

• EKS cluster
• NGINX ingress controller

Proposed changes(optional)

• Add SH_DISABLE_MANUAL_DETENTION environment to allow users opt out of putting search heads in manual detention during updates

Additional context(optional)

• This issue relates to Bug report draft: SHC ManualDetention + Kubernetes readiness gating can deadlock operator-driven rollouts (and can strand users with sticky ingress) #1676