OCPBUGS-13553: Trigger NodePool rollout on in-place pull secret changes#8700
OCPBUGS-13553: Trigger NodePool rollout on in-place pull secret changes#8700vsolanki12 wants to merge 1 commit into
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
Skipping CI for Draft Pull Request. |
|
@vsolanki12: This pull request references Jira Issue OCPBUGS-13553, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: vsolanki12 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/jira refresh |
|
@vsolanki12: This pull request references Jira Issue OCPBUGS-13553, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (19)
📒 Files selected for processing (9)
✅ Files skipped from review due to trivial changes (3)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughThis PR modifies NodePool rollout identity to include content-derived hashes of the HostedCluster pull secret and optional additionalTrustBundle. ConfigGenerator now fetches and hashes those contents and incorporates them into Hash()/HashWithoutVersion(). The NodePool controller watches Secrets and enqueues NodePools when a referenced pull Secret changes. Token no longer computes/stores those hashes itself; reconcile reads hashes from ConfigGenerator. Tests and a small documentation comment were updated accordingly. Sequence Diagram(s)sequenceDiagram
participant Secret
participant NodePoolReconciler
participant KubeAPI
participant ConfigGenerator
Secret->>NodePoolReconciler: Secret change event
NodePoolReconciler->>KubeAPI: Get Secret annotation -> Get(NodePool) (if annotated)
NodePoolReconciler->>KubeAPI: List HostedClusters(namespace)
KubeAPI-->>NodePoolReconciler: HostedClusters
NodePoolReconciler->>KubeAPI: List NodePools and match spec.clusterName
NodePoolReconciler-->>NodePoolReconciler: Enqueue reconcile.Requests for matching NodePools
ConfigGenerator->>KubeAPI: Get(Secret)/Get(ConfigMap) to read pullSecret/additionalTrustBundle
KubeAPI-->>ConfigGenerator: Secret/ConfigMap data
ConfigGenerator-->>ConfigGenerator: Compute pullSecretHash and additionalTrustBundleHash
Suggested reviewers
🚥 Pre-merge checks | ✅ 10 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (10 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@vsolanki12: This pull request references Jira Issue OCPBUGS-13553, which is valid. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8700 +/- ##
==========================================
+ Coverage 41.43% 41.48% +0.05%
==========================================
Files 756 756
Lines 93658 93675 +17
==========================================
+ Hits 38807 38864 +57
+ Misses 52128 52069 -59
- Partials 2723 2742 +19
... and 3 files with indirect coverage changes
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hypershift-operator/controllers/nodepool/config.go`:
- Around line 106-113: The current code computes additionalTrustBundleHash from
only the ConfigMap data, but without tying it to the ConfigMap's metadata
changes it won't reliably trigger rollouts on in-place edits; update the logic
in the block that sets cg.rolloutConfig.additionalTrustBundleHash (and the
analogous block at the other occurrence) to incorporate the ConfigMap's
ResourceVersion (or UID+ResourceVersion) into the value hashed (e.g.,
concatenate additionalTrustBundle.Data["ca-bundle.crt"] with
additionalTrustBundle.ResourceVersion before calling supportutil.HashSimple) so
the hash changes on edits and becomes an effective rollout trigger.
- Around line 106-113: The code currently hashes
additionalTrustBundle.Data["ca-bundle.crt"] without verifying the key exists or
is non-empty; update the block that handles
hostedCluster.Spec.AdditionalTrustBundle (around the client.Get for
additionalTrustBundle and assignment to
cg.rolloutConfig.additionalTrustBundleHash) to validate that the "ca-bundle.crt"
key is present and non-empty in additionalTrustBundle.Data and return a
descriptive error if missing/empty, instead of calling supportutil.HashSimple on
a nil/empty value; ensure the error message references the
hostedCluster.Spec.AdditionalTrustBundle.Name and namespace for easier
debugging.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 2de1b436-b36f-4629-935c-d8e94db65bd3
⛔ Files ignored due to path filters (19)
api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/AAA_ungated.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterUpdateAcceptRisks.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterVersionOperatorConfiguration.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDC.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUIDAndExtraClaimMappings.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUpstreamParity.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/GCPPlatform.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HCPEtcdBackup.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HyperShiftOnlyDynamicResourceAllocation.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ImageStreamImportMode.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/KMSEncryptionProvider.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/OpenStack.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/TLSAdherence.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-CustomNoUpgrade.crd.yamlis excluded by!**/zz_generated.crd-manifests/**,!cmd/install/assets/**/*.yamlcmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-Default.crd.yamlis excluded by!**/zz_generated.crd-manifests/**,!cmd/install/assets/**/*.yamlcmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-TechPreviewNoUpgrade.crd.yamlis excluded by!**/zz_generated.crd-manifests/**,!cmd/install/assets/**/*.yamldocs/content/reference/aggregated-docs.mdis excluded by!docs/content/reference/aggregated-docs.mddocs/content/reference/api.mdis excluded by!docs/content/reference/api.mdvendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.gois excluded by!vendor/**,!**/vendor/**
📒 Files selected for processing (8)
api/hypershift/v1beta1/hostedcluster_types.gohypershift-operator/controllers/nodepool/conditions_test.gohypershift-operator/controllers/nodepool/config.gohypershift-operator/controllers/nodepool/config_test.gohypershift-operator/controllers/nodepool/nodepool_controller.gohypershift-operator/controllers/nodepool/secret_janitor_test.gohypershift-operator/controllers/nodepool/token.gohypershift-operator/controllers/nodepool/token_test.go
2ad1771 to
fe44067
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
hypershift-operator/controllers/nodepool/config.go (1)
142-151: Run the rollout-upgrade e2e before merge.These hash inputs now include referenced Secret/ConfigMap contents, and the PR notes this will trigger a one-time rollout for existing NodePools. Please make sure
e2e-aws-upgrade-hypershift-operatoris part of validation for this PR before merge. As per coding guidelines, "If it does, the PR must passe2e-aws-upgrade-hypershift-operatorto prove the rollout is safe."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hypershift-operator/controllers/nodepool/config.go` around lines 142 - 151, The change to ConfigGenerator.Hash (and the related HashWithoutVersion) now includes referenced Secret/ConfigMap contents (cg.pullSecretHash, cg.additionalTrustBundleHash, etc.), which will trigger a one-time rollout for existing NodePools; before merging, run and ensure the e2e-aws-upgrade-hypershift-operator job passes to validate the rollout is safe and acceptable. Locate ConfigGenerator.Hash and ConfigGenerator.HashWithoutVersion (and their use of supportutil.HashSimple and cg.globalConfig) and run the full e2e-aws-upgrade-hypershift-operator test suite against this PR, addressing any failures or flakiness found by the test prior to merge. Ensure CI/job results are green and document the passing run in the PR.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hypershift-operator/controllers/nodepool/nodepool_controller_test.go`:
- Around line 3408-3410: In the TestEnqueueNodePoolsForSecret setup, don't
ignore errors from hyperv1.AddToScheme and corev1.AddToScheme; capture their
returned errors and fail the test if non-nil (e.g., use t.Fatalf or test
assertions like require.NoError) so scheme registration problems surface
immediately; update the calls that use the local variable scheme and replace the
ignored assignments with error checks for hyperv1.AddToScheme(scheme) and
corev1.AddToScheme(scheme).
---
Nitpick comments:
In `@hypershift-operator/controllers/nodepool/config.go`:
- Around line 142-151: The change to ConfigGenerator.Hash (and the related
HashWithoutVersion) now includes referenced Secret/ConfigMap contents
(cg.pullSecretHash, cg.additionalTrustBundleHash, etc.), which will trigger a
one-time rollout for existing NodePools; before merging, run and ensure the
e2e-aws-upgrade-hypershift-operator job passes to validate the rollout is safe
and acceptable. Locate ConfigGenerator.Hash and
ConfigGenerator.HashWithoutVersion (and their use of supportutil.HashSimple and
cg.globalConfig) and run the full e2e-aws-upgrade-hypershift-operator test suite
against this PR, addressing any failures or flakiness found by the test prior to
merge. Ensure CI/job results are green and document the passing run in the PR.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: e05f9a1b-7e2f-4d8c-b871-3435bce6c467
⛔ Files ignored due to path filters (19)
api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/AAA_ungated.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterUpdateAcceptRisks.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterVersionOperatorConfiguration.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDC.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUIDAndExtraClaimMappings.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUpstreamParity.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/GCPPlatform.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HCPEtcdBackup.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HyperShiftOnlyDynamicResourceAllocation.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ImageStreamImportMode.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/KMSEncryptionProvider.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/OpenStack.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**api/hypershift/v1beta1/zz_generated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/TLSAdherence.yamlis excluded by!**/zz_generated.featuregated-crd-manifests/**cmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-CustomNoUpgrade.crd.yamlis excluded by!**/zz_generated.crd-manifests/**,!cmd/install/assets/**/*.yamlcmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-Default.crd.yamlis excluded by!**/zz_generated.crd-manifests/**,!cmd/install/assets/**/*.yamlcmd/install/assets/crds/hypershift-operator/zz_generated.crd-manifests/hostedclusters-Hypershift-TechPreviewNoUpgrade.crd.yamlis excluded by!**/zz_generated.crd-manifests/**,!cmd/install/assets/**/*.yamldocs/content/reference/aggregated-docs.mdis excluded by!docs/content/reference/aggregated-docs.mddocs/content/reference/api.mdis excluded by!docs/content/reference/api.mdvendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.gois excluded by!vendor/**,!**/vendor/**
📒 Files selected for processing (9)
api/hypershift/v1beta1/hostedcluster_types.gohypershift-operator/controllers/nodepool/conditions_test.gohypershift-operator/controllers/nodepool/config.gohypershift-operator/controllers/nodepool/config_test.gohypershift-operator/controllers/nodepool/nodepool_controller.gohypershift-operator/controllers/nodepool/nodepool_controller_test.gohypershift-operator/controllers/nodepool/secret_janitor_test.gohypershift-operator/controllers/nodepool/token.gohypershift-operator/controllers/nodepool/token_test.go
✅ Files skipped from review due to trivial changes (3)
- api/hypershift/v1beta1/hostedcluster_types.go
- hypershift-operator/controllers/nodepool/conditions_test.go
- hypershift-operator/controllers/nodepool/secret_janitor_test.go
🚧 Files skipped from review as they are similar to previous changes (4)
- hypershift-operator/controllers/nodepool/nodepool_controller.go
- hypershift-operator/controllers/nodepool/token.go
- hypershift-operator/controllers/nodepool/token_test.go
- hypershift-operator/controllers/nodepool/config_test.go
Previously, modifying the pull secret or additional trust bundle data in-place (without changing the Secret/ConfigMap reference name) did not trigger a NodePool rollout because the rollout hash only included the resource name, not a content hash. This consolidates content hash computation into ConfigGenerator.rolloutConfig so both Hash() and HashWithoutVersion() reflect data changes. The redundant hash computation in Token is removed, and a secret watcher is added to enqueue NodePools when their HostedCluster's pull secret is modified. Signed-off-by: Vimal Solanki <vsolanki@redhat.com>
|
Now I have the complete picture. The workflow runs The log line Test Failure Analysis CompleteJob Information
Test Failure AnalysisErrorSummaryThe Root CauseThe newly added testCases := []struct {
name string
secret client.Object
objects []client.Object
expectedCount int
expectedNames []string
}{The The CI workflow in
Since Recommendations
Evidence
|
fe44067 to
481df5f
Compare
|
I have tried to reproduce and tested the fix in my test cluster as below: Environment: KubeVirt HCP cluster dhruvhcp-new500 Custom operator image deployed: Test scenario: Modified pull secret data in-place (without changing the Secret reference name) to verify that a NodePool rollout is now triggered. Before fix: in-place pull secret change did NOT trigger rollout: After fix: in-place pull secret change triggers rollout: Note: Deploying this fix causes a one time rollout for all existing |
|
@vsolanki12: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/hold |
Agree with @cblecker. I don't think this bug ticket is valid anymore. I see it was created from 2023. It was a conscious decision NOT to cause a rollout when the pull secret content changed. You must make a new secret and change it in the ref to change the pull secret. |
|
Closing based on reviewer feedback. Both reviewers confirmed this is by design — pull secret changes should never cause a NodePool rollout. The supported workflow is to create a new Secret and update the reference name. The original bug (OCPBUGS-13553, filed 2023) is no longer valid. |
|
@vsolanki12: This pull request references Jira Issue OCPBUGS-13553. The bug has been updated to no longer refer to the pull request using the external bug tracker. All external bug links have been closed. The bug has been moved to the NEW state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
What this PR does / why we need it:
Previously, modifying the pull secret or additional trust bundle data in-place (without changing
the Secret/ConfigMap reference name) did not trigger a NodePool rollout because the rollout hash
only included the resource name, not a content hash.
This PR:
pullSecretHashandadditionalTrustBundleHashfields toConfigGenerator.rolloutConfig, computed from actual Secret/ConfigMap dataHash()andHashWithoutVersion()so in-place data changes produce a different rollout hashTokenstruct (previously duplicated with TODO comments acknowledging the gap)enqueueNodePoolsForSecretwatcher so pull secret data changes trigger NodePool reconciliation (the pull secret lacks thenodePoolAnnotation, so the existingenqueueParentNodePooldidn't match it)PullSecretfield API doc to reflect the new behaviorNote: Deploying this fix will cause a one-time rollout for all existing NodePools on first reconciliation, because the hash now includes content hashes that were not previously part of the computation. This is expected and unavoidable.
Which issue(s) this PR fixes:
Fixes https://issues.redhat.com/browse/OCPBUGS-13553
Special notes for your reviewer:
additionalTrustBundleHashusesHashSimple(string)(not[]byte) to match the ignition server'sfetchAdditionalTrustBundlewhich returnsstring. ThepullSecretHashusesHashSimple([]byte)to match the ignition server'sfetchPullSecretwhich returns[]byte.dhruvhcp-new500): confirmed that modifying pull secret in-place now triggersUpdatingConfig=Truewith a new target hash.Checklist:
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests