Skip to content

OCPBUGS-84327: Product-cli missing controller-runtime logger initialization causes noisy warning#8491

Open
amogh-redhat wants to merge 1 commit into
openshift:mainfrom
amogh-redhat:fix-OCPBUGS-84327
Open

OCPBUGS-84327: Product-cli missing controller-runtime logger initialization causes noisy warning#8491
amogh-redhat wants to merge 1 commit into
openshift:mainfrom
amogh-redhat:fix-OCPBUGS-84327

Conversation

@amogh-redhat

@amogh-redhat amogh-redhat commented May 12, 2026

Copy link
Copy Markdown

What this PR does / why we need it:

product-cli/main.go does not initialize the controller-runtime logger, while the hypershift CLI (main.go) properly initializes it at line 43 with ctrl.SetLogger(zap.New(...)). This causes a noisy warning to be emitted during operations like create infra when using the product-cli:

[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
The hypershift CLI does not produce this warning because the logger is properly initialized.

Added "ctrl.SetLogger(zap.New(...))." to the product-cli/main.go to fix the above mentioned issue.

Which issue(s) this PR fixes:

Fixes : https://redhat.atlassian.net/browse/OCPBUGS-84327

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Chores
    • CLI now initializes structured JSON logging at startup, improving machine-readability and consistency of log output.
    • Timestamps in logs use RFC3339 formatting for better interoperability and easier correlation across systems.
    • Result: clearer, more parseable logs to aid debugging and monitoring of CLI behavior.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 12, 2026
@openshift-ci

openshift-ci Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The pull request updates product-cli/main.go to configure controller-runtime logging at startup. It adds imports for sigs.k8s.io/controller-runtime and Zap logging (including zapcore) and, in main(), initializes a Zap JSON logger with RFC3339 time encoding and sets it as the controller-runtime logger via ctrl.SetLogger().

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Custom check is not applicable. PR modifies only product-cli/main.go for logger initialization. No Ginkgo tests are involved—tests in product-cli use standard Go testing package, not Ginkgo.
Test Structure And Quality ✅ Passed Custom check for Ginkgo test code quality is not applicable. This PR only modifies product-cli/main.go to add controller-runtime logger initialization with no test code changes.
Microshift Test Compatibility ✅ Passed PR modifies product-cli/main.go to initialize controller-runtime logger. No Ginkgo e2e tests are added. Custom check for MicroShift test compatibility is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed This PR only modifies product-cli/main.go to initialize controller-runtime logging. No Ginkgo e2e tests are added, so the SNO test compatibility check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed PR modifies only product-cli/main.go to initialize controller-runtime logger. No deployment manifests, operators, or controllers introducing scheduling constraints are added/modified.
Ote Binary Stdout Contract ✅ Passed The PR initializes zap logging with default stderr output. No WriteTo() option or stdout configuration exists. Initialization produces no output. Matches reference implementation in main.go.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Custom check does not apply. This PR does not add any Ginkgo e2e tests—it only modifies product-cli/main.go to initialize controller-runtime logger.
Title check ✅ Passed The title clearly and specifically describes the main change: adding controller-runtime logger initialization to product-cli to eliminate a warning message.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented May 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.49%. Comparing base (ac1a1c2) to head (33a8aac).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
product-cli/main.go 0.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8491      +/-   ##
==========================================
- Coverage   41.50%   41.49%   -0.01%     
==========================================
  Files         758      758              
  Lines       93689    93692       +3     
==========================================
  Hits        38882    38882              
- Misses      52070    52073       +3     
  Partials     2737     2737              
Files with missing lines Coverage Δ
product-cli/main.go 0.00% <0.00%> (ø)
Flag Coverage Δ
cmd-support 34.86% <ø> (ø)
cpo-hostedcontrolplane 43.59% <ø> (ø)
cpo-other 43.17% <ø> (ø)
hypershift-operator 51.57% <ø> (ø)
other 31.63% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@amogh-redhat amogh-redhat marked this pull request as ready for review May 12, 2026 14:23
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 12, 2026
@openshift-ci openshift-ci Bot requested review from muraee and sdminonne May 12, 2026 14:24
@muraee

muraee commented May 12, 2026

Copy link
Copy Markdown
Contributor

/approve

@openshift-ci

openshift-ci Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amogh-redhat, muraee

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 12, 2026
@vsolanki12

Copy link
Copy Markdown
Contributor

/retitle OCPBUGS-84327: Product-cli missing controller-runtime logger initialization causes noisy warning

@openshift-ci openshift-ci Bot changed the title OCPBUGS-84327 : Product-cli missing controller-runtime logger initialization causes noisy warning OCPBUGS-84327: Product-cli missing controller-runtime logger initialization causes noisy warning May 13, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 13, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@amogh-redhat: This pull request references Jira Issue OCPBUGS-84327, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

product-cli/main.go does not initialize the controller-runtime logger, while the hypershift CLI (main.go) properly initializes it at line 43 with ctrl.SetLogger(zap.New(...)). This causes a noisy warning to be emitted during operations like create infra when using the product-cli:

[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
The hypershift CLI does not produce this warning because the logger is properly initialized.

Added "ctrl.SetLogger(zap.New(...))." to the product-cli/main.go to fix the above mentioned issue.

Which issue(s) this PR fixes:

Fixes : https://redhat.atlassian.net/browse/OCPBUGS-84327

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Chores
  • CLI now initializes structured JSON logging at startup, improving machine-readability and consistency of log output.
  • Timestamps in logs use RFC3339 formatting for better interoperability and easier correlation across systems.
  • Result: clearer, more parseable logs to aid debugging and monitoring of CLI behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label May 13, 2026
@vsolanki12

Copy link
Copy Markdown
Contributor

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 13, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@vsolanki12: This pull request references Jira Issue OCPBUGS-84327, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@csrwng

csrwng commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

/lgtm
/override "codecov/patch"

@openshift-ci

openshift-ci Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

@csrwng: Overrode contexts on behalf of csrwng: codecov/patch

Details

In response to this:

/lgtm
/override "codecov/patch"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 8, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@csrwng

csrwng commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

/area cli

@openshift-ci openshift-ci Bot added area/cli Indicates the PR includes changes for CLI and removed do-not-merge/needs-area labels Jun 8, 2026
@hypershift-jira-solve-ci

Copy link
Copy Markdown

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-azure-self-managed | Build: 2064006990471892992 | Cost: $2.3434012500000003 | Failed step: hypershift-azure-run-e2e-self-managed

View full analysis report


Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6

@amogh-redhat

Copy link
Copy Markdown
Author

/retest

1 similar comment
@amogh-redhat

Copy link
Copy Markdown
Author

/retest

product-cli's main.go lacks controller-runtime logger initialization,
unlike the hypershift CLI. This generates an unnecessary warning:
"log.SetLogger(...) was never called; logs will not be displayed."
Add structured logging with JSON output and RFC3339 timestamps to
align with the hypershift CLI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 9, 2026
@sdminonne

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 10, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@hypershift-jira-solve-ci

Copy link
Copy Markdown

Test Failure Analysis Complete

Job Information

Test Failure Analysis

Error

failed to wait for DaemonSet kubelet-config-verifier to be ready: context deadline exceeded

Pod kubelet-config-verifier-9v7nr on node ip-10-0-38-193.ec2.internal (us-east-1c):
  State: CrashLoopBackOff (16 restarts)
  Exit message: "ERROR: Pull secrets are different
    Node pull secret MD5: 57c5c0f175824217fa4c01d17747945d
    Cluster pull secret MD5: 937ccfb8f29c3359171725fe5d1075ec"

Summary

The TestCreateCluster/Main/EnsureGlobalPullSecret test failed because the kubelet-config-verifier DaemonSet never reached 3/3 ready pods within its 20-minute timeout. The test patches the management-cluster pull secret and then verifies that the updated pull secret propagates to all guest cluster nodes. On node ip-10-0-38-193.ec2.internal (us-east-1c), the global-pull-secret-syncer DaemonSet did not sync the updated pull secret to the node's /var/lib/kubelet/config.json before the verifier pod started checking. The verifier pod detected a mismatch and exited with code 1, entering CrashLoopBackOff with 16 restarts. The other 2 pods (on us-east-1a and us-east-1b nodes) eventually succeeded. This is a pre-existing flaky test unrelated to PR #8491, which only modifies product-cli/main.go (controller-runtime logger initialization). All other 588 tests in the run passed.

Root Cause

Race condition in pull secret propagation to one guest cluster node.

The EnsureGlobalPullSecret test performs these steps:

  1. Patches the management-cluster pull secret with a dummy auth entry
  2. Waits for the secret to propagate to guest cluster openshift-config/pull-secret and kube-system/original-pull-secret
  3. Creates a kubelet-config-verifier DaemonSet that checks whether each node's /var/lib/kubelet/config.json matches the cluster pull secret
  4. Waits (up to 20 minutes) for ovnkube-node, global-pull-secret-syncer, konnectivity-agent, and kubelet-config-verifier DaemonSets to all be ready

The global-pull-secret-syncer DaemonSet is responsible for syncing the updated pull secret to each node's kubelet config file. On the failing node (ip-10-0-38-193.ec2.internal, us-east-1c), the syncer pod (global-pull-secret-syncer-dsxnm) only started at 14:48:19 UTC and synced the pull secret at that time. However, the kubelet-config-verifier pod on that same node had been crash-looping since much earlier, continuously finding that the node pull secret (MD5: 57c5c0f175824217fa4c01d17747945d) didn't match the cluster pull secret (MD5: 937ccfb8f29c3359171725fe5d1075ec).

The critical timing issue:

  • Pod kubelet-config-verifier-xbs2z (ip-10-0-30-194, us-east-1b): Failed 3 times, then succeeded at 14:27:05 — the syncer had already updated this node
  • Pod kubelet-config-verifier-spnjp (ip-10-0-0-217, us-east-1a): Succeeded immediately with 0 restarts
  • Pod kubelet-config-verifier-9v7nr (ip-10-0-38-193, us-east-1c): Never succeeded — 16 restarts over 20+ minutes, the syncer only reached this node at 14:48:19 (after the test had already timed out)

The global-pull-secret-syncer on the us-east-1c node appears to have been delayed or restarted, causing an asymmetric propagation delay across the three guest cluster nodes. By the time the syncer finally updated the node's pull secret, the 20-minute DaemonSet readiness timeout had already been exceeded.

This failure is not caused by PR #8491, which only touches product-cli/main.go to add controller-runtime logger initialization. The failure is a pre-existing race condition in the EnsureGlobalPullSecret test's assumption that pull secret propagation will complete uniformly across all nodes within the wait window.

Recommendations
  1. Retry / retest: This is a flaky test failure unrelated to PR OCPBUGS-84327: Product-cli missing controller-runtime logger initialization causes noisy warning #8491. The PR can be safely retested with /retest.

  2. Test improvement (for hypershift test owners):

    • The EnsureGlobalPullSecret test should wait for the global-pull-secret-syncer DaemonSet to complete its sync cycle on all nodes before creating the kubelet-config-verifier DaemonSet. Currently, it waits for the syncer DaemonSet to be "ready" (pods running), but "running" doesn't mean "has completed a sync cycle."
    • Consider adding a pre-check that verifies each node's /var/lib/kubelet/config.json MD5 matches the expected value before deploying the verifier DaemonSet, or increase the timeout to account for slow syncer starts.
  3. Cleanup bug: The second subtest Check_if_the_config.json_is_correct_in_all_of_the_nodes fails with daemonsets.apps "kubelet-config-verifier" already exists — the first subtest's DaemonSet is not cleaned up before the second subtest tries to create it again.

Evidence
Evidence Detail
Failed test TestCreateCluster/Main/EnsureGlobalPullSecret/When_management-cluster_hostedCluster.Spec.PullSecret_is_updated_in-place_it_should_propagate_to_guest_without_rollout
Error failed to wait for DaemonSet kubelet-config-verifier to be ready: context deadline exceeded
Duration 1250.08s (20.8 min), DaemonSet timeout is 20 min
Failing pod kubelet-config-verifier-9v7nr on node ip-10-0-38-193.ec2.internal (us-east-1c)
Pod state CrashLoopBackOff, 16 restarts, exit code 1
Pod error Pull secret MD5 mismatch: node=57c5c0f175824217fa4c01d17747945d, cluster=937ccfb8f29c3359171725fe5d1075ec
Global-pull-secret-syncer on failing node Pod dsxnm started syncing at 14:48:19 UTC — after the test timeout
Syncer log "file content is different, updating it" at 14:48:19 (too late)
Successful pod (us-east-1b) xbs2z: 3 restarts then succeeded at 14:27:05
Successful pod (us-east-1a) spnjp: 0 restarts, succeeded immediately
PR #8491 changes Only product-cli/main.go — completely unrelated to globalps/pull-secret test
Overall test results 588 passed, 5 failed (all 5 are parent/child of the same EnsureGlobalPullSecret failure)
Cascade failure Check_if_the_config.json_is_correct_in_all_of_the_nodes failed with "already exists" due to leftover DaemonSet from timed-out subtest

@openshift-ci

openshift-ci Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

@amogh-redhat: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws 33a8aac link true /test e2e-aws
ci/prow/e2e-azure-v2-self-managed 33a8aac link false /test e2e-azure-v2-self-managed

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/cli Indicates the PR includes changes for CLI jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants