Skip to content

fix(provider): disable workload healthcheck lease ejection#418

Open
chalabi2 wants to merge 2 commits into
mainfrom
fix/disable-healthcheck-lease-ejection
Open

fix(provider): disable workload healthcheck lease ejection#418
chalabi2 wants to merge 2 commits into
mainfrom
fix/disable-healthcheck-lease-ejection

Conversation

@chalabi2

@chalabi2 chalabi2 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Default provider monitor health checks to disabled after a deployment becomes healthy.
  • Stop exposing the monitor healthcheck period and jitter provider-service flags.
  • Treat a zero healthcheck period as no schedule instead of panicking.

Root cause

The deployment monitor kept scheduling periodic health checks after a workload
had already become healthy. If later status checks failed enough times, the
provider broadcast MsgCloseBid with LeaseClosedReasonUnstable. The healthcheck
period was exposed as operator configuration, so providers could keep this lease
ejection behavior enabled unintentionally.

Verification

Failing-before output from the regression test:

=== RUN   TestMonitorHealthcheckDisabledWhenPeriodZero
    monitor_test.go:62:
        Error:       func (assert.PanicTestFunc)(...) should not panic
        Panic value: invalid argument to Int63n
--- FAIL: TestMonitorHealthcheckDisabledWhenPeriodZero

Passing checks:

GOWORK=off go test ./cluster -run TestMonitorHealthcheckDisabledWhenPeriodZero -count=1 -v
GOWORK=off go test ./cmd/provider-services/cmd -run TestRunCmdDoesNotExposeMonitorHealthcheckFlags -count=1 -v
GOWORK=off go test ./cluster -count=1
GOWORK=off go test ./cmd/provider-services/cmd -count=1
git diff --check

chalabi2 added 2 commits July 6, 2026 10:50
Add coverage for disabling monitor health checks with a zero period.
The regression captures the old panic from scheduling zero jitter and
asserts that the provider run command does not expose healthcheck flags.

Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
Default post-deploy monitor health checks to off and stop exposing the
healthcheck period knobs through provider-service flags. A zero period now
returns no schedule instead of panicking or closing stable workloads later.

Signed-off-by: Joseph Chalabi <chalabi.joseph@gmail.com>
@chalabi2 chalabi2 requested a review from a team as a code owner July 6, 2026 17:51
@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e5eaacfd-d068-49f3-8ca1-c207b68639f9

📥 Commits

Reviewing files that changed from the base of the PR and between 498f8ec and 65c9505.

📒 Files selected for processing (6)
  • cluster/config.go
  • cluster/monitor.go
  • cluster/monitor_test.go
  • cmd/provider-services/cmd/flags.go
  • cmd/provider-services/cmd/helpers_test.go
  • cmd/provider-services/cmd/run.go
💤 Files with no reviewable changes (3)
  • cmd/provider-services/cmd/flags.go
  • cluster/config.go
  • cmd/provider-services/cmd/run.go

Walkthrough

This PR removes default healthcheck period/jitter configuration and their associated CLI flags, replacing healthcheck-based monitor scheduling with retry-based scheduling. scheduleHealthcheck now returns nil when the period is non-positive, and jitter is only applied when positive. Provider run wiring is updated accordingly.

Changes

Healthcheck to retry-based monitor scheduling

Layer / File(s) Summary
Monitor scheduling core changes
cluster/config.go, cluster/monitor.go, cluster/monitor_test.go
Removes default healthcheck period/jitter values; scheduleHealthcheck returns nil when period is non-positive; schedule only applies jitter when positive; new test verifies disabled scheduling behavior.
CLI flags and run wiring migration
cmd/provider-services/cmd/flags.go, cmd/provider-services/cmd/run.go, cmd/provider-services/cmd/helpers_test.go
Removes FlagMonitorHealthcheckPeriod/FlagMonitorHealthcheckPeriodJitter flags and constants; validation and provider config wiring switch to FlagMonitorMaxRetries, FlagMonitorRetryPeriod, and FlagMonitorRetryPeriodJitter; new test confirms removed flags are not exposed.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant RunCmd
  participant Config as ProviderConfig
  participant Monitor as deploymentMonitor

  User->>RunCmd: run with monitor-retry-period flags
  RunCmd->>RunCmd: validate FlagMonitorRetryPeriod >= 4s
  RunCmd->>Config: set MonitorMaxRetries, MonitorRetryPeriod, MonitorRetryPeriodJitter
  Config->>Monitor: initialize with retry-based config
  Monitor->>Monitor: scheduleHealthcheck() returns nil (period unset)
Loading

Poem

Hop, hop, the old flags fade away,
Retry periods now hold sway,
No more ticks when zero's the sum,
Jitter waits till periods come,
A rabbit cheers this cleaner code today! 🐇✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: disabling workload healthcheck lease ejection in the provider.
Description check ✅ Passed The description directly matches the implemented changes and regression coverage around disabled healthchecks and removed flags.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/disable-healthcheck-lease-ejection

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant