Skip to content

fix(controller): stop reconcile hot loop - #14

Merged
birdmanmandbir merged 4 commits into
mainfrom
fix/status-phase-hot-loop
Jul 22, 2026
Merged

fix(controller): stop reconcile hot loop#14
birdmanmandbir merged 4 commits into
mainfrom
fix/status-phase-hot-loop

Conversation

@birdmanmandbir

@birdmanmandbir birdmanmandbir commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • preserve Running during later secret reconciliation
  • ignore status-only parent events while keeping owned-resource watches for self-healing
  • update owned resources and project status only when their operator-owned state drifts
  • make the manager/envtest startup assertion tolerant of informer startup time

Production evidence

A ready Sliqs project triggered 239 reconciles in 30 seconds while its resourceVersion changed continuously.

Verification

  • regression tests prove unchanged owned resources and unchanged project status emit zero updates while real drift is repaired
  • manager/envtest proves status-only parent updates stay quiet and spec updates still reconcile
  • go test ./... -count=1 (also passed in three separate full runs)
  • make lint with repo-pinned golangci-lint v2.5.0
  • make test-tanka test-delivery
  • go build -o /tmp/cloudnative-supabase-pr14-manager ./cmd

@birdmanmandbir

Copy link
Copy Markdown
Contributor Author

Triage Update

Fixed

  • P1 steady-state hot loop — 9e10c56 filters parent events with GenerationChangedPredicate, so status-only updates do not enqueue the project. Owned Secret, ConfigMap, Service, Deployment, Job, CronJob, CNPG, and ObjectStore watches remain active.
  • Manager/envtest regression — starts the real controller manager, proves a status update stays quiet, then proves a spec generation change still reconciles.

False Positive

  • None.

Deferred

  • None.

Remaining

  • GitHub checks for 9e10c56 are running.

Verification: focused envtest observed the extra reconcile before the fix and passed after it; full make test test-tanka test-delivery, golangci-lint, manager build, and shellcheck pass locally.

@birdmanmandbir

Copy link
Copy Markdown
Contributor Author

Triage Complete

Fixed

  • Parent status-only updates no longer enqueue reconcile — 9e10c56
  • Spec generation changes still enqueue reconcile — covered by manager/envtest
  • Running phase is not downgraded during secret reconciliation — 848ceb6

False Positive

  • None.

Deferred

  • None.

Remaining

  • None in code. GitHub build, lint, and amd64 Docker checks pass at 9e10c56.

@birdmanmandbir

Copy link
Copy Markdown
Contributor Author

Triage: both findings were actionable and are fixed in 8df65c8.

  • P1 [FIXED]: ConfigMaps, Deployments, Services, ObjectStores, ScheduledBackups, CronJobs, and PowerSync Publications now update only when operator-owned fields drift. Project status writes now compare the stored and desired status first. Added regression coverage proving unchanged owned resources and unchanged status emit zero updates while real drift still updates all resource types.
  • P2 [FIXED]: the manager/envtest assertions now use an explicit 10-second startup timeout. The full suite passed in three separate runs.

Verification:

  • go test ./... -count=1 — passed
  • three separate full go test ./... -count=1 runs — passed
  • make lint (repo-pinned golangci-lint v2.5.0) — 0 issues
  • make test-tanka test-delivery — passed
  • go build -o /tmp/cloudnative-supabase-pr14-manager ./cmd — passed

@birdmanmandbir birdmanmandbir changed the title fix(controller): stop status phase reconcile loop fix(controller): stop reconcile hot loop Jul 22, 2026
@birdmanmandbir

Copy link
Copy Markdown
Contributor Author

Triage Update

Fixed

  • P1 — replaced every controller DeepDerivative comparison with exact operator-owned field comparison in 3037d35.
  • Deployment and CronJob comparisons normalize the kube-apiserver defaults observed by envtest before exact comparison, so defaulting stays quiet while removed env vars, image pull secrets, resources, volumes, and list entries converge.
  • ObjectStore compares and updates configuration and retentionPolicy exactly while preserving CRD-defaulted instanceSidecarConfiguration.
  • Service ports/selectors, ScheduledBackup spec, and PowerSync Publication spec now converge when fields or map entries are removed.
  • Added enabled → disabled and non-empty → empty regressions. The Auth email-hook env is removed in one update and the next reconcile emits zero updates; stale Service ports, backup retention, ScheduledBackup fields, CronJob fields, and Publication parameters are also removed.

False Positive

  • None.

Deferred

  • None.

Remaining

  • CI is running for 3037d35.

Verification:

  • go test ./... -count=1 — passed
  • make lint — 0 issues
  • make test-tanka test-delivery — passed
  • go build -o /tmp/cloudnative-supabase-pr14-manager ./cmd — passed
  • git diff --check — passed

@birdmanmandbir
birdmanmandbir merged commit 80bdd96 into main Jul 22, 2026
3 checks passed
@birdmanmandbir
birdmanmandbir deleted the fix/status-phase-hot-loop branch July 22, 2026 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant