fix(chart): reconcile the telemetry Collector in place once its token Secret exists

[LukasWodka]

Summary

The <release>-telemetry-status record for the skipped-no-token state promised that the skip "resolves itself once jobs-manager writes the Secret". Nothing performed that: the lookup deciding the Collector's three-state rule runs only inside a helm upgrade, and the hourly auto-upgrade tick only upgraded onto a newer published chart (already at latest; nothing to do / deployed version is ahead of repo … skipping). Measured on a fresh installer edge: the token Secret was present within minutes, three ticks passed, no DaemonSet — the Collector rendered a day later only because a new chart happened to ship. Tracked internally.

This PR does two things together:

1. Mechanism — same-version reconcile in the tick. When the tick has no newer chart to install, it now asks one question and acts on one answer: if the stored release manifest records the Collector as skipped-no-token (which, by the decider's construction, means telemetryCollector.enabled was unset), the stored manifest carries no Collector DaemonSet, the repo serves the deployed chart version, and a server-side dry-run of that same version now resolves enabled (the token Secret has arrived), the tick re-runs helm upgrade pinned to --version "$CURRENT" with the exact flags and values discipline of the normal upgrade (--reset-then-reuse-values --atomic --cleanup-on-fail --timeout). It logs one line naming why. Any other state behaves exactly as before.
2. Honest record. The reason text now says what actually triggers rendering: the next auto-upgrade tick (CronJob schedule read from autoUpgrade.schedule, repo from autoUpgrade.repoUrl — never restated), or, with autoUpgrade.enabled: false, "run helm upgrade on this release". The skipped-incomplete-values branch loses its "resolves itself" as well. No branch of the record promises a self-resolution any more.

The condition, in one place

telemetry_reconcile_verdict() in the rendered auto-upgrade.sh is the only decider — tri-state, and 2 never acts:

• 0 reconcile: stored state skipped-no-token and no DaemonSet named $TELEMETRY_COLLECTOR_NAME in the stored manifest and the repo serves $CURRENT and helm upgrade … --version "$CURRENT" --reset-then-reuse-values --dry-run=server --hide-secret resolves enabled.
• 1 nothing to do: any other stored state (enabled, disabled-by-operator, skipped-incomplete-values, no record at all — a release rendered by a chart predating it), or a dry-run that still says skipped-no-token. Quiet, as today.
• 2 cannot tell: helm get manifest unreadable/empty, dry-run failed or rendered no record, the record contradicts the manifest (says skipped, DaemonSet present), the TELEMETRY_* names missing from env, or the deployed version is not served by the repo. Logged as reconcile: NOT re-rendering, cannot tell …, no upgrade.

Derived, not restated. No Secret name, Class A rule or DaemonSet gate is copied into the script. The decision is read from the chart's own decider twice — the stored state out of helm get manifest, the would-be state out of --dry-run=server (helm 3.13+; the image is pinned at 3.16.4), which runs the templates' real lookups (release-scoped, legacy and pre-override Secret names alike) against the cluster. The three names the awk matches on (TELEMETRY_STATUS_NAME, TELEMETRY_STATUS_ANNOTATION, TELEMETRY_COLLECTOR_NAME) arrive in the CronJob env from two new helpers (tracebloc.telemetryStatusName, tracebloc.telemetryStateAnnotation) plus the existing tracebloc.telemetryCollectorName — the same helpers the status template and the DaemonSet render with, pinned by tests/telemetry_status_wiring_test.yaml (including under fullnameOverride). The image has no kubectl, and a bare curl against the API is exactly what the house style forbids, so helm is the only client used.

Deployed ahead of the index (current > latest)

This is the state the ticket caught, and it deserves a plain statement: an edge running a chart version the repo does not serve (a dev chart installed from a local path) cannot be re-rendered by the tick, because there is nothing it can fetch — helm upgrade needs a chart, and the deployed one exists only inside the release. The verdict names it every tick (deployed chart X is not served by <repo> (newest there: Y): a same-version re-render is impossible, the Collector renders on the next published chart), and the status record says the same. The reconcile fixes the fleet case — an edge on the published chart whose Secret arrived after install, which until now logged already at latest; nothing to do forever.

Type

• Bug fix
• Chart change (Chart.yaml version and appVersion 1.9.116 → 1.9.117)

Test plan

Everything CI runs for the chart tier, locally at CI severity (outputs kept):

• make lint (bash -n on every script + shellcheck -S error) — green; shellcheck -S warning -x on both new gates — clean.
• make drift — drift: all 50 guards green (48 existing + the 2 new ones; auto-upgrade-inflight-vs-wedge.sh unchanged and green) (includes the two new guards).
• make helm-lint helm-vocab — 5 lints green, chart-env-vocabulary: all 177 checks passed.
• make helm-template with kubeconform — aks/bm/eks/oc all Invalid: 0, Errors: 0.
• make helm-unittest — Test Suites: 42 passed, 42 total / Tests: 750 passed, 750 total (was 743; +7 here).
• make bats — bats scripts/tests/*.bats 2067 ok, 0 not ok, rc=0.
• Rendered script parses under the image's own busybox sh -n (alpine/helm:3.16.4).

New checks

• scripts/tests/auto-upgrade-telemetry-reconcile.sh (registered in make drift) — drives the rendered script with a stub helm on PATH, 23 cases: the full tick end to end (Secret arrived → reconcile pinned to $CURRENT with all flags; Secret still absent / enabled / disabled-by-operator / incomplete-values / no record → no upgrade; contradictory record / unreadable manifest / failed dry-run / dry-run without record / names missing → no upgrade and the log says cannot tell; ahead-of-index → named, no upgrade; newer chart → normal upgrade, dry-run never consulted) and the extracted telemetry_reconcile_verdict function directly for the rc 0/1/2 table. Fixtures are real helm template renders of this chart, not typed YAML; the names come from the rendered CronJob's env.
• scripts/tests/auto-upgrade-telemetry-reconcile-mutations.sh (registered in make drift) — the permanent mutation proof, each asserting the specific finding: (a) drop the DaemonSet-absent check → contradictory case reconciles, gate red; (b) accept a skipped-no-token dry-run → Secret-still-absent case reconciles, gate red; (c) unhook the reconcile from the at-latest branch → Secret-arrived case does not reconcile, gate red; (d) restore "this resolves itself once it does" in the status template → telemetry_collector_test.yaml reddens on "the no-token skip names the trigger that actually re-renders it, not a self-resolution". All four seen red, tree as shipped seen green first.
• helm-unittest: 4 status-record cases in telemetry_collector_test.yaml (via kubernetesProvider, which is what lets the skipped-no-token state be reached offline — a kube-system Namespace and no Secret), 1 script-shape case in auto_upgrade_test.yaml, and the 2-case wiring suite.

Deliberately not changed

• telemetryCollector.enabled stays tri-state with no default; the collector is not defaulted on (option 4 of the internal ticket). The refusal-when-enabled-and-no-token path is untouched.
• The existing auto-upgrade-inflight-vs-wedge.sh gate still passes unchanged: it runs the script without the TELEMETRY_* env, which now takes the fail-closed 2 branch and logs, rather than erroring under set -u.
• No installer script touched, so manifest.sha256 is unchanged.
• Log text already at latest; nothing to do became already at latest; nothing to upgrade (and … skipping → … skipping the upgrade) so the line is not contradicted by a reconcile that follows it; nothing in the repo asserted the old strings.

Checklist

• Version bumped (Chart.yaml version + appVersion)
• Tests added / updated, mutation-proved
• shellcheck -S warning -x + bash -n clean on every shell touched
• No customer names, internal paths or internal tracker refs in this title/body (tracked internally)

---

Note

Medium Risk
Changes production auto-upgrade behavior (same-version helm upgrades and server-side dry-runs) and telemetry operator messaging; heavily gated by new drift and helm-unittest checks.

Overview
Fixes telemetry Collector not appearing after the token Secret arrives when the fleet is already on the latest published chart. Collector state comes from template lookup, which only runs on helm upgrade, so hourly auto-upgrade previously did nothing on “already at latest” ticks even after jobs-manager wrote the Secret.

When there is no newer chart to install, the auto-upgrade script now runs telemetry_reconcile_verdict(): if the deployed manifest records skipped-no-token, has no Collector DaemonSet, the repo serves the current chart version, and a server-side dry-run of that version would resolve enabled, it performs a same-version helm upgrade (--reset-then-reuse-values --atomic, etc.). Tri-state behavior fails closed—ambiguous cases log and do not upgrade.

Shared naming via new helpers tracebloc.telemetryStatusName and tracebloc.telemetryStateAnnotation, with CronJob env vars wired to the status ConfigMap writer. The telemetry status reason text no longer claims the skip “resolves itself”; it names the real trigger (next auto-upgrade tick with schedule/repo from values, or manual helm upgrade when auto-upgrade is off).

Chart bumps to 1.9.118. CI/drift adds auto-upgrade-telemetry-reconcile.sh (23 stub-helm cases) and mutation proofs; Helm unit workflow runs mutation (d) for the honesty unit test.

^{Reviewed by Cursor Bugbot for commit 53ac19e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[saqlainsyed007]

The chart change is correct and approvable on the merits — I traced it and found no blocking defect. The reconcile is the right shape: not a checksum/secret annotation (there's no rendered Collector to annotate yet, and the enable decision is a template lookup that only runs inside helm upgrade), but a tri-state, fail-closed same-version helm upgrade in the auto-upgrade tick, gated on a server-side --dry-run=server that runs the chart's real cluster lookups to detect the token Secret's arrival. No restart loop (post-reconcile the stored state flips to enabled, so the next tick is a no-op), no race with the normal upgrade path (reconcile runs only in the two exit 0 branches), Secret value never logged (--hide-secret), and the reader/writer names share single helpers pinned by a new wiring suite. Test coverage is excellent — a 23-case harness that drives the rendered script plus four mutation proofs.

The only red checks are PATH persist — ubuntu:22.04 and Prereqs — ubuntu:22.04, which are the known apt-mirror flake (remediated separately in the already-approved #1051), not a defect in this PR. Once those are re-run green (or #1051 lands), this is a clean approve. I left three low-severity inline notes (activeDeadlineSeconds buffer vs. the added dry-run, a trunc 63 omission in the new name helper, and a duplicated version-parse) — all optional cleanup, none blocking.

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 53ac19e. Configure here.}