Skip to content

fix(bundler): stamp AICR and payload versions into generated wrappers - #2571

Merged
mchmarny merged 1 commit into
mainfrom
fix/wrapper-chart-versions
Sep 4, 2026
Merged

fix(bundler): stamp AICR and payload versions into generated wrappers#2571
mchmarny merged 1 commit into
mainfrom
fix/wrapper-chart-versions

Conversation

@mchmarny

@mchmarny mchmarny commented Sep 4, 2026

Copy link
Copy Markdown
Member

Summary

Generated wrapper charts hardcoded version: 0.1.0, so every Kustomize-derived component, manifest-only component, and injected -pre / -post / -readiness release reported the same fictional version for both the artifact and its payload. Per ADR-021 Decision 7, version: now carries the AICR build that produced the wrapper while appVersion and a new aicr.run/component-version annotation carry the payload version.

Motivation / Context

helm list -A enumerates every component of an AICR-deployed stack by name but reports 0.1.0 for the generated ones — the real payload version existed only inside a vendored wrapper's dependencies: entry, and nowhere at all for the rest. The single version: field was being asked two questions at once and answering neither.

Fixes: #2526
Related: #2424

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • Build/CI/tooling

Component(s) Affected

  • CLI (cmd/aicr, pkg/cli)
  • API server (cmd/aicrd, pkg/server)
  • Recipe engine / data (pkg/recipe)
  • Bundlers (pkg/bundler, pkg/component/*)
  • Collectors / snapshotter (pkg/collector, pkg/snapshotter)
  • Validator (pkg/validator)
  • Core libraries (pkg/errors, pkg/k8s) — pkg/defaults
  • Docs/examples (docs/, examples/)

Implementation Notes

Both questions get their own field. A generated Chart.yaml now emits:

version: "1.4.0"            # the AICR build that produced the wrapper
appVersion: "v25.3.0"       # the payload pin
annotations:
  aicr.run/component-version: "v25.3.0"
  aicr.run/generated-by: "1.4.0"

localformat.stampFor derives the payload version per folder: the upstream chart pin for Helm components, the git ref for Kustomize ones, and — for a component with no upstream pin (manifest-only, and the injected releases belonging to one) — AICR's own version, since that content is AICR's. Injected -pre / -post / -readiness wrappers inherit their parent's pin, so a gpu-operator-post release reports v25.3.0. The annotation is therefore always populated, and a reader never has to distinguish "absent" from "empty".

The payload version is deliberately not normalized — a Kustomize ref like release-1.4 must not have to masquerade as SemVer, and matching the recipe pin byte-for-byte is what makes it comparable to one.

The dev-build fold is load-bearing, and I confirmed the failure it prevents. NormalizeChartVersion folds any non-SemVer build version to defaults.DevChartVersion (0.0.0-dev). Against a real helm binary:

$ helm lint <chart with version: dev>
[ERROR] templates/: validation: chart.metadata.version "dev" is invalid
Error: 1 chart(s) linted, 1 chart(s) failed

The parse is the lenient semver.NewVersion — the same call Helm makes when loading a chart — rather than StrictNewVersion, so anything the helper returns will load.

Scope beyond the issue's two files, and why. The issue names the two localformat templates. Two adjacent charts have the same defect and would otherwise be left reporting 0.1.0 (or failing to load) on a subset of deployers:

  • Flux's own manifest-chart template (flux/templates/chart.yaml.tmpl) also hardcoded 0.1.0. Without it, --deployer flux bundles would be the one place the acceptance criterion did not hold.
  • The argocd-helm root bundle chart already derived its version from the recipe's AICR version, but through NormalizeVersionWithDefault, which passes dev straight through. I verified on main that aicr bundle --deployer argocd-helm from a dev build emits version: "dev" and that the README's own helm install aicr-bundle . then fails to load the chart. One-line switch to NormalizeChartVersion; a regression case is added to TestGenerate_ChartVersion.

The argocd-helm root chart deliberately does not join this contract. Its version: comes from the recipe's metadata.version — the AICR build that generated the recipe — while the wrappers inside the same bundle are stamped from the build that ran bundle. Those differ on the ordinary two-step workflow (generate a recipe once, bundle it later with a newer CLI), so one argocd-helm bundle can show two AICR versions. I left the behavior alone rather than unifying it: that field also defaults the Argo CD targetRevision and the OCI push tag, so repointing it is a user-visible change outside this issue. The root chart carries neither annotation, so nothing reads it as the wrapper-provenance signal. Both the docs and chartVersion()'s comment now state the distinction explicitly.

No deployer path read the wrapper version: before this repurposed it (the issue's last acceptance bullet). Confirmed for all five: helmfile sets Release.Version only for KindUpstreamHelm folders (f.Upstream.Version) and emits local charts as chart: ./NNN-dir with no version; Argo CD path-based Applications use the git targetRevision; Flux HelmReleases for generated charts leave HelmReleaseData.Version empty and reference chart: ./<name>; install-local-helm.sh installs ./ with no --version; and the only code that parses a Chart.yaml version (pkg/oci, for the OCI push tag) reads the argocd-helm root chart, not a wrapper.

Annotation keys derive from header.Domain per ADR-013 rather than being free-standing literals. The templates spell the keys out, because a templated key reads worse than the value it labels; TestAnnotationKeysMatchTemplates is what keeps the two in sync, and fails loudly if a domain migration updates only one side.

Quoting. q (a strconv.Quote template func, now on the shared deployer.TemplateFuncs) emits YAML double-quoted scalars. This is not cosmetic: an unquoted appVersion: 1.2 parses as a float and fails to unmarshal into Helm's string field. Same reasoning as the %q quoting already in the argocd-helm Chart.yaml writer (#1034).

The 0.0.0-dev fold is lossy, so it logs. Any non-SemVer build version folds to the same placeholder, which means a corrupted or hand-edited metadata.version becomes indistinguishable from a genuine dev build. Erroring would block bundling over cosmetic metadata, so NormalizeChartVersion emits slog.Warn for the unexpected shapes while staying quiet for the two documented sentinels ("" and dev).

Reproducibility is unaffected in the sense the issue describes — the AICR version already travels in every bundle via recipe.yaml. Worth noting explicitly: helmfile does not emit recipe.yaml, so for that deployer, bundles containing generated wrappers now vary by AICR version where they previously did not. TestStockRenderParityGolden pins the builder and bundler versions, so the golden digest stays a pure function of the catalog and the render.

Testing

make qualify   # exit 0 on the final tree

New coverage:

  • TestNormalizeChartVersion — table over release tags, pre-releases, build metadata, dev, empty, and branch-style refs. Every case additionally asserts the result parses as SemVer, so the invariant holds independently of the expectation.
  • TestStampFor — payload derivation per component shape.
  • TestWrite_StampsGeneratedWrappers — all five folder shapes in one Write, asserting the parsed Chart.yaml fields.
  • TestWrite_DevBuildProducesHelmValidCharts — walks every emitted Chart.yaml and fails on any non-SemVer version:.
  • TestAnnotationKeysMatchTemplates — template/constant sync guard across both localformat templates.
  • TestRenderWrapperChartYAML_QuotesHostileVersion — a payload version carrying " and a newline must not escape its scalar, and must round-trip verbatim into appVersion and the annotation.
  • TestGenerate_StampsGeneratedCharts / _StampsVendoredWrapper / _DevBuildProducesHelmValidCharts (flux).
  • TestGenerate_ChartVersion gains the unstamped-dev case (argocd-helm).
  • The helmfile manifest_only golden now pins 001-node-prep/Chart.yaml, which was the one deployer whose wiring no golden covered.

Each guard was mutation-tested: reverting NormalizeChartVersion to a pass-through, dropping AICRVersion from helmfile's localformat.Options, and renaming the annotation key in a template each produce the expected failure.

Beyond the unit tests, I generated real bundles from a dev build and ran the local helm binary against them: helm lint and helm template both succeed on the generated helm wrapper and on the argocd-helm root chart, where the latter fails on main.

Coverage (per-package, vs origin/main):

Package Before After
pkg/bundler/deployer 93.5% 93.8%
pkg/bundler/deployer/localformat 81.3% 81.9%
pkg/bundler/deployer/flux 88.9% 89.0%
pkg/bundler/deployer/{argocd,argocdhelm,helm,helmfile} unchanged
pkg/defaults 100% 100%

ADR-021's ## Status now records that Decision 7 shipped and marks step 2 of its Implementation Plan, so a contributor scoping the remaining decisions is not misled. Its problem statements are left as written at proposal time — they describe the state the ADR reasoned about, not the state of main.

Risk Assessment

  • Low — Isolated change, well-tested, easy to revert
  • Medium — Touches multiple components or has broader impact
  • High — Breaking change, affects critical paths, or complex rollout

Every deployer's generated-chart output changes, so bundle bytes and checksums.txt move for any recipe with a generated wrapper. Rendered Kubernetes manifests are byte-identical, so there is no resource churn on redeploy.

Rollout notes: One consequence the issue calls out and accepts: with version: tracking the AICR version, a release that changes nothing material still shows a chart-version bump in helm diff. Since rendered manifests are unchanged, that surfaces as a metadata-only diff, not a resource replacement. No migration steps; the annotations are additive and nothing reads them yet — upgrade-check and the rest of #2424 remain post-v1.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

Generated wrapper charts hardcoded version 0.1.0, so every Kustomize-derived,
manifest-only, and injected -pre/-post/-readiness release reported the same
fictional version for both the artifact and its payload.

Per ADR-021 Decision 7, version: now carries the AICR build that produced the
wrapper, while appVersion and the new aicr.run/component-version annotation
carry the payload version. Non-SemVer build versions fold to 0.0.0-dev so an
unstamped build still produces a chart Helm will load.

Signed-off-by: Mark Chmarny <mark@chmarny.com>
@mchmarny mchmarny added the theme/deployer Helm, ArgoCD, and deployment bundle generation label Sep 4, 2026
@mchmarny mchmarny self-assigned this Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Recipe evidence check

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 83c234d4-79b2-4c60-89a1-42f5131d376c

📥 Commits

Reviewing files that changed from the base of the PR and between 5cda08f and 1c1eda0.

📒 Files selected for processing (41)
  • docs/design/021-component-upgrade-safety.md
  • docs/user/bundling.md
  • pkg/bundler/deployer/argocd/argocd.go
  • pkg/bundler/deployer/argocd/testdata/helm_and_manifest_only/002-nodewright-customizations/Chart.yaml
  • pkg/bundler/deployer/argocd/testdata/mixed_component/002-gpu-operator-post/Chart.yaml
  • pkg/bundler/deployer/argocd/testdata/mixed_with_pre/001-gpu-operator-pre/Chart.yaml
  • pkg/bundler/deployer/argocd/testdata/mixed_with_pre/003-gpu-operator-post/Chart.yaml
  • pkg/bundler/deployer/argocd/testdata/readiness_gate/002-gpu-operator-readiness/Chart.yaml
  • pkg/bundler/deployer/argocdhelm/argocdhelm.go
  • pkg/bundler/deployer/argocdhelm/argocdhelm_test.go
  • pkg/bundler/deployer/argocdhelm/testdata/helm_and_manifest_only/002-nodewright-customizations/Chart.yaml
  • pkg/bundler/deployer/flux/flux.go
  • pkg/bundler/deployer/flux/helm.go
  • pkg/bundler/deployer/flux/stamp_test.go
  • pkg/bundler/deployer/flux/templates/chart.yaml.tmpl
  • pkg/bundler/deployer/helm/helm.go
  • pkg/bundler/deployer/helm/testdata/manifest_only/001-skyhook-customizations/Chart.yaml
  • pkg/bundler/deployer/helm/testdata/mixed_gpu_operator/001-gpu-operator-pre/Chart.yaml
  • pkg/bundler/deployer/helm/testdata/mixed_gpu_operator/003-gpu-operator-post/Chart.yaml
  • pkg/bundler/deployer/helm/testdata/mixed_with_pre/001-foo-pre/Chart.yaml
  • pkg/bundler/deployer/helmfile/helmfile.go
  • pkg/bundler/deployer/helmfile/helmfile_test.go
  • pkg/bundler/deployer/helmfile/testdata/manifest_only/001-node-prep/Chart.yaml
  • pkg/bundler/deployer/helpers.go
  • pkg/bundler/deployer/helpers_test.go
  • pkg/bundler/deployer/localformat/doc.go
  • pkg/bundler/deployer/localformat/empty_render_test.go
  • pkg/bundler/deployer/localformat/local_helm.go
  • pkg/bundler/deployer/localformat/stamp.go
  • pkg/bundler/deployer/localformat/stamp_test.go
  • pkg/bundler/deployer/localformat/stamp_write_test.go
  • pkg/bundler/deployer/localformat/templates/chart.yaml.tmpl
  • pkg/bundler/deployer/localformat/templates/wrapper-chart.yaml.tmpl
  • pkg/bundler/deployer/localformat/testdata/local_helm_manifest_only/001-skyhook-customizations/Chart.yaml
  • pkg/bundler/deployer/localformat/vendor_folder.go
  • pkg/bundler/deployer/localformat/vendor_wrapper.go
  • pkg/bundler/deployer/localformat/vendor_wrapper_test.go
  • pkg/bundler/deployer/localformat/writer.go
  • pkg/bundler/deployer/localformat/writer_test.go
  • pkg/bundler/testdata/stock_render_golden.yaml
  • pkg/defaults/agent.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The bundlers now stamp generated Helm charts with a normalized AICR version and a separate payload version. Local, vendored, Flux, Argo CD, Helm, and Helmfile generation paths pass this metadata through chart templates. Development builds use 0.0.0-dev for Helm compatibility. Tests, golden outputs, ADR content, and user documentation cover the new metadata and fallback behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to 1c1ed

Generated wrapper metadata consistently separates Helm-valid AICR versions from payload versions across supported deployers and tests. No actionable merge risk remains.

Suggested reviewers: almaslennikov

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes satisfy issue #2526 by separating AICR and payload versions, adding required annotations, normalizing non-SemVer builds to 0.0.0-dev, updating deployer paths, tests, golden files, and docu…
Out of Scope Changes check ✅ Passed The additional Flux, argocd-helm, documentation, ADR, defaults, test, and golden-file changes support the stated versioning requirements and are not unrelated.
Title check ✅ Passed The title clearly and concisely summarizes the main change: stamping AICR and payload versions into generated wrappers.
Description check ✅ Passed The description directly explains the version-stamping changes, motivation, implementation, testing, documentation updates, and affected bundlers.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/wrapper-chart-versions

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 84.2%
Threshold 80%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-84.2%25-brightgreen)

Merging this branch will increase overall coverage

Impacted Packages Coverage Δ 🤖
github.com/NVIDIA/aicr/pkg/bundler/deployer 93.94% (+0.39%) 👍
github.com/NVIDIA/aicr/pkg/bundler/deployer/argocd 88.26% (ø)
github.com/NVIDIA/aicr/pkg/bundler/deployer/argocdhelm 89.28% (ø)
github.com/NVIDIA/aicr/pkg/bundler/deployer/flux 88.96% (+0.07%) 👍
github.com/NVIDIA/aicr/pkg/bundler/deployer/helm 88.55% (ø)
github.com/NVIDIA/aicr/pkg/bundler/deployer/helmfile 88.85% (ø)
github.com/NVIDIA/aicr/pkg/bundler/deployer/localformat 81.92% (+0.63%) 👍
github.com/NVIDIA/aicr/pkg/defaults 100.00% (ø)

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/NVIDIA/aicr/pkg/bundler/deployer/argocd/argocd.go 88.26% (ø) 264 233 31
github.com/NVIDIA/aicr/pkg/bundler/deployer/argocdhelm/argocdhelm.go 89.28% (ø) 886 791 95
github.com/NVIDIA/aicr/pkg/bundler/deployer/flux/flux.go 88.30% (ø) 359 317 42
github.com/NVIDIA/aicr/pkg/bundler/deployer/flux/helm.go 89.02% (+0.17%) 264 (+4) 235 (+4) 29 👍
github.com/NVIDIA/aicr/pkg/bundler/deployer/helm/helm.go 88.55% (ø) 131 116 15
github.com/NVIDIA/aicr/pkg/bundler/deployer/helmfile/helmfile.go 84.24% (ø) 184 155 29
github.com/NVIDIA/aicr/pkg/bundler/deployer/helpers.go 93.10% (+0.51%) 87 (+6) 81 (+6) 6 👍
github.com/NVIDIA/aicr/pkg/bundler/deployer/localformat/doc.go 0.00% (ø) 0 0 0
github.com/NVIDIA/aicr/pkg/bundler/deployer/localformat/local_helm.go 86.05% (ø) 86 74 12
github.com/NVIDIA/aicr/pkg/bundler/deployer/localformat/stamp.go 100.00% (+100.00%) 6 (+6) 6 (+6) 0 🌟
github.com/NVIDIA/aicr/pkg/bundler/deployer/localformat/vendor_folder.go 75.00% (ø) 64 48 16
github.com/NVIDIA/aicr/pkg/bundler/deployer/localformat/vendor_wrapper.go 88.89% (+11.97%) 18 (+5) 16 (+6) 2 (-1) 🎉
github.com/NVIDIA/aicr/pkg/bundler/deployer/localformat/writer.go 79.49% (+1.10%) 273 217 (+3) 56 (-3) 👍
github.com/NVIDIA/aicr/pkg/defaults/agent.go 100.00% (ø) 8 8 0

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

@mchmarny
mchmarny marked this pull request as ready for review September 4, 2026 10:20
@mchmarny
mchmarny requested a review from a team as a code owner September 4, 2026 10:20
@mchmarny
mchmarny enabled auto-merge (squash) September 4, 2026 10:21
@mchmarny
mchmarny merged commit 7469fdb into main Sep 4, 2026
76 of 77 checks passed
@mchmarny
mchmarny deleted the fix/wrapper-chart-versions branch September 4, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/bundler area/docs size/XL theme/deployer Helm, ArgoCD, and deployment bundle generation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stamp AICR and payload versions into generated wrapper charts

2 participants