feat(recipes): add NVIDIA Cluster Readiness Engine (nvcre) component

[xdu31]

Summary

Adds the NVIDIA Cluster Readiness Engine (NVCRE) to the component registry as nvcre, with default Helm values and a chainsaw health check. Registry-only — no recipe references it yet.

Motivation / Context

NVCRE (formerly Excalibur) shipped its first public release, v0.1.0, and the chart and controller image are now public on GHCR. It validates GPU clusters with real workloads before production: it runs training and NCCL communication workloads across topology-aware node groups, measures goodput and bandwidth, and bisects failing groups to isolate individual bad nodes.

Registering it here is the prerequisite for recipes to pull it. The component is deliberately unreferenced in this PR so the registry entry, values, and health check can land and be reviewed independently of any recipe wiring.

Fixes: N/A
Related: N/A

Type of Change

• New feature (non-breaking change that adds functionality)
• Documentation update

Component(s) Affected

• Recipe engine / data (pkg/recipe)
• Docs/examples (docs/, examples/)

Implementation Notes

Naming. The entry is nvcre, matching the chart's own nameOverride, install namespace, and CRD group (nvcre.nvidia.com), rather than the longer chart name cluster-readiness-engine. Both cre and cluster-readiness-engine are valueOverrideKeys, so --set cre:... works.

hasSelfRefCRDs: true. The chart ships nvcre.nvidia.com_logprofiles.yaml in crds/ and four LogProfile CRs of that kind in templates/ (nccl-bandwidth, nccl-loopback, megatron-training, megatron-bridge). On a fresh cluster helm-diff's render pass fails with no matches for kind LogProfile because only helm install applies crds/ before rendering. Same pattern as kai-scheduler and gpu-operator (#914).

nodeScheduling carries tolerationPaths only. The chart's Deployment template renders affinity and tolerations but has no nodeSelector block, so a system node selector written into values would be silently dropped. Listing manager.nodeSelector would advertise placement control that does not exist. The toleration path still narrows placement rather than widening it: the chart default is a blanket tolerations: [{operator: Exists}], which tolerates every taint in the cluster including tainted GPU nodes; replacing it with the recipe's system tolerations keeps the manager on the intended node class. Only the manager is covered — NVCRE schedules its own benchmarks through Kubeflow Trainer, and those belong on GPU nodes.

Consumer prerequisites, documented on the registry entry and in the catalog, for whoever wires the first recipe:

• NVCRE orchestrates benchmarks by creating TrainJobs against a TrainingRuntime. The chart does not install Kubeflow Trainer, so a referencing recipe must also carry kubeflow-trainer.
• The chart renders a ServiceMonitor unconditionally under metrics.serviceMonitor.enabled (default true), so prometheus-operator-crds must land first — as a dependencyRef or inherited from base.yaml — or the value must be set to false.

fullnameOverride: nvcre in the component values keeps resource names at nvcre-* instead of the aicr-stack- release prefix. The health check asserts the Deployment by name, so the two are coupled; both files note it.

Health check asserts what a caller actually depends on rather than inferring it from pod health: the manager Deployment is ready, the certifications / workloadruns / bandwidthmeasurements CRDs are Established (without them a create fails with no matches for kind), and the cluster-scoped nccl-bandwidth LogProfile is present (without it the BandwidthMeasurement controller has no parse rules and status.results[] stays empty). readyReplicas is omitempty, so readyReplicas > 0 is used to fail closed rather than passing vacuously the way unavailableReplicas == 0 would.

Testing

make bom-docs   # +1 component, +1 image: ghcr.io/nvidia/cluster-readiness-engine/manager:v0.1.0
make qualify

Rendered the chart at the pinned version against recipes/components/nvcre/values.yaml and confirmed every name the health check asserts:

Deployment      nvcre-manager           ns=nvcre   app.kubernetes.io/name=nvcre
ServiceMonitor  nvcre-metrics-monitor   ns=nvcre
LogProfile      nccl-bandwidth, nccl-loopback, megatron-training, megatron-bridge  (cluster-scoped)

make qualify: all change-relevant packages pass — pkg/recipe 90.0%, pkg/bundler 86.4%, pkg/component 78.6%, recipes ok. Two packages fail on my macOS workstation for reasons unrelated to this change and independent of it: pkg/oci (TestHelmPinnedVersionExplicitVersionPull — local helm is v4.2.0 against the v4.2.4 pin) and tests/releasepolicy (timeout: command not found and declare: -A: invalid option under bash 3.2). No Go source changed in this PR.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert

Rollout notes: Additive and inert. No overlay or mixin references nvcre, so no existing recipe changes shape and no bundle gains a release. The only effect on existing output is the BOM's component count and one new image row.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)

[github-actions]

🌿 Preview your docs: https://nvidia-preview-feat-nvcre-component.docs.buildwithfern.com/aicr

[github-actions]

Recipe evidence check

Registry change: scoped to recipes that reference a changed component
entry in recipes/registry.yaml (not every leaf).

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

[coderabbitai]

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: aa5477f0-1e2f-409e-b4c6-bf197d09727a

📥 Commits

Reviewing files that changed from the base of the PR and between a3075b0 and 7310093.

📒 Files selected for processing (5)

• docs/user/component-catalog.md
• docs/user/container-images.md
• recipes/checks/nvcre/health-check.yaml
• recipes/components/nvcre/values.yaml
• recipes/registry.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

Adds the opt-in nvcre component with Helm chart cluster-readiness-engine:v0.1.0 in the nvcre namespace. Adds component values, scheduling configuration, and a Chainsaw health check. The check validates manager readiness, CRD establishment, the nccl-bandwidth LogProfile, and pod health. Updates the component catalog and container image inventory.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 73100

This adds an unreferenced component registry entry, values, documentation, and health check without changing existing recipe behavior; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: arangogutierrez

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description clearly explains the addition of the NVIDIA Cluster Readiness Engine component, its configuration, health check, prerequisites, testing, and rollout scope.
Title check	✅ Passed	The title clearly and concisely identifies the main change: adding the NVIDIA Cluster Readiness Engine component to recipes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[mchmarny]

Request changes: 1 MAJOR, 1 MINOR against 7310093. Reviewed-SHA CI has two KWOK scenario failures and their summary failure; the scenario logs show Go-proxy HTTP/2 download errors outside the changed component paths.

[mchmarny]

MAJOR: This implements the registry decision before that decision is recorded or accepted. A registry entry is a stable AICR support commitment: the NVCRE chart and image, seven CRDs, cluster-scoped LogProfiles and RBAC, health contract, mirror and BOM coverage, dependencies, and future upgrades all become ours to qualify, but this PR has no ADR, named consumer, adoption path, ownership boundary, or acceptance gates. Because no recipe references nvcre, merging creates the same "ghost component" state identified in PR #2500. ADR-019 establishes the registry-admission precedent.

Minimum correction: keep only a proposed NVCRE ADR in this PR. Define the intended recipe or consumer, upstream/AICR ownership, non-goals, and proportional gates for the coherent chart/image/CRD/RBAC set, Kubeflow and ServiceMonitor ordering, mirror/BOM coverage, representative certification and cleanup behavior, lifecycle, and requalification. Move the registry, values, health check, catalog, and BOM implementation to a follow-up after the ADR is accepted.

[mchmarny]

MINOR: The placement guidance uses scalar-only --set for the object-valued manager.affinity field. An invocation such as --set cre:manager.affinity=system is accepted into the bundle but renders spec.template.spec.affinity: system, which Kubernetes rejects because affinity must be an object. The documented hard-placement path therefore cannot converge.

Minimum correction: replace it with a valid structured --set-json example (or a component-values example), and add a render assertion that the resulting Deployment affinity is an object.

[mchmarny]

Request changes: 1 MAJOR new and 1 prior MAJOR, 1 prior MINOR remain against f47021f. One KWOK lane failed after a GitHub release-asset download reset; required checks pass and the branch is behind base.

[mchmarny]

MAJOR — Enroll NVCRE in the audited CRD-owner set. This component ships and owns its NVCRE CRDs, but it sets only hasSelfRefCRDs; AICR emits Flux spec.upgrade.crds: CreateReplace only when ownsCRDs is true. A Flux upgrade can therefore leave installed CRDs at the old schema while the controller and CR instances move forward. Minimum correction: set ownsCRDs: true, add the pinned chart version to the ownership audit, and cover the generated Flux upgrade policy.

[xdu31]

Superseded by #2524, which absorbs this PR in full.

Verified file by file: recipes/checks/nvcre/health-check.yaml and the regenerated docs/user/container-images.md are byte-identical; recipes/components/nvcre/values.yaml is this branch plus metrics.serviceMonitor.enabled: false; the registry entry parses to the same values field for field, including hasSelfRefCRDs: true and nodeScheduling with tolerationPaths only and no nodeSelectorPaths. #2524 adds ownsCRDs: true, nvcre as an override key, and TestNVCRERegisteredWithoutOverlay on top.

The --set / --set-json finding from the review here is also fixed in #2524's catalog row. Design rationale now lives in ADR-024 (#2541).