RFE: Declarative NIC configuration CRD for Pollara 400 (auto-neg, PFC, QoS, DCQCN)

[leo8a]

Problem

Configuring Pollara 400 NICs for RDMA workloads (auto-negotiation, PFC, QoS, DCQCN) requires imperative nicctl commands per node. These settings don't persist across reboots unless NIC personas are used — and even with personas, applying them is still an imperative, per-node operation that is error-prone at scale.

Proposal

A new CRD (e.g., NicPolicy, NicConfig) that allows users to declaratively define NIC parameters and lets the operator reconcile them. Scope could include:

• Port: auto-negotiation, MTU, FEC, pause type
• QoS: DSCP/PCP classification, PFC no-drop priorities, scheduling (DWRR/SPQ)
• DCQCN: congestion control parameters per RDMA device

Benefits

• Persistence — operator reconciles the desired state, after reboot or firmware changes for persistence
• Validation — reject invalid combinations before applying (e.g., no-drop on an unmapped priority)
• Observability — CRD status reports applied state per node and drift detection
• Day-2 ops — config changes via kubectl apply, not SSH into each node
• Composability — different policies for different node groups (training vs inference)

Context

Running RDMA benchmarks on Pollara 400 requires several nicctl commands per node covering auto-neg, PFC, QoS, and DCQCN configuration. The full set of required parameters is documented in the AMD AI NIC Pollara 400 Ops Guide (UG1801, Sections: QoS Configuration, DCQCN Configuration) and the AMD AI NIC Benchmarking Guide (UG1813, Section: Host Configuration).

[leo8a]

/cc @yansun1996 @sajmera-pensando

[leo8a]

One additional benefit worth calling out for the CRD approach: admission webhook validation.

With a dedicated CRD, the operator can implement validating admission webhooks that intercept resources at creation and update time. Users get immediate feedback — right at kubectl apply — when they submit an invalid or unsupported NIC configuration, rather than discovering the problem later in controller logs or after a partially-applied state leaves the NIC inconsistent.

From my experience with other hardware-level operators, CRDs backed by admission webhooks have proven significantly more robust against user configuration errors than imperative tooling or ConfigMap-based approaches:

• Instant feedback — invalid values or unsupported parameter combinations (e.g., enabling PFC on a priority with no QoS mapping) are rejected before reaching the reconciler.
• Reduced blast radius — catching errors at admission time prevents degraded retry loops or half-configured nodes.
• Self-documenting constraints — the webhook logic codifies which configurations the hardware/firmware actually supports, evolving alongside NIC firmware releases.

This complements the validation point already in the proposal and, in practice, is one of the strongest arguments for a proper CRD over alternative configuration mechanisms.