Skip to content

fix: forward hostNQN/DHCHAP secrets on connect and fix allowed-node scheduling - #417

Merged
boddumanohar merged 1 commit into
mainfrom
fix/dhchap-hostnqn-connect
Aug 17, 2026
Merged

boddumanohar merged 1 commit into
mainfrom
fix/dhchap-hostnqn-connect

Conversation

@boddumanohar

@boddumanohar boddumanohar commented Aug 12, 2026 •

Copy link
Copy Markdown
Member

The bug

NodeStageVolume already computes a per-Kubernetes-node host NQN (nqn.2014-08.io.simplyblock:uuid:<Node UID>) and sends it to the control plane's /connect endpoint as host_nqn — the exact NQN the operator registers into a DHCHAP pool's allowed_hosts. The control plane resolves that host's DHCHAP secret and bakes --hostnqn=/--dhchap-secret=/--dhchap-ctrl-secret= into LvolConnectResp.Connect for it.

connectViaNVMe never used any of that — it built the real nvme connect invocation from only transport/IP/port fields, with no --hostnqn and no DHCHAP secret. Without an explicit --hostnqn, nvme-cli fell back to /etc/nvme/hostnqn, seeded once per node by the CSI DaemonSet's postStart hook with an unrelated legacy NQN — a different identity than what the operator registers as allowed. Net effect: DHCHAP-gated (and any allowed_hosts-restricted) volumes could not be mounted through the CSI driver at all.

The fix

dhchapAuthArgs (csi-driver/pkg/util/initiator.go) extracts the host-identity/DHCHAP flags the control plane already computed out of LvolConnectResp.Connect and connectViaNVMe appends them to the real connect command. Pure addition of flags — non-gated pools and existing connections are unaffected.

Two more issues surfaced while verifying this live, both fixed in the same PR:

  • hostid/hostnqn kernel collision. nvme-cli falls back to the node's shared, file-seeded default hostid whenever --hostid isn't passed explicitly, regardless of --hostnqn. The kernel enforces a strict per-node 1:1 hostid↔hostnqn consistency (nvme_fabrics: found same hostid ... but different hostnqn), so a node with any pre-existing default-identity connection rejected every later connect naming an explicit, different hostnqn — exactly what DHCHAP needs. hostIDFromHostNQN now derives --hostid from --hostnqn's own UUID, so every connect on a node converges to one internally-consistent pair instead of colliding with the shared default.
  • Allowed-node scheduling gap. A DHCHAP pool's first-ever allowed node didn't get correct PersistentVolume.spec.nodeAffinity, because that value used to be read out of CreateVolume's AccessibilityRequirements — which Kubernetes only populates for topology keys already registered in the node's CSINode object at CSI plugin registration time. dhchapAllowedNodeSegment now builds the segment directly from the StorageClass's dhchap_node_label parameter instead, since the value is always the same fixed constant and the key is already known — no dependency on CSI topology registration, and no need to restart the csi-node DaemonSet pod to force it (which would have been node-wide disruptive: it services every pool's volumes on that node, not just the one that changed).

Guardian's reconnect path (resolveExpectedPathCount, recoverPathsWithANA) resolves the same per-node hostNQN via NodeHostNQN, so a DHCHAP volume's automatic reconnect after a transient path drop authenticates correctly too — not just the initial connect.

See operator/docs/designs/design-dhchap.md for the full design, worked examples (StoragePool → generated StorageClass → nvme connect command), and the exact error a future scheduling bug would produce.

e2e coverage

SPDKCSI-DHCHAP in the csi-driver e2e suite (e2e/dhchap.go), two specs:

  • Connect/auth: a DHCHAP pool allowing one host NQN — the allowed host mounts and writes successfully, an unauthorized host NQN is genuinely rejected by /connect, and the guardian reconnects a dropped path using the same authorized identity with data intact.
  • Scheduling: labels a node the way the operator labels a DHCHAP pool's allowed nodes — with no pod restart — sets the matching StorageClass parameter, pins the first pod there, asserts the PV got the expected nodeAffinity, then strips the pin and recreates the pod to confirm nodeAffinity alone keeps it on the same node.

Not yet wired into CI (no csi-driver e2e job exists yet) — for manual verification against a live cluster.

Test plan

  • go build ./..., go vet ./..., gofmt -l clean in csi-driver and operator
  • go test (both modules, envtest included for operator) passes, including TestNodeHostNQNComputesAndCaches, TestNodeHostNQNRetriesAfterFailure, TestDHCHAPAllowedNodeSegment
  • golangci-lint run clean in both modules
  • make helm-sync clean
  • Verified end-to-end on two live clusters: both SPDKCSI-DHCHAP e2e specs pass, including the no-restart scheduling fix confirmed on a fresh cluster — labeling a node with zero pod restarts still produced correct PV nodeAffinity on the first attempt

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@boddumanohar
boddumanohar marked this pull request as draft August 13, 2026 15:58
@boddumanohar
boddumanohar force-pushed the fix/dhchap-hostnqn-connect branch from a162267 to fad4a04 Compare August 13, 2026 19:42
@boddumanohar boddumanohar changed the title fix(csi): forward host NQN and DHCHAP secrets to the actual nvme connect fix: forward hostNQN/DHCHAP secrets on connect and fix allowed-node scheduling Aug 13, 2026
@boddumanohar
boddumanohar marked this pull request as ready for review August 13, 2026 19:48
Comment thread csi-driver/e2e/reconnect.go
…cheduling

DHCHAP-gated (and any allowed_hosts-restricted) volumes could not be
mounted through the CSI driver: connectViaNVMe never passed --hostnqn
or any DHCHAP secret to the real nvme connect, even though
NodeStageVolume already computed the correct per-node NQN and sbcli's
/connect response already carried those flags in
LvolConnectResp.Connect. Fixed by dhchapAuthArgs extracting them.

Also fixed two related issues found while verifying that live:

- nvme-cli falls back to the node's shared default hostid whenever
  --hostid isn't passed explicitly, regardless of --hostnqn. The
  kernel enforces a strict per-node 1:1 hostid/hostnqn consistency, so
  a node with any pre-existing default-identity connection rejected
  every later connect naming an explicit, different hostnqn.
  hostIDFromHostNQN now derives --hostid from --hostnqn's own UUID, so
  every connect on a node converges to one internally-consistent pair.
- PersistentVolume.spec.nodeAffinity for a DHCHAP pool's allowed nodes
  is now built directly from the StorageClass's dhchap_node_label
  parameter, independent of Kubernetes' CSINode-registered topology
  keys — avoids a registration-timing gap on a pool's first-ever
  allowed node without needing to restart the csi-node DaemonSet pod
  to work around it (which would have been node-wide disruptive).

Guardian's reconnect path (resolveExpectedPathCount, recoverPathsWithANA)
resolves the same per-node hostNQN via NodeHostNQN, so a DHCHAP volume's
automatic reconnect after a transient path drop authenticates correctly
too.

Adds SPDKCSI-DHCHAP e2e coverage (connect/auth + allowed-node
scheduling) and a design doc (operator/docs/designs/design-dhchap.md).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0191TsKdkt9Uj3VLNcrkecE9
@boddumanohar
boddumanohar force-pushed the fix/dhchap-hostnqn-connect branch from fad4a04 to e3b8751 Compare August 17, 2026 09:48
@boddumanohar
boddumanohar merged commit d2a9fce into main Aug 17, 2026
17 checks passed
@boddumanohar
boddumanohar deleted the fix/dhchap-hostnqn-connect branch August 17, 2026 12:24
noctarius pushed a commit that referenced this pull request Aug 17, 2026
…cheduling (#417)

DHCHAP-gated (and any allowed_hosts-restricted) volumes could not be
mounted through the CSI driver: connectViaNVMe never passed --hostnqn
or any DHCHAP secret to the real nvme connect, even though
NodeStageVolume already computed the correct per-node NQN and sbcli's
/connect response already carried those flags in
LvolConnectResp.Connect. Fixed by dhchapAuthArgs extracting them.

Also fixed two related issues found while verifying that live:

- nvme-cli falls back to the node's shared default hostid whenever
  --hostid isn't passed explicitly, regardless of --hostnqn. The
  kernel enforces a strict per-node 1:1 hostid/hostnqn consistency, so
  a node with any pre-existing default-identity connection rejected
  every later connect naming an explicit, different hostnqn.
  hostIDFromHostNQN now derives --hostid from --hostnqn's own UUID, so
  every connect on a node converges to one internally-consistent pair.
- PersistentVolume.spec.nodeAffinity for a DHCHAP pool's allowed nodes
  is now built directly from the StorageClass's dhchap_node_label
  parameter, independent of Kubernetes' CSINode-registered topology
  keys — avoids a registration-timing gap on a pool's first-ever
  allowed node without needing to restart the csi-node DaemonSet pod
  to work around it (which would have been node-wide disruptive).

Guardian's reconnect path (resolveExpectedPathCount, recoverPathsWithANA)
resolves the same per-node hostNQN via NodeHostNQN, so a DHCHAP volume's
automatic reconnect after a transient path drop authenticates correctly
too.

Adds SPDKCSI-DHCHAP e2e coverage (connect/auth + allowed-node
scheduling) and a design doc (operator/docs/designs/design-dhchap.md).

Claude-Session: https://claude.ai/code/session_0191TsKdkt9Uj3VLNcrkecE9

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit d2a9fce)
boddumanohar added a commit that referenced this pull request Sep 4, 2026
…geClass

The StorageClass the operator generates for a `dhchap: true` pool with a
non-empty `allowedNodes` cannot provision a single volume. It carries an
allowedTopologies term on the pool's node label, and external-provisioner
matches that term against the topology keys cached in the selected node's
CSINode object. Those keys are frozen when the CSI node plugin registers
(NodeGetInfo reads Node labels once), so a pool label written afterwards by
syncNodeLabels is not among them and every PVC fails, even on an allowed node:

    ProvisioningFailed: error generating accessibility requirements: topology
    map[...] from selected node "worker-1" is not in requisite:
      [map[simplyblock.io/pool.simplyblock.simplyblock-cluster.pool-a:allowed]]

Reproduced on Talos 1.12 and OpenShift RHCOS 9.6. It clears only when the
csi-node DaemonSet restarts, and forcing that re-registration was already
rejected in #417 as node-wide disruptive.

Stop setting allowedTopologies. The restriction is already carried by the
dhchap_node_label parameter, which CreateVolume turns into the provisioned PV's
nodeAffinity — what the scheduler filters Pods on and what kubelet re-checks at
every mount. Nothing else about the class changes, binding mode included.

No migration and no new RBAC: a generated DHCHAP class has never provisioned
anything, so there is no install base to repair, and this ships in the same
release as the feature.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
boddumanohar added a commit that referenced this pull request Sep 8, 2026
…geClass (#484)

* fix(operator): drop allowedTopologies from the generated DHCHAP StorageClass

The StorageClass the operator generates for a `dhchap: true` pool with a
non-empty `allowedNodes` cannot provision a single volume. It carries an
allowedTopologies term on the pool's node label, and external-provisioner
matches that term against the topology keys cached in the selected node's
CSINode object. Those keys are frozen when the CSI node plugin registers
(NodeGetInfo reads Node labels once), so a pool label written afterwards by
syncNodeLabels is not among them and every PVC fails, even on an allowed node:

    ProvisioningFailed: error generating accessibility requirements: topology
    map[...] from selected node "worker-1" is not in requisite:
      [map[simplyblock.io/pool.simplyblock.simplyblock-cluster.pool-a:allowed]]

Reproduced on Talos 1.12 and OpenShift RHCOS 9.6. It clears only when the
csi-node DaemonSet restarts, and forcing that re-registration was already
rejected in #417 as node-wide disruptive.

Stop setting allowedTopologies. The restriction is already carried by the
dhchap_node_label parameter, which CreateVolume turns into the provisioned PV's
nodeAffinity — what the scheduler filters Pods on and what kubelet re-checks at
every mount. Nothing else about the class changes, binding mode included.

No migration and no new RBAC: a generated DHCHAP class has never provisioned
anything, so there is no install base to repair, and this ships in the same
release as the feature.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(e2e): drive the DHCHAP scheduling spec through a StoragePool CR

The spec built its own StorageClass with createStorageClassWithParams,
which clones the plain pool StorageClass (AllowedTopologies empty) and
injects dhchap_node_label. So it reconstructed half of what the operator
generates and never touched the other half — which is how a class that
could not provision a single volume passed CI for a release.

Create a StoragePool CR with dhchap + allowedNodes instead and use the
StorageClass the operator generates from it, asserting that class carries
dhchap_node_label and no allowedTopologies before provisioning through
it. The operator now supplies the node label and the backend allowed-host
entry too, so the sbctl pool/add-host and setNodeLabel scaffolding goes
away with it.

Skipped outside OPERATOR_MODE, where nothing reconciles the CR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore(operator): revert the dhchap CRD description change

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor: rename the DHCHAP StorageClass parameter to dhchap_node_selector

The operator now writes dhchap_node_selector on the classes it generates,
and CreateVolume prefers it. dhchap_node_label stays readable as a
deprecated alias so a StorageClass written before the rename keeps gating
its volumes: StorageClass parameters are immutable, so such a class cannot
be migrated in place, only replaced.

dhchap_node_selector wins when a class carries both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor: drop the dhchap_node_label alias

dhchap_node_label is in no released tag (v26.2.7, v26.2.8, v26.2.8-3 all
lack it), so nothing installable reads it. A class the operator generated
before the rename carries the unsatisfiable allowedTopologies term too and
never provisioned a volume, and being create-only it is not rewritten on
upgrade, so the alias rescued nothing there either.

That leaves only hand-written classes on a main build, which cost one
re-apply. Renaming before the first release avoids owing a deprecation
cycle for a name that never shipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants