fix: forward hostNQN/DHCHAP secrets on connect and fix allowed-node scheduling - #417
Merged
Merged
Conversation
3 tasks done
boddumanohar
marked this pull request as draft
August 13, 2026 15:58
boddumanohar
force-pushed
the
fix/dhchap-hostnqn-connect
branch
from
August 13, 2026 19:42
a162267 to
fad4a04
Compare
boddumanohar
marked this pull request as ready for review
August 13, 2026 19:48
noctarius
requested changes
Aug 17, 2026
…cheduling DHCHAP-gated (and any allowed_hosts-restricted) volumes could not be mounted through the CSI driver: connectViaNVMe never passed --hostnqn or any DHCHAP secret to the real nvme connect, even though NodeStageVolume already computed the correct per-node NQN and sbcli's /connect response already carried those flags in LvolConnectResp.Connect. Fixed by dhchapAuthArgs extracting them. Also fixed two related issues found while verifying that live: - nvme-cli falls back to the node's shared default hostid whenever --hostid isn't passed explicitly, regardless of --hostnqn. The kernel enforces a strict per-node 1:1 hostid/hostnqn consistency, so a node with any pre-existing default-identity connection rejected every later connect naming an explicit, different hostnqn. hostIDFromHostNQN now derives --hostid from --hostnqn's own UUID, so every connect on a node converges to one internally-consistent pair. - PersistentVolume.spec.nodeAffinity for a DHCHAP pool's allowed nodes is now built directly from the StorageClass's dhchap_node_label parameter, independent of Kubernetes' CSINode-registered topology keys — avoids a registration-timing gap on a pool's first-ever allowed node without needing to restart the csi-node DaemonSet pod to work around it (which would have been node-wide disruptive). Guardian's reconnect path (resolveExpectedPathCount, recoverPathsWithANA) resolves the same per-node hostNQN via NodeHostNQN, so a DHCHAP volume's automatic reconnect after a transient path drop authenticates correctly too. Adds SPDKCSI-DHCHAP e2e coverage (connect/auth + allowed-node scheduling) and a design doc (operator/docs/designs/design-dhchap.md). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0191TsKdkt9Uj3VLNcrkecE9
boddumanohar
force-pushed
the
fix/dhchap-hostnqn-connect
branch
from
August 17, 2026 09:48
fad4a04 to
e3b8751
Compare
noctarius
approved these changes
Aug 17, 2026
geoffrey1330
approved these changes
Aug 17, 2026
noctarius
pushed a commit
that referenced
this pull request
Aug 17, 2026
…cheduling (#417) DHCHAP-gated (and any allowed_hosts-restricted) volumes could not be mounted through the CSI driver: connectViaNVMe never passed --hostnqn or any DHCHAP secret to the real nvme connect, even though NodeStageVolume already computed the correct per-node NQN and sbcli's /connect response already carried those flags in LvolConnectResp.Connect. Fixed by dhchapAuthArgs extracting them. Also fixed two related issues found while verifying that live: - nvme-cli falls back to the node's shared default hostid whenever --hostid isn't passed explicitly, regardless of --hostnqn. The kernel enforces a strict per-node 1:1 hostid/hostnqn consistency, so a node with any pre-existing default-identity connection rejected every later connect naming an explicit, different hostnqn. hostIDFromHostNQN now derives --hostid from --hostnqn's own UUID, so every connect on a node converges to one internally-consistent pair. - PersistentVolume.spec.nodeAffinity for a DHCHAP pool's allowed nodes is now built directly from the StorageClass's dhchap_node_label parameter, independent of Kubernetes' CSINode-registered topology keys — avoids a registration-timing gap on a pool's first-ever allowed node without needing to restart the csi-node DaemonSet pod to work around it (which would have been node-wide disruptive). Guardian's reconnect path (resolveExpectedPathCount, recoverPathsWithANA) resolves the same per-node hostNQN via NodeHostNQN, so a DHCHAP volume's automatic reconnect after a transient path drop authenticates correctly too. Adds SPDKCSI-DHCHAP e2e coverage (connect/auth + allowed-node scheduling) and a design doc (operator/docs/designs/design-dhchap.md). Claude-Session: https://claude.ai/code/session_0191TsKdkt9Uj3VLNcrkecE9 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> (cherry picked from commit d2a9fce)
1 task done
boddumanohar
added a commit
that referenced
this pull request
Sep 4, 2026
…geClass
The StorageClass the operator generates for a `dhchap: true` pool with a
non-empty `allowedNodes` cannot provision a single volume. It carries an
allowedTopologies term on the pool's node label, and external-provisioner
matches that term against the topology keys cached in the selected node's
CSINode object. Those keys are frozen when the CSI node plugin registers
(NodeGetInfo reads Node labels once), so a pool label written afterwards by
syncNodeLabels is not among them and every PVC fails, even on an allowed node:
ProvisioningFailed: error generating accessibility requirements: topology
map[...] from selected node "worker-1" is not in requisite:
[map[simplyblock.io/pool.simplyblock.simplyblock-cluster.pool-a:allowed]]
Reproduced on Talos 1.12 and OpenShift RHCOS 9.6. It clears only when the
csi-node DaemonSet restarts, and forcing that re-registration was already
rejected in #417 as node-wide disruptive.
Stop setting allowedTopologies. The restriction is already carried by the
dhchap_node_label parameter, which CreateVolume turns into the provisioned PV's
nodeAffinity — what the scheduler filters Pods on and what kubelet re-checks at
every mount. Nothing else about the class changes, binding mode included.
No migration and no new RBAC: a generated DHCHAP class has never provisioned
anything, so there is no install base to repair, and this ships in the same
release as the feature.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
boddumanohar
added a commit
that referenced
this pull request
Sep 8, 2026
…geClass (#484) * fix(operator): drop allowedTopologies from the generated DHCHAP StorageClass The StorageClass the operator generates for a `dhchap: true` pool with a non-empty `allowedNodes` cannot provision a single volume. It carries an allowedTopologies term on the pool's node label, and external-provisioner matches that term against the topology keys cached in the selected node's CSINode object. Those keys are frozen when the CSI node plugin registers (NodeGetInfo reads Node labels once), so a pool label written afterwards by syncNodeLabels is not among them and every PVC fails, even on an allowed node: ProvisioningFailed: error generating accessibility requirements: topology map[...] from selected node "worker-1" is not in requisite: [map[simplyblock.io/pool.simplyblock.simplyblock-cluster.pool-a:allowed]] Reproduced on Talos 1.12 and OpenShift RHCOS 9.6. It clears only when the csi-node DaemonSet restarts, and forcing that re-registration was already rejected in #417 as node-wide disruptive. Stop setting allowedTopologies. The restriction is already carried by the dhchap_node_label parameter, which CreateVolume turns into the provisioned PV's nodeAffinity — what the scheduler filters Pods on and what kubelet re-checks at every mount. Nothing else about the class changes, binding mode included. No migration and no new RBAC: a generated DHCHAP class has never provisioned anything, so there is no install base to repair, and this ships in the same release as the feature. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(e2e): drive the DHCHAP scheduling spec through a StoragePool CR The spec built its own StorageClass with createStorageClassWithParams, which clones the plain pool StorageClass (AllowedTopologies empty) and injects dhchap_node_label. So it reconstructed half of what the operator generates and never touched the other half — which is how a class that could not provision a single volume passed CI for a release. Create a StoragePool CR with dhchap + allowedNodes instead and use the StorageClass the operator generates from it, asserting that class carries dhchap_node_label and no allowedTopologies before provisioning through it. The operator now supplies the node label and the backend allowed-host entry too, so the sbctl pool/add-host and setNodeLabel scaffolding goes away with it. Skipped outside OPERATOR_MODE, where nothing reconciles the CR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(operator): revert the dhchap CRD description change Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: rename the DHCHAP StorageClass parameter to dhchap_node_selector The operator now writes dhchap_node_selector on the classes it generates, and CreateVolume prefers it. dhchap_node_label stays readable as a deprecated alias so a StorageClass written before the rename keeps gating its volumes: StorageClass parameters are immutable, so such a class cannot be migrated in place, only replaced. dhchap_node_selector wins when a class carries both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: drop the dhchap_node_label alias dhchap_node_label is in no released tag (v26.2.7, v26.2.8, v26.2.8-3 all lack it), so nothing installable reads it. A class the operator generated before the rename carries the unsatisfiable allowedTopologies term too and never provisioned a volume, and being create-only it is not rewritten on upgrade, so the alias rescued nothing there either. That leaves only hand-written classes on a main build, which cost one re-apply. Renaming before the first release avoids owing a deprecation cycle for a name that never shipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
NodeStageVolumealready computes a per-Kubernetes-node host NQN (nqn.2014-08.io.simplyblock:uuid:<Node UID>) and sends it to the control plane's/connectendpoint ashost_nqn— the exact NQN the operator registers into a DHCHAP pool'sallowed_hosts. The control plane resolves that host's DHCHAP secret and bakes--hostnqn=/--dhchap-secret=/--dhchap-ctrl-secret=intoLvolConnectResp.Connectfor it.connectViaNVMenever used any of that — it built the realnvme connectinvocation from only transport/IP/port fields, with no--hostnqnand no DHCHAP secret. Without an explicit--hostnqn,nvme-clifell back to/etc/nvme/hostnqn, seeded once per node by the CSI DaemonSet'spostStarthook with an unrelated legacy NQN — a different identity than what the operator registers as allowed. Net effect: DHCHAP-gated (and anyallowed_hosts-restricted) volumes could not be mounted through the CSI driver at all.The fix
dhchapAuthArgs(csi-driver/pkg/util/initiator.go) extracts the host-identity/DHCHAP flags the control plane already computed out ofLvolConnectResp.ConnectandconnectViaNVMeappends them to the real connect command. Pure addition of flags — non-gated pools and existing connections are unaffected.Two more issues surfaced while verifying this live, both fixed in the same PR:
nvme-clifalls back to the node's shared, file-seeded default hostid whenever--hostidisn't passed explicitly, regardless of--hostnqn. The kernel enforces a strict per-node 1:1 hostid↔hostnqn consistency (nvme_fabrics: found same hostid ... but different hostnqn), so a node with any pre-existing default-identity connection rejected every later connect naming an explicit, different hostnqn — exactly what DHCHAP needs.hostIDFromHostNQNnow derives--hostidfrom--hostnqn's own UUID, so every connect on a node converges to one internally-consistent pair instead of colliding with the shared default.PersistentVolume.spec.nodeAffinity, because that value used to be read out ofCreateVolume'sAccessibilityRequirements— which Kubernetes only populates for topology keys already registered in the node'sCSINodeobject at CSI plugin registration time.dhchapAllowedNodeSegmentnow builds the segment directly from the StorageClass'sdhchap_node_labelparameter instead, since the value is always the same fixed constant and the key is already known — no dependency on CSI topology registration, and no need to restart the csi-node DaemonSet pod to force it (which would have been node-wide disruptive: it services every pool's volumes on that node, not just the one that changed).Guardian's reconnect path (
resolveExpectedPathCount,recoverPathsWithANA) resolves the same per-node hostNQN viaNodeHostNQN, so a DHCHAP volume's automatic reconnect after a transient path drop authenticates correctly too — not just the initial connect.See
operator/docs/designs/design-dhchap.mdfor the full design, worked examples (StoragePool→ generatedStorageClass→nvme connectcommand), and the exact error a future scheduling bug would produce.e2e coverage
SPDKCSI-DHCHAPin the csi-driver e2e suite (e2e/dhchap.go), two specs:/connect, and the guardian reconnects a dropped path using the same authorized identity with data intact.nodeAffinity, then strips the pin and recreates the pod to confirmnodeAffinityalone keeps it on the same node.Not yet wired into CI (no csi-driver e2e job exists yet) — for manual verification against a live cluster.
Test plan
go build ./...,go vet ./...,gofmt -lclean incsi-driverandoperatorgo test(both modules, envtest included foroperator) passes, includingTestNodeHostNQNComputesAndCaches,TestNodeHostNQNRetriesAfterFailure,TestDHCHAPAllowedNodeSegmentgolangci-lint runclean in both modulesmake helm-synccleanSPDKCSI-DHCHAPe2e specs pass, including the no-restart scheduling fix confirmed on a fresh cluster — labeling a node with zero pod restarts still produced correct PVnodeAffinityon the first attempt🤖 Generated with Claude Code