Summary
A volume created in a dhchap-enabled pool that has no entries in allowed_hosts is created with allow_any_host=True and no DHCHAP material on its subsystem — so any host may connect, with no authentication. That is the ordinary default posture for an unrestricted pool, but here the pool reports dhchap: true, so the volume looks protected and is not.
Worse, the state is sticky: registering hosts later does not fix those volumes, while an unrelated storage-node restart silently does.
Found while investigating a Kubernetes-side report where a Pod on a deliberately excluded node mounted a DHCHAP pool's volume and reached Running. This is the most plausible mechanical explanation.
Trace
Volume creation, with pool.dhchap = True and pool.allowed_hosts == []:
create_lvol — if pool.dhchap: standalone_allowed_hosts = [{"nqn": h} for h in pool.allowed_hosts] → [] (simplyblock_core/controllers/lvol_controller.py:696-700)
claim_lvol_ns_slot — lvol.allowed_hosts = standalone_allowed_hosts → [] (simplyblock_core/db_controller.py:907-908)
add_lvol_on_node — allow_any = not bool(lvol.allowed_hosts) → True, so subsystem_create(..., allow_any_host=True) and the whole if lvol.allowed_hosts: block (pool key registration + subsystem_add_host with dhchap_key/dhchap_ctrlr_key) is skipped (lvol_controller.py:1216-1266)
/connect — HostConnectAuth.resolve returns None on an empty allowed_hosts (simplyblock_core/utils/nvme.py:75-76), so the connect command carries no DHCHAP secrets
Why it does not self-correct
allow_any_host is only ever supplied at subsystem_create. Nothing patches it afterwards:
add_host_to_lvol (lvol_controller.py:4222) calls subsystem_add_host with the pool's keys but never clears allow_any_host. Since the host ACL is only consulted when allow_any_host is false, adding hosts to such a subsystem changes nothing about who can connect.
remove_host_from_lvol (lvol_controller.py:4333) likewise never sets it.
So pool add-host (or the operator adding a node to StoragePool.spec.allowedNodes) leaves every volume created during the empty window permanently open.
Meanwhile the recreate paths do recompute it from the current list — recreate_lvol_on_node (lvol_controller.py:1414) and storage_node_ops.py:9027, :10090, :11297. A node restart, failover or migration therefore flips such a volume to allow_any_host=False and enforced. A volume's security posture ends up depending on whether its subsystem has been rebuilt since it was created.
Reproduce
sbctl pool add p-open <cluster-id> --dhchap # dhchap on, no hosts registered
sbctl volume add v1 1G p-open ... # created allow_any_host=True
# from a host whose NQN was never registered anywhere:
nvme connect -t tcp -a <ip> -s <port> -n <lvol-nqn> --hostnqn=nqn.2014-08.io.simplyblock:uuid:00000000-0000-0000-0000-000000000000
# connects
sbctl pool add-host <pool-id> nqn.2014-08.io.simplyblock:uuid:<real-node-uid>
# same unregistered host still connects — allow_any_host is still True
# restart the volume's storage node, then retry the unregistered host:
# now refused, because the subsystem was recreated with allow_any_host=False
Suggested fix
Fail closed rather than open, and keep the flag maintained rather than frozen:
- A volume in a
dhchap pool should be created allow_any_host=False even when pool.allowed_hosts is empty. The window then means "volumes exist but nothing may connect until a host is registered" — safe, reversible, and obvious from a failed connect rather than invisible.
add_host_to_lvol / remove_host_from_lvol should set allow_any_host to match bool(lvol.allowed_hosts) (for a dhchap pool, keep it false regardless), so the create path and the recreate paths agree and posture no longer depends on restart history.
Non-DHCHAP pools with no allowed_hosts should keep today's allow_any_host=True default — that case is unrestricted by design.
Kubernetes-side context
The state is reachable in normal operator flow, not only by hand. simplyblock-operator creates the pool in one reconcile (handleStoragePoolCreation returns as soon as Status.UUID is set) and registers hosts in a later one (syncStoragePoolHosts). If host registration errors and requeues — API unreachable, an unresolvable node name — the pool is active with dhchap: true and an empty allowed_hosts for as long as that lasts, and any volume provisioned in the meantime is permanently open. The operator cannot repair this from its side; dhchap is immutable on the CR, and allowedNodes is deliberately mutable so the node set can be populated later.
Related operator work: simplyblock/simplyblock-operator#484.
Observed on main @ 76ad8cbf7.
Summary
A volume created in a
dhchap-enabled pool that has no entries inallowed_hostsis created withallow_any_host=Trueand no DHCHAP material on its subsystem — so any host may connect, with no authentication. That is the ordinary default posture for an unrestricted pool, but here the pool reportsdhchap: true, so the volume looks protected and is not.Worse, the state is sticky: registering hosts later does not fix those volumes, while an unrelated storage-node restart silently does.
Found while investigating a Kubernetes-side report where a Pod on a deliberately excluded node mounted a DHCHAP pool's volume and reached
Running. This is the most plausible mechanical explanation.Trace
Volume creation, with
pool.dhchap = Trueandpool.allowed_hosts == []:create_lvol—if pool.dhchap: standalone_allowed_hosts = [{"nqn": h} for h in pool.allowed_hosts]→[](simplyblock_core/controllers/lvol_controller.py:696-700)claim_lvol_ns_slot—lvol.allowed_hosts = standalone_allowed_hosts→[](simplyblock_core/db_controller.py:907-908)add_lvol_on_node—allow_any = not bool(lvol.allowed_hosts)→True, sosubsystem_create(..., allow_any_host=True)and the wholeif lvol.allowed_hosts:block (pool key registration +subsystem_add_hostwithdhchap_key/dhchap_ctrlr_key) is skipped (lvol_controller.py:1216-1266)/connect—HostConnectAuth.resolvereturnsNoneon an emptyallowed_hosts(simplyblock_core/utils/nvme.py:75-76), so the connect command carries no DHCHAP secretsWhy it does not self-correct
allow_any_hostis only ever supplied atsubsystem_create. Nothing patches it afterwards:add_host_to_lvol(lvol_controller.py:4222) callssubsystem_add_hostwith the pool's keys but never clearsallow_any_host. Since the host ACL is only consulted whenallow_any_hostis false, adding hosts to such a subsystem changes nothing about who can connect.remove_host_from_lvol(lvol_controller.py:4333) likewise never sets it.So
pool add-host(or the operator adding a node toStoragePool.spec.allowedNodes) leaves every volume created during the empty window permanently open.Meanwhile the recreate paths do recompute it from the current list —
recreate_lvol_on_node(lvol_controller.py:1414) andstorage_node_ops.py:9027,:10090,:11297. A node restart, failover or migration therefore flips such a volume toallow_any_host=Falseand enforced. A volume's security posture ends up depending on whether its subsystem has been rebuilt since it was created.Reproduce
Suggested fix
Fail closed rather than open, and keep the flag maintained rather than frozen:
dhchappool should be createdallow_any_host=Falseeven whenpool.allowed_hostsis empty. The window then means "volumes exist but nothing may connect until a host is registered" — safe, reversible, and obvious from a failed connect rather than invisible.add_host_to_lvol/remove_host_from_lvolshould setallow_any_hostto matchbool(lvol.allowed_hosts)(for adhchappool, keep it false regardless), so the create path and the recreate paths agree and posture no longer depends on restart history.Non-DHCHAP pools with no
allowed_hostsshould keep today'sallow_any_host=Truedefault — that case is unrestricted by design.Kubernetes-side context
The state is reachable in normal operator flow, not only by hand.
simplyblock-operatorcreates the pool in one reconcile (handleStoragePoolCreationreturns as soon asStatus.UUIDis set) and registers hosts in a later one (syncStoragePoolHosts). If host registration errors and requeues — API unreachable, an unresolvable node name — the pool is active withdhchap: trueand an emptyallowed_hostsfor as long as that lasts, and any volume provisioned in the meantime is permanently open. The operator cannot repair this from its side;dhchapis immutable on the CR, andallowedNodesis deliberately mutable so the node set can be populated later.Related operator work: simplyblock/simplyblock-operator#484.
Observed on
main@76ad8cbf7.