Skip to content

DHCHAP pool with no allowed hosts creates volumes with allow_any_host=True, and they stay that way #1310

Description

@boddumanohar

Summary

A volume created in a dhchap-enabled pool that has no entries in allowed_hosts is created with allow_any_host=True and no DHCHAP material on its subsystem — so any host may connect, with no authentication. That is the ordinary default posture for an unrestricted pool, but here the pool reports dhchap: true, so the volume looks protected and is not.

Worse, the state is sticky: registering hosts later does not fix those volumes, while an unrelated storage-node restart silently does.

Found while investigating a Kubernetes-side report where a Pod on a deliberately excluded node mounted a DHCHAP pool's volume and reached Running. This is the most plausible mechanical explanation.

Trace

Volume creation, with pool.dhchap = True and pool.allowed_hosts == []:

  1. create_lvol — if pool.dhchap: standalone_allowed_hosts = [{"nqn": h} for h in pool.allowed_hosts] → [] (simplyblock_core/controllers/lvol_controller.py:696-700)
  2. claim_lvol_ns_slot — lvol.allowed_hosts = standalone_allowed_hosts → [] (simplyblock_core/db_controller.py:907-908)
  3. add_lvol_on_node — allow_any = not bool(lvol.allowed_hosts) → True, so subsystem_create(..., allow_any_host=True) and the whole if lvol.allowed_hosts: block (pool key registration + subsystem_add_host with dhchap_key/dhchap_ctrlr_key) is skipped (lvol_controller.py:1216-1266)
  4. /connect — HostConnectAuth.resolve returns None on an empty allowed_hosts (simplyblock_core/utils/nvme.py:75-76), so the connect command carries no DHCHAP secrets

Why it does not self-correct

allow_any_host is only ever supplied at subsystem_create. Nothing patches it afterwards:

  • add_host_to_lvol (lvol_controller.py:4222) calls subsystem_add_host with the pool's keys but never clears allow_any_host. Since the host ACL is only consulted when allow_any_host is false, adding hosts to such a subsystem changes nothing about who can connect.
  • remove_host_from_lvol (lvol_controller.py:4333) likewise never sets it.

So pool add-host (or the operator adding a node to StoragePool.spec.allowedNodes) leaves every volume created during the empty window permanently open.

Meanwhile the recreate paths do recompute it from the current list — recreate_lvol_on_node (lvol_controller.py:1414) and storage_node_ops.py:9027, :10090, :11297. A node restart, failover or migration therefore flips such a volume to allow_any_host=False and enforced. A volume's security posture ends up depending on whether its subsystem has been rebuilt since it was created.

Reproduce

sbctl pool add p-open <cluster-id> --dhchap     # dhchap on, no hosts registered
sbctl volume add v1 1G p-open ...               # created allow_any_host=True

# from a host whose NQN was never registered anywhere:
nvme connect -t tcp -a <ip> -s <port> -n <lvol-nqn> --hostnqn=nqn.2014-08.io.simplyblock:uuid:00000000-0000-0000-0000-000000000000
# connects

sbctl pool add-host <pool-id> nqn.2014-08.io.simplyblock:uuid:<real-node-uid>
# same unregistered host still connects — allow_any_host is still True

# restart the volume's storage node, then retry the unregistered host:
# now refused, because the subsystem was recreated with allow_any_host=False

Suggested fix

Fail closed rather than open, and keep the flag maintained rather than frozen:

  1. A volume in a dhchap pool should be created allow_any_host=False even when pool.allowed_hosts is empty. The window then means "volumes exist but nothing may connect until a host is registered" — safe, reversible, and obvious from a failed connect rather than invisible.
  2. add_host_to_lvol / remove_host_from_lvol should set allow_any_host to match bool(lvol.allowed_hosts) (for a dhchap pool, keep it false regardless), so the create path and the recreate paths agree and posture no longer depends on restart history.

Non-DHCHAP pools with no allowed_hosts should keep today's allow_any_host=True default — that case is unrestricted by design.

Kubernetes-side context

The state is reachable in normal operator flow, not only by hand. simplyblock-operator creates the pool in one reconcile (handleStoragePoolCreation returns as soon as Status.UUID is set) and registers hosts in a later one (syncStoragePoolHosts). If host registration errors and requeues — API unreachable, an unresolvable node name — the pool is active with dhchap: true and an empty allowed_hosts for as long as that lasts, and any volume provisioned in the meantime is permanently open. The operator cannot repair this from its side; dhchap is immutable on the CR, and allowedNodes is deliberately mutable so the node set can be populated later.

Related operator work: simplyblock/simplyblock-operator#484.

Observed on main @ 76ad8cbf7.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions