A storage-node restart no longer takes the worker's spdk-proxy DNS record with it

[noctarius]

The bug

A storage-node restart aborts with a fatal name-resolution error:

2026-09-06 22:13:23,920: ERROR: restart_storage_node raised unexpectedly
socket.gaierror: [Errno -2] Name or service not known
  ... HTTPSConnection(host='worker-1.simplyblock-spdk-proxy.simplyblock.svc.cluster.local', port=4422)

worker-N.simplyblock-spdk-proxy.<namespace>.svc.cluster.local is a per-pod record of a
selectorless headless Service whose EndpointSlices this operator writes by hand. It was
addressed from pod.Status.PodIP, so the pod replacement at the heart of every storage-node
restart deleted the record, and the control plane — which RPCs that name a fixed five seconds
after "Pod created" — got NXDOMAIN.

NXDOMAIN is the worst failure mode available here. Its RPC client raises a fatal
socket.gaierror rather than the retryable connection error a record still pointing at a
down proxy would give it, and cluster DNS caches the denial for longer than the proxy is
actually absent.

Why it is this and not cluster DNS

From the k8s_native_resilient_failover-20260906-112437 run:

• All 126 resolution failures were on *.simplyblock-spdk-proxy names. Zero on anything
  else.
• worker-1.simplyblock-storage-node-api.simplyblock.svc.cluster.local:5000 answered 200 at
  22:13:01,527 — same client pod, same node, same namespace, 20 seconds earlier — and never
  once failed to resolve in twelve hours. That slice is filled from stable node internal
  IPs, so it never goes away.
• The failures come in ten short bursts, each landing within seconds of the SPDK pod for that
  exact RPC port being replaced, and every burst matches an entry in
  outage_log_20260906_112437.log.

Impact in that run: six restart attempts aborted, and three nodes reset to OFFLINE
(22:13:23, 22:23:15, 22:23:18).

The fix

The record's lifetime now belongs to the worker rather than to whichever pod currently serves
it. The spdk-proxy pod is host-networked — its PodIP is 10.0.0.11, identical to the
node's own InternalIP — so the operator was already publishing a node address; it was just
reading it off an object with a pod's lifetime.

reconcileSpdkProxyEndpointSlices now gathers (nodeName, rpcPort) from the StorageNode CRs
(spec.workerNode plus status.ports.rpc, both readable without a live pod) and addresses
them from the Node. A ready pod still overrides that address, which keeps the pod as
ground truth should it ever stop being host-networked, and as the only source for a worker
whose CR has not reported a port yet. A slice is deleted only once its port has left the set
entirely.

The property that matters: across a pod replacement the desired slice content is now
identical, so the reconcile writes nothing at all — no delete, no recreate, no DNS churn,
and no window in which a resolver can cache a denial.

One deliberate behavior change: the name resolves even when no proxy is listening. Callers
see a connection error instead of NXDOMAIN. simplyblock-storage-node-api already behaves
this way, so the pattern is established.

Secondary: the watch predicate

The Pod watch passed phase transitions only:

return oldPod.Status.Phase != newPod.Status.Phase

A pod reaches Running while a container is still unready, and the later
ContainerStatuses[].Ready flip — the change that makes isSpdkProxyPodReady true — leaves
the phase alone, so it fired no event at all and republishing waited for the periodic
requeue. It is now the named spdkProxyPodPublishStateChanged, keyed on what the reconciler
actually reads.

Also

• SpdkProxyEndpoint.PodIP → Address, since it no longer holds a pod IP.
• reconcileSpdkProxyEndpointSlices split into an orchestrator plus four helpers, to stay
  under the gocyclo limit.

A trap worth knowing

StorageNodeSet.status.nodes[].hostname is deliberately not used as the node name. One
write path fills it from the Kubernetes node name
(simplyblockstoragenodeset_controller.go:1463) and another overwrites it with the control
plane's sn.Status.Hostname (simplyblockstoragenodeset_storagenode.go:493) — the short
worker-1, not the FQDN worker-1.ocp.simplyblock.ai that is the real node name.
getNodeInternalIP would have failed on exactly the clusters in this incident.

Tests

All three were proven red against the unfixed tree first, and are rows U-264 through U-266 in
test-plan-storagenode.md:

Test	Red evidence
..._SurvivesSpdkPodReplacement	node-a.simplyblock-spdk-proxy.ns.svc.cluster.local resolves to nothing: no endpoint for node-a on port 9001 (slices present: [])
..._RemovedWorkerLosesRecord	Negative counterpart — surviving a pod replacement must not mean surviving forever
TestSpdkProxyPodPublishStateChanged	Red on all four dropped-update cases under the old predicate

Gates: go test ./... -shuffle=on green including envtest, make lint clean,
make manifests generate produces no drift, house-style table gate clean. No API types,
markers, or RBAC changed.

Still open upstream (sbcli, not this repo)

Neither change removes the reason a two-second DNS gap became three nodes reset to OFFLINE:

• set_opts_rpc.bdev_nvme_set_options() at storage_node_ops.py:5933 has no connect
  retries — it raised MaxRetryError 5 ms after its request with no retry warnings, while
  every neighboring RPC client rode out the identical flapping.
• _restart_storage_node_impl sleeps a fixed 5 s after "Pod created" and then RPCs the name;
  it should poll until the name resolves and the proxy answers.

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

The behavioral change is well-scoped, appears correct, and is backed by targeted unit tests; only a minor documentation comment nit was found.

Pull request overview

This PR fixes spdk-proxy per-worker DNS instability during storage-node restarts by decoupling EndpointSlice publishing from the lifetime of the spdk-proxy pod object, preventing transient NXDOMAIN responses and making the record stable across pod replacement.

Changes:

• Publish spdk-proxy EndpointSlice addresses from StorageNode/Node data (stable across pod replacement), with ready pods overriding as ground truth when available.
• Improve the spdk-proxy pod watch predicate so reconciles trigger on readiness/address/scheduling changes (not only Pod phase transitions).
• Rename the published endpoint field from PodIP to Address, and add unit tests + test-plan entries covering the regressions and negative case.

File summaries

File	Description
operator/internal/utils/storage_nodeset_ds.go	Renames SpdkProxyEndpoint field to Address and uses it when building EndpointSlice endpoints.
operator/internal/utils/storage_nodeset_ds_test.go	Updates EndpointSlice builder tests to use Address instead of PodIP.
operator/internal/controller/simplyblockstoragenodeset_controller.go	Refactors and hardens spdk-proxy EndpointSlice reconcile logic; adds improved pod update predicate and deterministic endpoint ordering.
operator/internal/controller/simplyblockstoragenodeset_controller_unit_test.go	Adds regression/negative tests for surviving pod replacement, record deletion after worker removal, and the new predicate behavior.
operator/docs/tests/test-plan-storagenode.md	Documents the new regression/negative test scenarios and links them to the added tests.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.