[Bug]: salt-call causes ~60s hangs on minions due to ZMQ identity collision with running salt-minion daemon (3006.27 regression)

[jzandbergen]

What happened?

What happened?

Pillar (and any other) requests made via salt-call on a minion take ~60 seconds
(the full REQUEST_TIMEOUT) to complete on 3006.27, when the same command
completes in a few seconds on 3006.26. The affected code path is unchanged
in 3006.26, so this is a regression introduced in 3006.27.

[06:31:40] alma98-p3g ~ # time salt-call pillar.data dontcarejustatest
local:
    ----------
    dontcarejustatest:

real    1m3.404s
user    0m1.045s
sys    0m0.137s

Trace logging on the minion shows the request hangs for ~60s waiting on a
reply from the master before finally timing out and retrying successfully:

[DEBUG   ] Initializing new AsyncAuth for (...)
[TRACE   ] ext_pillar_extra_data = {}
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key

<-- ~60s stall -->

[TRACE   ] Failed to send msg SaltReqTimeoutError('Message timed out')
[TRACE   ] Request timed out while waiting for a response. reconnecting.

Root cause

3006.27 includes commit 36e914a72af330e8e0c09bfea5f43893f5e972b1
("Use stable ZMQ identity for minion + syndic daemon ret-port REQ
sockets"), which sets a stable ZMQ IDENTITY on the AsyncReqMessageClient
REQ socket for long-lived minion/syndic daemons:

elif _role in ("minion", "syndic") and _minion_id:
    identity = "salt-req/{role}/{minion_id}/{slot}".format(
        role=_role,
        minion_id=_minion_id,
        slot=next(_REQ_IDENTITY_SLOT),
    )
    self.socket.setsockopt(zmq.IDENTITY, identity.encode("utf-8"))

_REQ_IDENTITY_SLOT is a module-level itertools.count() that starts at 0
per process. A running salt-minion daemon and a one-off salt-call
invocation both set opts["__role"] = "minion" and share the same
minion_id, so both processes independently compute the identity
salt-req/minion/<minion_id>/0 for their first REQ socket.

Because the master's ROUTER socket has ROUTER_HANDOVER=1 set (see
salt/transport/zeromq.py, MWorkerQueue), when the second process
(salt-call) connects with the same identity as the first (salt-minion),
libzmq hands the peer slot over to the newcomer and silently drops any
reply still in flight to the original owner. This causes whichever process
was waiting on that reply to stall for the full REQUEST_TIMEOUT (60s)
before retrying.

This directly contradicts the stated goal of the commit itself, which
explicitly calls out that giving two sockets the same identity causes
exactly this failure mode ("Giving them all the same stable identity would
cause ROUTER_HANDOVER on the upstream ROUTER to silently drop any reply
still in flight..."), but the fix only accounted for multiple sockets
within one process, not multiple processes (salt-minion and
salt-call) sharing the same minion_id and hence the same identity
namespace.

Suggested fix

Include something process-unique (e.g. os.getpid()) in the identity
string for the minion/syndic branch, so that salt-call,
salt-minion, and any other short-lived process using the minion config
never collide with the long-running daemon's identity slots.

Type of salt install

Official rpm

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

almalinux-9

salt --versions-report output

salt-minion 3006.27 (Sulfur)

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here's some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Salt's Contributor Guide
• Join our Community Discord
• Salt Project YouTube channel
• GitHub Discussions

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject.pdl@broadcom.com. We're glad you've joined our community and look forward to doing awesome things with you!