[Bug]: 3008.2 Migrating from a shared-filesystem cluster to isolated filesystem

[plakun]

What happened?

After migrating to an isolated filesystem following the instructions at https://docs.saltproject.io/en/latest/topics/tutorials/master-cluster.html#migrating-from-a-shared-filesystem-cluster , I encountered an issue where minions are unable to connect to the masters due to a key-related problem.

[root@salt-01 salt]# cat master.d/cluster.conf
id: 10.249.207.118
cluster_id: master_cluster
cluster_peers:
  - 10.249.207.111
cluster_pki_dir: /etc/salt/cluster.d/pki
cachedir: /etc/salt/cluster.d/cache
pki_dir: /etc/salt/cluster.d/pki
file_roots:
  base:
    - /srv/salt
pillar_roots:
  base:
    - /srv/pillar
cluster_isolated_filesystem: True
keys.cache_driver: mmap_key
cluster_secret: "secret"

Aug 19 13:01:09 salt-01.infra.example.com systemd[1]: Starting salt-master.service - The Salt Master Server...
Aug 19 13:01:10 salt-01.infra.example.com systemd[1]: Started salt-master.service - The Salt Master Server.
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]: [ERROR   ] SaltPeer: failed to send cluster/raft/pre-request-vote to 10.249.207.111
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]: Traceback (most recent call last):
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]:   File "/usr/lib/python3.11/site-packages/salt/cluster/consensus/peer.py", line 155, in _send
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]:     await _publish(self._pusher, raw)
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]:   File "/usr/lib/python3.11/site-packages/salt/cluster/consensus/peer.py", line 101, in _publish
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]:     await client.send(raw)
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]:   File "/usr/lib/python3.11/site-packages/salt/transport/tcp.py", line 2013, in send
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]:     await self.connect()
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]:   File "/usr/lib/python3.11/site-packages/salt/transport/tcp.py", line 1943, in _connect
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]:     await self.stream.connect(sock_addr)
Aug 19 13:01:13 salt-01.infra.example.com salt-master[2705565]: tornado.iostream.StreamClosedError: Stream is closed
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]: [CRITICAL] Unhandled error while polling master events
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]: Traceback (most recent call last):
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]:   File "/usr/lib/python3.11/site-packages/salt/channel/server.py", line 3726, in handle_pool_publish
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]:     key_str = self.master_key.master_key.decrypt(
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]:   File "/usr/lib/python3.11/site-packages/salt/crypt.py", line 375, in decrypt
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]:     return self.key.decrypt(
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]:            ^^^^^^^^^^^^^^^^^
Aug 19 13:01:14 salt-01.infra.example.com salt-master[2705565]: ValueError: Decryption failed
Aug 19 13:02:05 salt-01.infra.example.com salt-master[2705692]: [ERROR   ] Unable to decrypt token: Unable to decrypt message
Aug 19 13:02:05 salt-01.infra.example.com salt-master[2705694]: [ERROR   ] Unable to decrypt token: Unable to decrypt message
Aug 19 13:02:25 salt-01.infra.example.com salt-master[2705692]: [ERROR   ] Unable to decrypt token: Unable to decrypt message
Aug 19 13:02:25 salt-01.infra.example.com salt-master[2705694]: [ERROR   ] Unable to decrypt token: Unable to decrypt message

on minion:

авг 19 13:02:40 traefik-lb-01.infra.example.com systemd[1]: Started The Salt Minion.
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]: [CRITICAL] The payload signature did not validate.
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]: [ERROR   ] Exception getting pillar:
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]: Traceback (most recent call last):
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/pillar/__init__.py", line 273, in compile_pillar
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     ret_pillar = yield self.channel.crypted_transfer_decode_dictentry(
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/ext/tornado/gen.py", line 1056, in run
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     value = future.result()
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     raise_exc_info(self._exc_info)
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "<string>", line 4, in raise_exc_info
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/ext/tornado/gen.py", line 1064, in run
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     yielded = self.gen.throw(*exc_info)
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/channel/client.py", line 235, in crypted_transfer_decode_dictentry
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     yield self.auth.authenticate()
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/ext/tornado/gen.py", line 1056, in run
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     value = future.result()
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     raise_exc_info(self._exc_info)
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "<string>", line 4, in raise_exc_info
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/crypt.py", line 778, in _authenticate
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     creds = yield self.sign_in(channel=channel)
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/ext/tornado/gen.py", line 1056, in run
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     value = future.result()
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     raise_exc_info(self._exc_info)
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "<string>", line 4, in raise_exc_info
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/ext/tornado/gen.py", line 1070, in run
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     yielded = self.gen.send(value)
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/crypt.py", line 927, in sign_in
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     ret = self.handle_signin_response(sign_in_payload, payload)
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:   File "/usr/lib/python3/site-packages/salt/crypt.py", line 979, in handle_signin_response
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]:     raise SaltClientError("Invalid signature")
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]: salt.exceptions.SaltClientError: Invalid signature
авг 19 13:02:54 traefik-lb-01.infra.example.com salt-minion[2413443]: [ERROR   ] Error while bringing up minion for multi-master. Is master at 10.249.207.231 responding? The error message was Exception getting pill>
авг 19 13:03:14 traefik-lb-01.infra.example.com salt-minion[2413443]: [CRITICAL] The payload signature did not validate.
авг 19 13:03:14 traefik-lb-01.infra.example.com salt-minion[2413443]: [ERROR   ] Error while bringing up minion for multi-master. Is master at 10.249.207.231 responding? The error message was Unable to sign_in to m>
lines 1-50/50 (END)

i'v already tried to clean /etc/salt/minion/*, deleted minion key from master, nothing helps

Type of salt install

other (please specify in bug details)

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

rhel-8

salt --versions-report output

Salt Version:
          Salt: 3008.2

Python Version:
        Python: 3.11.15 (main, May  6 2026, 00:00:00) [GCC 13.3.1 20250418 (RED SOFT 13.3.1-3)]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: unknown
  cryptography: 46.0.6
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.9
     gitpython: 3.1.50
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 25.0
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.15.0
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: Not Installed
         smmap: 5.0.0
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.5

Salt Package Information:
  Package Type: Not Installed

System Versions:
          dist: redos 8.0 n/a
        locale: utf-8
       machine: x86_64
       release: 6.12.87-2.red80.x86_64
        system: Linux
       version: RED OS 8.0 n/a

[plakun]

the same problem occures when i deploy fresh salt-master cluster with isolated fs.
Minion -> Haproxy -> Salt cluster
Minion gets error:

[root@test-salt-01 salt]# salt-call pillar.items
[ERROR   ] The master key has changed, the salt master could have been subverted, verify salt master's public key
[CRITICAL] The Salt Master server's public key did not authenticate!
The master may need to be updated if it is a version of Salt lower than 3008.2, or
If you are confident that you are connecting to a valid Salt Master, then remove the master public key and restart the Salt Minion.
The master public key can be found at:
/etc/salt/pki/minion/minion_master.pub
[ERROR   ] Exception getting pillar:
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 243, in wrap
    asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/salt/pillar/__init__.py", line 359, in compile_pillar
    ret_pillar = self.channel.crypted_transfer_decode_dictentry(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 250, in wrap
    return self.io_loop.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/tornado/ioloop.py", line 527, in run_sync
    return future_cell[0].result()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/channel/client.py", line 221, in crypted_transfer_decode_dictentry
    await self.auth.authenticate()
  File "/usr/lib/python3.11/site-packages/salt/crypt.py", line 1100, in _authenticate
    creds = await self.sign_in(channel=channel)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/crypt.py", line 1261, in sign_in
    return self.handle_signin_response(sign_in_payload, payload)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/salt/crypt.py", line 1297, in handle_signin_response
    raise SaltClientError("Invalid master key")
salt.exceptions.SaltClientError: Invalid master key
Exception getting pillar.