[Bug]: deltaproxy: targeting two sub-proxies in one job makes them render the same SLS identity, and it persists

[RobertArndtEllerhold]

What happened?

Description

On a deltaproxy minion, targeting two or more sub-proxies in a single job
makes all of them render an SLS with the same sub-proxy's grains. Each
result is returned under the correct minion's key, so nothing looks wrong at a
glance.

The effect outlives the job: afterwards, even a single-target render
against the affected sub-proxy keeps returning the other one's identity, until
the deltaproxy is restarted.

Rendering the same SLS one sub-proxy at a time is correct and stays correct.

Execution modules are correct throughout, including in multi-target jobs.

Setup

• deltaproxy, ~70 sub-proxies, parallel_startup: True
• Salt 3008.2, Python 3.14, Debian
• No custom modules involved in the reproducer

Reproducer

probe.sls in the sub-proxies' file_roots environment -- three lines, nothing
executed:

probe:
  test.nop:
    - name: {{ grains['id'] }}

Restart the deltaproxy and wait until the sub-proxies answer, then:

salt -C 'minionA'             state.show_sls probe    # 1
salt -C 'minionB'             state.show_sls probe    # 2
salt -C 'minionA or minionB'  state.show_sls probe    # 3
salt -C 'minionA'             state.show_sls probe    # 4

Observed

1)  minionA -> name: minionA          correct
2)  minionB -> name: minionB          correct
3)  minionA -> name: minionB          WRONG
    minionB -> name: minionB
4)  minionA -> name: minionB          WRONG -- persists after the job

Step 4 keeps failing until the deltaproxy is restarted. state.show_low_sls
and state.apply behave the same. state.show_sls only renders -- it does not
call verify_high -- so test.nop does not even have to be loadable.

With state.apply, everything opts-derived is the other sub-proxy's

A state module executed this way sees, consistently:

value	reports
__grains__['id']	the other sub-proxy
__pillar__	the other sub-proxy's pillar
__opts__['id'], ['cachedir'], ['proxyid']	the other sub-proxy
__salt__['grains.get']('id')	the other sub-proxy

A state module cannot detect this from the inside: the wrong values are
mutually consistent. __grains__['id'] and __pillar__ describe the same
(wrong) device, so a self-check comparing them passes.

The same sub-proxies in the same multi-target job return correct values through
execution modules -- grains.get id, config.get cachedir, and a custom module
reading __grains__['id'] are all right.

What we ruled out

• Not a race. Deterministic: five identical runs, identical output; the
  sub-proxy whose identity wins is stable.
• Not --batch-size 1. Batching does not help.
• Not the cached SLS file. Renders are correct both with and without the
  cached copy under /var/cache/salt/proxy/<id>/files/<saltenv>/; the file is
  written by a render, not required for one.
• Not custom modules. The reproducer above uses test.nop.

Not the same as #61297

#61297 (closed, fixed by #61732) reported the same symptom class for
configuration and grains. That fix works here -- config.get id and
grains.get id return the correct value per sub-proxy, including in
multi-target jobs. The SLS rendering path appears not to have been covered.

Impact

Silent and plausible. A state.highstate across a deltaproxy fleet reports
device A's configuration under device B's name, with no error and no log entry
-- and every subsequent run against A stays wrong until the proxy is restarted.
For network devices that means acting on the wrong switch. We disabled
state.highstate for our proxies; the execution-module path is unaffected and
remains usable.

Related

• deltaproxy overriding minion configuration settings #61297 -- same symptom class for config/grains, closed, fixed by Ensure config module loads opts from the named_context. #61732.
• # [BUG] deltaproxy: sub-proxies raise AttributeError 'tok' when dispatching events (3008 asyncio port) #70071, # [BUG] deltaproxy: sub-proxy init failures are logged at INFO and invisible at default log level #70072 -- other deltaproxy issues we filed on 3008.2.

Type of salt install

Official deb

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-13

salt --versions-report output

Salt Version:
           Salt: 3008.2

Python Version:
         Python: 3.14.6 (main, Jun 11 2026, 02:19:05) [GCC 11.2.0]

Dependency Versions:
           cffi: 2.0.0
       cherrypy: 18.10.0
   cryptography: 48.0.0
       dateutil: 2.9.0.post0
      docker-py: Not Installed
          gitdb: 4.0.12
      gitpython: 3.1.50
         Jinja2: 3.1.6
        libgit2: Not Installed
   looseversion: 1.3.0
       M2Crypto: Not Installed
           Mako: Not Installed
        msgpack: 1.1.2
   msgpack-pure: Not Installed
   mysql-python: Not Installed
      packaging: 24.0
      pycparser: 3.00
       pycrypto: Not Installed
   pycryptodome: 3.23.0
         pygit2: Not Installed
   python-gnupg: 0.5.6
         PyYAML: 6.0.3
          PyZMQ: 27.1.0
         relenv: 0.22.14
          smmap: 5.0.2
        timelib: 0.3.0
        Tornado: 6.5.7
            ZMQ: 4.3.5

Salt Extensions:
 saltext.apache: 1.0.1
  saltext.vault: 1.6.0
    saltext.zfs: 1.0.0

Salt Package Information:
   Package Type: onedir

System Versions:
           dist: debian 13.6 trixie
         locale: utf-8
        machine: x86_64
        release: 6.12.101+deb13-amd64
         system: Linux
        version: Debian GNU/Linux 13.6 trixie