tests/pytests/integration/netapi/rest_tornado/test_minions_api_handler.py::test_mem_leak_in_event_listener intermittently fails with AssertionError: assert 2 == 0 for app.event_listener.tag_map.
Observed on the 3008.x nightly (Amazon Linux 2 integration zeromq 4, run 32913591224 job 98023096704) — first attempt fails with two salt/job/<jid>/ret prefix-matcher futures left in tag_map, retry passes. Same pattern reproduced on merge-forward PR #70132.
Root cause: saltnado.py::_disbatch_local spawns job_not_running via ioloop.spawn_callback to ping saltutil.find_job. When the outer job's is_finished completes first, job_not_running calls event.set_result(None) on the in-flight ping future but does not remove it from event_listener.tag_map / timeout_map. Because spawn_callback runs independently of the handler, the ping may be registered after on_finish -> clean_by_request has already emptied request_map for the handler, so the leaked entry only clears when the gather_job_timeout callback fires (10 s default, 30 s under the netapi test fixture).
The test only allows a 1 s wait before asserting the maps are empty, so any leaked ping trips the assertion.
tests/pytests/integration/netapi/rest_tornado/test_minions_api_handler.py::test_mem_leak_in_event_listenerintermittently fails withAssertionError: assert 2 == 0forapp.event_listener.tag_map.Observed on the 3008.x nightly (Amazon Linux 2 integration zeromq 4, run 32913591224 job 98023096704) — first attempt fails with two
salt/job/<jid>/retprefix-matcher futures left intag_map, retry passes. Same pattern reproduced on merge-forward PR #70132.Root cause:
saltnado.py::_disbatch_localspawnsjob_not_runningviaioloop.spawn_callbackto pingsaltutil.find_job. When the outer job'sis_finishedcompletes first,job_not_runningcallsevent.set_result(None)on the in-flight ping future but does not remove it fromevent_listener.tag_map/timeout_map. Becausespawn_callbackruns independently of the handler, the ping may be registered afteron_finish -> clean_by_requesthas already emptiedrequest_mapfor the handler, so the leaked entry only clears when thegather_job_timeoutcallback fires (10 s default, 30 s under the netapi test fixture).The test only allows a 1 s wait before asserting the maps are empty, so any leaked ping trips the assertion.