test_runner: fix suite rerun edge case #5819

Job	Run time
Build slim tarball	3m 11s
aarch64-darwin: with shared libraries	24m 29s
aarch64-linux: with shared libraries	19m 16s
x86_64-darwin: with shared libraries	36m 41s
x86_64-linux: with shared libraries	21m 50s
	1h 45m 27s

Provide feedback