Timing-sensitive KUnit suites flake on loaded hosts (drm_sched, lib_ratelimit)

[tamnd]

The 7.2.8 x86_64 defconfig+gk cell with gcc-8.5.0 (cell 34d78cb6) is flaky: boots 2 and 3 reached L8, and boot 1 never finished its KUnit run.

In boot 1, drm_sched_basic_priority_tests passed drm_sched_priorities and then the machine stopped making progress. RCU reported a stall on CPU 1 at 234s, and the NMI backtrace shows CPU 1 spinning in hrtimer_cancel. The call comes from dma_fence_release, through drm_sched_fence_release_scheduled, inside drm_mock_sched_job_signal_timer, which is the timer itself. A timer callback that ends up waiting for its own hrtimer to finish never returns. The same trace was still there at 1119s, when the boot budget ran out.

This looks like a race in the drm_sched mock scheduler and not something gcc-8.5.0 did, since the same build passed the suite twice. TCG on a loaded host makes the timing wide enough to hit it. To check whether it's specific to the GCC, compare how often it shows up in other columns as the stripe fills in. If it shows up in other columns too, drm_sched_basic_priority_tests is a candidate for the known flaky list instead of a per-cell flake.

The log is kunit-1.log in the cell directory.

[tamnd]

More of the same kind on the 7.2.8 x86_64 defconfig row from server3, which runs at a load average around 60 on 8 cores:

• gcc-10.5.0 (a88d054e), boot 3: the console stops inside the lib_ratelimit suite at 258s and nothing more is printed until the budget runs out, so every suite after it counts as failed. Boots 1 and 2 reach L8.
• gcc-12.2.0, boot 3: WARNING: lib/tests/test_ratelimit.c:87 at test_ratelimit_stress_child. Boots 1 and 2 reach L8.
• gcc-8.5.0, boot 1: the drm_sched hang above.

Each of these misses in only one of the three boots, in a timing-sensitive suite, on a host that's badly overloaded. Under spec 02.6 that's what the flaky mark is for, and gk follows it: these cells keep the lowest rung and are marked flaky. I'm leaving grading as it is. If the census shows the same suites flaking on most columns, the next step is to list lib_ratelimit and drm_sched_basic_priority_tests as timing-sensitive, so their misses don't hide what the GCC did.

[tamnd]

#61 skips both modules in the M14 KUnit boot with kunit.filter_action=skip, since KUNIT_ALL_TESTS leaves no way to turn them off in the fragment. They still show in the log as skipped. The M14 cells run again with the new digest. I am leaving this open until a few rows confirm the flakes are gone.

[tamnd]

The command line skip from #61 did not work: on 7.2.8, kunit.filter=module!=... panics in attr_string_filter, which calls strcmp on a test case with no module name, so every KUnit boot died before a test ran. #65 drops it and turns the two tests off in the configuration instead, with a second olddefconfig pass that has KUNIT_ALL_TESTS off. The two cells made with the broken filter are moved out of the store.