test(BACKEND-ROCM): reconcile the Strix full-suite failures

[localai-org-maint-bot]

Row: BACKEND-ROCM

The operator's HIP full-suite gate for PR #3010 is unresolved. Strix lease d7a790d5-158e-46d8-9e32-1284f4c41a6c recorded 18 failed targets and then lost its host to a reboot at 741/745 completed tests. It supplies no completed full-suite verdict. Issues #3066 and #3068 own the diffusion fixture failures; this issue owns the remaining 17 targets and the required baseline pairing.

The failed targets fall into these groups:

• test_qwen3_dflash2_draft, test_dflash2_runner_reach, test_dflash2_concurrency, test_dflash2_ctx_capacity, and test_dflash2_prefix_cache: explicit missing ROCm IndexSelect; host reference fallback is refused because device memory is not host-addressable.
• test_dflash2_exl3_reach and test_dflash2_draft_phase_trace: caught exceptions and missing draft outputs; the log does not print their exception text, so the operation cause is not established.
• test_mtp_depth: explicit missing CausalConv1dSpecUpdate, plus an async-capability expectation that needs separate diagnosis.
• test_qwen4_exp_gguf_load_plan, test_qwen4_exp_gguf_weights, and test_qwen4_exp_runner: ROCm keep-quant/gather expectations or explicit refusal. The block-decoding gather arms for METAL, VULKAN, ROCM and TENSTORRENT #2394 owns missing block-decoding gather arms; ROCm ports zero I-quant formats, and porting a complex-vec_dot one would test whether #1910's nwarps split transfers #1940 owns broader I-quant coverage.
• test_placement_reach and test_engine_scratch_steady_state: fixtures appear to assume CPU placement or a CPU pool while the runtime selects ROCm. This is not yet a runtime-paired diagnosis.
• test_loaded_engine_dense and test_llm_engine: expected async scheduling/multiple concurrent batches, but the ROCm backend does not advertise sampled-token readback support.
• test_cuda_quant_dot: a CUDA-named residency fixture routes through CurrentPlatform() and expects formats not admitted by ROCm.
• test_glm_moe_dsa_forward: explicit missing ROCm DsaIndexerLogits; the related W2 source/spec and historical closed issues need reconciliation.

The affected tests and relevant registration, loader, and runner sources are byte-identical between main 08a34c3a74d78046f83886f242d07110a70ff45e and sampling repair 2bd0ca67f051d7ef7d48b52ad75a8729fe30d0b3. That is supporting evidence only. None of these 17 targets has a paired main runtime result in the supplied log.

Next gate: reproduce each target on pinned main under the same leased device, build, environment, and serial scheduling. Capture omitted exception text in diagnostic scratch. Then distinguish test-fixture assumptions from actual missing backend capabilities, and assign each required implementation to its owning row/spec before changing code. A change in another row is not authorized merely to make this report green.

Do not enable unsafe managed allocation to hide missing native operations. Do not expand keep-quant tables to bypass residency refusals. Do not call this a passing or narrowly waived gate. Local evidence remains /tmp/pr3010-full-lease.log; the separate focused sampling tests do not replace this full-suite obligation.

[localai-org-maint-bot]

The paired runtime check completed on strix:gpu0 under lease 0ed62196-7b2a-42eb-9738-90cb550b632a on 8 September 2026.

Main: 08a34c3a74d78046f83886f242d07110a70ff45e.
Sampling candidate: c10ebbd3801a902d4794abe43e18496fed75d54e.

All 17 selected CTest targets failed with exit 8 on both revisions. The script recorded all 34 runs and ended with SCRIPT_EXIT=1 PAIRING_COMPLETE=1. The build used HIP 7.2.53211, ROCm clang 22.0.0git, Release, and gfx1151. Targets ran serially with the same configuration. The recorded execution overrides, including VT_ROCM_MANAGED_ALLOC, VT_FAST_RANDOM_SAMPLE, and VT_SAMPLE_SPLIT, were unset.

The comparison checked complete numbered child transcripts, not just exit codes. CTest prints a second, unnumbered copy after failure; that duplicate was excluded. Twelve target transcripts are byte-identical. Five differ only in process IDs inside temporary DFlash checkpoint paths: draft, runner-reach, prefix-cache, EXL3-reach, and draft-phase-trace. Replacing only the final PID in /tmp/vllmcpp_dflash2_{seam,reach}_<fixture-number>_<PID> makes all 17 transcripts identical. No assertion values, numerical results, operation IDs, counters, or exception text were removed. No substantive numerical difference was observed in the emitted transcripts.

The counts and locations below apply to both revisions. Locations are in each target's corresponding .cpp test file. “Cases” and “assertions” mean failed/total, not passed/total. Uncaught exceptions can fail a case without incrementing the failed-assertion count.

Target	Cases	Assertions	Failure locations	Observed signature on both revisions
test_qwen3_dflash2_draft	3/44	3/1491	2812, 2825, 2852	Nonempty generation errors; explicit missing IndexSelect (90).
test_dflash2_runner_reach	9/10	22/81	87, 90, 136–138, 209–211, 246–248, 279, 281, 347–348, 393–394, 399, 457–458, 537, 540	Explicit missing IndexSelect (90), empty blocks, absent phase/proposal markers, zero seam calls and steps.
test_dflash2_concurrency	2/2	2/14	162, 170	Concurrent and solo calls report missing IndexSelect (90).
test_dflash2_ctx_capacity	3/6	12/71	198–199, 201, 209–210, 212, 230, 274, 276–277, 423–424	Explicit missing IndexSelect (90); unfinished requests and empty tokens or blocks.
test_dflash2_prefix_cache	5/5	15/56	218–219, 221, 224–225, 236, 264–266, 279, 325, 327–328, 330, 340	Explicit missing IndexSelect (90); unfinished requests, empty blocks, and missing cache marker.
test_dflash2_exl3_reach	2/3	5/106	97, 101, 104, 164–165	Nonempty caught exceptions, absent proposal marker, empty drafts. Exception strings are suppressed; exact exception identity remains unknown.
test_dflash2_draft_phase_trace	3/3	7/23	91, 96, 167, 169, 203–205	Nonempty caught exceptions and zero recorded device-phase entries. Exception strings are suppressed; exact exception identity remains unknown.
test_mtp_depth	6/10	2/29	Exceptions: 480, 539, 592, 671, 830. Assertions: 814, 817	Five explicit missing CausalConv1dSpecUpdate (7) exceptions. Async scheduling is false and concurrent batches are 1, expected 2.
test_qwen4_exp_gguf_load_plan	2/10	2/7462	418, 452	IQ4_NL embedding and stacked-expert routing returns residency 0, expected keep-quant 1.
test_qwen4_exp_gguf_weights	1/14	1/3093	646	N-gram table dtype is 2, expected Q8_0 value 8.
test_qwen4_exp_runner	1/5	1/120	480	Explicit refusal: ROCm lacks the block-decoding gather needed to keep per_layer_token_embd.weight block-resident. The message names the owed arm in #2394.
test_placement_reach	1/5	1/20	193	Placement origin is 1, expected kNone value 0.
test_engine_scratch_steady_state	1/3	1/15	455	after_peak.retained_bytes is 0, required greater than 0.
test_loaded_engine_dense	1/30	5/128	455–458, 483	Runner async support and scheduling are false; batches are 1 rather than 2; AsyncScheduler cast is null.
test_llm_engine	1/24	2/546	1210–1211	Async scheduling is false; batches are 1 rather than 2.
test_cuda_quant_dot	1/14	2/169	1270 twice	Residency is 0 rather than keep-quant 1 for blk.3.ffn_gate_exps.weight and blk.11.ffn_down_exps.weight.
test_glm_moe_dsa_forward	1/8	0/5332	Exception: 258	Explicit missing DsaIndexerLogits (34) exception.

Every printed missing-operation exception names ROCm device type 5 and src/vt/op_provider.cpp:615. Each rejects the portable CPU tier because the backend does not claim host-addressable device memory. The diagnostic reports hipDeviceAttributePageableMemoryAccess = 0 and the safe hipMalloc path. This check did not enable managed allocation or alter that refusal.

Evidence is retained at /workspace/pr3070-paired.lcY4qm/. Each full revision directory contains <target>.log, <target>.LastTest.log, binary hashes, the CTest manifest, and the build configuration. results.tsv records all 34 exits. The local combined transcript is /tmp/pr3070-paired-lease.log; the comparison script and outputs are /tmp/pr3070-compare.py, /tmp/pr3070-comparison.txt, and /tmp/pr3070-comparison-summary.txt.

This establishes paired baseline reproduction of the observed failure signatures for these 17 targets, rather than an inference from source identity. It does not establish the identity of the suppressed EXL3 and phase-trace exceptions. A separately labelled diagnostic scratch rerun must print those caught strings before assigning their exact cause.

The full suite remains failing, and model acceptance remains unestablished by this run. This is not a waiver, a successful full-suite result, or evidence that unexecuted paths work. No fixes, capability changes, or test exceptions are proposed by this diagnostic comment.