fix(BACKEND-GATE-ROCM-SGLANG): diagnose Qwen3-4B first-inference GPU hang on Strix

[localai-org-maint-bot]

Row: BACKEND-GATE-ROCM-SGLANG

Owner: current Strix campaign operator. Parent: #3076, #3053. Existing campaign spec: .agents/specs/strix-qwen3-4b-c4-performance.md.

In leased production qualification cfebd7eb-2612-455c-bd67-32bd28b5c800, pinned vLLM completed c1/c4 qualification (c1 repeatability FAIL, normal teardown). The native adapter returned a successful configure reply after model loading. Its first c1 qualification RUN then aborted with status -6 before an aggregate run result. The earlier configure-failure description was incorrect. stderr reports Asynchronous scheduling is disabled (max_concurrent_batches=1), then HW Exception by GPU node-1 reason GPU Hang. Baseline native revision6e3cbfb940be89e28d1d71c264fd8c3a4e44afeb; engine/adapter/model bindings are the unchanged qualification-manifest-12.json. Controller194a changes teardown only; do not infer it caused the GPU fault.

Root cancelled the job on observing the fault; terminal state killed/143 at09:02:07UTC. Later engine output is not accepted. Read-only leased health job69b0ea7a-9899-4f8b-99d3-5c46d333b48c retained kernel messages showing MES unrecoverable-state warning, failed process-queue eviction, automatic GPU reset at09:01:15 and reset success09:01:18 with VRAM lost. No manual reset or unhealthy-device clearing was performed. This is not evidence of the offending kernel.

NAS evidence: /mnt/nas_share/rc/strix-four-engine-3053.X94a3J/child3108-production-qualification.jGml1g/run/vllm.cpp-qualification/{stderr.log,result.json}; kernel evidence native-hang-health.6z03Bv/health.log in the same campaign directory.

Required: establish post-reset device state before model work, preserve logs and exact bindings, inspect native startup source and loader provenance, then scope a bounded standalone same-binary reproducer under a lease. Stop on further device fault; no automatic recovery commands, pin changes, or performance acceptance. Distinguish sequencing after vLLM from standalone startup only with measured controls. A root cause requires source and executed-path evidence; do not blame the last logged line or an unrelated historical Strix hang.

[localai-org-maint-bot]

Boundary correction verified against the actual JSON: configuration.status=ok, configuration.id=1 and canonical prompt IDs are present. native_adapter.cpp emits that reply only after vllm_engine_load returns success. runs=[] means the first c1 corpus did not return an aggregate result, not that configure failed or that zero internal tokens executed. The issue title/body now reflect first-inference failure. The root operator previously summarized this incorrectly as startup/configure failure; this correction supersedes that statement.

[localai-org-maint-bot]

Fresh read-only Strix health checks completed at 11:30-11:31 UTC on 9 September. Jobs 54e87e01-ac4e-4f49-bbcf-a2eaa917e58b, a1d0ec62-7afd-4084-8ac1-8ee207181500, and 6523609e-c9b1-42c1-ace5-8f436a29696b each exited 0 under bounded leases. No model was loaded and no reset was requested.

The retained model directory, vLLM/SGLang environments, adapters, and native library still exist. This checks presence, not current file hashes. Initial GPU utilization was16%; five follow-up samples from11:30:53 through11:31:01 showed0% and no visible KFD process. The kernel query at11:31:21 returned only the known09:01 reset and no newer matching fault. The first activity job lacked rg, so a separate grep-based job completed that query; its exit0 is not inferred from the failed query.

This establishes availability for a bounded diagnostic, not reliable inference or a fault repair. Full binding checks remain required before model work. Evidence: campaign strix-four-engine-3053.X94a3J/resume-health.jZORkQ on the resolved NAS. health.log SHA25622ae0916982917024cf069f7511b7304a11356050304f2192de80f389723ef08; activity.log SHA2564fd0a3184a07b06f5976161c7b526c7c9f640db37dee3ed669863057237fbcfa; kernel-fault-history.log SHA25660fa57b821c1fdd466e2ff02bcdda21bc052e6c6bc9c52fc2c5b037a254e22f1.

[localai-org-maint-bot]

Proposed native-only first-c1 control spec committed locally as 0a5399ad09b49d5e7b4576daee698bd449b1b104, at .agents/specs/strix-native-first-c1-control.md. It is not pushed or landed. The spec binds retained canonical input IDs, preserves model and binary identities, runs one c1 corpus, and stops on faults without reset or retry. It does not grant token tolerance or performance acceptance.

Pre-edit and corrected staged gates each reached actual exit 0. The corrected staged log SHA256 is b0ef6619979cb833bada0b540202f71ae8a4491b117f89bf97f26a6eddc9e5c6. Exact parent/head size, style, and trailer checks passed; four artifact-dependent checks remain unmeasured. An earlier staged run was deliberately cancelled with exit 143 to remove a blank line at EOF; it is not credited as passing.

Implementation is pending approval of the proposed design and the reviewed #3110 command. #3110 now has independent CPU/source PASS; operator verification is running. Hardware diagnostic, fault attribution, and benchmark acceptance remain pending.

[localai-org-maint-bot]

Developer approved the bounded native-only c1 control in the current session: unchanged binaries and model, six prompts,128 tokens,15-minute lease,no GPU resets. Committed design0a5399ad09b49d5e7b4576daee698bd449b1b104 now has implementation authority. Fresh implementer dispatched to extend reviewed #3110 head656ce703 using the committed spec, test-first and scoped files only. Independent review and operator gate precede any hardware run. Approval does not change SGLang signal-status policy, numerical tolerance, engine pins, or benchmark acceptance. #3111 stays open because a successful control would not itself resolve the GPU fault.

[localai-org-maint-bot]

Independent review of919ae3ca found one P2 regression-coverage gap. Production correctly validates top-level canonical token arrays, but the boolean fixture corrupts both top-level and configuration arrays. Removing only the top-level validator therefore survives the existing suite because configuration validation still refuses the input. An asymmetric top-level True/configuration1 probe detects the removed guard before adapter launch; Python list equality would otherwise treat those values as equal. Reviewer probe baseline SHA66af08f5adf21d84448e4efac29774d22dd341df215db3b3233e58adaf87206a, mutant SHA4781bd0f8a5638c1d6f556272eea5262bd484bfe664a35fbd351671efa7b52f7. Fresh implementer assigned tests plus owning-spec repair only, preserving production and all acceptance rules. Author and reviewer full gates remain on immutable919ae3ca; no hardware launch occurs on the failed review.

[localai-org-maint-bot]

Original-head verification completed at 919ae3ca49bc355948598f5d2305c2533c44c6d2. The author frozen gate and independent reviewer full gate both returned actual exit 0. Review remains FAIL for one P2 coverage gap: top-level boolean token IDs paired with configuration integer IDs lack an independent regression. Production validation is unchanged and correct. A fresh implementer is repairing tests and scoped evidence. The operator gate and approved native-only hardware control remain pending. Four artifact checks remain unmeasured. Current rc ps is empty and strix:gpu0 reports ready/free. No throughput result or speed advantage is established.

[localai-org-maint-bot]

The fresh coverage repair is committed at 80f1277784abe5c647fab6ccddea39dade2a267b. It adds one independent public-entry regression and scoped evidence notes; production code remains byte-identical to 919ae3ca. The original 24-test suite passed with the top-level guard deleted. The new asymmetric boolean/integer test passes correct production and fails that deletion at the intended no-engine-construction assertion. Restored focused suites pass 63 tests. Pre-edit and staged full gates returned actual exit 0 before commit. Immutable author verification and fresh scoped review are running. An earlier ENOSPC mutation attempt is explicitly excluded. Operator verification and the bounded native hardware control remain pending; no parity or speed claim is made.

[localai-org-maint-bot]

Fresh scoped review now reports PASS at 80f1277784abe5c647fab6ccddea39dade2a267b. Independent focused tests pass 63 cases, the exact missing guard fails only the new regression, and byte-restored diagnostic tests pass 25 cases. The author immutable gate returned exit 0. Reviewer full run 98321 exited 1 because the reviewer checkout was detached; its log is preserved. After branch-only setup correction at unchanged HEAD, targeted role verification and corrected full run 97131 returned exit 0. Four artifact checks remain unmeasured. The operator full gate is now running on the exact repaired commit, original session 38786. The bounded native c1 hardware control remains pending; no token-parity or performance result is established.

[localai-org-maint-bot]

Operator full gate 38786 completed with actual exit 0 at 15:15:37 UTC on unchanged 80f1277784abe5c647fab6ccddea39dade2a267b; worktree clean and image released. Log SHA256 4d6657688941a9d0a9859c8715180125c9e68a3ba2653b1c1f19714a4cd5dbb2. Four artifact omissions remain explicit. Approved native-first-c1 control is now running as rc job 52d7df93-eac5-4ac3-b429-0a2b519188fe, started 15:16:10 UTC, maximum lease 900 seconds. Source archive SHA256 787c8ff1b1d6bbdea69d2146cea87ff51f7f08aa4bd631a4528889f16f303eca. Output: native3111-first-c1.o1NINF under the resolved NAS campaign. Startup health reports no KFD processes and 13% GPU use; retained kernel history contains the earlier 09:01 reset and no later reset at startup. This is not an idle performance measurement. No resets or retries are authorized in this control.

[localai-org-maint-bot]

The approved isolated native-first-c1 control completed: rc job 52d7df93-eac5-4ac3-b429-0a2b519188fe succeeded with exit 0, 15:16:10 to 15:20:28 UTC. Aggregate diagnostic PASS, SHA256 5b4f75c0b7e6cbe3e2455fc561cecd2660660950716b9b6486244b54a0382f6e. Operator assertions confirm aggregate engine equals its separately published record; exactly configure, qualification c1 repetition 0, shutdown; six complete 128-token requests with canonical inputs; parent exit 0; no adopted children; absent process group; restored lifecycle; unchanged before/after bindings; no errors. Kernel logs before and after are byte-identical. After-run health reports GPU use 0% and no KFD processes. All health commands returned 0. No retry or reset occurred. Evidence is native3111-first-c1.o1NINF under the resolved NAS campaign. The earlier GPU hang did not reproduce with native first on this run. This does not establish the fault cause, output-token parity, or a performance advantage. The six-request cold diagnostic is not the approved 96-request c1/c4/c32 benchmark. Keep the GPU fault and benchmark acceptance gaps open.