Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
ID: ISSUE-LOCAL-01M2CWFFYH3N4HSD6RR3EC3M7Z
Title: the Strix vLLM oracle suite turns a low-disk box into 49 opaque test failures
Row: BACKEND-GATE-ROCM-VLLM
State: OPEN
Kind: bug
GitHub: -
Mirror: PENDING
Availability: FULL
Created: 2026-09-13
Updated: 2026-09-13
Closed: -

## Problem

On a host with little free disk, tests/tools/test_strix_vllm_oracle.py reports 49 failures and 1 error, and scripts/agent-preflight.sh fails its 'tools suites' gate with them. The cause is not the tree: tools/bench/strix_vllm_oracle/worker.py:257 raises ValueError('disk headroom exhausted') from session.headroom(), the test's built() helper asserts returncode == 0, and every case that builds a session fails with that traceback buried in an assertEqual message. Reproduced 2026-09-13 on a dev box at 97% full (15 GiB free of 447 GiB) against a PRISTINE git archive of origin/main ee0644eab, 49 failures and 1 error, byte-identical to what the same suite reports on row/MODEL-MM-QWEN4-EXP-ROCM-CHUNKED-H2D, whose diff touches no file under tools/ or tests/tools/. So this is an environment condition that an unrelated row's preflight reads as its own red. A headroom guard that cannot run should SKIP with the named reason, the way preflight already reports its five argument-starved gates, rather than fail 49 cases whose messages do not say 'disk'. Found while repairing MODEL-MM-QWEN4-EXP-ROCM-CHUNKED-H2D; not fixed there, because the fix is this row's tooling and needs its own scope.

## Resolution

-
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
ID: ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW
Title: OWED: chunked H2D through a pinned bounce buffer, llama.cpp's 4 x 64 MiB shape
Row: MODEL-MM-QWEN4-EXP
State: OPEN
State: CLOSED
Kind: task
GitHub: -
Mirror: PENDING
Availability: FULL
Created: 2026-09-12
Updated: 2026-09-12
Closed: -
Updated: 2026-09-13
Closed: 2026-09-13

## Problem

ResidentWeight stages every weight with one hipMemcpyAsync straight out of a PAGEABLE, file-backed mmap. llama.cpp does not: llama-model-loader.cpp uses a set of pinned host buffers (4 x 64 MiB) and copies the weight through them in chunks, so the driver never has to pin or stage an arbitrarily large pageable range itself. If the gfx1151 svm_range_set_attr stall survives the host-residency fixes in .agents/specs/rocm-host-residency-after-upload.md, the pageable file-backed source is itself the trigger and this is the next change. It is deliberately NOT built there: it touches every staged weight on every backend and needs its own measurement.

## Resolution

-
Built and measured 2026-09-13. RocmBackend::Copy now stages a host-to-device copy of 64 MiB or more, out of unregistered host storage, into device memory, on a non-capturing stream, through four pinned 64 MiB buffers with an event each -- llama.cpp's shape at 10bf611e5, src/llama-model-loader.cpp:1440 and :1449. Spec .agents/specs/rocm-chunked-pinned-h2d.md; decision and loop in the HIP-free include/vt/rocm/rocm_pinned_h2d.h, table-tested in tests/vt/test_rocm_pinned_h2d.cpp (9 cases / 84 assertions) and reached through Backend::Copy in tests/vt/test_backend_cross_device.cpp (61 cases / 84841 assertions on strix:gpu0, rc job e8bf3b66). It is the change that makes the row's model gate pass: on strix:gpu0 the 67.56 GiB Qwen3.8-Flash-Next UD-IQ1_S produced three generations of 32 tokens with the ring on, and the SAME BINARY with VT_ROCM_PINNED_H2D_MIB=0 wedged in svm_range_set_attr for 135 of 197 wchan samples and was killed at 1200 s (rc job 672093bc).
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
ID: ISSUE-LOCAL-01M2CV6HSEM3QD2H37G9DTBJ0S
Title: ROCm pinned H2D ring is allocated before the decision that would use it
Row: MODEL-MM-QWEN4-EXP
State: CLOSED
Kind: bug
GitHub: -
Mirror: PENDING
Availability: FULL
Created: 2026-09-13
Updated: 2026-09-13
Closed: 2026-09-13

## Problem

RocmBackend::StagedCopy evaluated in.ring_available = EnsureRing(chunk) before ShouldStageH2D, so the first copy of 64 MiB or more on a non-capturing stream allocated 256 MiB of pinned host memory and four events whatever the pointer kinds turned out to be. Measured on gfx1151 with VT_ROCM_MANAGED_ALLOC=1: staged=0 direct=3 chunks=0 ring_bytes=268435456, a ring built and never touched, on the arm whose managed ceiling .agents/environment.md measures as bounded by host RAM. The device case printed the number and asserted nothing about it. A second, narrower defect rides with it: in.stream_capturing was hardcoded false while a separate early return did the work, so the truth table gated a value production never supplied.

## Resolution

Fixed 2026-09-13. EnsureRing is now the LAST term StagedCopy evaluates: the header splits the four cheap terms out as StagingTermsExceptRing, production asks those first and calls the allocator only when they pass, and ShouldStageH2D is spelled as StagingTermsExceptRing(in) && in.ring_available so it stays the single authority the truth table gates. A new 480-input case in tests/vt/test_rocm_pinned_h2d.cpp asserts the two agree over the whole table so the split cannot drift. The capture probe now feeds in.stream_capturing instead of a separate early return, so the guard is the predicate term the table describes; StreamIsCapturing's own return value stays ungated on a board and .agents/specs/rocm-chunked-pinned-h2d.md section 6 says why. RED on strix:gpu0 at c794b5dda with VT_ROCM_MANAGED_ALLOC=1: FAILURE, 1 of 4 assertions failed, ring_bytes=268435456. GREEN at 33a1eaaf8: 4/4, ring_bytes=0. Gates: test_rocm_pinned_h2d 10/1527, cross-device 61/84841, -tc=*DSA* 2/273, -tc=*pinned bounce* 1/8, every selector printing the ROCm-board line. Evidence /workspace/vtchunked/repair-20260913-073233/.
Loading
Loading