From b1b0716e92e4f881927e3cee7e19fe48c04963f1 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 23:38:35 +0000 Subject: [PATCH 01/10] spec(MODEL-MM-QWEN4-EXP): host residency after a ROCm staging upload The 67.56 GiB Qwen3.8-Flash-Next UD-IQ1_S loads on gfx1151 with zero op refusals and then wedges forever in svm_range_set_attr on a 31 GiB host. We call no SVM API ourselves, so the stall is host residency, and two sites compound to make it: PrefaultBorrowedSpan faults in every borrowed span synchronously, and nothing releases those pages once the device copy exists. llama.cpp does the opposite on both counts at the recorded pin. The spec also records a reversal. RocmPlatform::residency_policy() still carries the reason "on a unified part freeing the host copy after upload would free the ONLY copy". #2511 falsified that: gfx1151 reports pageableMemoryAccess = 0, the host-alias arm is not taken, and the staging branch always makes a real second copy. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M2BZ5DZ2710201WMH9TNSVH3.md | 19 ++ .../ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW.md | 19 ++ .../specs/rocm-host-residency-after-upload.md | 216 ++++++++++++++++++ 3 files changed, 254 insertions(+) create mode 100644 .agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2BZ5DZ2710201WMH9TNSVH3.md create mode 100644 .agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW.md create mode 100644 .agents/specs/rocm-host-residency-after-upload.md diff --git a/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2BZ5DZ2710201WMH9TNSVH3.md b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2BZ5DZ2710201WMH9TNSVH3.md new file mode 100644 index 000000000..a4d249ece --- /dev/null +++ b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2BZ5DZ2710201WMH9TNSVH3.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2BZ5DZ2710201WMH9TNSVH3 +Title: gfx1151: the whole GGUF is prefaulted and never released, so a 67.56 GiB load wedges in svm_range_set_attr +Row: MODEL-MM-QWEN4-EXP +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +On gfx1151 the 67.56 GiB Qwen3.8-Flash-Next UD-IQ1_S loads with zero op refusals and then wedges forever in svm_range_set_attr (uninterruptible, gpu_busy 0) on a ~31 GiB host. Two host-residency defects compound. (1) Every borrowed GGUF span is synchronously prefaulted -- madvise(WILLNEED) plus a one-byte-per-page touch loop over the whole span (PrefaultBorrowedSpan, qwen3_5_gguf_weights.cpp) -- faulting in 65.488 GiB that is read exactly once. (2) Nothing releases those pages after the device upload: ResidentWeight's staging arm copies the borrow to the device and leaves every source page mapped for the process lifetime, and OwnedTensor::ReleaseHost() refuses to madvise a BORROWED buffer on the argument that clean file-backed pages need no help. That argument holds against the page reclaimer and fails against the KFD's resident-system-memory accounting. llama.cpp (pin 10bf611e5) does the opposite on both counts: MAP_POPULATE plus an advisory posix_madvise with no synchronous touch loop, and unmap_fragment(0, mmap_used.first) after load, which for a fully offloaded model munmaps the entire mapping. + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW.md b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW.md new file mode 100644 index 000000000..8893e41c1 --- /dev/null +++ b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW +Title: OWED: chunked H2D through a pinned bounce buffer, llama.cpp's 4 x 64 MiB shape +Row: MODEL-MM-QWEN4-EXP +State: OPEN +Kind: task +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-12 +Updated: 2026-09-12 +Closed: - + +## Problem + +ResidentWeight stages every weight with one hipMemcpyAsync straight out of a PAGEABLE, file-backed mmap. llama.cpp does not: llama-model-loader.cpp uses a set of pinned host buffers (4 x 64 MiB) and copies the weight through them in chunks, so the driver never has to pin or stage an arbitrarily large pageable range itself. If the gfx1151 svm_range_set_attr stall survives the host-residency fixes in .agents/specs/rocm-host-residency-after-upload.md, the pageable file-backed source is itself the trigger and this is the next change. It is deliberately NOT built there: it touches every staged weight on every backend and needs its own measurement. + +## Resolution + +- diff --git a/.agents/specs/rocm-host-residency-after-upload.md b/.agents/specs/rocm-host-residency-after-upload.md new file mode 100644 index 000000000..c70e3fda5 --- /dev/null +++ b/.agents/specs/rocm-host-residency-after-upload.md @@ -0,0 +1,216 @@ +# ROCm host residency after upload — stop prefaulting, and release the source + +Row: `MODEL-MM-QWEN4-EXP` +Issues: `ISSUE-LOCAL-01M2BZ5DZ2710201WMH9TNSVH3` (the defect), +`ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW` (owed: chunked H2D) + +## 1. The defect + +On `strix:gpu0` (gfx1151) the 67.56 GiB `Qwen3.8-Flash-Next UD-IQ1_S` loads with +zero op refusals and then wedges forever inside `svm_range_set_attr`. The thread +is uninterruptible and `gpu_busy` reads 0. The host has about 31 GiB of RAM. We +call no SVM API ourselves: the staging path is `hipMalloc` plus +`hipMemcpyAsync` (`src/vt/rocm/rocm_backend.hip`). + +The stall is a host-residency problem, and two sites compound to make it. + +**One. Every borrowed GGUF span is synchronously prefaulted.** +`PrefaultBorrowedSpan` (`src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp`) +issues `madvise(MADV_WILLNEED)` and then reads one byte per page across the whole +span. The two keep-quant call sites pass `prefault = true` unconditionally. On +this checkpoint that faults in 65.488 GiB of file pages that each weight reads +exactly once, on a box with half that much RAM. + +The prefault has a good reason on the CPU tier, and only there: a weight left +BORROWED in the mapping is not resident until first touch, and without the +prefault those faults land in the timed prefill. On a device that stages, the +weight is copied to the device once at load and the host pages are never read +again. There is nothing to keep warm. + +**Two. Nothing releases the source pages after the device upload.** +`ResidentWeight` (`src/vllm/model_executor/models/qwen3_5.cpp`) stages the weight +with `Alloc` + `Copy` and then calls `AdoptDeviceBytesAsHost`, which returns +immediately for a GGUF borrow. `OwnedTensor::ReleaseHost()` likewise refuses to +`madvise` a BORROWED buffer, arguing that the pages are clean and file-backed and +the kernel can reclaim them unaided. + +That argument holds against the page reclaimer and fails against the KFD. The +resident pages are still mapped into this process, and that is what the +accounting in `svm_range_set_attr` walks. Existing release machinery does not +reach this path either: `GgufFile::DropSpanResidency` is this tree's port of +llama.cpp's `unmap_fragment` and is called only for tensors the loader COPIED or +EXPANDED; `ReleaseDirectUploadSource` is inert because the GGUF loader never sets +`mmap_src`; and the bf16 MoE release loop in `qwen3_5.cpp` has no counterpart on +the keep-quant tower path, which is the entire 67.56 GiB. + +## 2. #2511 falsified the premise the ROCm policy still states + +`RocmPlatform::residency_policy()` (`src/vllm/platforms/rocm.cpp`) leaves +`release_host_weights_after_upload` false and its comment gives the reason: + +> on a unified part (780M, Strix Halo) freeing the host copy after "upload" +> would free the ONLY copy + +That is no longer true, and a stale comment is what kept 65 GiB pinned. Since +**#2511**, gfx1151 reports `pageableMemoryAccess = 0`. `HostMemoryIsDeviceAddressable` +therefore answers false (`src/vt/rocm/rocm_backend.hip`, `src/vllm/platforms/rocm.cpp`), +`ResidentWeight`'s host-alias arm is not taken (`qwen3_5.cpp`), and the staging +branch below it always makes a real second copy on the device. The host copy has +not been the only copy on this part since that change landed. + +This spec records the reversal and repairs the comment. It does NOT flip +`release_host_weights_after_upload`: that flag is read only through +`ShouldReleaseHostWeights` / `ShouldInterleaveLoadStream`, both of which also +require `marlin_committed`, which is false on ROCm, so flipping it would change +no behaviour and would assert a pool/release measurement nobody has taken. The +release this spec adds is gated on the property that is actually load-bearing and +is checkable at the call site — the device cannot dereference host memory, and +the borrow is a re-faultable read-only file mapping — not on a policy bit whose +other consumers are dead here. + +## 3. Oracle + +`llama-cpp` pin `10bf611e5` (b10451), the recorded pin in +`.agents/oracles/llama-cpp.md`. + +- `src/llama-model-loader.cpp:1567-1577` grows `mmap_used` ONLY for a tensor + that lands in a HOST buffer. A tensor going to a device buffer takes + `ggml_backend_tensor_set` and grows nothing. +- `src/llama-model-loader.cpp:1683-1694` then calls + `unmap_fragment(0, mmap_used.first)`. For a fully offloaded model + `mmap_used.first` is the whole mapping, so the whole mapping goes. +- `src/llama-mmap.cpp:492-507` shows `unmap_fragment` is a real `munmap`, not an + advisory hint. +- `src/llama-mmap.cpp:449-461` is the prefetch: `MAP_POPULATE` plus an advisory + `posix_madvise(POSIX_MADV_WILLNEED)`. There is **no synchronous touch loop**. + +Peak host residency for a fully offloaded model is therefore O(one tensor) +during load and zero after it. Ours is O(whole model) for the process lifetime. + +## 4. Scope — two fixes + +### Fix 1 — release the source pages after a staging upload + +In `ResidentWeight`'s staging arm, once the device copy exists and has completed, +drop the resident interior pages of a BORROWED, file-backed source span when the +platform is NOT host-addressable. The borrow itself is untouched and stays a +valid, re-faultable `PROT_READ MAP_PRIVATE` view, so a later read re-faults from +the file and nothing observes a byte difference. This is `unmap_fragment`'s +intent expressed as `MADV_DONTNEED`, which is what this tree already uses for the +same job in `DropSpanResidency`, `ReleaseHost` and `AdoptDeviceBytesAsHost`. + +Two things are load-bearing. + +**It must be memoized, and #1299 is why.** `ResidentWeight` is called about 1,361 +times per forward step on this checkpoint. A release that re-tests its condition +on every call would `MADV_DONTNEED` the very pages the GPU is about to read, on +every step, and the kernel would fault them straight back in. Correctness +survives that; throughput does not. The release therefore goes inside +`AdoptDeviceBytesAsHost`, which is reached only from behind `if (!w.d_dev)` — the +same memo that made the aligned-borrow branch of `MakeHostBytesDeviceAliasable` +a repeat hazard when it had none. The red test covers the REPEAT call, not only +the first. + +**The copy must have completed.** `RocmBackend::Copy` is `hipMemcpyAsync` on a +stream. Dropping the source pages while the copy may still be reading them is a +correctness bug, so the staging arm synchronizes the queue before the release. +It is behind the same `d_dev` memo, so it costs one synchronize per weight on +the first forward and nothing thereafter. + +### Fix 2 — make the prefault device-aware + +`prefault` is decided at the two keep-quant call sites in +`qwen3_5_gguf_weights.cpp` and passed as a literal `true`. Give it the same +device term `quant_repack` got in #2406: default OFF when the resolved device +cannot dereference host memory, because on that device the prefault reads the +whole tower off disk into pages exactly one `memcpy` then reads. An explicit +`VT_GGUF_PREFAULT=1` (or `vllm_cpp.mmap.prefault: true`) still wins, so the A/B +stays available in the same binary; `ResolveGgufPrefault` remains the sole reader +of that variable and the new helper only decides the DEFAULT. + +### Out of scope + +Chunked H2D through a pinned bounce buffer — llama.cpp's 4 x 64 MiB shape — is +NOT built here. See `## Owed`. + +### Also in scope + +The stale comment at `src/vllm/platforms/rocm.cpp` (§2). + +## 5. Tests — red first + +`tests/vllm/model_executor/test_resident_weight_host_addressable.cpp` already +carries the fake platform and fake backend this needs, and it enters through +`Qwen3_5EmbeddingTable`, the named production bridge over `ResidentWeight`. New +cases there: + +1. **The pages actually go.** Build a real temporary file, `mmap` it + `PROT_READ MAP_PRIVATE`, touch every page so `RssFile` in + `/proc/self/status` grows by the span, borrow it into an `OwnedTensor`, and + stage it through `Qwen3_5EmbeddingTable` on a platform whose + `host_memory_is_device_addressable()` is false. Assert the resident set drops + back. This is the assertion a counter cannot make: the release either unmaps + the pages or it does not. +2. **Once, not 1,361 times.** Call the same bridge repeatedly on the same weight + and assert the release instrument counts exactly one release. Mutating the + memo away must fail this case. +3. **A host-addressable platform is unchanged.** The alias arm never reaches the + release, and the existing cases in this file must stay green. + +For fix 2 the instrument already exists: `NoteGgufPrefaultedSpan` / +`GgufPrefaultSnapshot` in `include/vllm/config/weight_residency.h` count spans +actually prefaulted, and were added precisely because a prefault changes no byte +and a byte-transparency case cannot see it. A truth-table case over the new +device helper pins the default per device and pins that an explicit knob wins. + +Every added assertion is mutation-proven: delete the release, delete the memo, +delete the synchronize, delete the device term, and the focused suite must go +red for each. + +## 6. Gates + +- Focused: `-tc=*resident*`, `-tc=*prefault*`, `-tc=*DSA*` (273 assertions). +- Full ROCm cross-device suite: 60 cases / 84833 assertions, unchanged. +- `scripts/check-agent-record.py`, `scripts/check-commit-style.py`, + `scripts/check-commit-trailers.py`, `scripts/check-pr-size.py`. +- The model gate: does the 67.56 GiB UD-IQ1_S at + `/workspace/ckpt/qwen4exp-flash-next-iq1s` forward on `strix:gpu0` and produce + a token? If it does, that is this row's G3, and load time, peak host and device + memory, prefill and decode tok/s, recipe, revisions, model sha256, environment + and contention are recorded with it. gfx1151 greedy decode fails about two runs + in five with an illegal GPU memory access + (`ISSUE-LOCAL-01M2BY2M2ATNVR3XQKV2DB1BJD`), so every measurement is repeated at + least three times and reported as a spread. If it does not forward, NO number + is recorded and the thread's `/proc//stat` state and `wchan` are reported + instead. + +## 7. Risks + +- **A released page that something still reads.** The borrow stays valid, so a + read re-faults from the file and the bytes are identical. The cost of being + wrong is a page fault, not a wrong token. This is the same contract + `DropSpanResidency` has carried since it landed. +- **A repeat release on the hot path.** #1299's shape exactly. Held off by the + `d_dev` memo and pinned by case 2 above. +- **A dropped page mid-DMA.** Held off by the synchronize, which is itself + behind the memo. +- **The stall survives both fixes.** Then the pageable file-backed source is + itself the trigger and the owed chunked-H2D change is required. That is a + legitimate result and is reported with `wchan` evidence rather than papered + over. + +## Owed + +- `ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW` — chunked H2D through a pinned bounce + buffer, llama.cpp's 4 x 64 MiB shape. Needed only if the stall survives fixes 1 + and 2. Medium-size and touches every staged weight on every backend, so it gets + its own row, spec and measurement. + +## 8. Stop conditions + +- The release cannot be placed behind an existing memo without a new one: STOP + and return `NEEDS_DECISION` rather than adding an unmemoized hot-path + `madvise`. +- `strix:gpu0` is unreachable or the controller is down: the model gate is + reported UNVERIFIED. It is never replaced by an `ssh` plus a file mutex the + fleet cannot see. From b786fdf67a1a01775689846d28ade844114bf00f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 23:38:48 +0000 Subject: [PATCH 02/10] fix(MODEL-MM-QWEN4-EXP): release a staged borrow's source pages, and stop prefaulting for a staging device ResidentWeight's staging arm copies a weight to the device and then leaves every source page of it mapped for the process lifetime. On a GGUF keep-quant load those pages are the whole model, and the load-time prefault has already faulted all of them in. On gfx1151 that is 65.488 GiB resident against a 31 GiB host, and the load wedges in svm_range_set_attr. MaybeReleaseStagedBorrowSource drops the spent source pages once the device copy exists, on a device whose kernels cannot read host memory, for a borrow whose mmap_fd says the pages are file-backed and therefore re-faultable. It is called from behind the d_dev memo, which is #1299's lesson: this function runs about 1,361 times per forward step, and an unmemoized release would madvise away the pages the GPU is about to read on every one of them. It synchronizes the queue first, because the staging copy is an async stream copy. GgufPrefaultForDevice gives the prefault the same device term QuantRepackForDevice got in #2406, and GgufLoadPolicy::FromEnv carries it into the field the loader reads. VT_GGUF_PREFAULT and vllm_cpp.mmap.prefault are answered before the device is consulted, so the same-binary A/B stays reachable on the device the default narrows. The stale reason in RocmPlatform::residency_policy() is corrected in the same change. #2511 falsified it, and leaving it in place is how the defect survived. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- include/vllm/config/weight_residency.h | 12 + .../model_loader/gguf_keep_quant.h | 37 ++ .../model_executor/models/qwen3_5_weights.h | 67 ++++ src/vllm/config/weight_residency.cpp | 10 + .../model_loader/gguf_keep_quant.cpp | 20 + src/vllm/model_executor/models/qwen3_5.cpp | 19 + .../models/qwen3_5_gguf_weights.cpp | 9 +- .../model_executor/models/qwen3_5_weights.cpp | 45 +++ .../models/qwen4_exp_weights.cpp | 14 +- src/vllm/platforms/rocm.cpp | 33 +- .../test_resident_weight_host_addressable.cpp | 347 ++++++++++++++++++ 11 files changed, 598 insertions(+), 15 deletions(-) diff --git a/include/vllm/config/weight_residency.h b/include/vllm/config/weight_residency.h index 09699c53d..3a88170b4 100644 --- a/include/vllm/config/weight_residency.h +++ b/include/vllm/config/weight_residency.h @@ -487,6 +487,18 @@ void NoteExpertStreamGeometry(int64_t slots, int64_t slot_bytes); // `setenv` could not affect anything, so both of its arms ran the same way. bool ResolveGgufPrefault(); +// Whether the prefault answer above was DECIDED by an operator -- the +// `VT_GGUF_PREFAULT` knob or a `vllm_cpp.mmap.prefault` key -- rather than +// falling through to the built-in default. +// +// It exists so that a caller may give the DEFAULT a device term without taking +// the decision away from whoever set the knob (`GgufPrefaultForDevice`, +// gguf_keep_quant.h). It lives HERE, beside the resolver, so that +// `VT_GGUF_PREFAULT` still has exactly one reader in the tree: a second +// `getenv` of that name in the model layer is how the two would come to +// disagree about one load. +bool GgufPrefaultIsExplicit(); + // Called by the prefault site after it has actually faulted a span in. Feeds // GgufPrefaultedSpanCount above; nothing else reads it. void NoteGgufPrefaultedSpan(); diff --git a/include/vllm/model_executor/model_loader/gguf_keep_quant.h b/include/vllm/model_executor/model_loader/gguf_keep_quant.h index 978f78066..500bf11d7 100644 --- a/include/vllm/model_executor/model_loader/gguf_keep_quant.h +++ b/include/vllm/model_executor/model_loader/gguf_keep_quant.h @@ -218,6 +218,34 @@ bool GgufNvfp4ComputeAvailable(vt::DeviceType dev); bool QuantRepackForDevice(bool keep_quant, bool cpu_ref, bool host_repack_active, vt::DeviceType dev); +// Whether a BORROWED span should be PREFAULTED at load, for a load the engine +// resolved onto `dev`. +// +// The prefault (`PrefaultBorrowedSpan`, qwen3_5_gguf_weights.cpp) faults a +// borrowed span in at load with `madvise(MADV_WILLNEED)` plus a synchronous +// one-byte-per-page read, so the page traps land off the timed prefill instead +// of inside it. That is worth paying for exactly when the borrowed pages are +// what the FORWARD reads -- the CPU tier, and a device whose kernels can +// dereference host storage. +// +// On a device that STAGES, they are not. `ResidentWeight` copies the weight to +// the device once and the host pages are never read again, so the prefault +// reads the whole model off disk into pages one `memcpy` then consumes. On +// gfx1151 that is 65.488 GiB of resident file pages on a 31 GiB host, and the +// load wedges in `svm_range_set_attr` +// (.agents/specs/rocm-host-residency-after-upload.md). +// +// THIS DECIDES THE DEFAULT ONLY. `VT_GGUF_PREFAULT` and +// `vllm_cpp.mmap.prefault` still win, because the A/B they exist for has to +// stay available in the same binary on the very device this narrows. +// `ResolveGgufPrefault` remains the sole reader of that variable; this asks +// `GgufPrefaultIsExplicit()` whether it was set at all. +// +// `dev` is a PARAMETER for the same reason `QuantRepackForDevice`'s is: a +// decision that takes its inputs can be checked from a host that is not the one +// it decides for. +bool GgufPrefaultForDevice(vt::DeviceType dev); + // Loader-wide residency policy. struct GgufLoadPolicy { // Master switch for keep-quant residency. The STRUCT default stays false so @@ -343,6 +371,15 @@ struct GgufLoadPolicy { // until then a wrong default would be a silent correctness bug, not a slow // path. bool elem_kn_repack = false; + // Whether a BORROWED span is PREFAULTED at load. `FromEnv` resolves it through + // `GgufPrefaultForDevice(dev)` above, exactly as `quant_repack` is resolved + // through `QuantRepackForDevice(..., dev)`, so the decision carries the + // ENGINE's device rather than being a literal at each call site. + // + // The STRUCT default is `true`, which is what every call site passed before + // this field existed, so a hand-built policy and a caller that still passes + // the argument itself are both unchanged. + bool prefault = true; // Optional observer; null in production. GgufRoutingAudit audit; diff --git a/include/vllm/model_executor/models/qwen3_5_weights.h b/include/vllm/model_executor/models/qwen3_5_weights.h index dde6ac5f9..50686ff33 100644 --- a/include/vllm/model_executor/models/qwen3_5_weights.h +++ b/include/vllm/model_executor/models/qwen3_5_weights.h @@ -225,6 +225,73 @@ struct OwnedTensor { // behavior (house convention for a default-on residency change). void AdoptDeviceBytesAsHost(vt::Backend& backend, const OwnedTensor& w); +// Drop the resident host pages a just-STAGED weight's borrowed source span still +// holds, on a device whose kernels cannot read host memory. +// +// WHAT IT IS FOR. `ResidentWeight`'s staging arm copies the weight to the device +// and leaves every source page mapped for the process lifetime. For a GGUF +// keep-quant load that is the WHOLE MODEL: 65.488 GiB on the 67.56 GiB +// `Qwen3.8-Flash-Next UD-IQ1_S`, faulted in by the load-time prefault and read +// exactly once. `ReleaseHost()`'s borrowed branch declines to `madvise` those +// pages, arguing that clean file-backed pages are reclaimable without our help. +// That argument holds against the page reclaimer and FAILS against the KFD, +// which walks what this process has resident: on gfx1151 the load then wedges +// forever in `svm_range_set_attr` with a 31 GiB host +// (.agents/specs/rocm-host-residency-after-upload.md). +// +// This is llama.cpp's `unmap_fragment` (`src/llama-mmap.cpp:492-507`, called at +// `src/llama-model-loader.cpp:1683-1694` after the offloaded tensors are set) +// expressed as `MADV_DONTNEED` rather than `munmap`, which is the spelling this +// tree already uses for the same job in `GgufFile::DropSpanResidency`, +// `ReleaseHost` and `AdoptDeviceBytesAsHost`. The borrow itself is untouched and +// stays a valid, re-faultable `PROT_READ MAP_PRIVATE` view, so a later read +// re-faults the identical bytes from the file. The cost of being wrong is a page +// fault, never a wrong token. +// +// THREE PRECONDITIONS, ALL CHECKED HERE. +// +// 1. The device cannot dereference host storage +// (`vt::Backend::DeviceMemoryIsHostAddressable()` false). Where it can, the +// bytes ARE the weight and `AdoptDeviceBytesAsHost` handles it instead. +// 2. `bytes` is BORROWED. An owned buffer is `ReleaseHost`'s business. +// 3. `mmap_fd >= 0`. This is the discriminator that makes the call SAFE, and it +// is not a convenience. `MADV_DONTNEED` on a file-backed private mapping +// drops re-faultable pages; on ANONYMOUS memory it ZEROES them. The other +// borrow producer in this tree is a tied pair's shared bf16 expansion, which +// is anonymous and whose keep-alive both tensors share. Only the mmap-borrow +// path sets `mmap_fd` (`SourceOfSpan`, qwen3_5_gguf_weights.cpp), so asking +// for it is asking "are these pages backed by a file I can re-read". +// +// IT MUST BE CALLED FROM BEHIND THE `d_dev` MEMO, and #1299 is why: the caller +// reaches `ResidentWeight` about 1,361 times per forward step on that +// checkpoint, so a release that re-tested its condition on every call would +// `MADV_DONTNEED` the pages the GPU is about to read on every step and the +// kernel would fault them straight back in. Correctness survives that; +// throughput does not. The one production call site is inside +// `if (!w.d_dev)` and `BorrowReleaseSnapshot().calls` is what makes that +// checkable rather than asserted. +// +// It SYNCHRONIZES `queue` before releasing anything, because the staging copy is +// `hipMemcpyAsync` on a stream and dropping the source pages under a live DMA is +// a correctness bug rather than a residency one. The synchronize is skipped +// entirely when the preconditions do not hold, so a backend this does not apply +// to pays nothing. +// +// Returns true when pages were released. +bool MaybeReleaseStagedBorrowSource(vt::Backend& backend, vt::Queue& queue, + const OwnedTensor& w); + +// What `MaybeReleaseStagedBorrowSource` has done in this process. `calls` counts +// the releases that HAPPENED, not the invocations that declined, for the same +// reason `NoteGgufPrefaultedSpan` counts after the fact: an `madvise` changes no +// byte, so a count of the ones that ran is the only thing that separates a +// release from a skip. Read it at a stated point; it is cumulative. +struct BorrowReleaseStats { + uint64_t calls = 0; + uint64_t bytes = 0; +}; +BorrowReleaseStats BorrowReleaseSnapshot(); + // The whole of `src` as a ZERO-COPY view: same bytes, shape, dtype, `nk` and // layout markers, with a keep-alive on `src`'s buffer. // diff --git a/src/vllm/config/weight_residency.cpp b/src/vllm/config/weight_residency.cpp index 560423d51..e67ff2016 100644 --- a/src/vllm/config/weight_residency.cpp +++ b/src/vllm/config/weight_residency.cpp @@ -1033,6 +1033,16 @@ bool ResolveGgufPrefault() { /*builtin_default=*/true); } +// The header states why this is not a second `getenv` in the model layer. The +// PRESENCE of the variable is what makes the answer explicit, empty string +// included: `ResolveResidencyBool` reads an empty value as an explicit OFF, so +// treating "" as unset here would let the device default overrule an operator +// who deliberately emptied the knob. +bool GgufPrefaultIsExplicit() { + return std::getenv("VT_GGUF_PREFAULT") != nullptr || + ActiveWeightResidencyConfig().prefault.has_value(); +} + bool ExpertStreamRequestedFrom(const char* env_value, std::optional configured) { // THE FIRST-CHARACTER RULE, transcribed rather than normalised. The site this diff --git a/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp b/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp index f5096530e..bed8ce7d3 100644 --- a/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp +++ b/src/vllm/model_executor/model_loader/gguf_keep_quant.cpp @@ -8,6 +8,7 @@ #include "vllm/config/weight_residency.h" #include "vllm/model_executor/device_placement.h" +#include "vllm/platforms/interface.h" #include "vt/ops.h" #include "vt/quant.h" @@ -98,6 +99,20 @@ bool QuantRepackForDevice(bool keep_quant, bool cpu_ref, dev == vt::DeviceType::kCPU; } +// See the header. The ORDER of the terms is the contract: an explicit knob is +// answered before the device is consulted, so the same-binary A/B still reaches +// a staging device. +bool GgufPrefaultForDevice(vt::DeviceType dev) { + if (!ResolveGgufPrefault()) return false; + if (GgufPrefaultIsExplicit()) return true; + if (dev == vt::DeviceType::kCPU) return true; + // A device with no registered platform cannot be asked, and this function is + // not the place to refuse a load. Answering ON leaves such a caller with + // exactly the behaviour it had before this term existed. + if (!vllm::platforms::HasPlatform(dev)) return true; + return vllm::platforms::GetPlatform(dev).host_memory_is_device_addressable(); +} + const char* Name(GgufTensorRole role) { switch (role) { case GgufTensorRole::kMatmulWeight: return "matmul_weight"; @@ -431,6 +446,11 @@ GgufLoadPolicy GgufLoadPolicy::FromEnv( // rather than called there. p.quant_repack = QuantRepackForDevice(p.keep_quant, p.cpu_ref, vt::cpu::QuantRepackActive(), dev); + // The load-time prefault, with the SAME device term and for a reason of the + // same shape: the transform is worth paying for only where the forward reads + // the borrowed pages. See `GgufPrefaultForDevice`. `VT_GGUF_PREFAULT` and + // `vllm_cpp.mmap.prefault` still win over the device. + p.prefault = GgufPrefaultForDevice(dev); // KERNEL-GEMM-CPU-TILED lever 2, elementwise [N,K] -> [K,N] repack-at-load. // OPT-IN ONLY (default false) because the repacked bytes are transposed and // only the CPU MatmulBTKernel honours Tensor.elem_kn_repacked today; see the diff --git a/src/vllm/model_executor/models/qwen3_5.cpp b/src/vllm/model_executor/models/qwen3_5.cpp index 5eeac9291..257f5a352 100644 --- a/src/vllm/model_executor/models/qwen3_5.cpp +++ b/src/vllm/model_executor/models/qwen3_5.cpp @@ -1298,6 +1298,25 @@ Tensor ResidentWeight(Dev d, const OwnedTensor& w, std::vector shape = d.b.Copy(d.q, p, w.bytes.data(), nb); Backend* bk = &d.b; w.d_dev = std::shared_ptr(p, [bk](void* q) { bk->Free(q); }); + // THE SOURCE PAGES ARE SPENT, AND ON A STAGING DEVICE NOTHING WAS DROPPING + // THEM (.agents/specs/rocm-host-residency-after-upload.md). The copy above + // is the only read of this weight's host bytes that will ever happen there, + // and for a GGUF keep-quant load those bytes are the whole model -- 65.488 + // GiB on `Qwen3.8-Flash-Next UD-IQ1_S`, faulted in by the load-time prefault + // and then held resident for the process lifetime, which is what wedges a + // 31 GiB gfx1151 host inside `svm_range_set_attr`. llama.cpp + // `unmap_fragment`s the equivalent range after its offloaded tensors are set + // (llama-model-loader.cpp:1683-1694). `AdoptDeviceBytesAsHost` below cannot + // do this job: it returns immediately for a GGUF borrow, by design, because + // the borrow must NOT be re-pointed at a device allocation the kernels are + // the only readers of. + // + // INSIDE THE `d_dev` MEMO ON PURPOSE, which is the whole of #1299's lesson: + // this function runs about 1,361 times per forward step, and a release that + // re-tested its condition each time would madvise away the pages the GPU is + // about to read, every step. The helper states its other two preconditions + // and synchronizes the queue before it touches anything. + vllm::MaybeReleaseStagedBorrowSource(d.b, d.q, w); // Same adoption as the dense block's ResidentWeight: on a host-addressable // device the uploaded buffer IS the host buffer, so keeping the mirror // costs a second full copy of the model out of the same unified RAM. diff --git a/src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp b/src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp index 0f55e3f8a..de2f4a293 100644 --- a/src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp @@ -298,13 +298,14 @@ OwnedTensor OwnGgufKeptSlice(const GgufFile& g, const GgufLoadPolicy& pol, int64_t k, int64_t row_offset) { if (r == GgufResidency::kKeepQuant) { return OwnGgufQuantBlocks(t, n, k, row_offset, MmapSrc(g, pol), - pol.quant_repack); + pol.quant_repack, /*cuda_align=*/false, + pol.prefault); } VT_CHECK(r == GgufResidency::kKeepF16, "qwen3_5 gguf: OwnGgufKeptSlice called for a non-keep residency on " + t.name); return OwnGgufF16(t, n, k, row_offset, MmapSrc(g, pol), /*nk=*/true, - pol.elem_kn_repack, /*prefault=*/true, pol.weight_value_dtype); + pol.elem_kn_repack, pol.prefault, pol.weight_value_dtype); } bool HasTensor(const GgufFile& g, const std::string& name) { @@ -816,14 +817,14 @@ void LoadEmbedAndHead(const GgufFile& g, const GgufLoadPolicy& pol, // it does on the f16 arm — this tensor is a table, not a [N,K] GEMM weight. *embed = OwnGgufQuantBlocks(et, et.shape[0], et.shape[1], /*row_offset=*/0, MmapSrc(g, pol), /*repack=*/false, - /*cuda_align=*/false, /*prefault=*/true, + /*cuda_align=*/false, pol.prefault, GgufTensorRole::kEmbeddingTable); embed->nk = false; } else if (embed_r == GgufResidency::kKeepF16) { VT_CHECK(et.shape.size() == 2, "qwen3_5 gguf: token_embd must be 2-D"); // Embedding gather table: never repacked (EmbeddingKernel reads it row-wise). *embed = OwnGgufF16(et, et.shape[0], et.shape[1], 0, MmapSrc(g, pol), - /*nk=*/false, /*elem_kn_repack=*/false, /*prefault=*/true, + /*nk=*/false, /*elem_kn_repack=*/false, pol.prefault, pol.weight_value_dtype); } else { VT_CHECK(embed_r == GgufResidency::kExpandBf16, diff --git a/src/vllm/model_executor/models/qwen3_5_weights.cpp b/src/vllm/model_executor/models/qwen3_5_weights.cpp index d0846fd02..29b4f7c75 100644 --- a/src/vllm/model_executor/models/qwen3_5_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_5_weights.cpp @@ -368,6 +368,51 @@ OwnedTensor BorrowWholeOwnedTensor(OwnedTensor& src) { return v; } +namespace { + +struct AtomicBorrowReleaseStats { + std::atomic calls{0}; + std::atomic bytes{0}; +}; + +AtomicBorrowReleaseStats& BorrowReleaseStatsRef() { + static AtomicBorrowReleaseStats s; + return s; +} + +} // namespace + +BorrowReleaseStats BorrowReleaseSnapshot() { + const AtomicBorrowReleaseStats& s = BorrowReleaseStatsRef(); + BorrowReleaseStats out; + out.calls = s.calls.load(std::memory_order_relaxed); + out.bytes = s.bytes.load(std::memory_order_relaxed); + return out; +} + +bool MaybeReleaseStagedBorrowSource(vt::Backend& backend, vt::Queue& queue, + const OwnedTensor& w) { + // See the header for each of the three. The ORDER matters only in that the + // cheap, allocation-free tests come before the synchronize: a backend this + // does not apply to must not pay a stream sync per weight to find that out. + if (backend.DeviceMemoryIsHostAddressable()) return false; + if (w.bytes.empty() || !w.bytes.borrowed()) return false; + if (w.mmap_fd < 0) return false; + // THE STAGING COPY IS ASYNCHRONOUS. `RocmBackend::Copy` is `hipMemcpyAsync` on + // `queue`'s stream, and the CUDA backend's is the same shape. Releasing the + // source under a live DMA would be a correctness bug, not a residency one, so + // the copy is waited for here. This runs once per weight (see the `d_dev` + // memo note in the header), so it is a first-forward cost and not a per-step + // one. + backend.Synchronize(queue); + const size_t nb = w.bytes.size(); + DropResidentInteriorPages(w.bytes.data(), nb); + AtomicBorrowReleaseStats& st = BorrowReleaseStatsRef(); + st.calls.fetch_add(1, std::memory_order_relaxed); + st.bytes.fetch_add(static_cast(nb), std::memory_order_relaxed); + return true; +} + void AdoptDeviceBytesAsHost(vt::Backend& backend, const OwnedTensor& w) { if (w.d_dev == nullptr) return; // ENG-LOAD-DIRECT-UPLOAD: a direct-upload borrow is the ONE borrow that may be diff --git a/src/vllm/model_executor/models/qwen4_exp_weights.cpp b/src/vllm/model_executor/models/qwen4_exp_weights.cpp index a0c921137..82fd982b5 100644 --- a/src/vllm/model_executor/models/qwen4_exp_weights.cpp +++ b/src/vllm/model_executor/models/qwen4_exp_weights.cpp @@ -134,10 +134,11 @@ OwnedTensor LoadMatmul(const GgufFile& g, const GgufLoadPolicy& pol, const GgufResidency r = pol.Route(t, GgufTensorRole::kMatmulWeight); if (r == GgufResidency::kKeepQuant) return OwnGgufQuantBlocks(t, n, k, /*row_offset=*/0, MmapSrc(g, pol), - pol.quant_repack); + pol.quant_repack, /*cuda_align=*/false, + pol.prefault); if (r == GgufResidency::kKeepF16) return OwnGgufF16(t, n, k, /*row_offset=*/0, MmapSrc(g, pol), /*nk=*/true, - pol.elem_kn_repack); + pol.elem_kn_repack, pol.prefault); return ExpandBf16(g, name, {n, k}, /*nk=*/true); } @@ -156,7 +157,8 @@ OwnedTensor LoadStackedExperts(const GgufFile& g, const GgufLoadPolicy& pol, // [E*N, K] and reshaped back. The bytes are identical either way; only the // recorded shape differs, and the consumer slices by expert. OwnedTensor o = OwnGgufQuantBlocks(t, e * n, k, /*row_offset=*/0, - MmapSrc(g, pol), pol.quant_repack); + MmapSrc(g, pol), pol.quant_repack, + /*cuda_align=*/false, pol.prefault); o.rank = 3; o.shape[0] = e; o.shape[1] = n; @@ -722,14 +724,16 @@ Qwen4ExpWeights LoadQwen4ExpFromGguf(const GgufFile& gguf, if (r == GgufResidency::kKeepQuant) { w.ngram_table = OwnGgufQuantBlocks(t, rows, cols, /*row_offset=*/0, MmapSrc(gguf, pol), - /*repack=*/false); + /*repack=*/false, + /*cuda_align=*/false, pol.prefault); // A gather table is read row-wise, never dotted, so it must NOT carry the // matmul orientation flag: `nk` is what tells a consumer this is a // MatmulBT operand. w.ngram_table.nk = false; } else if (r == GgufResidency::kKeepF16) { w.ngram_table = OwnGgufF16(t, rows, cols, /*row_offset=*/0, - MmapSrc(gguf, pol), /*nk=*/false); + MmapSrc(gguf, pol), /*nk=*/false, + /*elem_kn_repack=*/false, pol.prefault); } else { w.ngram_table = ExpandBf16(gguf, nm, {rows, cols}, /*nk=*/false); } diff --git a/src/vllm/platforms/rocm.cpp b/src/vllm/platforms/rocm.cpp index 6567f7be2..78faf7b20 100644 --- a/src/vllm/platforms/rocm.cpp +++ b/src/vllm/platforms/rocm.cpp @@ -125,12 +125,33 @@ class RocmPlatform final : public Platform { // probe) — the ONE field the device-fit check reads. The other two fields // stay DEFAULT/false, unlike CUDA's: `release_host_weights_after_upload` // and `uses_device_memory_pool` are separate policy questions (a discrete - // card's host-copy release and DevicePool reuse) this row does not touch, - // because on a unified part (780M, Strix Halo) freeing the host copy after - // "upload" would free the ONLY copy — the same answer CPU, Metal and Vulkan - // give for the same reason, and per-DEVICE (not per-DEVICE-TYPE) besides. - // Flip those when a discrete board's release/pool behavior is actually - // measured, not as a side effect of making the budget check reachable. + // card's host-copy release and DevicePool reuse) this row does not touch. + // + // THE REASON THIS COMMENT USED TO GIVE IS FALSE, AND SAYING SO IS THE POINT. + // It read: "on a unified part (780M, Strix Halo) freeing the host copy after + // 'upload' would free the ONLY copy — the same answer CPU, Metal and Vulkan + // give for the same reason". #2511 falsified that premise. gfx1151 reports + // `pageableMemoryAccess = 0`, so `HostMemoryIsDeviceAddressable` answers + // false (`vt/rocm/rocm_backend.hip`, and `host_memory_is_device_addressable` + // above), `ResidentWeight`'s host-alias arm is not taken, and its staging + // branch always makes a real second copy on the device. The host copy has not + // been the only copy on this part since that change landed. Left uncorrected, + // that sentence is what kept 65.488 GiB of spent GGUF source pages resident + // on a 31 GiB host until the load wedged in `svm_range_set_attr` + // (.agents/specs/rocm-host-residency-after-upload.md). + // + // THE FLAG NONETHELESS STAYS FALSE, for a different and checkable reason. + // Nothing reads it except `ShouldReleaseHostWeights` and + // `ShouldInterleaveLoadStream` (`platforms/interface.h`), and BOTH also + // require `marlin_committed`, which no ROCm path sets. Flipping it here would + // therefore change no behaviour while asserting a release/pool measurement + // nobody has taken on this board. The host-residency release that defect + // needed is gated instead on the property that is load-bearing and checkable + // at the call site — the device cannot dereference host memory and the source + // is a re-faultable read-only file mapping — in + // `MaybeReleaseStagedBorrowSource` (qwen3_5_weights.h). Flip these two when a + // board's release/pool behavior is actually measured, not as a side effect of + // making the budget check reachable. ResidencyPolicy residency_policy() const override { ResidencyPolicy p; p.device_memory_total_bytes = device_memory_total_bytes_; diff --git a/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp b/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp index d3dd8d846..109d9f663 100644 --- a/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp +++ b/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp @@ -42,15 +42,26 @@ // staging flag — is what selects it. #include +#include #include +#include #include #include #include +#include #include #include #include #include "vllm/model_executor/models/owned_bytes.h" +#if defined(__linux__) +#include +#include +#endif + +#include "vllm/config/weight_residency.h" +#include "vllm/model_executor/model_loader/gguf_keep_quant.h" +#include "vllm/model_executor/models/qwen3_5.h" #include "vllm/model_executor/models/qwen3_5_dense.h" #include "vllm/model_executor/models/qwen3_5_internal.h" #include "vllm/model_executor/models/qwen3_5_weights.h" @@ -739,3 +750,339 @@ TEST_CASE("stage-vs-retag: a model larger than the whole box REFUSES without wra CHECK_FALSE(vllm::StagingFitsModel(200ull << 30, 119ull << 30, 12ull << 30)); CHECK_FALSE(vllm::StagingFitsModel(1ull << 30, 8ull << 30, 12ull << 30)); } + + +// --------------------------------------------------------------------------- +// THE SPENT SOURCE PAGES OF A STAGED BORROW (.agents/specs/ +// rocm-host-residency-after-upload.md). +// +// `ResidentWeight`'s staging arm copies a weight to the device and then leaves +// every source page of it mapped for the process lifetime. For a GGUF keep-quant +// load those source pages ARE the whole model: 65.488 GiB on the 67.56 GiB +// `Qwen3.8-Flash-Next UD-IQ1_S`, faulted in by the load-time prefault and read +// exactly once by the copy. `ReleaseHost()`'s borrowed branch declines to touch +// them on the argument that clean file-backed pages are the kernel's problem, +// which holds against the page reclaimer and fails against the KFD: on gfx1151 +// the load wedges forever inside `svm_range_set_attr` with a 31 GiB host. +// +// WHY THIS IS MEASURED AS RESIDENT PAGES AND NOT AS A COUNTER ALONE. An +// `madvise(MADV_DONTNEED)` changes no byte, moves no pointer and allocates +// nothing, so every observable a staging case already has stays identical +// whether it ran or not. A counter says the call happened; only the kernel's own +// accounting says the pages went. Both are asserted below, because a counter +// that is the ONLY instrument is the shape that has produced false greens here +// before. +#if defined(__linux__) +namespace { + +// `VmRSS` in KiB, the figure the KFD's resident-system-memory accounting is +// about. Returns 0 when it cannot be read, which a case treats as "cannot +// measure" rather than as a pass. +size_t VmRssKib() { + std::FILE* f = std::fopen("/proc/self/status", "r"); + if (f == nullptr) return 0; + char line[256]; + size_t kib = 0; + while (std::fgets(line, sizeof(line), f) != nullptr) { + if (std::strncmp(line, "VmRSS:", 6) == 0) { + kib = static_cast(std::strtoull(line + 6, nullptr, 10)); + break; + } + } + std::fclose(f); + return kib; +} + +// A real file, mapped PROT_READ MAP_PRIVATE, standing in for the GGUF mapping. +// It has to be a REAL file: the whole safety argument for the release is that +// the borrow stays a re-faultable view, and a mapping with nothing behind it +// cannot prove that. +class MappedFile { + public: + explicit MappedFile(size_t bytes) : bytes_(bytes) { + std::snprintf(path_, sizeof(path_), "/tmp/vt_borrow_release_XXXXXX"); + fd_ = ::mkstemp(path_); + if (fd_ < 0) return; + ::unlink(path_); // the descriptor keeps it alive; nothing is left behind + std::vector chunk(1u << 20); + for (size_t i = 0; i < chunk.size(); ++i) + chunk[i] = static_cast((i * 31 + 7) & 0xFF); + for (size_t off = 0; off < bytes_; off += chunk.size()) { + const size_t n = std::min(chunk.size(), bytes_ - off); + if (::write(fd_, chunk.data(), n) != static_cast(n)) return; + } + void* p = ::mmap(nullptr, bytes_, PROT_READ, MAP_PRIVATE, fd_, 0); + if (p == MAP_FAILED) return; + addr_ = static_cast(p); + } + ~MappedFile() { + if (addr_ != nullptr) ::munmap(addr_, bytes_); + if (fd_ >= 0) ::close(fd_); + } + MappedFile(const MappedFile&) = delete; + MappedFile& operator=(const MappedFile&) = delete; + + // Make every page RESIDENT, which is what the load-time prefault does and what + // gives this case something to watch go away. + void Prefault() const { + volatile uint8_t sink = 0; + const size_t ps = static_cast(::sysconf(_SC_PAGESIZE)); + for (size_t off = 0; off < bytes_; off += ps) sink = sink ^ addr_[off]; + (void)sink; + } + + bool ok() const { return addr_ != nullptr; } + const uint8_t* data() const { return addr_; } + size_t size() const { return bytes_; } + int fd() const { return fd_; } + + private: + char path_[64] = {}; + int fd_ = -1; + uint8_t* addr_ = nullptr; + size_t bytes_ = 0; +}; + +// A weight that BORROWS `f`, shaped exactly as the GGUF keep-quant borrow the +// loader builds: a keep-alive on the mapping, and `mmap_fd` set, which is the +// discriminator that says these pages are backed by a file and may be dropped. +OwnedTensor BorrowWeight(const MappedFile& f, int64_t vocab, int64_t hidden) { + OwnedTensor w; + w.dtype = DType::kBF16; + w.rank = 2; + w.shape[0] = vocab; + w.shape[1] = hidden; + w.nk = false; // a gather table, which is what the bridge below binds + std::shared_ptr keep(static_cast(f.data()), + [](const void*) {}); // owned by MappedFile + w.bytes = vllm::OwnedBytes::Borrow(f.data(), f.size(), std::move(keep)); + w.mmap_fd = f.fd(); + w.mmap_file_offset = 0; + return w; +} + +constexpr int64_t kBigVocab = 4096; +constexpr int64_t kBigHidden = 8192; // 4096 * 8192 * 2 = 64 MiB + +} // namespace + +TEST_CASE("a STAGED borrow's source pages are released, and the RSS says so") { + const PlatformArm arm(false); // a device that cannot read host memory + MappedFile f(static_cast(kBigVocab * kBigHidden) * 2); + REQUIRE(f.ok()); + f.Prefault(); + + const size_t rss_resident = VmRssKib(); + REQUIRE(rss_resident > 0); // unreadable /proc is "cannot measure", not a pass + + const OwnedTensor w = BorrowWeight(f, kBigVocab, kBigHidden); + const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); + + // THROUGH THE PRODUCTION BRIDGE, not through the test seam. `Qwen3_5EmbeddingTable` + // is the call the forward makes; a case that hand-built the operand one step + // later would prove the helper works and not that anything reaches it. + Queue q = XpuQueue(); + const Tensor t = vllm::Qwen3_5EmbeddingTable(Fake(), q, w, kBigVocab, kBigHidden); + + REQUIRE(w.d_dev != nullptr); // it really staged + REQUIRE(t.data == w.d_dev.get()); + + const vllm::BorrowReleaseStats after = vllm::BorrowReleaseSnapshot(); + CHECK(after.calls == before.calls + 1); + CHECK(after.bytes == before.bytes + f.size()); + + // THE ASSERTION THE COUNTER CANNOT MAKE. The 64 MiB of file pages this case + // faulted in are gone from this process's resident set. The bar is HALF the + // span rather than all of it, because the staging copy itself allocated 64 MiB + // of device (here: malloc'd) memory that is also resident and is supposed to + // stay: what is asserted is that the SOURCE went, against that background. + const size_t rss_after = VmRssKib(); + CHECK(rss_resident > rss_after); + CHECK(rss_resident - rss_after >= (f.size() / 2) / 1024); + + // ...AND THE BORROW IS STILL A VALID VIEW, which is the entire safety + // argument: MADV_DONTNEED on a private file mapping drops re-faultable pages, + // so reading them back re-faults the identical bytes from the file. If this + // ever reads differently, the release is touching something anonymous. + CHECK(w.bytes.data() == f.data()); + CHECK(std::memcmp(w.bytes.data(), t.data, f.size()) == 0); +} + +TEST_CASE("the source release happens ONCE, not on every step (#1299's shape)") { + // #1299 IS THE HAZARD, AND IT IS NOT HYPOTHETICAL HERE. `ResidentWeight` runs + // about 1,361 times per forward step on the target checkpoint. An unmemoized + // release would `MADV_DONTNEED` the pages the GPU is about to read on every + // one of them, and the kernel would fault them straight back in: correct + // tokens, destroyed throughput, and nothing in a token gate to see it. The + // release is inside `if (!w.d_dev)` for exactly this reason, and this case is + // what makes that placement checkable. Move it outside the memo and the count + // below becomes 8. + const PlatformArm arm(false); + MappedFile f(1u << 20); + REQUIRE(f.ok()); + f.Prefault(); + const int64_t vocab = 64; + const int64_t hidden = (1 << 20) / (64 * 2); + const OwnedTensor w = BorrowWeight(f, vocab, hidden); + + const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); + Queue q = XpuQueue(); + for (int i = 0; i < 8; ++i) + (void)vllm::Qwen3_5EmbeddingTable(Fake(), q, w, vocab, hidden); + const vllm::BorrowReleaseStats after = vllm::BorrowReleaseSnapshot(); + + CHECK(after.calls == before.calls + 1); + CHECK(after.bytes == before.bytes + f.size()); +} + +TEST_CASE("a HOST-ADDRESSABLE device releases nothing: those bytes are the weight") { + // The other side of the predicate. Where the kernels can follow a host + // pointer the source pages are not spent at all -- they are what the weight + // IS -- so dropping them would cost a fault on the very next read. + // `AdoptDeviceBytesAsHost` owns that case and declines it for a borrow. + const PlatformArm arm(true); + MappedFile f(1u << 20); + REQUIRE(f.ok()); + f.Prefault(); + const int64_t vocab = 64; + const int64_t hidden = (1 << 20) / (64 * 2); + const OwnedTensor w = BorrowWeight(f, vocab, hidden); + + const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); + Queue q = XpuQueue(); + (void)vllm::Qwen3_5EmbeddingTable(Fake(), q, w, vocab, hidden); + CHECK(vllm::BorrowReleaseSnapshot().calls == before.calls); +} + +TEST_CASE("an ANONYMOUS borrow is never released: MADV_DONTNEED would ZERO it") { + // THE DISCRIMINATOR, AND WHY IT IS NOT A CONVENIENCE. `MADV_DONTNEED` on a + // private FILE mapping drops re-faultable pages; on ANONYMOUS memory it + // zeroes them. The tree's other borrow producer is a tied pair's shared bf16 + // expansion, which is anonymous and which both tensors read. `mmap_fd` is set + // only on the mmap-borrow path, so requiring it is what keeps the release off + // those pages. Clear the field and this case is what goes red -- with the + // wrong bytes, not with a count. + const PlatformArm arm(false); + const size_t nb = static_cast(kN * kK) * 2; + auto* block = new uint8_t[nb]; + for (size_t i = 0; i < nb; ++i) block[i] = static_cast(i & 0xFF); + const std::vector expect(block, block + nb); + std::shared_ptr keep(static_cast(block), + [](const void* p) { + delete[] static_cast(p); + }); + OwnedTensor w; + w.dtype = DType::kBF16; + w.rank = 2; + w.shape[0] = kN; + w.shape[1] = kK; + w.nk = false; + w.bytes = vllm::OwnedBytes::Borrow(block, nb, std::move(keep)); + // mmap_fd deliberately LEFT AT -1: this is anonymous memory. + REQUIRE(w.mmap_fd == -1); + + const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); + Queue q = XpuQueue(); + (void)vllm::Qwen3_5EmbeddingTable(Fake(), q, w, kN, kK); + CHECK(vllm::BorrowReleaseSnapshot().calls == before.calls); + CHECK(std::memcmp(w.bytes.data(), expect.data(), nb) == 0); +} +#endif // __linux__ + + +// --------------------------------------------------------------------------- +// FIX 2: THE LOAD-TIME PREFAULT'S DEVICE TERM (`GgufPrefaultForDevice`). +// +// The prefault faults a borrowed span in at load so its page traps land off the +// timed prefill. That is worth paying for where the FORWARD reads the borrowed +// pages: the CPU tier, and a device whose kernels can dereference host storage. +// On a device that stages, the copy is the only read there will ever be, so the +// prefault reads the whole model off disk to populate pages one `memcpy` then +// consumes -- 65.488 GiB on gfx1151, against a 31 GiB host. +// +// This fake platform is the only way to check that from a host with no such +// device: the decision takes `dev`, so it can be asked about a device this +// machine is not. +namespace { + +// setenv/unsetenv around one case, restored on every exit path including a +// REQUIRE that aborts the body -- the same discipline `PlatformArm` above +// exists for, and for the same reason. +struct EnvArm { + EnvArm(const char* name, const char* value) : name_(name) { + const char* prev = std::getenv(name); + had_ = prev != nullptr; + if (had_) prev_ = prev; + if (value == nullptr) ::unsetenv(name); + else ::setenv(name, value, 1); + } + ~EnvArm() { + if (had_) ::setenv(name_, prev_.c_str(), 1); + else ::unsetenv(name_); + } + const char* name_; + bool had_ = false; + std::string prev_; +}; + +} // namespace + +TEST_CASE("the prefault DEFAULT follows the device, and the knob still wins") { + // With no knob set, the answer is the platform's own + // `host_memory_is_device_addressable()`. + { + const EnvArm knob("VT_GGUF_PREFAULT", nullptr); + { + const PlatformArm arm(true); // kernels can read host memory + CHECK(vllm::GgufPrefaultForDevice(DeviceType::kXPU)); + } + { + const PlatformArm arm(false); // a device that stages + CHECK_FALSE(vllm::GgufPrefaultForDevice(DeviceType::kXPU)); + } + // The CPU tier is where the prefault was measured and where it stays ON: + // there the borrowed pages ARE what the forward reads. + CHECK(vllm::GgufPrefaultForDevice(DeviceType::kCPU)); + } + + // AN EXPLICIT KNOB IS ANSWERED BEFORE THE DEVICE IS CONSULTED. The A/B this + // variable exists for has to stay reachable in the same binary on the very + // device the default narrows; a device term that could not be overridden + // would have removed the instrument along with the cost. + { + const PlatformArm arm(false); + const EnvArm on("VT_GGUF_PREFAULT", "1"); + CHECK(vllm::GgufPrefaultForDevice(DeviceType::kXPU)); + CHECK(vllm::GgufPrefaultIsExplicit()); + } + // ...and =0 still turns it off everywhere, including where the default is ON. + { + const PlatformArm arm(true); + const EnvArm off("VT_GGUF_PREFAULT", "0"); + CHECK_FALSE(vllm::GgufPrefaultForDevice(DeviceType::kXPU)); + CHECK_FALSE(vllm::GgufPrefaultForDevice(DeviceType::kCPU)); + } +} + +TEST_CASE("FromEnv carries the prefault decision into the policy the loader reads") { + // THE ROUTING HALF. The decision above is only worth anything if the loader + // asks it: before this change every call site passed a literal `true`. This + // is the case that would stay green if `FromEnv` stopped setting the field, + // so it asserts the FIELD, on a policy built the way a load builds one. + const EnvArm knob("VT_GGUF_PREFAULT", nullptr); + { + const PlatformArm arm(false); + const vllm::GgufLoadPolicy p = vllm::GgufLoadPolicy::FromEnv(DeviceType::kXPU); + CHECK_FALSE(p.prefault); + } + { + const PlatformArm arm(true); + const vllm::GgufLoadPolicy p = vllm::GgufLoadPolicy::FromEnv(DeviceType::kXPU); + CHECK(p.prefault); + } + const vllm::GgufLoadPolicy cpu = vllm::GgufLoadPolicy::FromEnv(DeviceType::kCPU); + CHECK(cpu.prefault); + // A hand-built policy is unchanged: the struct default is what every call site + // passed before the field existed. + CHECK(vllm::GgufLoadPolicy{}.prefault); +} From 7c462138f4f961f7df88195f4448360fa7c39fcf Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 23:43:16 +0000 Subject: [PATCH 03/10] record(MODEL-MM-QWEN4-EXP): repair the three anchors this change moved Adding the release helper to qwen3_5_weights.h and one call line to qwen3_5.cpp shifted three cited symbols. SPEC-MTP-GGUF and SPEC-DFLASH-GGUF each moved by one line in qwen3_5_gguf_weights.cpp, and QUANT-QWEN38-27B-NVFP4-ARM's Fp8Weight moved from 652 to 719. The last one is cited as a markdown link whose label and target must both say the same number, which is why repairing only the visible half left the checker red. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/engine-matrix.md | 4 ++-- .agents/quantization-matrix.md | 2 +- .agents/specs/rocm-host-residency-after-upload.md | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index fc8bce95e..4f8da9334 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -172,8 +172,8 @@ lifecycle are unchanged. |---|---|---|---|---|---|---|---|---| | `SPEC-MTP` | Qwen3.6 MTP heads, k=1 first. **M-mtp-0 CLOSED 2026-07-24: the standalone draft head is oracle-parity-proven on BOTH checkpoints** (27B dense + 35B MoE, k=1, vLLM 0.25.0 executable @ pin `e24d1b24`) - argmax exact on 26/26 unambiguous rows each; the one remaining row per checkpoint is an EXACT oracle top1==top2 tie where vLLM's own `argmax` and `topk` disagree and our pick is a tied maximum; logits within the whole-model bound (atol 0.05 + rtol 0.05), 0/216 out-of-tol on both; shared lm_head isolated is bit-exact on the 35B NVFP4 head. **I2 scheduler-half LANDED (2026-07-24)**: host-side spec plumbing + the FROZEN spec-metadata ABI (spec §2.7) - `SpeculativeConfig`, `DraftTokenIds`, `Request::spec_token_ids`/`NumTokensWithSpec`, populated `scheduled_spec_decode_tokens`, `Scheduler::update_draft_token_ids`, `take_draft_token_ids` seam, `EngineCore::post_step`, `InputBatch::num_accepted_tokens`/`update_req_spec_token_ids`; DEFAULT-OFF and INERT (no `SpeculativeConfig` => `num_lookahead_tokens == 0`). **I3 verify-half LANDED (2026-07-24)**: greedy rejection sampler + per-request logits expansion (see `SPEC-REJECTION`, now `ACTIVE`). **I4 GDN-half LANDED (2026-07-24)**: the GDN speculative slot path + bit-exact state rollback, the piece BOTH GDN-hybrid gate checkpoints need (see `SPEC-GDN-SEGMENTS`, now `ACTIVE`). **I5a GDN LAYER ROUTING + runner spec-metadata upload LANDED (2026-07-24, `CLAIM-SPEC-MTP-I5A`)**: `GdnBlockPaged` now routes a pure-spec batch through `vt::GdnSpecDecode`/`vt::CausalConv1dSpecUpdate` and the runner uploads I4's six spec device tensors — first sub-increment of the scoped M-mtp-1 (I5a GDN wiring → I5b prepare_prefill → I5c MTP paged propose → I5d config+runner-loop+the 27B token gate, spec §5). DEFAULT-OFF INERT, bit-exact vs the I4 ops, no e2e loop yet. **I5b `prepare_prefill_inputs` LANDED (2026-07-24, `CLAIM-SPEC-MTP-I5B`, recorded under `SPEC-REJECTION`)**: the drafter prefill input-prep host routine (shift-splice + `query_len -= num_rejected` + last-token index / metadata) — second scoped M-mtp-1 sub-increment, DEFAULT-OFF INERT, unit-gated RED-first, additive. **I5d CONFIG + RUNNER LOOP LANDED, PARTIAL (2026-07-25, `CLAIM-SPEC-MTP-I5D`)**: `--speculative-config` JSON parse -> `EngineParams::speculative_config`; `LoadedEngine` resolution (`ResolveSpecConfig`/`ResolveMtp`, widened KV `MakeQwen3_5KVCacheSpec(num_spec>0)`, `BuildMtpDraft`, forced sync scheduling, `MakeScheduler(spec)`, `EngineCore(check_for_draft=true)`); the full runner verify/propose loop (draft splice, hidden-tap capture, GDN builder spec-overload feed, k+1 GDN state-slot remap + widened conv cache + draft-KV alloc, `MtpProposePrefill` post-sampling, `take_draft_token_ids`, acceptance telemetry). CUDA `-Werror` 0 warnings, cutlass-ON banner. SPEC-OFF BYTE-IDENTICAL (all gated on `spec_on()`): SACRED 27B 235/235, 35B 315/315, Coder 138/138 + unit test_runner 257 / test_mtp_speculator 169 / test_gdn_metadata_builder 483 / test_ops_gdn 3630 ALL PASS. **The three-way 27B token gate is NOT yet passing** (`tests/parity/test_qwen27_spec_decode.cpp` RUNS the loop + MEASURES the blocker): the spec-ON engine throws on the FIRST prefill step at `gdn_state_gather: working/cache row shapes must match` (`src/vt/ops.cpp:1773`) — I4's spec conv rollback needs the conv row widened to `(K-1)+num_spec` but the non-spec GDN conv ops assume `(K-1)`. Closing needs widened-cache-aware non-spec GDN conv ops + the MIXED `GdnBlockPaged` split/merge. Row LEFT `GATING` at I5e. **I5e LANDED 2026-07-25 (`CLAIM-SPEC-MTP-I5E`) — `SPEC-MTP` LEAVES `GATING`.** Made the non-spec GDN conv ops widened-cache-aware (mirror vLLM `state_len=KERNEL_WIDTH-1` + physical `stride_conv_state_tok`; leading `(K-1)` sub-window; byte-identical at `num_spec==0`, contiguous fast path kept) AND RCA'd the resulting 0-acceptance dead-drafter to the async input-combine overwriting the verify batch's draft position with the committed token (forced off under spec, nullopt-guarded). **THREE-WAY 27B GATE PASSES** (single-request greedy): our-ON == vLLM `--speculative-config mtp` greedy == our-OFF token-for-token; **acceptance 16/16 drafts accepted**, ~16 target steps saved. Spec-OFF SACRED byte-identical (27B 235/235, 35B 315/315, Coder 138/138), `test_ops_gdn` 3678, compute-sanitizer 0 on the spec step. NOT `DONE`: MIXED `GdnBlockPaged` split/merge (concurrency) + throughput A/B are I6. **I6 LANDED 2026-07-25 (`CLAIM-SPEC-MTP-I6`), `benchmark_binding=true` — the §5 c1 THROUGHPUT GATE, first spec-decode speed number:** OURS spec-ON (`examples/vllm-bench` + an additive `--speculative-config` flag, production config) vs pinned vLLM 0.25.0 spec-ON (graphed `vllm serve --speculative-config mtp` + `vllm bench serve`, `enforce_eager=False`/`FULL_AND_PIECEWISE`/inductor; MTP confirmed `Resolved architecture: Qwen3_5MTP`), SAME `{"method":"mtp","num_speculative_tokens":1}`, 27B `~/bench/q36-27b-nvfp4-vllm`, c1, greedy, 8 real prompts x 256 out, prose + code, idle box one-engine-at-a-time under one `flock`, 3 reps (cold TTFT discarded), token-identity re-confirmed FIRST (`test_qwen27_spec_decode` PASS 16/16). RESULT — **ours AT/ABOVE vLLM on EVERY measured axis** (prose / code): TPOT 66.2/62.95 vs 69.1/65.3 ms (ours ~1.04x faster), output tput 15.10/15.72 vs 14.43/15.13 tok/s (+4.6%/+3.9%), ITL 121.6/121.1 vs 123.2 ms, TTFT(warm) 131/131 vs 151.5/181 ms, acceptance ours 0.85/0.92 vs vLLM 0.838 overall (within noise, live drafter both), peak RSS 28.4 GB ON / 24.8 GB OFF (both inside the 119 GiB pool). Spec helps both (ours 1.52x/1.59x, vLLM 1.51x/1.60x TPOT); ours already ~4% faster spec-OFF. STAYS `ACTIVE`: the c>1 mixed spec+non-spec `GdnBlockPaged` split/merge is still refused (needs a row `IndexSelect`/`IndexCopy` vt op) + owes a c>1 A/B, and no user-facing supported `--speculative-config` on the OpenAI server yet (bench flag example-only/additive). Raw logs dgx `~/work/mtp-bench-i6/{results,vresults}`. **I7 LANDED 2026-07-25 (`CLAIM-SPEC-MTP-I7`, `benchmark_binding=true`) — the MIXED spec+non-spec GDN batch (concurrency), the server/CLI `--speculative-config`, and the c>1 A/B — implementation COMPLETE + at vLLM parity; STAYS `ACTIVE` for one honest reason (below), NOT a lag.** New row op `vt::IndexSelect`/`vt::IndexCopy` (CUDA==CPU bit-exact at GDN widths, RED-first); `GdnBlockPagedMixedSpec` split/merge (mirror `qwen_gdn_linear_attn.py:1329-1576`) proven MODEL-INDEPENDENTLY bit-exact (mixed == pure spec + pure prefill, 27B/35B, `test_qwen3_5_gdn_spec_routing`, RED-first by a broken merge); compute-sanitizer 0 on the mixed step + op; server (I5d) + CLI (ABI v6) `--speculative-config`. **c>1 A/B (both spec-ON, same config):** ours ON-PAR-OR-ABOVE vLLM at c2/c4/c8 (output tput within ~+/-2%, ours +1.6%/+2.5% c2, +0.9%/+1.7% c4, +0.9%/-1.1% c8 within noise, prose/code; both ~1.5x spec speedup — does NOT go neutral; acceptance 0.84-0.92 vs vLLM 0.835). **Why STAYS `ACTIVE` (honest, not a lag):** the DONE criterion's strict `token-exact at c>1` clause is a proven MODEL impossibility — the 27B greedy is bf16-batch-nondeterministic (spec-OFF max_seqs 4-vs-1 differs 2/3 short prompts, NO spec involved), affecting vLLM identically, so exact c>1 token identity cannot be met by any correct implementation; c>1 correctness is instead established by the model-independent bit-exact split/merge proof + acceptance parity (near-tie-distributional-gate), with token-exact strict at c1 (I6). No missing work, no lever — the DONE final call is deferred to the user given this criterion ambiguity. SACRED spec-OFF byte-identical 27B 235/235, 35B 315/315, Coder 138/138; CUDA `-Werror` 0 warnings. Raw logs dgx `~/work/mixed-batch/{cN_results,cN_vresults}`. **I8 — `SPEC-MTP` → `DONE` 2026-07-26 (`CLAIM-SPEC-MTP-DONE`, records-only, ZERO code):** the user RATIFIED the deferred c>1 criterion — at concurrency > 1 the DONE bar is the near-tie-distributional form (ours ∈ vLLM's batch-nondeterministic set) + the SPEED delta, NOT strict token-exact (a proven bf16-batch-nondeterminism MODEL impossibility that affects vLLM identically). Both I6-owed DONE items are therefore CLOSED: (1) the MIXED spec+non-spec `GdnBlockPaged` split/merge (I7, model-independently bit-exact + compute-sanitizer 0) with the c2-c8 A/B on-par-or-above vLLM, and (2) the server + CLI + C-ABI(v6) `--speculative-config` flag (I5d/I7, `examples/server/main.cpp`+`examples/cli/main.cpp`+`src/capi/vllm_c.cpp`). MTP k=1 spec-decode is COMPLETE and gated: 27B three-way token-exact at c1 (I5e), c1 above vLLM on every axis (I6), c2-c8 on-par-or-above (I7), spec-OFF byte-identical SACRED (27B 235/235, 35B 315/315, Coder 138/138). This transition is byte-identical BY CONSTRUCTION (`git diff --stat` = records only; ZERO `src/`/`include/`/`examples/` touched, so the I5d/I6/I7 GPU gates stand on this exact code). Tracked follow-ons: the 35B `Qwen3_5MoeMTP` full e2e token gate (M-mtp-2) is now **CLOSED — `DONE` 2026-07-26 (`CLAIM-SPEC-MTP-M-MTP-2`)**: three-way token-exact 16/16 vs the live vLLM 0.25.0 oracle (spec-ON AND spec-OFF), acceptance 16/16 both sides, c1 spec-ON 1.19x TPOT / +16.3% output-tput vs spec-OFF (0.908) — `MODEL-SPEC-qwen3-5-mtp-qwen3-5-moe-mtp` `GATING`→`DONE`, so MTP is `DONE` on BOTH gate models. Remaining spec-decode follow-on: `SPEC-DFLASH` (oracle-BLOCKED, vllm#40898) | T1 | `vllm/v1/worker/gpu/spec_decode/mtp/speculator.py:12`; `vllm/model_executor/models/qwen3_5_mtp.py:63,129-165,192-301`; **I5d** `vllm/engine/arg_utils.py` (`--speculative-config`); `vllm/v1/worker/gpu/model_runner.py:1455-1489` | `include/vllm/config/speculative.h`; `include/vllm/v1/core/sched/scheduler.h`; `src/vllm/v1/core/sched/scheduler.cpp`; `include/vllm/v1/worker/gpu/input_batch.h`; `include/vllm/model_executor/models/qwen3_5_mtp.h:23,58`; `src/vllm/model_executor/models/qwen3_5_mtp.cpp:271`; `src/vllm/model_executor/models/qwen3_5.cpp:3336,3359`; **I5d** `src/vllm/config/speculative.cpp`; `src/vllm/entrypoints/model_loader.cpp` (`ResolveSpecConfig`/`MakeKVCacheMaybeSpec`/ctor wiring); `src/vllm/v1/worker/gpu/runner.cpp` (splice/tap/GDN spec feed/`propose_drafts`/`take_draft_token_ids`/spec-slot remap/draft-KV alloc); `examples/server/main.cpp` | `tests/vllm/v1/test_scheduler.cpp:1135,1238,1272,1316`; `tests/vllm/v1/worker/test_input_batch.cpp`; `tests/vllm/v1/spec_decode/test_mtp_speculator.cpp:201,225,263,299,331` (7/7 cases, 141 assertions); oracle runner `tests/parity/test_op_parity.cpp:1373` + focused case `:1914` (20/20 assertions, both checkpoints, `VLLM_MTP_REQUIRE_CHECKPOINTS=1`); goldens `tests/parity/goldens/qwen3_5_mtp_head_{27b,35b}/`; dump `tools/parity/dump_qwen3_5_mtp.py:144`; **I5d** `tests/parity/test_qwen27_spec_decode.cpp` (three-way gate, RUNS + measures the RCA blocker); **I6** `examples/bench/{main.cpp,bench_core.h}` (additive `--speculative-config` bench flag + acceptance telemetry); **I7** `tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp` (mixed == pure spec + prefill bit-exact), `tests/parity/test_qwen27_spec_decode_concurrent.cpp`, `tests/vt/test_ops_gdn.cpp` (IndexSelect/IndexCopy); DONE closure [ledger](parity-ledger.md#L714) | [mtp-spec-decode.md](specs/mtp-spec-decode.md) | `DONE` | `72f9fb1` | | `SPEC-MTP-K-GT-1` | **MTP speculation DEPTH (`num_speculative_tokens` > 1).** Ports the autoregressive multi-step propose the k=1 early exit sits in front of, so a configured depth is SERVED instead of silently degraded. Before it, `--num-speculative-tokens 3` reserved KV for 3, captured the verify shape at T=4 and stashed ONE draft per request, with no error and no log; a refusal by name landed first and this row removed it in the same flow. `MtpProposeDrafts` runs the prefill, the k=1 early exit, then `prepare_decode_inputs` and the k-1 single-token draft decode steps over the draft's own paged KV, with `update_draft_inputs` recording each step and feeding it forward. Greedy plus accept-if-equal makes the emitted sequence INDEPENDENT of k, so a token-identity gate cannot see a clamped drafter and every depth assertion needs a positive witness beside the identity. The per-depth counters were the FIRST witness and a fresh review proved them BLIND: they report the LENGTH of the emitted draft list, so a propose that runs one forward and pads all k columns satisfies them, and acceptance is zero at every depth on the CPU model, so no acceptance figure separates the arms either. TWO witnesses survive, because one does not cover both failures. `spec_mtp_draft_decode_forwards() == spec_mtp_propose_calls() * (k - 1)`, counted after each draft decode forward RETURNS and guarded by a non-zero call count, catches a propose that SHORT-CIRCUITS or CLAMPS. A third fresh review then proved it does NOT catch PADDING, since a loop that runs every forward and then discards what it sampled increments it honestly. `spec_mtp_proposals_with_varied_drafts()`, read at the CONSUMER on the array the propose delivered, catches exactly that. NEITHER shows per-column provenance, and neither does a non-zero acceptance count AT DEPTH, which a padded row earns whenever the target repeats a token. The owed DGX gate closes it with a per-depth acceptance RATE against a PADDED CONTROL. The CPU tier therefore proves k drafts are PROPOSED and VERIFIED, never ACCEPTED at depth. DEFAULT unchanged at k=1 (both checkpoints' `n_predict`). **NO speed number at any k>1**: the GPU was held by another session for the whole flow, so the DGX three-way at k=2..4 on the 27B and 35B and the matched-k throughput A/B are OWED, as is the bf16 GDN-state arm (the CPU gate runs the f32 arm because `vt::CausalConv1dSpecUpdate` rejects bf16 off CUDA). Also owed and filed: [#1020](https://github.com/mudler/vllm.cpp/issues/1020), a step whose ACTUAL draft count differs from the configured k leaves the captured verify graph silently. | T1 | `vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py:129-274,335-371,374-419,426-471,597-671,674-771` @ `555967922`; `vllm/config/speculative.py:967-991` | [`src/vllm/v1/worker/gpu/spec_decode/mtp/speculator.cpp`](../src/vllm/v1/worker/gpu/spec_decode/mtp/speculator.cpp) (`MtpProposeDrafts`); [`prepare_decode_inputs.cpp`](../src/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_decode_inputs.cpp); `Qwen3_5MTPModel::GatherHiddenRows` ([qwen3_5.cpp](../src/vllm/model_executor/models/qwen3_5.cpp)); `GPUModelRunner::propose_drafts` + the per-depth counters ([runner.cpp](../src/vllm/v1/worker/gpu/runner.cpp), [runner.h](../include/vllm/v1/worker/gpu/runner.h)); the in-memory `mtp_weights` seam ([model_loader.h](../include/vllm/entrypoints/model_loader.h)) | [`test_mtp_depth`](../tests/vllm/v1/spec_decode/test_mtp_depth.cpp) 5/5, 63 assertions (k=1,2,3,4 through `LoadedEngine`, greedy tokens identical to spec-OFF, each arm witnessed BOTH by the draft decode forwards the propose RAN and by whether the DELIVERED draft row varied with depth; neither witness shows per-column provenance, which is owed to the DGX gate); [`test_prepare_decode_inputs`](../tests/vllm/v1/spec_decode/test_prepare_decode_inputs.cpp) 8/8, 33 (both kernel ports + both `max_model_len` clamps, 5 mutations caught); [`test_speculative_mtp_depth`](../tests/vllm/config/test_speculative_mtp_depth.cpp) 4/4, 20; full CPU suite ctest 493 passed / 0 failed / 2 skipped of 495 (the two skips checkpoint-gated and unrelated) | [mtp-k-gt-1.md](specs/mtp-k-gt-1.md) | `ACTIVE` | `CLAIM-SPEC-MTP-K-GT-1` ([#81](https://github.com/mudler/vllm.cpp/issues/81)) | -| `SPEC-MTP-GGUF` | MTP speculative decoding from a GGUF TARGET. Today `FromModelDir` refuses `mtp`+GGUF outright (`src/vllm/entrypoints/model_loader.cpp:717-723`) on the original spike's assumption that GGUF exports carry no `mtp.*` ([mtp-spec-decode.md](specs/mtp-spec-decode.md):979-980, "until we re-export GGUFs with the head"). That is stale: llama.cpp's Qwen3.5 converter DOES emit the head, under layer-indexed `nextn` naming, and our own `HfConfigFromGguf` ALREADY reads `nextn_predict_layers` (it just discards the value into the trunk layer count). Gap is a `TensorResolver` over `GgufFile` mapping `mtp.*` onto `blk.{L+i}.nextn.*` with dequant-to-bf16, one config field, and narrowing the rejection to `dflash`. `ngram`+GGUF already works and is untouched. Qwen3.5/3.6 only (the widened spec KV path serves no other arch). NO ABI change | T2 | llama.cpp (the producer contract; vLLM has no GGUF MTP path) `conversion/qwen.py:535-604` `_Qwen35MtpMixin` (the authoritative `mtp.*`->`nextn` remapper + `add_nextn_predict_layers`); `gguf-py/gguf/constants.py:129,910-917,1494-1501`; `gguf-py/gguf/tensor_mapping.py` `NEXTN_*` | **G1-G3 LANDED 2026-07-28.** `HfConfigFromGguf` republishes the head depth `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:598` (`c.raw["mtp_num_hidden_layers"] = nextn`, previously read then discarded); the head loader `LoadQwen3_5MTPFromGguf` `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:1457` (+ decl `include/vllm/model_executor/models/qwen3_5_gguf_weights.h:143`) reusing the TRUNK helpers `OwnNormMinus1`/`OwnMatmulWeight`/`OwnBf16`/`LoadAttnGguf`/`LoadMoeGguf` so the head inherits the GGUF (w+1) norm storage, quantization/residency routing and torch [N,K] shapes; `NumMtpLayers`/`UsesDedicatedEmbeddings` exported out of the anon namespace `include/vllm/model_executor/models/qwen3_5_mtp.h`; rejection narrowed to dflash + a head-less-GGUF check `src/vllm/entrypoints/model_loader.cpp` and the head attached in the GGUF branch; **G4 GREEN + `CPU-SPEC-DIVERGENCE` FIXED 2026-07-28**: root cause `src/vllm/model_executor/models/qwen3_5.cpp:3616` sized the GDN state gather/scatter row by `(Kw-1)` while the speculative persistent row is `(Kw-1)+num_spec`, so `GatherRows`/`ScatterRows` mis-strode the slot AND every channel past the first, corrupting post-prefill recurrent state. Fix = `CopyStateRowsStrided` (same TU) used by `GatherStateF32`/`ScatterStateF32` when `cache.shape[2] != work.shape[2]`; the contiguous helpers are kept when the widths agree, so every non-spec path is byte-identical by construction. CPU-only in effect (the fp16/bf16 arm routes through the `GdnStateGather`/`Scatter` ops, so CUDA was never exposed; no GPU result affected) | `tests/vllm/models/test_qwen3_5_gguf_mtp.cpp:109,146,156,184` **4 cases, and the split is the 2026-08-21 repair** ([#1454](https://github.com/mudler/vllm.cpp/issues/1454)): the file used to be the env-gated pair ALONE, each opening on a bare `return`, so with `VLLM_MTP_GGUF_MODEL` unset it reported `test cases: 2 \| 2 passed`, **`assertions: 0`**, `Status: SUCCESS!`, exit 0 - which is every CI run of this repository, the variable being set nowhere in `.github/workflows/`. The `18 assertions` this cell used to record was the LIVE count and was never once reached in CI. Now `:109` and `:146` are **HERMETIC** (KV-only synthetic GGUFs, no weight bytes, 18 assertions on any machine) and pin the arithmetic the old file only NAMED in a comment above `CHECK(c.num_hidden_layers > 0)`: `num_hidden_layers + mtp_num_hidden_layers == block_count` over 65/1, 25/1 and 28/3 - the third arm separating `- nextn` from `- 1` - plus the head-less arm, where the key is NOT published and `NumMtpLayers` answering 1 for an absent key is exactly why the invariant cannot be written with that helper alone. `:156` and `:184` stay env-gated on `VLLM_MTP_GGUF_MODEL` (so CI stays asset-free) and now SKIP LOUDLY with a `MESSAGE` naming the variable, as `tests/vllm/entrypoints/test_gguf_mmproj_reach.cpp` does; `:156` re-derives the same invariant from the file's OWN `block_count` kv. Unset: **4 cases / 18 assertions / `Status: SUCCESS!` / rc 0**. Live on `Qwen3.8-27B-Q4_K_M.gguf` (`block_count` 65, `nextn_predict_layers` 1): **4 cases / 38 assertions / `Status: SUCCESS!` / rc 0**. Mutation-proved on the production line `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:889`, both compiling clean and both restored against a pre-taken sha256: `= block_count` (drop the subtraction) 3/4 cases, 9/18 red, exit 1; `= block_count - 1` (the wrong constant) 2/4 cases, 5/18 red, exit 1. The SAME mutations left the PREVIOUS file at 2/2 cases, 0 assertions, `SUCCESS!`, exit 0. Correctness of the production line is unchanged and was never in question (`1a4db5c3c`, `493327b4e`); this was a test defect. Live-arm content unchanged: depth reaches config.raw; fc is [H,2H] verbatim; 3 norms [H]; head block is full-attention. **RED-first BEHAVIOURAL** (reverting only the G1 line fails both cases 2/2). Trunk inertness: `test_gguf` 103, `test_gguf_qwen36_loader` 99, `test_gguf_keep_quant` 5958, `test_gguf_dequant` 215, `test_capi` 33/232 all unchanged; `tests/parity/test_qwen35_gguf_spec_decode.cpp:74,139` - spec-ON == spec-OFF token-exact with 13 proposed/11 accepted, plus an `ngram` regression guard (widens the cache, never runs the spec conv update) that was token-exact throughout and pinned the widening as innocent. Regression sweep all unchanged: ops_gdn 1825, gdn_metadata_builder 483, gdn_prefill_conv 28, gdn_spec_routing 12, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, gguf_dequant 215, llm_engine 196, input_batch 163, runner 257, capi 232 **GPU CLOSE-OUT + DEVICE-DELTA ATTRIBUTION 2026-07-28 (`G5`-`G7`), ledger [parity-ledger.md#L800](parity-ledger.md#L800).** The GPU end-to-end gate re-run on a from-scratch RELEASE-TARGET build (`-DVLLM_CPP_CUDA_ARCHITECTURES=121a`, build dir DELETED first; arch VERIFIED by `build-cuda/CMakeFiles/vllm.dir/flags.make` `--generate-code=arch=compute_121a,code=[compute_121a,sm_121a]` and by `cuobjdump -lelf` 20 cubins ALL `sm_121a` zero sm_75, NOT by `CMakeCache.txt`, whose `CMAKE_CUDA_ARCHITECTURES:STRING=75` is the `enable_language(CUDA)` compiler-probe default shadowed by the normal variable at `CMakeLists.txt:186` - the prior wrong-arch conclusion was that decoy): dgx.casa GB10 under `flock $HOME/gpu.lock`, 35B A3B NVFP4 GGUF, **2/2 cases, 10/10 assertions, exit 0**, spec-ON token-identical to spec-OFF, 13 proposed / 11 accepted, 90.2 GiB peak RSS, 8m01s; re-run on the EXACT committed source **3/3 cases, 10/10 assertions, exit 0**, 7m25s, the new probe case SKIPping and adding zero assertions. **The CPU-vs-GPU token delta is a MEASURED near-tie, not a defect** (it was never this row's bar - spec-ON == spec-OFF WITHIN a device is): NEW double-gated spec-OFF-only probe `tests/parity/test_qwen35_gguf_spec_decode.cpp:217` (asset + `VLLM_MTP_GGUF_PROBE=1`, 20 alternatives per position, 484/484 assertions per arm, GPU then `CUDA_VISIBLE_DEVICES=` in one `flock` series) shows both arms picking `11751` at position 0 and forking at position 1 on a BIT-IDENTICAL prefix: GPU rank1 `13` -0.773180 over rank2 `11` -0.847055 (margin 0.0739 nats), CPU rank1 `11` -0.765499 over rank2 `13` -0.830374 (margin 0.0649 nats). Each device's pick is the other's rank 2, both ~7x inside the ratified 0.5-nat band, and the cross-device disagreement on the SAME token (0.057 and 0.082 nats) EXCEEDS the margin being decided, so rounding settles it; the 24 texts look unrelated only because positions 2+ cascade off that one coin flip. Margin sweep over all 24 positions: **GGUF GPU and GGUF CPU carry ZERO exact ties**, minimum margins 0.0482 and 0.0649 nats, and both arms reproduced their sequence across every run. **Gate 4 MET on the safetensors sibling of the same quantization run** (`FromModelDir` takes it unchanged): acceptance 12 proposed / 11 accepted vs the GGUF's 13 / 11. That arm, however, FAILS spec-ON == spec-OFF at concurrency 1 and does not reproduce its own spec-OFF sequence run to run, and the probe attributes both to THREE EXACT ties (positions 7, 10, 16, bit-identical logprobs) produced by its 1/16-grid quantized-GEMM logits - which EXONERATES the GGUF arm and opens a recorded, not-root-caused `SPEC-MTP` item on the safetensors NVFP4 path, not on this row. Gate 3 is NOT APPLICABLE twice over: no F16/F32 head-carrying export exists, and the only same-weights sibling is not token-stable against itself. **EVIDENCE RE-ANCHORED 2026-07-29 to a PRODUCTION-CONFIGURED build, because every GPU number above came from a build configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR` and WITHOUT `-DVLLM_CPP_TRITON=ON`** (the defect `CLAIM-27B-GATE-RCA` proved, which runs the emulation fp4 GEMM + hand GDN kernels). Re-run from a clean `git archive` tree of `main` `3f34534d`, build proven correct three ways (configure log has ZERO `CUTLASS not found` and prints `CUTLASS found ... sm120a NVFP4 cutlass GEMM` + `FlashAttention-2 ... ENABLED for arch(es) [121a]` + the vendored `sm_121a` Triton-AOT lines with `MANIFEST hashes OK`; `cuobjdump -lelf` 40 cubins ALL `sm_121a`, zero `sm_75`; SACRED `test_qwen27_paged_engine` **235/235 exit 0**, and the build precondition proven to FIRE by recompiling only that TU without the two defines against the same `libvllm.a`, which throws and exits 1 with 0 assertions). **The row PASSES UNCHANGED:** `tests/parity/test_qwen35_gguf_spec_decode` **3/3 cases, 10/10 assertions, exit 0**, spec-ON token-identical to spec-OFF, **13 proposed / 11 accepted (identical to the recorded number)**, 90.26 GiB, 7m13.59s; loader gate 19 assertions on the Qwen3.5-2B and 18 on the 35B A3B, unchanged. **ONE recorded finding is RETRACTED by the re-measurement: the CPU-vs-GPU token delta was a BUILD artifact, not a device near-tie cascade.** On the production build both devices emit the SAME 24 tokens; the probe shows GPU rank1 `11` -0.763897 over rank2 `13` -0.824083 where the defective build had rank1 `13` -0.773180 over rank2 `11` -0.847055, while the CPU arm is bit-identical to the earlier measurement (CUTLASS and Triton are CUDA-only). Zero exact ties in either arm, min margins 0.060186 GPU / 0.064875 CPU, 484/484 assertions per arm. Evidence: [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) top section, [parity-ledger.md](parity-ledger.md) | [specs/gguf-mtp-spec-decode.md](specs/gguf-mtp-spec-decode.md) | `DONE` | `edf91449` | -| `SPEC-DFLASH-GGUF` | DFlash speculative decoding from GGUF, two axes: (A) GGUF DRAFT + safetensors target, (B) GGUF target too. llama.cpp master carries a full `dflash` GGUF contract (arch string `dflash`, tensors `fc`/`enc.output_norm`/`output_norm`/`blk.N.*`, KVs `dflash.target_layers` + `dflash.target_hidden_size`); the arch is ABSENT from checkouts older than ~2026-07, so a stale tree reads as "no contract exists". The GGUF tensor set omits `token_embd`/`output` because the draft SHARES the target's embed+lm_head, which is exactly what `LoadDflashDraft` already does. Blockers are in the loader, not the model: `MakeDflashDraftConfig` reads `draft_dir/config.json` (a GGUF has none), `ResolveDflashDraftDir` probes for `config.json` so it cannot see a `.gguf`, and `LoadDflashDraft` is typed on `std::vector` for the shared bf16 head (the axis-B blocker). Axis A independently shippable. NO ABI change | T2 | llama.cpp `origin/master` @ 2026-07-28 (tag era `b10158`): `gguf-py/gguf/constants.py:547,1151,4350`; `gguf-py/gguf/tensor_mapping.py:1297-1305` (`ENC_OUTPUT_NORM`<-`model.hidden_norm`, `FC`<-`model.fc`); `conversion/qwen.py:351` (mask token via the standard tokenizer KV); `convert_hf_to_gguf.py --target-model-dir` | **GD1-GD7 LANDED 2026-07-28 (BOTH AXES COMPLETE and PROVEN end to end on GB10)**: `MakeDflashGgufConfig` + `LoadQwen3DFlashFromGguf` `src/vllm/model_executor/models/qwen3_dflash_gguf.cpp:88,227` (+ header), `IsDflashGgufDraft` + the `.gguf` branch in `ResolveDflashDraftDir`/`LoadDflashDraft` `src/vllm/entrypoints/model_loader.cpp:121,222`. Goes through the `TensorResolver` seam (unlike `SPEC-MTP-GGUF`) because dflash norms are RAW, so the existing `LoadQwen3DFlash` qkv/gate_up concatenation is reused unchanged. **`GD4` defect FIXED** (`model_loader.cpp:238-249`): the GGUF branch left `config.vocab_size` 0 - correct for `MakeDflashGgufConfig` (the DFLASH arch has no vocab KV and no `token_embd`) but fatal for the forward, which sizes the shared embedding view as `{config.vocab_size, H}`, so the first propose threw `cuda embedding: empty table (vocab 0)`. Now back-filled from the target's `embed_tokens` rows (the condition is on the VALUE, not the draft source, so it generalizes to a GGUF target). Load-level green had hidden it; only GENERATING found it. **GD5-GD7 = axis B**: `SharedHeadSource` `src/vllm/entrypoints/model_loader.cpp` re-expresses the shared bf16 `embed_tokens`+`lm_head` seam as a SOURCE and re-types `LoadDflashDraft`'s second parameter - THAT TYPE was the whole axis-B blocker - with the GGUF arm `LoadGgufSharedEmbedAndHeadBf16` `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:1044` reusing the trunk loader's tied-embedding rule and sidecar-aware dequant instead of restating them; the shared-head load moved into ONE common tail so all four (draft format x target container) combinations run identical code; the `dflash` half of the GGUF-branch rejection `model_loader.cpp` is deleted (the `mtp` half untouched) and the draft load is wired into the GGUF branch | `tests/vllm/models/test_qwen3_dflash_gguf.cpp:36,84` 2 cases / 47 assertions against the REAL published Qwen3.6-27B DFlash draft (env-gated `VLLM_DFLASH_GGUF_MODEL`, CI asset-free): the +1 target-layer offset undone against the KV read back from the same file, block_size/mask_token present, vocab_size left 0, layer_types cover every block, fc `[H, H*num_taps]` with `nk` SET, qkv/gate_up row-concat shapes, embed/lm_head left EMPTY for the target. **RED-first BEHAVIOURAL** (dropping the `-1` fails the offset checks). **`GD4` e2e gate** `tests/parity/test_qwen27_dflash_spec_decode.cpp:343` (second case, draft source env-driven via `VLLM_DFLASH_DRAFT`/`_B`; asset-gated, CI-inert): on dgx GB10 sm_121a against the Qwen3.6-27B NVFP4 safetensors target, the Q4_K_M GGUF draft and the bf16 z-lab safetensors draft produce **token-for-token IDENTICAL** DFlash-ON continuations with **IDENTICAL** accepted/proposed (20/80 on a 24-token prompt, 42/96 on a 48-token prompt), spec-OFF self-reproducible 3/3 and 0 exact ties (min margin 0.197/0.400 nats). Regression: gguf_mtp 19, qwen35_gguf_spec_decode 10, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, ops_gdn 1825, llm_engine 196, capi 232, runner 257 all unchanged. **`GD5` unit gate** `tests/vllm/test_gguf_qwen36_loader.cpp` 3 new synthetic-GGUF cases (6 cases / 286 assertions total, CPU and the dgx CUDA build): the untied head really comes from `output.weight` and not the embedding (distinct fill values), the tied fallback aliases it onto `token_embd`, the `nk` flags separate the gather table from the MatmulBT weight, a file with no `token_embd` is refused. 3-mutant battery, 3 caught (`nk` flipped, head forced to the embedding, tied forced false). **`GD7` e2e gate** `tests/parity/test_qwen27_dflash_spec_decode.cpp` third case (targets env-driven via `VLLM_DFLASH_TARGET_B`; asset-gated, CI-inert): on dgx GB10 sm_121a the Qwen3.6-27B NVFP4 **GGUF** target + `Q4_K_M` GGUF draft loads, takes the shared head from the GGUF, generates, and its DFlash-ON continuation is **token-for-token IDENTICAL to that same target's spec-OFF** (24/24, the STRICT form) with acceptance ALIVE at 14/160; 1 case / 15 assertions, exit 0. **The spike's highest risk is EMPTY on this asset, proven not assumed**: the 27B NVFP4 GGUF stores `token_embd`/`output` as ggml BF16, byte-identical to the safetensors sibling (2,542,796,800 bytes each, ZERO differing), so B1's shared-head read is verbatim, not a dequant. Acceptance IS lower than the safetensors-target arm and is NOT chargeable to the head: the two containers diverge at index 4 with NO speculation, because `QUANT-GGUF-NVFP4` is dequant-only so the GGUF target computes in bf16 while the safetensors target runs the true W4A4 kernels. **RE-MEASURED 2026-07-29 on a PRODUCTION-CONFIGURED build (`CLAIM-GGUF-SPEC-REVERIFY`), because every GD4/GD7 GPU number above came from a build configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR` and WITHOUT `-DVLLM_CPP_TRITON=ON`.** Build proven correct three ways (see the `SPEC-MTP-GGUF` row; SACRED 27B **235/235**, `cuobjdump` 40 cubins all `sm_121a`). **AXIS B HOLDS EXACTLY**: `test_qwen27_dflash_spec_decode -tc="dflash axis-B*"` **15/15 assertions, exit 0**, GGUF-target DFlash-ON token-identical to that target's own spec-OFF 24/24, acceptance **14/160 unchanged**, cross-target spec-OFF divergence still at index 4, 81.01 GiB peak RSS, 6m53.08s. **AXIS A WAS RED ON THE 48-TOKEN PROMPT (reproducibly, 3 of 3 runs) AND IS NOW CLOSED.** The RED was real: cross-format TOKEN identity held on both prompts, but the exact accept-count half of bar (a) failed (`arm_a.proposed == arm_b.proposed` / `arm_a.accepted == arm_b.accepted`) because the Q4_K_M draft measured **46/112** against the bf16 z-lab draft's **47/96** (one extra 16-wide propose block, one fewer acceptance, zero token difference), 15/17, exit 1; the 24-token prompt stayed green at 17/17 with both drafts at 15/144. **`GD9` 2026-07-29 root-caused it IN WEIGHT SPACE as ordinary `Q4_K_M` cost, category (a), not a defect in our GGUF draft path - and the bar's own premise ("Same weights, two containers") was false for the asset it was pointed at.** The publishing repo also carries an UNQUANTIZED `BF16` GGUF (3,471,497,440 B) beside `Q8_0`/`Q6_K`/`Q5_K`/`Q4_K_M`, which the spec had recorded as nonexistent; that retired the `NOT APPLICABLE` on gate 2. CPU gate `tests/vllm/models/test_qwen3_dflash_gguf.cpp` third case (asset-gated `VLLM_DFLASH_GGUF_BF16_MODEL` + `VLLM_DFLASH_ST_DIR`): `LoadQwen3DFlashFromGguf(BF16)` is **BYTE-IDENTICAL to `LoadQwen3DFlash(z-lab shards)` on all 58 tensors, 302/302 assertions, exit 0**, and FUNCTIONALLY RED against the `Q4_K_M` file (21/302 red, exactly the 21 quantized matmul tensors), so not a vacuous pass. Supporting: our `DequantGgufRowToBf16` is bit-equal to `gguf-py`'s `gguf.quants.dequantize` on the real `fc.weight` (Q4_K), `blk.0.attn_q.weight` (Q4_K) and `blk.2.ffn_down.weight` (Q6_K), zero differing bf16 values; the ladder's mean relative weight error is monotone and uniform with NO outlier tensor (BF16 0, Q8_0 5.6e-3, Q6_K 1.85e-2, Q5_K 3.85e-2, Q4_K_M 7.6e-2); the only numeric config delta is `rms_norm_eps` at 2.5e-9 relative. Also landed: an off-by-default `VT_SPEC_TRACE=1` per-block propose/accept trace in `GPUModelRunner::sample_tokens_with_rejection` (`src/vllm/v1/worker/gpu/runner.cpp`). **`GD10` 2026-07-29 CONFIRMED IT END TO END ON GB10 and closed gates 3 and 5.** Build proven production-configured three ways (configure log 0 `CUTLASS not found`; `cuobjdump -lelf` 40 cubins ALL `sm_121a` zero `sm_75` on both binaries; SACRED `test_qwen27_paged_engine` **235/235, exit 0**, 31.34s, 23.67 GiB). The **`BF16` GGUF draft reads EXACTLY 47/96**, the safetensors draft's own number, at 48 tokens on the discriminating prompt - reproduced 2 of 2 - plus 27/64 = 27/64 at 24 tokens and 15/144 = 15/144 on the second prompt, tokens IDENTICAL throughout, 17/17 exit 0 each time; the `Q4_K_M` arm reads 46/112 on the SAME binary in the SAME `flock` series. Restoring only the draft's numeric precision restores the count, so quantization is the whole cause and nothing structural survives. Bar (a) is consequently SPLIT rather than relaxed (`tests/parity/test_qwen27_dflash_spec_decode.cpp`): tokens stay EXACT unconditionally; accept counts are EXACT on a cross-FORMAT arm and BANDED (`abs(d_accepted) <= 2`, `abs(d_proposed) <= k*2`) on a cross-QUANTIZATION one, with the arm chosen by `IsQuantizedGgufDraft` reading the draft file's ggml types (`GgmlTraits().block_elems > 1`) rather than by a flag. The band is derived, not picked: measured `d_accepted` is 0, 0, -1, so the bound is that maximum plus one quantum; and `d_proposed = -k * d_accepted` EXACTLY once the token streams match (confirmed at -1 / +16), so the proposed bound follows. **Mutation-proved non-vacuous**: rebuilt at band 0 the `Q4_K_M` arm is 15/17 exit 1 while the `BF16` arm stays 17/17 exit 0 on the exact branch. **AXIS B BROADENED from ONE prompt to THREE**, strict form green on all: "The capital of France is" IDENTICAL 14/160 (15/15), "Write a Python function that reverses a string:" IDENTICAL 24/64 (15/15), "Photosynthesis is the process by which" IDENTICAL 15/128 (9/9), all exit 0, ~6m30-6m52 and ~81 GiB peak RSS each. The second prompt REFINES the recorded acceptance claim: the safetensors-target arm is ALSO 24/64 there with the two containers' DFlash-ON streams IDENTICAL, so the GGUF target's lower acceptance is prompt-dependent (their spec-OFF streams diverge at index 4 on the first prompt, index 16 on the second) and not a standing penalty; the cause remains `QUANT-GGUF-NVFP4` being dequant-only, with the shared head excluded by a byte comparison. Gates 1-5 and 7 MET; gate 6 (speed) `PENDING` BY DESIGN and not owed - a DFlash-ON throughput A/B between the two target containers is not a fair comparison until a native NVFP4 GGUF GEMM exists. Evidence: [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) top section, [parity-ledger.md](parity-ledger.md#L845) | [specs/gguf-dflash-draft.md](specs/gguf-dflash-draft.md) | `DONE` | `c62f2fa3` | +| `SPEC-MTP-GGUF` | MTP speculative decoding from a GGUF TARGET. Today `FromModelDir` refuses `mtp`+GGUF outright (`src/vllm/entrypoints/model_loader.cpp:717-723`) on the original spike's assumption that GGUF exports carry no `mtp.*` ([mtp-spec-decode.md](specs/mtp-spec-decode.md):979-980, "until we re-export GGUFs with the head"). That is stale: llama.cpp's Qwen3.5 converter DOES emit the head, under layer-indexed `nextn` naming, and our own `HfConfigFromGguf` ALREADY reads `nextn_predict_layers` (it just discards the value into the trunk layer count). Gap is a `TensorResolver` over `GgufFile` mapping `mtp.*` onto `blk.{L+i}.nextn.*` with dequant-to-bf16, one config field, and narrowing the rejection to `dflash`. `ngram`+GGUF already works and is untouched. Qwen3.5/3.6 only (the widened spec KV path serves no other arch). NO ABI change | T2 | llama.cpp (the producer contract; vLLM has no GGUF MTP path) `conversion/qwen.py:535-604` `_Qwen35MtpMixin` (the authoritative `mtp.*`->`nextn` remapper + `add_nextn_predict_layers`); `gguf-py/gguf/constants.py:129,910-917,1494-1501`; `gguf-py/gguf/tensor_mapping.py` `NEXTN_*` | **G1-G3 LANDED 2026-07-28.** `HfConfigFromGguf` republishes the head depth `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:598` (`c.raw["mtp_num_hidden_layers"] = nextn`, previously read then discarded); the head loader `LoadQwen3_5MTPFromGguf` `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:1458` (+ decl `include/vllm/model_executor/models/qwen3_5_gguf_weights.h:143`) reusing the TRUNK helpers `OwnNormMinus1`/`OwnMatmulWeight`/`OwnBf16`/`LoadAttnGguf`/`LoadMoeGguf` so the head inherits the GGUF (w+1) norm storage, quantization/residency routing and torch [N,K] shapes; `NumMtpLayers`/`UsesDedicatedEmbeddings` exported out of the anon namespace `include/vllm/model_executor/models/qwen3_5_mtp.h`; rejection narrowed to dflash + a head-less-GGUF check `src/vllm/entrypoints/model_loader.cpp` and the head attached in the GGUF branch; **G4 GREEN + `CPU-SPEC-DIVERGENCE` FIXED 2026-07-28**: root cause `src/vllm/model_executor/models/qwen3_5.cpp:3616` sized the GDN state gather/scatter row by `(Kw-1)` while the speculative persistent row is `(Kw-1)+num_spec`, so `GatherRows`/`ScatterRows` mis-strode the slot AND every channel past the first, corrupting post-prefill recurrent state. Fix = `CopyStateRowsStrided` (same TU) used by `GatherStateF32`/`ScatterStateF32` when `cache.shape[2] != work.shape[2]`; the contiguous helpers are kept when the widths agree, so every non-spec path is byte-identical by construction. CPU-only in effect (the fp16/bf16 arm routes through the `GdnStateGather`/`Scatter` ops, so CUDA was never exposed; no GPU result affected) | `tests/vllm/models/test_qwen3_5_gguf_mtp.cpp:109,146,156,184` **4 cases, and the split is the 2026-08-21 repair** ([#1454](https://github.com/mudler/vllm.cpp/issues/1454)): the file used to be the env-gated pair ALONE, each opening on a bare `return`, so with `VLLM_MTP_GGUF_MODEL` unset it reported `test cases: 2 \| 2 passed`, **`assertions: 0`**, `Status: SUCCESS!`, exit 0 - which is every CI run of this repository, the variable being set nowhere in `.github/workflows/`. The `18 assertions` this cell used to record was the LIVE count and was never once reached in CI. Now `:109` and `:146` are **HERMETIC** (KV-only synthetic GGUFs, no weight bytes, 18 assertions on any machine) and pin the arithmetic the old file only NAMED in a comment above `CHECK(c.num_hidden_layers > 0)`: `num_hidden_layers + mtp_num_hidden_layers == block_count` over 65/1, 25/1 and 28/3 - the third arm separating `- nextn` from `- 1` - plus the head-less arm, where the key is NOT published and `NumMtpLayers` answering 1 for an absent key is exactly why the invariant cannot be written with that helper alone. `:156` and `:184` stay env-gated on `VLLM_MTP_GGUF_MODEL` (so CI stays asset-free) and now SKIP LOUDLY with a `MESSAGE` naming the variable, as `tests/vllm/entrypoints/test_gguf_mmproj_reach.cpp` does; `:156` re-derives the same invariant from the file's OWN `block_count` kv. Unset: **4 cases / 18 assertions / `Status: SUCCESS!` / rc 0**. Live on `Qwen3.8-27B-Q4_K_M.gguf` (`block_count` 65, `nextn_predict_layers` 1): **4 cases / 38 assertions / `Status: SUCCESS!` / rc 0**. Mutation-proved on the production line `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:889`, both compiling clean and both restored against a pre-taken sha256: `= block_count` (drop the subtraction) 3/4 cases, 9/18 red, exit 1; `= block_count - 1` (the wrong constant) 2/4 cases, 5/18 red, exit 1. The SAME mutations left the PREVIOUS file at 2/2 cases, 0 assertions, `SUCCESS!`, exit 0. Correctness of the production line is unchanged and was never in question (`1a4db5c3c`, `493327b4e`); this was a test defect. Live-arm content unchanged: depth reaches config.raw; fc is [H,2H] verbatim; 3 norms [H]; head block is full-attention. **RED-first BEHAVIOURAL** (reverting only the G1 line fails both cases 2/2). Trunk inertness: `test_gguf` 103, `test_gguf_qwen36_loader` 99, `test_gguf_keep_quant` 5958, `test_gguf_dequant` 215, `test_capi` 33/232 all unchanged; `tests/parity/test_qwen35_gguf_spec_decode.cpp:74,139` - spec-ON == spec-OFF token-exact with 13 proposed/11 accepted, plus an `ngram` regression guard (widens the cache, never runs the spec conv update) that was token-exact throughout and pinned the widening as innocent. Regression sweep all unchanged: ops_gdn 1825, gdn_metadata_builder 483, gdn_prefill_conv 28, gdn_spec_routing 12, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, gguf_dequant 215, llm_engine 196, input_batch 163, runner 257, capi 232 **GPU CLOSE-OUT + DEVICE-DELTA ATTRIBUTION 2026-07-28 (`G5`-`G7`), ledger [parity-ledger.md#L800](parity-ledger.md#L800).** The GPU end-to-end gate re-run on a from-scratch RELEASE-TARGET build (`-DVLLM_CPP_CUDA_ARCHITECTURES=121a`, build dir DELETED first; arch VERIFIED by `build-cuda/CMakeFiles/vllm.dir/flags.make` `--generate-code=arch=compute_121a,code=[compute_121a,sm_121a]` and by `cuobjdump -lelf` 20 cubins ALL `sm_121a` zero sm_75, NOT by `CMakeCache.txt`, whose `CMAKE_CUDA_ARCHITECTURES:STRING=75` is the `enable_language(CUDA)` compiler-probe default shadowed by the normal variable at `CMakeLists.txt:186` - the prior wrong-arch conclusion was that decoy): dgx.casa GB10 under `flock $HOME/gpu.lock`, 35B A3B NVFP4 GGUF, **2/2 cases, 10/10 assertions, exit 0**, spec-ON token-identical to spec-OFF, 13 proposed / 11 accepted, 90.2 GiB peak RSS, 8m01s; re-run on the EXACT committed source **3/3 cases, 10/10 assertions, exit 0**, 7m25s, the new probe case SKIPping and adding zero assertions. **The CPU-vs-GPU token delta is a MEASURED near-tie, not a defect** (it was never this row's bar - spec-ON == spec-OFF WITHIN a device is): NEW double-gated spec-OFF-only probe `tests/parity/test_qwen35_gguf_spec_decode.cpp:217` (asset + `VLLM_MTP_GGUF_PROBE=1`, 20 alternatives per position, 484/484 assertions per arm, GPU then `CUDA_VISIBLE_DEVICES=` in one `flock` series) shows both arms picking `11751` at position 0 and forking at position 1 on a BIT-IDENTICAL prefix: GPU rank1 `13` -0.773180 over rank2 `11` -0.847055 (margin 0.0739 nats), CPU rank1 `11` -0.765499 over rank2 `13` -0.830374 (margin 0.0649 nats). Each device's pick is the other's rank 2, both ~7x inside the ratified 0.5-nat band, and the cross-device disagreement on the SAME token (0.057 and 0.082 nats) EXCEEDS the margin being decided, so rounding settles it; the 24 texts look unrelated only because positions 2+ cascade off that one coin flip. Margin sweep over all 24 positions: **GGUF GPU and GGUF CPU carry ZERO exact ties**, minimum margins 0.0482 and 0.0649 nats, and both arms reproduced their sequence across every run. **Gate 4 MET on the safetensors sibling of the same quantization run** (`FromModelDir` takes it unchanged): acceptance 12 proposed / 11 accepted vs the GGUF's 13 / 11. That arm, however, FAILS spec-ON == spec-OFF at concurrency 1 and does not reproduce its own spec-OFF sequence run to run, and the probe attributes both to THREE EXACT ties (positions 7, 10, 16, bit-identical logprobs) produced by its 1/16-grid quantized-GEMM logits - which EXONERATES the GGUF arm and opens a recorded, not-root-caused `SPEC-MTP` item on the safetensors NVFP4 path, not on this row. Gate 3 is NOT APPLICABLE twice over: no F16/F32 head-carrying export exists, and the only same-weights sibling is not token-stable against itself. **EVIDENCE RE-ANCHORED 2026-07-29 to a PRODUCTION-CONFIGURED build, because every GPU number above came from a build configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR` and WITHOUT `-DVLLM_CPP_TRITON=ON`** (the defect `CLAIM-27B-GATE-RCA` proved, which runs the emulation fp4 GEMM + hand GDN kernels). Re-run from a clean `git archive` tree of `main` `3f34534d`, build proven correct three ways (configure log has ZERO `CUTLASS not found` and prints `CUTLASS found ... sm120a NVFP4 cutlass GEMM` + `FlashAttention-2 ... ENABLED for arch(es) [121a]` + the vendored `sm_121a` Triton-AOT lines with `MANIFEST hashes OK`; `cuobjdump -lelf` 40 cubins ALL `sm_121a`, zero `sm_75`; SACRED `test_qwen27_paged_engine` **235/235 exit 0**, and the build precondition proven to FIRE by recompiling only that TU without the two defines against the same `libvllm.a`, which throws and exits 1 with 0 assertions). **The row PASSES UNCHANGED:** `tests/parity/test_qwen35_gguf_spec_decode` **3/3 cases, 10/10 assertions, exit 0**, spec-ON token-identical to spec-OFF, **13 proposed / 11 accepted (identical to the recorded number)**, 90.26 GiB, 7m13.59s; loader gate 19 assertions on the Qwen3.5-2B and 18 on the 35B A3B, unchanged. **ONE recorded finding is RETRACTED by the re-measurement: the CPU-vs-GPU token delta was a BUILD artifact, not a device near-tie cascade.** On the production build both devices emit the SAME 24 tokens; the probe shows GPU rank1 `11` -0.763897 over rank2 `13` -0.824083 where the defective build had rank1 `13` -0.773180 over rank2 `11` -0.847055, while the CPU arm is bit-identical to the earlier measurement (CUTLASS and Triton are CUDA-only). Zero exact ties in either arm, min margins 0.060186 GPU / 0.064875 CPU, 484/484 assertions per arm. Evidence: [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) top section, [parity-ledger.md](parity-ledger.md) | [specs/gguf-mtp-spec-decode.md](specs/gguf-mtp-spec-decode.md) | `DONE` | `edf91449` | +| `SPEC-DFLASH-GGUF` | DFlash speculative decoding from GGUF, two axes: (A) GGUF DRAFT + safetensors target, (B) GGUF target too. llama.cpp master carries a full `dflash` GGUF contract (arch string `dflash`, tensors `fc`/`enc.output_norm`/`output_norm`/`blk.N.*`, KVs `dflash.target_layers` + `dflash.target_hidden_size`); the arch is ABSENT from checkouts older than ~2026-07, so a stale tree reads as "no contract exists". The GGUF tensor set omits `token_embd`/`output` because the draft SHARES the target's embed+lm_head, which is exactly what `LoadDflashDraft` already does. Blockers are in the loader, not the model: `MakeDflashDraftConfig` reads `draft_dir/config.json` (a GGUF has none), `ResolveDflashDraftDir` probes for `config.json` so it cannot see a `.gguf`, and `LoadDflashDraft` is typed on `std::vector` for the shared bf16 head (the axis-B blocker). Axis A independently shippable. NO ABI change | T2 | llama.cpp `origin/master` @ 2026-07-28 (tag era `b10158`): `gguf-py/gguf/constants.py:547,1151,4350`; `gguf-py/gguf/tensor_mapping.py:1297-1305` (`ENC_OUTPUT_NORM`<-`model.hidden_norm`, `FC`<-`model.fc`); `conversion/qwen.py:351` (mask token via the standard tokenizer KV); `convert_hf_to_gguf.py --target-model-dir` | **GD1-GD7 LANDED 2026-07-28 (BOTH AXES COMPLETE and PROVEN end to end on GB10)**: `MakeDflashGgufConfig` + `LoadQwen3DFlashFromGguf` `src/vllm/model_executor/models/qwen3_dflash_gguf.cpp:88,227` (+ header), `IsDflashGgufDraft` + the `.gguf` branch in `ResolveDflashDraftDir`/`LoadDflashDraft` `src/vllm/entrypoints/model_loader.cpp:121,222`. Goes through the `TensorResolver` seam (unlike `SPEC-MTP-GGUF`) because dflash norms are RAW, so the existing `LoadQwen3DFlash` qkv/gate_up concatenation is reused unchanged. **`GD4` defect FIXED** (`model_loader.cpp:238-249`): the GGUF branch left `config.vocab_size` 0 - correct for `MakeDflashGgufConfig` (the DFLASH arch has no vocab KV and no `token_embd`) but fatal for the forward, which sizes the shared embedding view as `{config.vocab_size, H}`, so the first propose threw `cuda embedding: empty table (vocab 0)`. Now back-filled from the target's `embed_tokens` rows (the condition is on the VALUE, not the draft source, so it generalizes to a GGUF target). Load-level green had hidden it; only GENERATING found it. **GD5-GD7 = axis B**: `SharedHeadSource` `src/vllm/entrypoints/model_loader.cpp` re-expresses the shared bf16 `embed_tokens`+`lm_head` seam as a SOURCE and re-types `LoadDflashDraft`'s second parameter - THAT TYPE was the whole axis-B blocker - with the GGUF arm `LoadGgufSharedEmbedAndHeadBf16` `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:1045` reusing the trunk loader's tied-embedding rule and sidecar-aware dequant instead of restating them; the shared-head load moved into ONE common tail so all four (draft format x target container) combinations run identical code; the `dflash` half of the GGUF-branch rejection `model_loader.cpp` is deleted (the `mtp` half untouched) and the draft load is wired into the GGUF branch | `tests/vllm/models/test_qwen3_dflash_gguf.cpp:36,84` 2 cases / 47 assertions against the REAL published Qwen3.6-27B DFlash draft (env-gated `VLLM_DFLASH_GGUF_MODEL`, CI asset-free): the +1 target-layer offset undone against the KV read back from the same file, block_size/mask_token present, vocab_size left 0, layer_types cover every block, fc `[H, H*num_taps]` with `nk` SET, qkv/gate_up row-concat shapes, embed/lm_head left EMPTY for the target. **RED-first BEHAVIOURAL** (dropping the `-1` fails the offset checks). **`GD4` e2e gate** `tests/parity/test_qwen27_dflash_spec_decode.cpp:343` (second case, draft source env-driven via `VLLM_DFLASH_DRAFT`/`_B`; asset-gated, CI-inert): on dgx GB10 sm_121a against the Qwen3.6-27B NVFP4 safetensors target, the Q4_K_M GGUF draft and the bf16 z-lab safetensors draft produce **token-for-token IDENTICAL** DFlash-ON continuations with **IDENTICAL** accepted/proposed (20/80 on a 24-token prompt, 42/96 on a 48-token prompt), spec-OFF self-reproducible 3/3 and 0 exact ties (min margin 0.197/0.400 nats). Regression: gguf_mtp 19, qwen35_gguf_spec_decode 10, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, ops_gdn 1825, llm_engine 196, capi 232, runner 257 all unchanged. **`GD5` unit gate** `tests/vllm/test_gguf_qwen36_loader.cpp` 3 new synthetic-GGUF cases (6 cases / 286 assertions total, CPU and the dgx CUDA build): the untied head really comes from `output.weight` and not the embedding (distinct fill values), the tied fallback aliases it onto `token_embd`, the `nk` flags separate the gather table from the MatmulBT weight, a file with no `token_embd` is refused. 3-mutant battery, 3 caught (`nk` flipped, head forced to the embedding, tied forced false). **`GD7` e2e gate** `tests/parity/test_qwen27_dflash_spec_decode.cpp` third case (targets env-driven via `VLLM_DFLASH_TARGET_B`; asset-gated, CI-inert): on dgx GB10 sm_121a the Qwen3.6-27B NVFP4 **GGUF** target + `Q4_K_M` GGUF draft loads, takes the shared head from the GGUF, generates, and its DFlash-ON continuation is **token-for-token IDENTICAL to that same target's spec-OFF** (24/24, the STRICT form) with acceptance ALIVE at 14/160; 1 case / 15 assertions, exit 0. **The spike's highest risk is EMPTY on this asset, proven not assumed**: the 27B NVFP4 GGUF stores `token_embd`/`output` as ggml BF16, byte-identical to the safetensors sibling (2,542,796,800 bytes each, ZERO differing), so B1's shared-head read is verbatim, not a dequant. Acceptance IS lower than the safetensors-target arm and is NOT chargeable to the head: the two containers diverge at index 4 with NO speculation, because `QUANT-GGUF-NVFP4` is dequant-only so the GGUF target computes in bf16 while the safetensors target runs the true W4A4 kernels. **RE-MEASURED 2026-07-29 on a PRODUCTION-CONFIGURED build (`CLAIM-GGUF-SPEC-REVERIFY`), because every GD4/GD7 GPU number above came from a build configured WITHOUT `-DVLLM_CPP_CUTLASS_DIR` and WITHOUT `-DVLLM_CPP_TRITON=ON`.** Build proven correct three ways (see the `SPEC-MTP-GGUF` row; SACRED 27B **235/235**, `cuobjdump` 40 cubins all `sm_121a`). **AXIS B HOLDS EXACTLY**: `test_qwen27_dflash_spec_decode -tc="dflash axis-B*"` **15/15 assertions, exit 0**, GGUF-target DFlash-ON token-identical to that target's own spec-OFF 24/24, acceptance **14/160 unchanged**, cross-target spec-OFF divergence still at index 4, 81.01 GiB peak RSS, 6m53.08s. **AXIS A WAS RED ON THE 48-TOKEN PROMPT (reproducibly, 3 of 3 runs) AND IS NOW CLOSED.** The RED was real: cross-format TOKEN identity held on both prompts, but the exact accept-count half of bar (a) failed (`arm_a.proposed == arm_b.proposed` / `arm_a.accepted == arm_b.accepted`) because the Q4_K_M draft measured **46/112** against the bf16 z-lab draft's **47/96** (one extra 16-wide propose block, one fewer acceptance, zero token difference), 15/17, exit 1; the 24-token prompt stayed green at 17/17 with both drafts at 15/144. **`GD9` 2026-07-29 root-caused it IN WEIGHT SPACE as ordinary `Q4_K_M` cost, category (a), not a defect in our GGUF draft path - and the bar's own premise ("Same weights, two containers") was false for the asset it was pointed at.** The publishing repo also carries an UNQUANTIZED `BF16` GGUF (3,471,497,440 B) beside `Q8_0`/`Q6_K`/`Q5_K`/`Q4_K_M`, which the spec had recorded as nonexistent; that retired the `NOT APPLICABLE` on gate 2. CPU gate `tests/vllm/models/test_qwen3_dflash_gguf.cpp` third case (asset-gated `VLLM_DFLASH_GGUF_BF16_MODEL` + `VLLM_DFLASH_ST_DIR`): `LoadQwen3DFlashFromGguf(BF16)` is **BYTE-IDENTICAL to `LoadQwen3DFlash(z-lab shards)` on all 58 tensors, 302/302 assertions, exit 0**, and FUNCTIONALLY RED against the `Q4_K_M` file (21/302 red, exactly the 21 quantized matmul tensors), so not a vacuous pass. Supporting: our `DequantGgufRowToBf16` is bit-equal to `gguf-py`'s `gguf.quants.dequantize` on the real `fc.weight` (Q4_K), `blk.0.attn_q.weight` (Q4_K) and `blk.2.ffn_down.weight` (Q6_K), zero differing bf16 values; the ladder's mean relative weight error is monotone and uniform with NO outlier tensor (BF16 0, Q8_0 5.6e-3, Q6_K 1.85e-2, Q5_K 3.85e-2, Q4_K_M 7.6e-2); the only numeric config delta is `rms_norm_eps` at 2.5e-9 relative. Also landed: an off-by-default `VT_SPEC_TRACE=1` per-block propose/accept trace in `GPUModelRunner::sample_tokens_with_rejection` (`src/vllm/v1/worker/gpu/runner.cpp`). **`GD10` 2026-07-29 CONFIRMED IT END TO END ON GB10 and closed gates 3 and 5.** Build proven production-configured three ways (configure log 0 `CUTLASS not found`; `cuobjdump -lelf` 40 cubins ALL `sm_121a` zero `sm_75` on both binaries; SACRED `test_qwen27_paged_engine` **235/235, exit 0**, 31.34s, 23.67 GiB). The **`BF16` GGUF draft reads EXACTLY 47/96**, the safetensors draft's own number, at 48 tokens on the discriminating prompt - reproduced 2 of 2 - plus 27/64 = 27/64 at 24 tokens and 15/144 = 15/144 on the second prompt, tokens IDENTICAL throughout, 17/17 exit 0 each time; the `Q4_K_M` arm reads 46/112 on the SAME binary in the SAME `flock` series. Restoring only the draft's numeric precision restores the count, so quantization is the whole cause and nothing structural survives. Bar (a) is consequently SPLIT rather than relaxed (`tests/parity/test_qwen27_dflash_spec_decode.cpp`): tokens stay EXACT unconditionally; accept counts are EXACT on a cross-FORMAT arm and BANDED (`abs(d_accepted) <= 2`, `abs(d_proposed) <= k*2`) on a cross-QUANTIZATION one, with the arm chosen by `IsQuantizedGgufDraft` reading the draft file's ggml types (`GgmlTraits().block_elems > 1`) rather than by a flag. The band is derived, not picked: measured `d_accepted` is 0, 0, -1, so the bound is that maximum plus one quantum; and `d_proposed = -k * d_accepted` EXACTLY once the token streams match (confirmed at -1 / +16), so the proposed bound follows. **Mutation-proved non-vacuous**: rebuilt at band 0 the `Q4_K_M` arm is 15/17 exit 1 while the `BF16` arm stays 17/17 exit 0 on the exact branch. **AXIS B BROADENED from ONE prompt to THREE**, strict form green on all: "The capital of France is" IDENTICAL 14/160 (15/15), "Write a Python function that reverses a string:" IDENTICAL 24/64 (15/15), "Photosynthesis is the process by which" IDENTICAL 15/128 (9/9), all exit 0, ~6m30-6m52 and ~81 GiB peak RSS each. The second prompt REFINES the recorded acceptance claim: the safetensors-target arm is ALSO 24/64 there with the two containers' DFlash-ON streams IDENTICAL, so the GGUF target's lower acceptance is prompt-dependent (their spec-OFF streams diverge at index 4 on the first prompt, index 16 on the second) and not a standing penalty; the cause remains `QUANT-GGUF-NVFP4` being dequant-only, with the shared head excluded by a byte comparison. Gates 1-5 and 7 MET; gate 6 (speed) `PENDING` BY DESIGN and not owed - a DFlash-ON throughput A/B between the two target containers is not a fair comparison until a native NVFP4 GGUF GEMM exists. Evidence: [docs/BENCHMARKS.md](../docs/BENCHMARKS.md) top section, [parity-ledger.md](parity-ledger.md#L845) | [specs/gguf-dflash-draft.md](specs/gguf-dflash-draft.md) | `DONE` | `c62f2fa3` | | `SPEC-REJECTION` | Rejection sampler. **I3 verify half LANDED (2026-07-24)**: per-request logits EXPANSION to `1 + k_i` rows (`StepInputs::cu_num_logits` / `num_draft_tokens_per_req` / expanded `logits_indices`) plus the GREEDY rejection sampler — accept a draft iff it equals the target argmax at its own position, emit the target argmax on the FIRST mismatch and stop, emit the bonus argmax when all `k_i` accept, `num_sampled = accepted + 1`, `num_rejected = k_i - accepted` (feeds I2's `num_computed_tokens` rollback and `InputBatch::num_accepted_tokens`). One additive vt op (`kGreedyRejectionSample`) with a CPU reference and a CUDA two-phase mirror of upstream's row-argmax + one-thread-per-request accept walk. DEFAULT-OFF and INERT: with no `SpeculativeConfig` no drafts are ever scheduled, `cu_num_logits` is `arange(num_reqs+1)`, `logits_indices` is the pre-change array and the runner never enters the rejection branch. STOCHASTIC/Gumbel, block verification, `apply_sampling_params` over the expanded batch, and the spec grammar bitmask stay DEFERRED (M-mtp-3). **I5b DRAFTER PREFILL INPUT-PREP LANDED (2026-07-24, `CLAIM-SPEC-MTP-I5B`)**: the draft-token input splice this row's I3 note deferred to I5 — `vllm::v1::prepare_prefill_inputs` + its `SpecPrefillInputs` output struct shift each request's `input_ids` left one within its query span, splice the just-sampled next token (`num_sampled>0 ? last_sampled[idx_mapping[r]] : next_prefill_tokens[...]`) into the freed slot, `query_len -= num_rejected`, and emit last-token index / query_start_loc / seq_lens + CG padding (mirror `speculator.py:469-588`, k=1 early-exit :236-238). A HOST routine in a NEW spec_decode-tree TU (no new CUDA kernel; mirrors the DEVICE-NEUTRAL `prepare_inputs`/`combine_sampled_and_draft_tokens` family — the DGX runner leaf ports the loop to the Triton kernel at I5d), unit-gated `test_prepare_prefill_inputs` 7 cases / 27 assertions RED-first, DEFAULT-OFF INERT (nothing calls it until I5d), additive by construction. Row stays `ACTIVE` — the e2e greedy token gate (M-mtp-1) is owed before `DONE` | T1 | `vllm/v1/worker/gpu/spec_decode/rejection_sampler.py:43,101-160`; `rejection_sampler_utils.py:524,564-585,628,828-841,846-849,863-1125`; `vllm/v1/worker/gpu/model_runner.py:866-898,1065-1077`; `vllm/v1/worker/gpu/input_batch.py:303-397,408-453`; **I5b** `vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py:469-588,236-238` | `include/vllm/v1/spec_decode/rejection_sampler.h`; `src/vllm/v1/spec_decode/rejection_sampler.cpp`; `include/vt/ops.h` (`kGreedyRejectionSample`, `vt::GreedyRejectionSample`); `src/vt/cpu/cpu_sample.cpp` (CPU reference); `src/vt/cuda/cuda_sample.cu` (`RejectionRowArgmaxKernel` + `GreedyRejectAcceptKernel`); `src/vt/ops.cpp`; `include/vllm/v1/worker/gpu/prepare_inputs.h` + `src/vllm/v1/worker/gpu/prepare_inputs.cpp` (the expansion); `include/vllm/v1/worker/gpu/runner.h` + `src/vllm/v1/worker/gpu/runner.cpp` (`step_num_logits`, `sample_tokens_with_rejection`); **I5b** `include/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_prefill_inputs.h` + `src/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_prefill_inputs.cpp` — anchor `include/vllm/v1/spec_decode/rejection_sampler.h:96` | `tests/vllm/v1/spec_decode/test_rejection_sampler.cpp`; `tests/vllm/v1/worker/test_prepare_inputs.cpp` (expansion + no-draft byte-identity); `tests/vt/test_cuda_ops.cpp` (CUDA==CPU bit-exact at vocab 248320); **I5b** `tests/vllm/v1/spec_decode/test_prepare_prefill_inputs.cpp` (7 cases / 27 assertions, RED-first) — anchor `tests/vllm/v1/spec_decode/test_rejection_sampler.cpp:128` | [mtp-spec-decode.md §2.4,§5](specs/mtp-spec-decode.md) | `ACTIVE` | `CLAIM-SPEC-REJECTION-I3`, `CLAIM-SPEC-MTP-I5B` | | `SPEC-GDN-SEGMENTS` | GDN speculative metadata and slot-snapshot rollback. **I4 LANDED (2026-07-24):** the spec/non-spec metadata split with decode→prefill reclassification (the #34845 case), the `T>1`/`IS_SPEC` GDN recurrence with per-timestep state snapshots, the conv sliding window advancing by the ACCEPTED count, and the k+1 state-slot allocation. DEFAULT-OFF and INERT (`num_spec==0` ⇒ `num_spec_decodes==0`, no shipped kernel branched — both spec kernels are NEW op ids). ROLLBACK PROVEN bit-exact: for every rejection point j the surviving SSM state and conv window are memcmp-identical to running only the accepted prefix through the shipped `vt::GdnDecode`/`CausalConv1dUpdate`, at the real 27B (Hv=48) and 35B (Hv=32) GDN dims on CPU and CUDA. MEASURED state cost: one f32 SSM slot = Hv·Dv·Dk·4B ⇒ 144 MiB/req (27B, 48 layers) / 60 MiB/req (35B, 30 layers) per extra slot; k=1 doubles the GDN SSM state. **I5a GDN LAYER ROUTING WIRED (2026-07-24, `CLAIM-SPEC-MTP-I5A`):** `GdnBlockPaged`'s `num_spec_decodes>0` branch now routes a PURE-spec batch through `vt::CausalConv1dSpecUpdate` + `vt::GdnSpecDecode` (mirror `qwen_gdn_linear_attn.py:1344-1357,1455-1475`), and the runner per-step upload (`StepDevInputs`/`BuildStepDevInputs` + the two decode-graph `Refresh` copies) now carries I4's six spec device tensors, gated by the extended `ValidateGdnAttentionMetadata` spec contract. DEFAULT-OFF INERT (`num_spec_decodes==0` ⇒ stub uploads + the identical non-spec branch). BIT-EXACT vs the I4 ops applied as a token-sequential decode chain, at the real 27B/35B GDN dims, via `GdnBlockPagedForTest` (`tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp`, CPU bit-exact + CUDA on-device); RED-first by a reverted stub (spec recurrence zeroed ⇒ 4/8 fail, maxΔ 1.3-1.6). MIXED spec+non-spec batch refused loudly — lands with I5d's runner loop. Row advances to `ACTIVE`: the M-mtp-1 e2e greedy token gate (verify/propose runner wiring) is owed before `DONE`, and `SPEC-MTP` STAYS `GATING` | T1 | `vllm/v1/attention/backends/gdn_attn.py:189-326,413-462`; `fla/ops/fused_sigmoid_gating.py:66-72,103-116,156-166`; `mamba/ops/causal_conv1d.py:818-1067,1181-1184`; `qwen_gdn_linear_attn.py:1329-1576`; `mamba_utils.py:213-234`; `mamba/abstract.py:55-59` | `include/vllm/v1/attention/backends/gdn_attn.h`; `src/vllm/v1/attention/backends/gdn_attn.cpp`; `include/vt/ops.h` (`kGdnSpecDecode`, `kCausalConv1dSpecUpdate`); `src/vt/ops.cpp`; `src/vt/cpu/cpu_ops.cpp`; `src/vt/cuda/cuda_gdn.cu`; `src/vllm/model_executor/models/qwen3_5_common.{h,cpp}` (`MakeQwen3_5KVCacheSpec`); **I5a:** `src/vllm/model_executor/models/qwen3_5.cpp` (`GdnBlockPaged` spec branch, `StepDevInputs`/`BuildStepDevInputs`, `ValidateGdnAttentionMetadata`), `src/vllm/model_executor/models/qwen3_5_internal.h` (`GdnBlockPagedForTest`) | `tests/vllm/v1/attention/test_gdn_metadata_builder.cpp` (20 cases / 483 assertions incl. the full upstream `GDN_BUILD_TEST_CASES` + default-off byte-identity); `tests/vt/test_ops_gdn.cpp` (reject-at-every-j rollback, CPU + CUDA, real dims); `tests/vllm/models/test_model_registry.cpp` (k+1 slot / widened-conv sizing + `num_spec==0` identity); **I5a** `tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp` (spec-routing bit-exact, RED-first) — anchor `tests/vllm/v1/attention/test_gdn_metadata_builder.cpp:83` | [mtp-spec-decode.md §3,§5](specs/mtp-spec-decode.md) | `ACTIVE` | `CLAIM-SPEC-GDN-I4`, `CLAIM-SPEC-MTP-I5A` | | `SPEC-DFLASH` | Block-diffusion drafter. **READINESS RE-ASSESSED 2026-07-25 (`CLAIM-SPEC-DFLASH-READINESS`, design-only, DONE) against the LANDED MTP machinery (`SPEC-MTP` I1..I7).** Verdict **GREEN, dispatch-ready, NO hardware/oracle/download blocker** (spec [§0](specs/dflash-spec-decode.md)). Refreshed reuse-vs-new map: DFlash gets FREE from landed MTP — the frozen spec-metadata ABI, the greedy rejection sampler (k-general, I3 tested k∈{1,3}), the GDN spec slot path + rollback + mixed spec/non-spec batch (`GdnBlockPagedMixedSpec`/`IndexSelect`/`IndexCopy`, general `num_spec`), the widened-cache-aware conv ops (I5e), the draft-KV layer pattern (`fa_draft`), the I5d/I7 runner verify/propose loop, and **`num_lookahead_tokens=k+1` ALREADY coded** (`speculative.h:91-108` `use_dflash()`); EXTENDS the single I5d-pre `hidden_tap` seam to multi-tap `[T,H×taps]`; builds NEW the `qwen3_dflash` drafter, the project's FIRST non-causal in-block attention primitive, context-KV precompute, `prepare_dflash_inputs`, and the uniform-1+k FULL CG. **k>1 verdict:** the landed rejection + GDN machinery is MECHANICALLY k-general (no `k==1` hardwiring) — DFlash's k=15 blocks need NO mechanism extension, only exercise/validation at scale (D4) + the k+1-slot memory measurement (~2.3 GiB/req 27B GDN state at block-16, the #1 risk, §5). **Checkpoint-fit:** both z-lab drafts EXIST on HF (27B 1.73 GB / 35B 368 MB bf16, DFlashDraftModel) and FIT the 119 GiB pool trivially (drafts NOT yet on dgx — D0 downloads ≤1.73 GB); the active dgx oracle `vllm-oracle-v0.25.0-stage` CONSTRUCTS DFlash (registry `DFlashDraftModel→qwen3_dflash`, speculator dir present) — soft D0 risk = confirm it SERVES DFlash+NVFP4 on sm_121 (non-causal backend; community `AEON-7/vllm-dflash` container proves the combination runs on GB10). W-plan D0-D6 in the spec. **D0+D1 LANDED 2026-07-26 (`CLAIM-DFLASH-D0D1`) on the ADVANCED pin `555967922`/vLLM 0.26.0.dev0 — `SPEC-DFLASH` → `ACTIVE`.** D0 UNBLOCKED (vllm#40898 resolved under `VLLM_USE_V2_MODEL_RUNNER=1`): the mixed-attn z-lab 27B draft CONSTRUCTS + the drafter is ALIVE (acceptance 2.21/8.80/4.75/4.57 > 1, `num_spec=16`, flashinfer-native fp8-KV, goldens committed); gate FORM measured STRICT MODE-MATCHED (vLLM-ON run-deterministic K>=3 but != vLLM-OFF — the k=16 block verify diverges at bf16 near-ties, so NOT the MTP three-way identity). D1 `DF-AUX-TAPS` DONE: `Qwen3_5AuxTaps` + `ModelForwardInput::aux_tap` route to `Qwen3_5{,Dense}Model::ForwardDeviceMultiTap` capturing `(hidden+res)` at `target_layer_ids` into `[T,H×taps]` (eagle3 `_maybe_add_hidden_state`, aux key L+1); config-gated byte-identical off. Unit gate 598 assertions (independent truncated-model reference, RED-first reversed-concat 384 fail); CUDA 697/697 + compute-sanitizer 0; INERTNESS PROVEN — 27B MTP e2e 9/9 + 27B text SACRED 235/235 byte-identical on the new oracle. **D2 `DF-DRAFT-MODEL` CODE LANDED + CPU-GATED 2026-07-26 (`CLAIM-DFLASH-D2`, kernel row `KERNEL-ATTN-DFLASH-BLOCK`):** the `qwen3_dflash` draft model (plain 5-layer Qwen3-dense reusing `dense_attn_block.h` ops), the project's FIRST non-causal / bidirectional attention primitive `vt::DFlashBlockAttention` (a SEPARATE op — causal `kAttention`/`kPagedAttention` byte-identical), the fc aux-combine, mask-embed, per-layer SWA/full resolution, and the z-lab loader. CPU gate GREEN (op 12/12 incl. RED non-causal; model forward 95/95 incl. RED full-layer-causal-flip + block isolation + fc RED); existing causal `test_ops_attention` 9/9 + `test_qwen3_forward` 1028 UNCHANGED. **D2 GPU PROMOTION GREEN on dgx (`CLAIM-DFLASH-D2`):** CUDA `-Werror` clean, CUDA==CPU 198412/198412 + compute-sanitizer 0, draft-forward parity vs the REAL vLLM draft (fc rel-L2 0.46%, hidden ≤1.3%, 11 STRICT + 5 near-tie ids), 27B SACRED 235/235 + MTP 9/9 byte-identical — **D2 DONE.** **D3 `DF-DRAFT-KV-PREP` DONE 2026-07-26 (`CLAIM-DFLASH-D3`):** `PrecomputeContextKV` + `PrepareDflashInputs` + `ForwardBlockLogitsWithContext` (reuse the UNCHANGED D2 kernel via [context;block]); GPU numeric-parity `test_qwen3_dflash_kvprep_parity` 61/61 (prepare INTEGER bit-exact vs vLLM's Triton kernel, context-KV K/V rel-L2 0.31%/0.26%, 13 STRICT + 3 near-tie = 16/16), CPU 114/114 RED-proven, inertness 235/235 + 9/9 + D2 37/37 byte-identical. **D4 `DF-ENGINE-INTEGRATION` propose brick + `dflash` config-select CODE LANDED + CPU-GATED 2026-07-26 (`CLAIM-DFLASH-D4D5`):** `DflashProposeBlock`/`SampleDflashBlockDrafts` (the non-autoregressive whole-block propose composing D3 `ForwardBlockLogitsWithContext` + greedy per-mask argmax, anchor not sampled, `dflash/speculator.py:300-413`) + `ParseSpeculativeConfigJson`/`ResolveDflash` accept `method:"dflash"`. CPU gate `test_dflash_propose` 5/19 GREEN (RED-first anchor-read fails 4/5; brick composes forward+sampler; empty-ctx degenerates to D2; config lookahead k+1). Additive + config-gated ⇒ MTP + non-spec byte-identical BY CONSTRUCTION (`git diff --stat` = new speculator TU + config accept-list + CMake + test, NO runner/model/loader/scheduler edit). **D5 `DF-ENGINE-INTEGRATION` runner-loop LANDED + e2e RUNS on dgx 2026-07-26 (`CLAIM-DFLASH-D5`):** full verify/propose loop wired — loader loads the SEPARATE z-lab draft (`LoadDflashDraft`, host bf16 + target-SHARED bf16 embed/lm_head) via a `--speculative-config` `model` key + `ResolveSpecConfig` dflash branch + `runner.set_dflash_draft`; the verify forward captures the D1 multi-tap (`aux_tap`→`ForwardDeviceMultiTap`) instead of the MTP single tap; `propose_drafts_dflash` ACCUMULATES the per-request combined-feature context (`CombineAuxFeatures(aux_tap)`) across steps and honors the `num_rejected` rollback by appending only the `(T_req−num_rejected)` accepted-prefix features, then runs `DflashProposeBlock` (k=16 GDN-spec exercised first time). **e2e (`test_qwen27_dflash_spec_decode`, 4 prompts×32 tok, our-DFlash-ON vs the committed vLLM-DFlash-ON golden): 2/4 STRICT token-exact (fibonacci, three-laws) + acceptance ~ vLLM on ALL 4 (accepted 19/39/29/25 vs golden 17/39/30/25, deltas +2/0/−1/0 — the MANDATORY dead-drafter-trap condition MET).** The 2 divergences (France tok11 `2972`↔`11751`, 17*23 tok12 `567`↔`488`) are SINGLE bf16 near-tie flips (17*23 RE-CONVERGES after one token = proven near-tie; France cascades from one flip) — the ratified near-tie ROOT the D0 gate-form anticipated, rooted in the D3-documented inline bf16 context-KV recompute envelope (~0.3-1.3% rel-L2), NOT a wiring bug (proven by the 2 exact prompts + near-exact acceptance + a non-trivial shared prefix). Inertness GREEN on this build: SACRED `test_qwen27_paged_engine` 235/235 + MTP `test_qwen27_spec_decode` 9/9 byte-identical; CUDA `-Werror` clean; NO new CUDA kernel (host orchestration reusing D1/D2/D3-sanitized ops). **NOT a clean strict-4/4 pass; STRICT 4/4 token-identity + the speed A/B = D6 (the persistent paged draft-KV bit-matching vLLM's fused context-KV projections + the uniform-1+k FULL CG).** Row STAYS `ACTIVE` (correctness at the ratified near-tie envelope; D6 remains) **D6 2026-07-27 (`CLAIM-DFLASH-D6`) — c1 SPEED A/B DONE + STRICT-irreducibility RCA + CG feasibility (records-only, NO source code):** (1) **c1 speed A/B** (`examples/vllm-bench` at `361189a7`, 8 prose+code prompts×256 tok greedy c1, 2 reps): our DFlash-ON = **2.50x TPOT (40.4 vs 101.2 ms) / 2.48x output-tput (24.4 vs 9.86 tok/s)** over our OFF, acceptance 0.22 (3.56/16), rep-stable <1.5%; `benchmark_binding=true`. vs vLLM-DFlash-ON graphed (same workload): vLLM-DFlash-ON graphed = 28.5 tok/s / 35.1 ms TPOT / acceptance_len 4.30 (same 8 prompts, `VLLM_USE_V2_MODEL_RUNNER=1`, mm-off, gpu_util 0.30), so OURS IS ~14% BELOW vLLM-DFlash-ON on output throughput (24.4 vs 28.5 tok/s) - both ~on-par at spec-OFF (9.86 vs 9.83 tok/s), but vLLM extracts a larger DFlash speedup (2.90x vs our 2.47x) because its draft step is fully device-resident + CUDA-graphed (ours host-orchestrates 13 downloads/step) + slightly higher acceptance (~4.3 vs ~3.6 draft tokens/step). The DONE speed bar (ours >= vLLM) is NOT met; closing it = the device-resident draft rewrite + FULL CG (D6 part 2). (2) **STRICT-4/4 proven bf16-IRREDUCIBLE** — the draft KV cache is bf16 not fp8 (`torch_utils.py:398` `auto`→model dtype; the D0 "fp8-KV" was the backend name, not the KV storage dtype), the D3 golden already compares pre-storage bf16 (residual K 0.31%/V 0.26% = sub-ULP kernel noise), and a fused multi-layer KV GEMM is per-element invariant to our per-layer GEMMs ⇒ bit-exact needs vLLM's exact kernels ⇒ the ratified near-tie gate is the FINAL correctness form (no fused-KV code landed). (3) **FULL CG BLOCKED** on a device-resident draft-path rewrite (the D5 path does 13 device→host downloads/step + host `[context;block]` interleaving) — the remaining throughput-parity increment (the perf form of persistent-paged-KV + the graph). Inertness by construction (the gated binary is the D5 binary; SACRED 235/235 + MTP 9/9 stand). Evidence tool `scripts/spec/vllm_dflash_timing.py`. **D7 2026-07-27 (`CLAIM-DFLASH-D7`) — within-step draft forward made DEVICE-RESIDENT (source-owning): `PrecomputeContextKVDevice` keeps per-layer K/V on device; `ForwardBlockLogitsWithContext` builds [context;block] with `vt::IndexCopy`/`IndexSelect` (removes ~30 D→H `Download`s/step). BIT-IDENTICAL (identity bf16↔f32 round-trips replaced) — e2e `test_qwen27_dflash_spec_decode` 27/27 SAME tokens (2/4 STRICT + 2/4 near-tie, acceptance 19/39/29/25), SACRED 235/235 + MTP 9/9, CUDA `-Werror` clean, compute-sanitizer 0 (198412). But the direct old-vs-new A/B = +2.0% output-tput (IN-NOISE) ⇒ D6's "downloads = the ~14% gap" REFUTED by measurement; ours 19.68 tok/s STILL ~33% BELOW vLLM-DFlash-ON 29.2 tok/s (reconstructed 8-prompt set, more prose-heavy); OFF parity our 9.97 ≥ vLLM 9.66. Residual re-attributed: acceptance (ours 2.49 vs vLLM ~3.13 accepted draft-tok/step, bf16-irreducible) + per-step context-KV RECOMPUTE (O(context²), needs the cross-step persistent paged draft-KV store) + eager-vs-graphed. SPEED BAR NOT met; SPEC-DFLASH stays `ACTIVE`; next = persistent paged draft-KV store → then FULL CG. **D9 2026-07-27 (`CLAIM-DFLASH-D9`) — PERSISTENT PAGED DRAFT-KV LANDED (bit-identical, +22.7% throughput, 0.69×→0.917×); D8 acceptance-ceiling REFUTED; residual = FULL CG ONLY:** `qwen3_dflash.cpp` `AppendContextKVHost` (project ONLY newly-accepted rows → per-layer bf16 K/V, append to `PrecomputedContextKV`) + `ForwardBlockLogitsWithPrecomputedKV` (upload the persistent store, NO re-projection) share the core `ForwardWithCtxKVDev` with the old recompute; `runner.cpp::propose_drafts_dflash` swaps the O(context²) per-step recompute (`dflash_ctx_feats_`) for an append-only per-request `dflash_kv_store_` (rollback=don't-append). NO new CUDA kernel; config-gated. BIT-IDENTICAL: CPU `test_dflash_propose` two new D9 cases = exact float equality vs full recompute; GPU e2e `test_qwen27_dflash_spec_decode` **27/27 SAME tokens** (acceptance 19/39/29/25, same divergences France@11/17×23@12); SACRED 235/235 + MTP 9/9 byte-identical; CUDA `-Werror` clean. **A/B (c1, 8 prose+code×256 tok input-len 512, 2 reps <0.1%, `benchmark_binding=true`):** ours-ON **25.75 tok/s** (was D8 20.99, +22.7%) / 38.40 ms TPOT / acc **3.68/step** vs vLLM-ON graphed **28.09** / 35.60 / acc 3.31 = **0.917×** (~8% below, was 0.69×). **Part 1 same-trajectory:** on the 2 token-identical-trajectory prompts ours per-step acceptance == vLLM's EXACTLY (fibonacci 7.80/7.80, three-laws 3.571/3.571, ratio 1.00) AND on the A/B ours acceptance (3.68) is HIGHER than vLLM's (3.31) ⇒ D8's 0.80–0.85× "bf16 acceptance ceiling" is a trajectory-divergence CONFOUND, REFUTED. Residual (~8%) = eager-vs-graphed ONLY (ours ON/OFF 2.60× vs vLLM 2.91×, OFF at parity, recompute eliminated, acceptance higher) — NOT an irreducible ceiling; the FULL uniform-(1+k) CG (device paged-KV store + paged attn, new-CUDA multi-file) is the SOLE un-landed increment. SPEC-DFLASH stays `ACTIVE` (speed not yet ≥ vLLM; residual isolated to FULL CG). **D12 2026-07-27 (`CLAIM-DFLASH-D12`) — A-wire + Part B LANDED + GPU-gated; Part C (capture) remaining; 0.917×:** A-wire makes the D11 Part-A device store the PRODUCTION path (`runner.{h,cpp}` `dflash_kv_store_`→`shared_ptr`, `MakeDeviceKVStore`/`AppendContextKVDevice`/`ForwardBlockLogitsWithDeviceKV`; GPU-gated e2e `test_qwen27_dflash_spec_decode` 27/27 all-exact acceptance 19/39/29/25 + SACRED 235/235 + MTP 9/9 byte-identical, `-Werror` clean). Part B adds `vt::DFlashPagedBlockAttention` (`OpId::kDFlashPagedBlockAttention`), the capture-safe paged kernel with EVERY metadata input a persistent DEVICE tensor and NO function-local host `cu_seqlens` upload (fixes the `cuda_ops.cu:1277-1280` capture-UAF class), gated CPU==CUDA + cross-check vs materialized `DFlashBlockAttention` `test_ops_dflash_paged_block_attn` 795648/795648 + compute-sanitizer 0. Speed 0.917× (A-wire eager + Part B not yet wired into the forward); `benchmark_binding=false`. Part C (static-shape capture + device mask-scatter + `BeginCapture`/replay + the ≥vLLM c1 A/B) is the SOLE remaining piece; if ours-ON-graphed ≥ vLLM-ON → SPEC-DFLASH DONE. Stays `ACTIVE`. **D13 2026-07-27 (`CLAIM-DFLASH-D13`) — Part C LANDED + GPU-GATED; capture-correctness PROVEN; c1 throughput NEAR-PARITY (ours 0.978x, ~2% below vLLM); gap CLOSED 0.917x→0.978x; STAYS `ACTIVE` (≥vLLM bar not yet met):** single-file additive change (`qwen3_dflash.cpp` +368/-58). (C.1) `DflashDeviceKVStore` → fixed-capacity PAGED cache (per-layer pool `[max_pages,16,Hkv,Dh]` + identity `block_table` + `seq_lens`; append = `vt::IndexCopy` scatter at slot==abs-pos, bit-identical to the D9/D11 store). (C.2) `ForwardPagedBody` runs the (1+k) block through the D12 `vt::DFlashPagedBlockAttention` reading the paged store (no `[context;block]` materialization, no function-local host uploads); runner P==1 propose routes through it, P>1 bit-identical materialized fallback. (C.3) per-request CUDA GRAPH over the paged draft step (warm-in-step repopulates the shared pool free-list right before `BeginCapture` — the fix for a `cudaMalloc`-in-capture `Get` miss from the intervening 27B target forward — then `BeginCapture → ForwardPagedBody → EndCaptureGraph`, replay with growing context entering only via in-place `seq_lens`). **Capture-correctness (MANDATORY): `test_qwen27_dflash_spec_decode` 27/27 with the graph (VT_DFLASH_GRAPH=1) BIT-IDENTICAL to eager (=0)** — same divergence tokens (France@11 got[…2972…], 17×23@12 got[…567…]), same acceptance 19/39/29/25 as D5/D7/D9/D12; graph ENGAGED (5 captures C=2048/5/4/15/6, 32+ replays); the token-diff is the capture-safety proof ([[cudagraph-capture-bakes-stack-addresses]]). **c1 A/B (one flock series, cold rep discarded, 8 prompts×256 tok):** our OFF 10.24 / our ON eager-paged 28.65 (28.69,28.61) / **our ON GRAPHED 28.70 (28.70,28.70), TPOT 34.40** / vLLM-ON graphed steady-state 29.35 (tight 3-rep 29.33/29.37/29.33, TPOT 34.07, acc_len 4.44); D9's 28.09 was a colder cross-session outlier — **NEAR-PARITY: ours 0.978× (~2% below) on the rigorous same-session band** (across sessions ours 28.70 falls inside vLLM's observed 28.09–29.37 range). ON/OFF 2.80× (vLLM ~2.98×), our OFF ≥ vLLM OFF. Per the acceptance rule ("below on any axis = an open gap; near-parity is NOT met"), the ≥vLLM bar is NOT met; STAYS `ACTIVE`. Residual (data-grounded): NOT acceptance (ours realized ~3.68 accepted draft-tok/step > vLLM's 3.44) and NOT launch/graph (both graphed, CG neutral) — per-step COMPUTE (~2% slower target-step); next lever = nsys both draft steps (`--cuda-graph-trace=node`), no premature ceiling. **ATTRIBUTION (supersedes D9):** the CUDA graph is perf-NEUTRAL (+0.3%); the ACTUAL lever was the paged context read (C.1/C.2) removing the D9/D12 per-layer `[context;block]` `IndexCopy` materialization of the whole growing context (25.75 D9 → 28.65 eager-paged, +11%) — the roadmap's "the full CG closes the gap" premise is corrected by measurement. Inertness VERIFIED on the capture binary: SACRED 235/235 + MTP 9/9 byte-identical, CUDA `-Werror` clean, no new kernel (D12 paged kernel already memcheck-0 795648), `check-device-leakage` not increased (paged path REMOVES the materialized-buffer allocs + host uploads). `benchmark_binding=true`. Correctness-complete (ratified near-tie); throughput NEAR-PARITY (0.978×, ~2% residual) ⇒ STAYS `ACTIVE` (the capture-correctness gate is MET; the ≥vLLM speed bar is the sole remaining item, a ~2% per-step-compute residual for an nsys). Anchors: `src/vllm/model_executor/models/qwen3_dflash.cpp` (`DflashDeviceKVStore` paged store, `ForwardPagedBody`, the per-request graph in `ForwardBlockLogitsWithDeviceKV`). **D14 2026-07-27 (`CLAIM-DFLASH-D14`) — SPEED GATE MET → SPEC-DFLASH `DONE`:** an nsys (`--cuda-graph-trace=node`) of the graphed spec-on step attributed the D13 ~2% residual to the from-scratch `DFlashPagedBlockAttentionKernel` draft attention (242.9 ms = 1.8% of GPU time, median ~460 us/call over context C~500-640, vs vLLM's fused flash draft-attn ~0.15%; BOTH engines run identical `cutlass_80_wmma` for the draft bf16 GEMMs, so the GEMMs were NOT the gap). Ported it to a WARP-scoped online-softmax variant `DFlashPagedBlockAttentionWarpKernel` (mirrors the shipped `AttentionWarpKernel`: one warp per (block-query,head), `__shfl_xor` butterfly reduction, register accumulator, NO `__syncthreads` storm; SAME paged/block combined-index read + causal/SWA mask + GQA; default ON, `VT_DFLASH_ATTN_BLOCK=1` keeps the bit-identical D12/D13 block kernel for A/B). Draft attn 242.9 → 77.9 ms (3.1x); our-ON c1 28.60 → 29.32 tok/s (+2.5%). **FINAL same-session 3-rep A/B (8 prompts×256 tok, cold leg discarded): our-ON graphed 29.42/29.27/29.32 (med 29.32) vs vLLM-ON graphed 29.240/29.247/29.233 (med 29.240) — our WORST rep (29.27) > vLLM's BEST (29.247), NON-OVERLAPPING bands, 1.003× ⇒ the ≥vLLM speed gate is MET.** Correctness UNCHANGED (output is exact by spec-decode construction — the target verify is untouched, only which draft proposals are accepted can shift): e2e `test_qwen27_dflash_spec_decode` 27/27 with graph==eager BIT-IDENTICAL, acceptance 19/39/29/25 unchanged (draft accepted 1629 identical warp-vs-block across the whole A/B set), 2/4 STRICT (France@11, 17×23@12 unchanged); CUDA==CPU `test_ops_dflash_paged_block_attn` 795648/795648 (warp within the f32 1e-4 / bf16 3e-2 envelope) + compute-sanitizer 0. Inertness SACRED 235/235 + MTP 9/9 byte-identical; CUDA `-Werror` clean; `check-device-leakage` not increased. `benchmark_binding=true`. Block-diffusion drafting is now correctness-complete (ratified near-tie) AND at/above vLLM throughput — this was the roadmap's FINAL open speed item. Anchors: `src/vt/cuda/cuda_ops.cu` (`DFlashPagedBlockAttentionWarpKernel` + `UseDflashAttnBlockKernel`; the D12 block kernel retained as the `VT_DFLASH_ATTN_BLOCK=1` reference). | T1 | `vllm/v1/worker/gpu/spec_decode/dflash/speculator.py`; `vllm/model_executor/models/qwen3_dflash.py`; `vllm/model_executor/models/interfaces.py:1382` (aux value); `eagle3_utils.py:41-56` (+1 shift) | `include/vllm/model_executor/models/qwen3_5.h` (`Qwen3_5AuxTaps`, `ForwardDeviceMultiTap`); `qwen3_5_dense.h`; `model_registry.h` (`aux_tap`); `src/vllm/model_executor/models/qwen3_5.cpp` (`MaybeCaptureAuxTap`/`ValidateAuxTapLayerIds`/`ForwardDeviceMultiTap`); `qwen3_5_moe.cpp`+`qwen3_5_dense.cpp` (routing); D2/D3 `include/vllm/model_executor/models/qwen3_dflash.h` + `src/vllm/model_executor/models/qwen3_dflash{,_weights}.cpp`; D4 `include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h` + `src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp` (`DflashProposeBlock`/`SampleDflashBlockDrafts`); D5 `src/vllm/entrypoints/model_loader.cpp` (`LoadDflashDraft`/`DflashDraft`) + `include/vllm/entrypoints/model_loader.h`; D5 `src/vllm/v1/worker/gpu/runner.cpp` (`set_dflash_draft`/`propose_drafts_dflash`/aux-tap capture) + `include/vllm/v1/worker/gpu/runner.h`; `src/vllm/config/speculative.cpp` + `include/vllm/config/speculative.h` (`ResolveDflash` + `dflash`/`model` parse); D14 warp kernel [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L1433) | `tests/vllm/models/test_qwen27_paged_forward.cpp` (multi-tap 598); `tests/vt/test_ops_dflash_block_attn.cpp`; `tests/vllm/models/test_qwen3_dflash_forward.cpp`; `tests/vllm/v1/spec_decode/test_dflash_kvprep.cpp`; `tests/parity/test_qwen3_dflash_{draft,kvprep}_parity.cpp`; D4 `tests/vllm/v1/spec_decode/test_dflash_propose.cpp` (5/19, RED-first); D5 `tests/parity/test_qwen27_dflash_spec_decode.cpp` (e2e 27/27, 2/4 strict + acceptance~vLLM); `scripts/spec/d{0,2,3}_dflash_*.py`; `tests/parity/goldens/dflash_27b{,_draft,_kvprep}/`; D6 `scripts/spec/vllm_dflash_timing.py` (vLLM-DFlash c1 timing); D7 device-resident `src/vllm/model_executor/models/qwen3_dflash.cpp` (`PrecomputeContextKVDevice` + `ForwardBlockLogitsWithContext` via `vt::IndexCopy`/`IndexSelect`); D9 persistent paged draft-KV `qwen3_dflash.{h,cpp}` (`AppendContextKVHost`/`ForwardBlockLogitsWithPrecomputedKV`/`ForwardWithCtxKVDev`/`PrecomputedContextKV`) + `runner.{h,cpp}` (`dflash_kv_store_`/`propose_drafts_dflash`) + `tests/vllm/v1/spec_decode/test_dflash_propose.cpp` (2 D9 bit-identity cases); D12 A-wire `runner.{h,cpp}` (device store as production path) + D12 Part B `include/vt/ops.h`/`src/vt/ops.cpp`/`src/vt/cpu/cpu_ops.cpp`/`src/vt/cuda/cuda_ops.cu` (`kDFlashPagedBlockAttention`) + `tests/vt/test_ops_dflash_paged_block_attn.cpp` (CPU==CUDA + cross-check, 795648/795648 + sanitizer-0); D13 `src/vllm/model_executor/models/qwen3_dflash.cpp` (fixed-capacity paged `DflashDeviceKVStore` + `ForwardPagedBody` + the per-request draft-step CUDA graph in `ForwardBlockLogitsWithDeviceKV`); D14 [test_ops_dflash_paged_block_attn](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) + [ledger](parity-ledger.md#L738) | [dflash-spec-decode.md](specs/dflash-spec-decode.md) | `DONE` | `489a7544` | diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md index debdffb26..a35886ce0 100644 --- a/.agents/quantization-matrix.md +++ b/.agents/quantization-matrix.md @@ -35,7 +35,7 @@ otherwise it remains `PARTIAL` or `INVENTORIED` even if parsing works. | `QUANT-GGUF-CIQ-GEMM` | Compute-in-quant GEMM: activation quant (Q8_0/Q8_K) + per-type vec_dot dispatch for Q8_0/Q4_K/Q5_K/Q6_K/Q3_K/Q4_0; portable C++ tier, then x86/Arm SIMD + repack tiers. **G1-G4 landed** — the portable tier-0 path is complete, gated at the OP level, and **ROUTED end to end**: `vt::MatmulBT` dispatches a block-dtype weight to `kMatmulBTQuant`, keep-quant is the production DEFAULT wherever that op is registered, and the six routed encodings compute in quant with **no token movement**. **G6 (2026-07-23)** added the Arm **i8mm mmla `nrc==2` tier** for q8_0/q4_0/q4_K/q6_K (q3_K/q5_K have no upstream mmla → stay portable), 2x2-tiled into `kMatmulBTQuant` at even M,N: op-level q4_K **7–8.4×** / q6_K **3.8–4.5×** / q8_0 ~1.2× over portable, e2e prefill +8.4 % on the q8_0-dominant bench file (1.44× behind llama.cpp), tokens byte-identical. **G7 (2026-07-23)** added q8_0 **repack-at-load** (the `q8_0_4x8` tier `ggml_repack_get_optimal_repack_type` picks on NEON+i8mm): the loader repacks each q8_0 weight once into the `block_q8_0x4` interleave and `kMatmulBTQuant` dispatches a pre-shuffled i8mm gemm/gemv with no per-block register shuffles — op-level q8_0 **3.7–5.9×** over the mmla tier, **E2E prefill 1.92× same-binary → 223.8 t/s vs llama.cpp pp128 177.3 = at/beyond parity** (was ~1.5× behind), decode at parity, tokens byte-identical. **CPU prefill parity reached; the prefill-lever search is closed** (remaining gap = peak RSS 1.39×, loader-bound). G5 (x86) + G8 open. **The FRESH op-dispatch profile this row owed is DONE (2026-08-06, dgx aarch64, `main` @`dfd29060`, same bench file; see `.agents/benchmark-record.md` 'FRESH op-dispatch profile'), and it does NOT support starting G5 next:** `QuantRepackMatmul` is 5.06 % of prefill and 15.99 % of decode on aarch64 where the i8mm tier already landed. The profile re-ranks the CPU levers to (1) threadpool synchronisation at 47 % of decode (`ThreadReady`+`PollForWork`+`Barrier`; M=1 cannot amortise the barrier) and (2) CPU paged attention at ~39 % of prefill, of which 20.68 % is a per-ELEMENT dtype switch in the attention dot loop (`cpu_paged_attn.cpp:29` called from `:143`), the same defect class E1 already removed from the elementwise GEMM. G5 stays a real x86 gap worth closing for x86 users, but it is not the top lever, and the x86 box is VOID for timing so it cannot be speed-gated here | llama.cpp `ggml/src/ggml-cpu/ggml-cpu.c:211-406` traits table, `ggml-cpu/quants.c:174-860` generic vec_dot, `arch/{x86,arm}/quants.c`, `ggml-cpu/repack.cpp:4153-4830` at `237ad9b96` | G1: [block dtypes + geometry](../src/vt/dtype.cpp#L32), [quant traits table](../src/vt/cpu/cpu_quant_traits.cpp#L1), [shared block decoders](../src/vt/cpu/cpu_quant_dequant.cpp#L1), [op surface](../include/vt/quant.h#L1). G2: [activation quant + scratch sizing](../src/vt/cpu/cpu_quant_act.cpp#L1) (`quantize_row_q8_0/q8_K`). G3: [the six generic vec_dot](../src/vt/cpu/cpu_quant_dot.cpp#L1), [block-struct mirror](../src/vt/cpu/cpu_quant_blocks.h#L1), [`kMatmulBTQuant` quantized path + composite fallback](../src/vt/cpu/cpu_quant_gemm.cpp#L1). G4: [the routing point](../src/vt/ops.cpp#L158) — `vt::MatmulBT` sends a block-dtype `b` to `MatmulBTQuant` and is otherwise unchanged, which is sufficient because every model matmul helper already routes an `nk=true` weight there ([qwen3_5.cpp:1067](../src/vllm/model_executor/models/qwen3_5.cpp#L1067)); plus [the default flip + `expand_nk`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) and [the untransposed expand path](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L194). G6: [Arm i8mm mmla tier](../src/vt/cpu/cpu_quant_dot_arm.cpp#L1) (q8_0/q4_0/q4_K/q6_K `vmmlaq_s32`, HWCAP2_I8MM-probed, `VT_CPU_QUANT_MMLA` defeat) + [2x2 tile in kMatmulBTQuant](../src/vt/cpu/cpu_quant_gemm.cpp#L85), per-file `+i8mm` in CMakeLists | [G1 traits cross-check + fallback units](../tests/vt/test_ops_quant_traits.cpp#L1) — 8 cases / 5,615 assertions green (was 5,694; its composite case now covers Q8_K alone because the six weight types legitimately no longer take that path): vt geometry vs the reader's `GgmlTraits` vs ggml-common.h arithmetic all agree, and the composite equals the loader dequant byte-for-byte. [G2/G3 units](../tests/vt/test_ops_quant_dot.cpp#L1) — 16 cases / 78,052 assertions green: every `vec_dot` gated against an INDEPENDENT f64 dequantize-then-dot reference (tolerance relative to the dot's L1 magnitude, actual agreement ~1e-6) over nblocks {1,2,3,5,7,16} incl. single-block and odd multiples; ragged K throws at every layer; upstream thresholds ported unwidened (test-quantize-fns:17-28, test-backend-ops:4277 NMSE ≤ 5e-4 at M {1,4,32,512} × N {1,7,16}); bit-exact run-to-run and across threads 1/2/4; byte-exact encoder gate pins the rounding rules; 14-mutant battery, 13 caught, the 1 uncaught mutant provably unreachable. [dequant units](../tests/vllm/test_gguf_dequant.cpp#L25) still green after the decoder move. DGX (G2/G3 re-confirmed, each gate STANDALONE, goldens md5 identical before/after): clean CUDA `-Werror` build 0 warnings + full regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16 on both 0.6B and 4B, OPT 6/6, DeepSeek-V2 8/8) + `test_qwen36_gguf_engine` 28/28 with 16/16 tokens on both APEX files + the new CPU units green on aarch64 with identical counts. **G4 (2026-07-22):** `test_qwen36_gguf_engine` PASSES STANDALONE on a CPU-only dgx build (where keep-quant is live) — 2/2 cases, 16/16 greedy tokens on APEX-Compact AND APEX-Balanced vs the same-file llama.cpp oracle, exercising 5 of the 6 routed encodings end to end; the CUDA regression set is UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, gguf 28/28 incl. `VT_CPU_REF=1`), goldens md5 identical. **Binding CPU A/B** (idle dgx aarch64, one flock, same binary, 3 reps, `Qwen3.5-2B-UD-Q8_K_XL`): decode 2.216 -> 7.650 t/s (**3.45x**), prefill 5.149 -> 21.44 t/s (**4.16x**), peak RSS 7.428 -> 6.401 GiB, output tokens byte-identical across the pre-G4, post-G4 and `VT_CPU_REF=1` arms. Still **3.38x / 8.20x / 2.29x behind llama.cpp** — the projected 9-17x did NOT hold because 60 % of that file's weight bytes are `f16`, which no block encoding covers. **That gap is now CLOSED by `KERNEL-GEMM-CPU-ELEM`** (2026-07-22, same box/recipe/binary discipline): the elementwise kernel went 18-24 -> 69-351 GFLOP/s bit-exactly, taking the CPU position to **decode 1.03x behind (parity within 3.1 %) and prefill 2.34x behind**, tokens unchanged (same md5). Its measured NEGATIVE re-ranks G5-G8 once more: M-blocking the elementwise GEMM bought 1.63x op-level and **0.0 % end-to-end**, so the 95.37 % `kMatmul` attribution these G-rows were ranked against is STALE and a FRESH op-dispatch profile is owed before G5/G6/G7 are started. **G6 (2026-07-23):** [Arm i8mm mmla tier](../src/vt/cpu/cpu_quant_dot_arm.cpp#L1) landed against the refreshed profile (kMatmulBTQuant 50 % + kMatmul 16 % + kMatmulBT 14 % = 80 % of prefill). [test_ops_quant_dot G6 cross-check](../tests/vt/test_ops_quant_dot.cpp#L1) — 19 cases / **78,162** assertions on dgx aarch64: q8_0/q4_0 mmla **BIT-IDENTICAL** to the portable/scalar tier (`vmlaq_f32` non-fused under `-ffp-contract=off`), q4_K/q6_K within NMSE ≤ 5e-4, mmla GEMM bit-identical across threads 1/2/4/20. `test_qwen36_gguf_engine` 2/2 · 16/16 on both APEX files with mmla live (q8_0/q4_K/q6_K at prefill), bench-file token md5 `d235db12f2cd304007530286a1755c95` byte-identical across mmla-OFF/ON/`VT_CPU_REF=1`. Op-level portable→i8mm: q8_0 ~1.2×, q6_K 3.8–4.5×, q4_K 7–8.4×; e2e prefill same-binary 1.084× (1.56×→1.44× behind llama.cpp pp128). CUDA `-Werror` 0-warn, regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138, Qwen3-dense 184, OPT, DeepSeek-V2 223), goldens untouched. **G7 (2026-07-23):** [q8_0 repack transform](../src/vt/cpu/cpu_quant_repack.cpp#L1) + [i8mm repack gemm/gemv](../src/vt/cpu/cpu_quant_repack_arm.cpp#L1) dispatched from [`kMatmulBTQuant`](../src/vt/cpu/cpu_quant_gemm.cpp#L151) on `b.repacked`; loader repacks via [`OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) under `GgufLoadPolicy::quant_repack`, flag carried to the kernel through [`ResidentWeight`](../src/vllm/model_executor/models/qwen3_5.cpp#L702). [test_ops_quant_repack](../tests/vt/test_ops_quant_repack.cpp#L1) — 305 assertions on dgx aarch64: repacked gemm/gemv `memcmp`-equal to plain `kMatmulBTQuant` across decode/leftover/prefill, f32+bf16 out, strided activations, threads 1/2/4/20; interleave matches `make_block_q8_0x4` byte-for-byte (110 on x86, numeric skip). `test_qwen36_gguf_engine` STANDALONE 2/2·16/16 on APEX Compact+Balanced (repack live), token md5 `d235db12f2cd304007530286a1755c95` byte-identical across repack-ON/OFF/`VT_CPU_REF=1`. Binding dgx aarch64 (idle, one flock, 6 interleaved reps): op-level q8_0 3.7–5.9× (518→2401/583→3456/514→1902 GFLOP/s); E2E prefill **1.92×** (1096→572 ms), **223.8 t/s vs llama.cpp pp128 177.3 = 1.26× at/beyond parity**, decode at parity, RSS unchanged; fresh profile q8_0 GEMM 55%→~21%, prefill-lever search CLOSED. CUDA `-Werror` 0-warn, regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, Llama 16/16), goldens content-hash identical . **P0 REGRESSION FOUND + FIXED (2026-08-06, `CLAIM-QUANT-GGUF-CIQ-GROUPED-DTYPE`):** the GROUPED provider `MatmulBTQuantGroupedKernel` was f32-ONLY — it advanced a `float*` by `act.stride[0]` and declared the row `kF32` whatever `act.dtype` said, so a bf16/f16 activation was mis-strode 2x AND mis-decoded. Every prior caller/test passed f32; qwen3_5 W3b `KqGrouped` (bf16 act, `b4f5610a`) was the first non-f32 caller, so CPU-only GGUF 35B decode became all-token-0 while the CUDA gate stayed byte-exact (CUDA always honoured `act.dtype`). Fixed at [`cpu_quant_gemm.cpp:220-268`](../src/vt/cpu/cpu_quant_gemm.cpp) (rows addressed by `SizeOf(act.dtype)`/`SizeOf(out.dtype)`; `repacked`/`q8_0_aligned` now propagate onto the per-expert slice — the CIQ-G7 all-zero mode). Gated per activation dtype + bf16-out by 2 NEW cases in [`test_ops_quant_dot.cpp`](../tests/vt/test_ops_quant_dot.cpp) (RED pre-fix on f16+bf16 for all 12 weight encodings, GREEN after; f32 unaffected either way) | [CIQ GEMM leaf](specs/gguf-compute-in-quant-gemm.md) | `ANCHOR-BACKFILL` | `CLAIM-QUANT-GGUF-CIQ-G7-1` | | `QUANT-GGUF-KEEPQ-LOADER` | Keep-quantized GGUF loader: block-resident 2-D matmul weights ([N,K], no transpose), per-tensor routing, `VT_CPU_REF` dequant-oracle switch, bench-branch `7c91a42` merge. **L1+L2+L3 landed** — block residency, the TOTAL per-tensor routing policy and the `VT_CPU_REF` oracle switch all exist and are gated. **Keep-quant is DEFAULT ON since CIQ G4** wherever the running device has a registered `kMatmulBTQuant` (CPU, and since 2026-07-29 also **CUDA** for the Q8_K family via the `KERNEL-QUANT-CIQ-GEMM-CUDA` kCUDA provider — a CUDA runner now keeps k-quant/i-quant blocks COMPRESSED instead of expanding), with `VT_GGUF_KEEP_QUANT=0` as the opt-out. L4 measured; **L5 LANDED** (mmap in-place residency + tied-head sharing + read-once page release) — peak RSS 6.401 -> **3.884 GiB**, 2.29x -> **1.39x** llama.cpp, byte-identical | llama.cpp `src/llama-model-loader.cpp:1047,1385` (file-typed residency), `:1676` + `ggml/src/llama-mmap.cpp:490` (`unmap_fragment`), `ggml/src/ggml-cpu/repack.cpp:4727` (repack-at-load hook) at `237ad9b96` | L1: dense-arch (`qwen35`) GGUF path on main via the registry — [dense GGUF load](../src/vllm/model_executor/models/qwen3_5_dense.cpp#L60), [arch->registered-ID map](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L212), [F16/BF16 row dequant](../src/vllm/model_executor/model_loader/gguf_dequant.cpp#L61). L2: [block residency `OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) — raw ggml blocks into an `OwnedTensor` with a block `vt::DType`, file `[N,K]` orientation, `nk=true`, no transpose; stacked experts split by byte range. L3: [routing policy + `VT_CPU_REF`/`VT_GGUF_KEEP_QUANT`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L1) (6 roles, no `default:` label so an unrouted role is a `-Werror=switch` build failure) wired at every loader call site via [`OwnMatmulWeight`/`RequireExpand`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L185). **Default now device-derived** (CIQ G4): [`GgufQuantComputeAvailable`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) gates it on `vt::OpRegistered(kMatmulBTQuant, CurrentPlatform().device_type())`, and the same condition drives `expand_nk`, which stops transposing a weight that must expand. **`expand_nk` now also covers the GDN split projections** (2026-07-23, `CLAIM-CPU-GDN-ORIENT-1`): a fresh op-dispatch profile found `LoadGdnGguf`'s `in_proj_qkv/z/b/a` + `out_proj` were the ONE expanded weight family still transposed to [K,N] (nk=false → slow `kMatmul`, 17.9 % of prefill); the new [`gdn_expand_nk` field](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) + [`MakeGdnProj`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L131) keep them [N,K] nk=true (V-head reorder applied first, orthogonal to orientation; `VT_GGUF_GDN_NK=0` A/B opt-out) → M-blocked `kMatmulBT`, same-binary prefill **1.090×** / decode 1.09×, byte-identical (`kMatmul` 72→0 calls in prefill) | [L2/L3 units](../tests/vllm/test_gguf_keep_quant.cpp#L1) — 17 cases / 5,574 assertions green. **Gate 1 (losslessness) proven PER ENCODING**, one case each for Q4_0/Q8_0/Q3_K/Q4_K/Q5_K/Q6_K: resident bytes `memcmp`-equal to the file span and resident-block dequant BYTE-IDENTICAL to the direct-from-file expansion (f32 and bf16), over pseudo-random block bytes constrained only to finite f16 scales; at loader level the kept weight rehydrates to the expanded `[K,N]` bf16 tensor byte for byte, per weight and per expert, on dense and MoE fixtures. **Totality**: the audit hook proves `routed == the file's complete tensor list` on both fixtures, plus 6 roles × 12 encodings × 6 shapes against a LONGHAND expectation (12 keep / 420 expand, so neither outcome is vacuous). **Gate 2 (oracle stability)**: `VT_CPU_REF=1` keeps nothing quantized and every weight is bit-identical to the historical load; on dgx [`test_qwen36_gguf_engine`](../tests/parity/test_qwen36_gguf_engine.cpp#L143) under `VT_CPU_REF=1` is 28/28 assertions, 16/16 tokens on both APEX files — same as without. 10-mutant battery, 10 caught (the expert-slice-offset mutant survived the first pass, exposed a real coverage hole, and drove the MoE fixture). DGX (each gate STANDALONE, production flags, goldens md5 identical before/after `2965ef5772b556d3f3f86fedf4221b2f`): clean CUDA `-Werror` 0 warnings + regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16 on both, OPT 6/6, DeepSeek-V2 8/8) + gguf units green on aarch64 with identical counts; full CPU ctest 154/154. **RSS at G4 was 6.401 GiB (2.29x); L5 took it to 3.884 GiB (1.39x)** — binding, idle dgx aarch64, same-binary 3-rep A/B: mmap in-place residency (borrow kept q8_0 blocks out of the mapping, refcounted, -0.998 GiB), tied-head sharing (one bf16 vocab matrix for embed+lm_head, -0.946 GiB), read-once page release (MADV_DONTNEED the expanded tensors' file pages, port of llama.cpp `unmap_fragment`, -0.573 GiB). Decode TPOT 41.7 ms UNCHANGED, prefill TTFT +4% (first-touch faults move into the timed window), output md5 `d235db12f2cd304007530286a1755c95` identical across BEFORE/AFTER/ORACLE. Lifetime safety tested explicitly (borrow outlives the GgufFile AND the on-disk file; shared head freed once either order). **L6 (2026-07-23, `CLAIM-QUANT-GGUF-KEEPF16-L6-1`) implemented keep-f16 residency and REFUTED the "remaining gap is the f16 expansion" attribution above.** New `kKeepF16` residency + [`OwnGgufF16`/`OwnGgufKeptSlice`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L79) keep F16 matmul weights (+ F16 embed/tied head, one shared f16 vocab matrix via [`OwnedBytes::KeepAlive`](../include/vllm/model_executor/models/owned_bytes.h)) resident as F16, consumed by the elementwise f16 GEMM. Binding A/B: peak RSS 3.884 → **3.832 GiB (−52 MB, RSS-NEUTRAL)** — L5's page-release ALREADY dropped the f16 file pages, so keep-f16 only swaps an anonymous bf16 buffer for equal-size file-backed f16 pages. smaps attribution: keep-f16 file-backed **2.634 GiB ≈ llama.cpp's 2.68 file** (weight residency AT PARITY), anon 1.20 GiB; the **remaining ~1.08 GiB gap is the engine's ANONYMOUS activation/KV workspace, NOT weights** — the real, separate CPU RSS lever. Also regresses prefill (TTFT 577 → ~1000 ms, first-touch faults into the timed window; decode at parity). Tokens byte-identical (md5 `d235db1…`). Ships DEFAULT OFF at L6. **L7 (2026-07-23, `CLAIM-QUANT-GGUF-RSS-L7-1`) REVERSED L6's refutation and CLOSED the CPU RSS gap to 1.01× llama.cpp.** The profile disproved the "workspace" attribution — DevicePool 20 MiB, whole KV 115 MiB, both ≤ llama.cpp. The 1 GiB residual was a q8_0 repack-source DOUBLE-COUNT: on aarch64 the G7 repack COPIES q8_0 into an anonymous buffer while the f16 borrows keep the mapping alive, so the DEAD source blocks stay file-backed. [`OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) now `DropSpanResidency`es the repack source (port of llama.cpp `unmap_fragment`), and [`PrefaultBorrowedSpan`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L21) faults borrowed weights at load (port of llama.cpp mmap prefetch), removing L6's prefill regression — so [keep-f16 flips DEFAULT ON](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L168) (`VT_GGUF_KEEP_F16=0` opt-out). Binding A/B (idle dgx aarch64, base-vs-L7 same-binary): peak RSS **3.884 → 2.832 GiB = 1.39× → 1.01× llama.cpp** (File 2.632 → 1.629, the released q8_0 source; anon 1.200 unchanged), prefill **1.18× AHEAD** (204 vs pp128 173.2, denominator SUPERSEDED by #1003), decode ~parity (24.4 vs 25.09), tokens BYTE-IDENTICAL (md5 `809f2d0…` base/L7/oracle). **Against our own keep-f16-off arm the default costs about 9% of prefill (224 → 204 t/s) and about 1.4% of decode (TPOT 40.4 → 40.95 ms) for 1.05 GiB, settled 2026-08-17 as a product decision, NOT by the competitor floor.** Anon 1.200 GiB is IRREDUCIBLE (repacked q8_0 1.06 + KV 0.115 + pool 0.02). Regressions UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, Llama 16/16, GGUF engine 28/28); `test_gguf_keep_quant` 36/36 (+1 L7 prefault byte-transparency case, x86+aarch64) | [keep-quant loader leaf](specs/gguf-keep-quant-loader.md) | `ANCHOR-BACKFILL` | `CLAIM-QUANT-GGUF-RSS-L7-1` | | `QUANT-QWEN38-27B-GGUF-ARM` | The `Qwen3.8-27B-Q4_K_M.gguf` arm end to end: tensor accounting, text decode, the multimodal legs, and this ARTIFACT's own tokenizer and chat template. The standing GGUF k-quant requirement for a model whose bf16 arm is already gated ([#915](https://github.com/mudler/vllm.cpp/issues/915)), and the arm `BACKEND-GATE-CUDA-LLAMACPP` in the [backend matrix](backend-matrix.md) is already recorded as blocked on. **Header-verified 2026-08-18** at `unsloth/Qwen3.8-27B-GGUF`@`fe1e2a23d973adb629709749dc4f6756df66ef10`: GGUF v3, arch `qwen35`, 866 tensors, F32 456 / Q4_K 294 / Q6_K 67 / Q5_K 48 / Q8_0 1, data end == file size 17,106,775,008. **Two facts [#821](https://github.com/mudler/vllm.cpp/issues/821) did not record and which change the scope:** `qwen35.block_count = 65` with `qwen35.nextn_predict_layers = 1`, so block 64 is the MTP/`nextn` DRAFTER (`blk.64.nextn.{eh_proj,enorm,hnorm,shared_head_norm}` plus a full-attention block and an FFN) — exactly the 15-tensor difference from the same model's 851-tensor BF16 GGUF, and a loader that reads `block_count` as decoder depth builds a 65-layer model out of a 64-layer checkpoint plus a drafter; and `tokenizer.ggml.padding_token_id = 248055` against 248044 in the BF16 GGUF and `null` in the official HF config, which is why the tokenizer gate belongs to the ARM. NOT blocked on kernels: every dtype this file carries is already computed natively on BOTH tiers. The CUDA tier really has no prefill/decode split (`LaunchGemm` [cuda_quant_dot.cu:1609](../src/vt/cuda/cuda_quant_dot.cu#L1609) sizes its grid `m*n` and the encoding switch at [:1864](../src/vt/cuda/cuda_quant_dot.cu#L1864) never sees `M`); the CPU tier DOES branch on `M` at [cpu_quant_gemm.cpp:190](../src/vt/cpu/cpu_quant_gemm.cpp#L190), which takes the Arm i8mm `mmla` 2x2 tile only for even `M` and `N` and sends decode (`M=1`) to the portable `nrc==1` path. That is a kernel-TIER split, NOT a coverage split -- no dtype gains or loses support at any `M`, both arms end in the same `BlockVecDot` table -- so the conclusion stands and it is a W3 speed fact rather than a W2 gap | llama.cpp `b10451` = `10bf611e5` ([pin](oracles/llama-cpp.md), **`gateable = yes`** since [#857](https://github.com/mudler/vllm.cpp/issues/857) landed 2026-08-22) is the arm's ORACLE and its only comparator — at the vLLM pin `555967922` there is no in-tree GGUF reader (`6635279d8` moved it out of tree) and SGLang's alias table does not reach `qwen3_5` ([#979](https://github.com/mudler/vllm.cpp/issues/979)). llama.cpp is never the MIRROR | the single-file GGUF entry [qwen3_5_gguf_weights.cpp:1474](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L1474); the CUDA quant enum [cuda_quant_dot.cu:700](../src/vt/cuda/cuda_quant_dot.cu#L700) and CPU [cpu_quant_dot.cpp:787](../src/vt/cpu/cpu_quant_dot.cpp#L787) already cover Q4_K/Q5_K/Q6_K, and Q8_0 has its own path [cuda_quant_dot.cu:1659](../src/vt/cuda/cuda_quant_dot.cu#L1659) | **W2 LANDED the accounting**, modelled on [muse_glimmer_gguf_manifest.inc](../tests/vllm/models/muse_glimmer_gguf_manifest.inc): committed header-only manifests [qwen38_27b_q4km_gguf_manifest.inc](../tests/vllm/models/qwen38_27b_q4km_gguf_manifest.inc) (866 names, 51 kv) and [qwen38_27b_mmproj_gguf_manifest.inc](../tests/vllm/models/qwen38_27b_mmproj_gguf_manifest.inc) (334 names, 35 kv), generated by [gen-qwen38-27b-gguf-manifest.py](../scripts/gen-qwen38-27b-gguf-manifest.py) from the mirrored bytes; the accounting gate [test_qwen38_27b_gguf_manifest.cpp:223](../tests/vllm/models/test_qwen38_27b_gguf_manifest.cpp#L223) (6 cases, 464 assertions hermetic, 4745 over the shipped bytes under `VLLM_CPP_QWEN38_27B_{GGUF,MMPROJ}`, ZERO unaccounted in BOTH directions on both files); and the reachability gate [test_gguf_accounting_reach.cpp:184](../tests/vllm/entrypoints/test_gguf_accounting_reach.cpp#L184) (6 cases, 22 assertions), which enters through `LoadedEngine::FromModelDir` and reds 3/6 when either refusal call site in `model_loader.cpp` is deleted while the manifest target stays green at 6/6. The `nextn` correction was a gap that DID NOT EXIST: [qwen3_5_gguf_weights.cpp:889](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L889) has taken `block_count - nextn_predict_layers` since `1a4db5c3c`, and `mtp_num_hidden_layers` has been republished since `493327b4e`; what was missing was a gate, because [test_qwen3_5_gguf_mtp.cpp:36](../tests/vllm/models/test_qwen3_5_gguf_mtp.cpp#L36) is asset-gated on `VLLM_MTP_GGUF_MODEL`, skips SILENTLY when it is unset, and checks only `num_hidden_layers > 0` rather than the arithmetic its own comment claims. **W3 RAN THE TOKEN GATE ON 2026-08-23 AND IT FAILED.** Two `rc run` jobs on `thor:gpu0` (`64f66cda`, `8e0d8e54`), same GGUF file both sides, greedy, 48 tokens, concurrency 1, MTP OFF so both engines decode the same 851 tensors and the same 64-layer trunk (llama.cpp ignores all 15 of `blk.64`, re-observed as exactly 15 `unused tensor` warnings). **Tokenizer EXACT 6/6** through three of our paths (`examples/tokenize`, `vllm-cli` prompt counts, and the agreeing generation prefixes), so the #1355 prompt-token undercount is absent here. **Generation DIVERGES 5/6**, first differing index 7/34/20/-/14/32 with prompt 3 token-exact 48/48. Teacher-forcing the oracle along OUR ids over all 288 steps puts our token at the oracle's **rank 1 on 282 and rank 2 on 6, never rank 3 or worse**, losing by 0.027-0.178 logits against absolute logits of 15.9-22.6 - a PRECISION difference in the quantized compute path, not a wiring defect. The near-tie band was NOT reached for: the oracle's greedy decode reproduced #857's text byte for byte from a different build, so it is deterministic and the band's premise fails. No speed or memory number is admissible from this arm; resident bytes were measured only to refuse a dequant hypothesis (ours 24.997 GiB vs the oracle's 30.917 GiB on the same box and file, so NO dequant-to-bf16 blow-up). **2026-09-02, the cause is FOUND and PARTLY FIXED and the gate still FAILS:** our final logits carried only bf16 RESOLUTION (288 of 288 top-1 logits exactly on the bf16 grid, ULP 0.125 at magnitude 16-32, against contested gaps of 0.027-0.178), because a GGUF keep-quant head reached the bf16-output logits helper on the `nk` LAYOUT flag. Routing a block-quant head to the f32-output GEMM takes the arm from **5 of 6 to 3 of 6** divergent prompts, measured as one tree built twice on `thor:gpu0` (`c0b3fc6d`) whose bf16 arm reproduces 2026-08-23 index for index. The two SMALLEST margins (0.027185, 0.058050) resolved; the three largest (0.085434, 0.115482, 0.178236) did not, at the same indices with the same agreeing prefixes, so the residual is a MAGNITUDE term rather than a resolution one. `TOKEN_GATE` stays `FAIL` and no speed or memory axis becomes admissible. [Evidence](../docs/bench-evidence/qwen38-27b-q4km-logits-f32-20260902.md), and the superseded 5-of-6 run [Evidence](../docs/bench-evidence/qwen38-27b-q4km-token-gate-20260823.md) | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | -| `QUANT-QWEN38-27B-NVFP4-ARM` | The `unsloth/Qwen3.8-27B-NVFP4` artifact, which is **not what its name says**. **Its pinned revision is GONE:** `a767244d27bd76589a3e3b2ab4e64032c4ebc7af`, the revision [#821](https://github.com/mudler/vllm.cpp/issues/821) names, answers HTTP 404 and `git ls-remote` reports one ref, `refs/heads/main` = `7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108` — the second in-place re-quantization this publisher has done in this family, after `unsloth/Qwen3.6-27B-NVFP4`. So the user-reported load failure on #821 is CORROBORATED at a different revision, never reproduced. At the live revision (header-verified 2026-08-18, 1953 + 15 tensors, `8 + header_len + max(data_offsets[1])` == file size 22,568,192,096) `quantization_config.format` is `mixed-precision`: `group_0` is FP8 W8A8 with **per-CHANNEL** weight scales and **DYNAMIC per-token** activations over `self_attn.(q\|k\|v\|o)_proj`, `linear_attn.(in_proj_qkv\|in_proj_z\|out_proj)`, `lm_head` and `layers.(56..63).mlp.*`; `group_1` is `nvfp4-pack-quantized` W4A4 over the remaining `mlp.*`; plus an 8-bit static `kv_cache_scheme` and an `ignore` list of **303 entries** -- not just the vision tower: 48 x `linear_attn`, `linear_attn.norm`, `linear_attn.in_proj_b` and `linear_attn.in_proj_a` (the GDN layer count), 27 x 4 vision blocks, 2 mergers, and `re:^mtp.*`. That list is what makes the predicate claim provable rather than asserted: `in_proj_a`/`in_proj_b` are IGNORED while `in_proj_qkv`/`in_proj_z`/`out_proj` are `group_0` TARGETS, so a resolver that reads the groups but not the `ignore` list gets the GDN block wrong in both directions. **`*.input_scale` appears ZERO times in the checkpoint.** Four independent blockers, each anchored in the spec: the unconditional `.input_scale` read, a per-channel BF16 `weight_scale` that `ReadF32Scalar` refuses on BOTH count and dtype, no representation for a dynamic per-token activation scheme, and a scheme that is never read from the config at all. The NVFP4 half is the half CLOSEST to working; the FP8 tower is the blocker. **A SECOND artifact of the same model is now in scope and it is a DIFFERENT FORMAT:** `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121`@`36f717a22990e82c54c1d48ee77c491b87825680`, needed by campaign [#1574](https://github.com/mudler/vllm.cpp/issues/1574), declares `quant_method: "modelopt"` and `quant_algo: "MIXED_PRECISION"` with a `quantized_layers` map of 401 EXACT module names, an EMPTY `ignore`, per-TENSOR STATIC FP8 (`weight_scale` and `input_scale` both `F32 []`, 208 modules) and ModelOpt-spelled W4A16_NVFP4 g16 weight-only (`weight` U8 + `weight_scale` F8_E4M3 + `weight_scale_2` `F32 []`, 193 modules including `lm_head`) over ALL 64 layers' MLP -- no layer-56 boundary. Header-verified 2026-08-21 over all four shards (970/976/40/15 = 2001 names; `8 + header_len + max(data_offsets[1])` == the size the hub reports for each; the four sizes exceed the index's `metadata.total_size` 21,921,427,300 by exactly the four headers plus their 8-byte prefixes). **NONE of W4's four blockers applies to it** -- they were properties of the unsloth artifact, not of the format -- so both halves LOAD. The fifth blocker W5 found instead: `ct::Config` stops at `quant_method != "compressed-tensors"`, so nothing in this tree read this config at all and every routing decision fell to the per-projection tensor-NAME probe, which is wrong in BOTH directions silently. `nvidia/Qwen3.6-27B-NVFP4`@`0893e160`, the #466 gate model, is the same ModelOpt shape (2194 names, same 208/193 split, `exclude_modules` `["mtp*","mtp.layers.0*"]`, a `kv_cache_scheme` with ZERO scales shipped) and reaches the same call site, which is why W5 refuses a DISAGREEMENT rather than routing by the declaration ([#1597](https://github.com/mudler/vllm.cpp/issues/1597) owes that) | vLLM `555967922` is the MIRROR and the primary oracle — it runs this format, so nothing here may diverge from its compressed-tensors semantics. A generic mixed-precision resolver already exists at [modelopt_mixed_precision.h](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h). Through W4 **no production file included it** -- the only two includes in the tree were its own two tests, and `nemotron_h_weights.cpp` (which this row previously named) includes `nemotron_h.h`, `nemotron_h_loader.h`, `nvfp4_dequant.h` and `vt/unaligned.h` and reads its quant config inline; a 33,575-byte header reachable only from tests is an `AGENTS.md` §"Nothing lands dead" item. W4 did NOT adopt it and argued the exception: the two headers resolve two DIFFERENT upstream formats that share only the English word "mixed", and its artifact is the compressed-tensors one. **W5 is the FIRST production wiring**, for the artifact that really is ModelOpt: [qwen3_5_dense_weights.cpp](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp) now includes it and `LoadQwen3_5Dense` calls [RefusalForQuantizationConfig](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h) once per checkpoint. So the tree ends with ONE resolver per format, which is upstream's own structure | the failing read [qwen3_5_weights.cpp:642](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L642) (`:457` was WRONG and is `OwnedBytes::Borrow`; re-derived by W4), the resolution [compressed_tensors_config.h:354](../src/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_config.h#L354) (`Resolve`, ignore-first then first-matching target) and its production call site [qwen3_5_dense_weights.cpp:875](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L875), its refusal [dense_weight_loaders.h:164](../include/vllm/model_executor/models/dense_weight_loaders.h#L164) (the count check at `:168`, the dtype check at `:172`; #1258 moved `ReadF32Scalar` down 63 lines and `:101` is now an unrelated `try`/`catch` probe), the scalar-only `Fp8Weight` [qwen3_5_weights.h:652](../include/vllm/model_executor/models/qwen3_5_weights.h#L652), the static activation quant [qwen3_5.cpp:3629](../src/vllm/model_executor/models/qwen3_5.cpp#L3629), and the stale predicate [qwen3_5_dense_weights.cpp:699](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L699) (the `.linear_attn.in_proj_` early-false at `:702`) that declares the GDN input projections never quantized — true for the 3.6 unsloth artifact ([hf_snapshot.h:287](../tests/parity/hf_snapshot.h#L287)) and false for this one | W4: [test_qwen38_27b_nvfp4_arm.cpp:208](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L208) (per-scheme composition), [:582](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L582) (the FP8 refusal through `LoadQwen3_5Dense`), [:666](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L666) (the env-gated live re-read); 8 cases / 190 assertions hermetic and 204 with `VLLM_CPP_QWEN38_27B_NVFP4_DIR` set, over the committed [qwen38_27b_nvfp4_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_manifest.inc) (1953) and [qwen38_27b_nvfp4_mtp_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_mtp_manifest.inc) (15), summing to the index's 1968; per-scheme composition 466/672/475/323/32 tensors over 233/168/317/267/16 modules, zero unclassified. Manifest-capture precedent [minimax_h3_nvfp4_manifest.inc](../tests/vllm/models/minimax_h3_nvfp4_manifest.inc), captured the same way this row's numbers were: an HTTP range read of the file's own header. W5: [test_qwen38_27b_modelopt_mtp_arm.cpp](../tests/vllm/models/test_qwen38_27b_modelopt_mtp_arm.cpp), 22 cases / 1687 assertions hermetic, over four committed header-only manifests ([s1](../tests/vllm/models/qwen38_27b_modelopt_mtp_s1_manifest.inc) 970, [s2](../tests/vllm/models/qwen38_27b_modelopt_mtp_s2_manifest.inc) 976, [s3](../tests/vllm/models/qwen38_27b_modelopt_mtp_s3_manifest.inc) 40, [s4](../tests/vllm/models/qwen38_27b_modelopt_mtp_s4_manifest.inc) 15) summing to the index's 2001; per-scheme composition 720/579/702 tensors over 256/193/536 modules with zero KV scales and zero unclassified, and the 256 split as 208 `kDirect` Linears plus 48 `kPrefix` `linear_attn` CONTAINERS so the count cannot be right for the wrong reason; 937 weight-bearing modules split 208/193/536. RED before wiring was 5 cases failing with an EMPTY refusal -- the loader accepted every config/tensor disagreement and both unloadable algorithms. Twelve negative mutations, every one detected, including the deleted production call site (8 cases red), a suffix list without `.weight_scale_2` (7), one manifest row removed with its count literal left behind (5, and all 22 cases still RUN -- deriving the row count with `std::size` turned what was an out-of-bounds read into an ordinary red), the NVFP4 refusal branch disabled (2) and an unseen operand family skipped rather than refused (2). The fresh review found the last two: the NVFP4 branch is the whole cross-check for 193 of the 401 declared modules and `if (false && ...)` on it left the suite fully green, and `Refusal` skipped a tensor name whose family `SplitOperand` has never seen, which its own contract forbids reading as unquantized. A separate case pins that the refusal is SILENT on the `nvidia/Qwen3.6-27B-NVFP4` shape -- wildcard `exclude_modules`, an `input_scale` on every NVFP4 module, and a `kv_cache_scheme` with zero scales -- and its own mutation reds it while every other case stays green. W4's gate is untouched and re-ran 9 cases / 194 assertions green on the same tree. The token gate is PENDING on [#1632](https://github.com/mudler/vllm.cpp/issues/1632): the pinned oracle DOES run a model inside an `rc` lease (2026-08-18), at `max_num_batched_tokens` 512 against a recorded denominator of 8192, and this artifact's ~20.4 GiB are not staged where a lease can read them. It supersedes #1185, closed 2026-08-18 as local-only | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | +| `QUANT-QWEN38-27B-NVFP4-ARM` | The `unsloth/Qwen3.8-27B-NVFP4` artifact, which is **not what its name says**. **Its pinned revision is GONE:** `a767244d27bd76589a3e3b2ab4e64032c4ebc7af`, the revision [#821](https://github.com/mudler/vllm.cpp/issues/821) names, answers HTTP 404 and `git ls-remote` reports one ref, `refs/heads/main` = `7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108` — the second in-place re-quantization this publisher has done in this family, after `unsloth/Qwen3.6-27B-NVFP4`. So the user-reported load failure on #821 is CORROBORATED at a different revision, never reproduced. At the live revision (header-verified 2026-08-18, 1953 + 15 tensors, `8 + header_len + max(data_offsets[1])` == file size 22,568,192,096) `quantization_config.format` is `mixed-precision`: `group_0` is FP8 W8A8 with **per-CHANNEL** weight scales and **DYNAMIC per-token** activations over `self_attn.(q\|k\|v\|o)_proj`, `linear_attn.(in_proj_qkv\|in_proj_z\|out_proj)`, `lm_head` and `layers.(56..63).mlp.*`; `group_1` is `nvfp4-pack-quantized` W4A4 over the remaining `mlp.*`; plus an 8-bit static `kv_cache_scheme` and an `ignore` list of **303 entries** -- not just the vision tower: 48 x `linear_attn`, `linear_attn.norm`, `linear_attn.in_proj_b` and `linear_attn.in_proj_a` (the GDN layer count), 27 x 4 vision blocks, 2 mergers, and `re:^mtp.*`. That list is what makes the predicate claim provable rather than asserted: `in_proj_a`/`in_proj_b` are IGNORED while `in_proj_qkv`/`in_proj_z`/`out_proj` are `group_0` TARGETS, so a resolver that reads the groups but not the `ignore` list gets the GDN block wrong in both directions. **`*.input_scale` appears ZERO times in the checkpoint.** Four independent blockers, each anchored in the spec: the unconditional `.input_scale` read, a per-channel BF16 `weight_scale` that `ReadF32Scalar` refuses on BOTH count and dtype, no representation for a dynamic per-token activation scheme, and a scheme that is never read from the config at all. The NVFP4 half is the half CLOSEST to working; the FP8 tower is the blocker. **A SECOND artifact of the same model is now in scope and it is a DIFFERENT FORMAT:** `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121`@`36f717a22990e82c54c1d48ee77c491b87825680`, needed by campaign [#1574](https://github.com/mudler/vllm.cpp/issues/1574), declares `quant_method: "modelopt"` and `quant_algo: "MIXED_PRECISION"` with a `quantized_layers` map of 401 EXACT module names, an EMPTY `ignore`, per-TENSOR STATIC FP8 (`weight_scale` and `input_scale` both `F32 []`, 208 modules) and ModelOpt-spelled W4A16_NVFP4 g16 weight-only (`weight` U8 + `weight_scale` F8_E4M3 + `weight_scale_2` `F32 []`, 193 modules including `lm_head`) over ALL 64 layers' MLP -- no layer-56 boundary. Header-verified 2026-08-21 over all four shards (970/976/40/15 = 2001 names; `8 + header_len + max(data_offsets[1])` == the size the hub reports for each; the four sizes exceed the index's `metadata.total_size` 21,921,427,300 by exactly the four headers plus their 8-byte prefixes). **NONE of W4's four blockers applies to it** -- they were properties of the unsloth artifact, not of the format -- so both halves LOAD. The fifth blocker W5 found instead: `ct::Config` stops at `quant_method != "compressed-tensors"`, so nothing in this tree read this config at all and every routing decision fell to the per-projection tensor-NAME probe, which is wrong in BOTH directions silently. `nvidia/Qwen3.6-27B-NVFP4`@`0893e160`, the #466 gate model, is the same ModelOpt shape (2194 names, same 208/193 split, `exclude_modules` `["mtp*","mtp.layers.0*"]`, a `kv_cache_scheme` with ZERO scales shipped) and reaches the same call site, which is why W5 refuses a DISAGREEMENT rather than routing by the declaration ([#1597](https://github.com/mudler/vllm.cpp/issues/1597) owes that) | vLLM `555967922` is the MIRROR and the primary oracle — it runs this format, so nothing here may diverge from its compressed-tensors semantics. A generic mixed-precision resolver already exists at [modelopt_mixed_precision.h](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h). Through W4 **no production file included it** -- the only two includes in the tree were its own two tests, and `nemotron_h_weights.cpp` (which this row previously named) includes `nemotron_h.h`, `nemotron_h_loader.h`, `nvfp4_dequant.h` and `vt/unaligned.h` and reads its quant config inline; a 33,575-byte header reachable only from tests is an `AGENTS.md` §"Nothing lands dead" item. W4 did NOT adopt it and argued the exception: the two headers resolve two DIFFERENT upstream formats that share only the English word "mixed", and its artifact is the compressed-tensors one. **W5 is the FIRST production wiring**, for the artifact that really is ModelOpt: [qwen3_5_dense_weights.cpp](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp) now includes it and `LoadQwen3_5Dense` calls [RefusalForQuantizationConfig](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h) once per checkpoint. So the tree ends with ONE resolver per format, which is upstream's own structure | the failing read [qwen3_5_weights.cpp:642](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L642) (`:457` was WRONG and is `OwnedBytes::Borrow`; re-derived by W4), the resolution [compressed_tensors_config.h:354](../src/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_config.h#L354) (`Resolve`, ignore-first then first-matching target) and its production call site [qwen3_5_dense_weights.cpp:875](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L875), its refusal [dense_weight_loaders.h:164](../include/vllm/model_executor/models/dense_weight_loaders.h#L164) (the count check at `:168`, the dtype check at `:172`; #1258 moved `ReadF32Scalar` down 63 lines and `:101` is now an unrelated `try`/`catch` probe), the scalar-only `Fp8Weight` [qwen3_5_weights.h:719](../include/vllm/model_executor/models/qwen3_5_weights.h#L719), the static activation quant [qwen3_5.cpp:3629](../src/vllm/model_executor/models/qwen3_5.cpp#L3629), and the stale predicate [qwen3_5_dense_weights.cpp:699](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L699) (the `.linear_attn.in_proj_` early-false at `:702`) that declares the GDN input projections never quantized — true for the 3.6 unsloth artifact ([hf_snapshot.h:287](../tests/parity/hf_snapshot.h#L287)) and false for this one | W4: [test_qwen38_27b_nvfp4_arm.cpp:208](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L208) (per-scheme composition), [:582](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L582) (the FP8 refusal through `LoadQwen3_5Dense`), [:666](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L666) (the env-gated live re-read); 8 cases / 190 assertions hermetic and 204 with `VLLM_CPP_QWEN38_27B_NVFP4_DIR` set, over the committed [qwen38_27b_nvfp4_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_manifest.inc) (1953) and [qwen38_27b_nvfp4_mtp_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_mtp_manifest.inc) (15), summing to the index's 1968; per-scheme composition 466/672/475/323/32 tensors over 233/168/317/267/16 modules, zero unclassified. Manifest-capture precedent [minimax_h3_nvfp4_manifest.inc](../tests/vllm/models/minimax_h3_nvfp4_manifest.inc), captured the same way this row's numbers were: an HTTP range read of the file's own header. W5: [test_qwen38_27b_modelopt_mtp_arm.cpp](../tests/vllm/models/test_qwen38_27b_modelopt_mtp_arm.cpp), 22 cases / 1687 assertions hermetic, over four committed header-only manifests ([s1](../tests/vllm/models/qwen38_27b_modelopt_mtp_s1_manifest.inc) 970, [s2](../tests/vllm/models/qwen38_27b_modelopt_mtp_s2_manifest.inc) 976, [s3](../tests/vllm/models/qwen38_27b_modelopt_mtp_s3_manifest.inc) 40, [s4](../tests/vllm/models/qwen38_27b_modelopt_mtp_s4_manifest.inc) 15) summing to the index's 2001; per-scheme composition 720/579/702 tensors over 256/193/536 modules with zero KV scales and zero unclassified, and the 256 split as 208 `kDirect` Linears plus 48 `kPrefix` `linear_attn` CONTAINERS so the count cannot be right for the wrong reason; 937 weight-bearing modules split 208/193/536. RED before wiring was 5 cases failing with an EMPTY refusal -- the loader accepted every config/tensor disagreement and both unloadable algorithms. Twelve negative mutations, every one detected, including the deleted production call site (8 cases red), a suffix list without `.weight_scale_2` (7), one manifest row removed with its count literal left behind (5, and all 22 cases still RUN -- deriving the row count with `std::size` turned what was an out-of-bounds read into an ordinary red), the NVFP4 refusal branch disabled (2) and an unseen operand family skipped rather than refused (2). The fresh review found the last two: the NVFP4 branch is the whole cross-check for 193 of the 401 declared modules and `if (false && ...)` on it left the suite fully green, and `Refusal` skipped a tensor name whose family `SplitOperand` has never seen, which its own contract forbids reading as unquantized. A separate case pins that the refusal is SILENT on the `nvidia/Qwen3.6-27B-NVFP4` shape -- wildcard `exclude_modules`, an `input_scale` on every NVFP4 module, and a `kv_cache_scheme` with zero scales -- and its own mutation reds it while every other case stays green. W4's gate is untouched and re-ran 9 cases / 194 assertions green on the same tree. The token gate is PENDING on [#1632](https://github.com/mudler/vllm.cpp/issues/1632): the pinned oracle DOES run a model inside an `rc` lease (2026-08-18), at `max_num_batched_tokens` 512 against a recorded denominator of 8192, and this artifact's ~20.4 GiB are not staged where a lease can read them. It supersedes #1185, closed 2026-08-18 as local-only | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | | `QUANT-GGUF-PRESETS` | Representative mixed-file gates for every llama.cpp output preset family | llama.cpp `tools/quantize/quantize.cpp:34-74` | only custom APEX mixed files are executable; no general preset dispatch | [APEX gates](../tests/parity/test_qwen36_gguf_engine.cpp#L143) do not prove llama.cpp preset breadth | [coverage spike](specs/quantization-coverage.md); split exact preset IDs before `READY` | `INVENTORIED` | - | ## 1. llama.cpp / GGUF encodings diff --git a/.agents/specs/rocm-host-residency-after-upload.md b/.agents/specs/rocm-host-residency-after-upload.md index c70e3fda5..3ebabe89a 100644 --- a/.agents/specs/rocm-host-residency-after-upload.md +++ b/.agents/specs/rocm-host-residency-after-upload.md @@ -131,7 +131,7 @@ of that variable and the new helper only decides the DEFAULT. ### Out of scope Chunked H2D through a pinned bounce buffer — llama.cpp's 4 x 64 MiB shape — is -NOT built here. See `## Owed`. +NOT built here. See `## Deferred, with an issue`. ### Also in scope @@ -199,9 +199,9 @@ red for each. legitimate result and is reported with `wchan` evidence rather than papered over. -## Owed +## Deferred, with an issue -- `ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW` — chunked H2D through a pinned bounce +- `ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW` (row-owned, not started) covers chunked H2D through a pinned bounce buffer, llama.cpp's 4 x 64 MiB shape. Needed only if the stall survives fixes 1 and 2. Medium-size and touches every staged weight on every backend, so it gets its own row, spec and measurement. From 87d96a8820082a387291e0ca4761345741c9df9d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 23:45:06 +0000 Subject: [PATCH 04/10] test(MODEL-MM-QWEN4-EXP): measure the released pages as RssFile, not as total RSS The residency case watched VmRSS, which cannot see this release. The staging branch allocates a device buffer the same size as the weight, and on the fake backend that is a malloc, so total RSS ends roughly where it started: the source pages went and an equal-sized anonymous copy arrived. The case would have been subtracting two large numbers and reading the remainder as zero. RssFile is the half a GGUF mapping contributes and the half the KFD accounting walks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../test_resident_weight_host_addressable.cpp | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp b/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp index 109d9f663..7e79d6ebe 100644 --- a/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp +++ b/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp @@ -775,17 +775,25 @@ TEST_CASE("stage-vs-retag: a model larger than the whole box REFUSES without wra #if defined(__linux__) namespace { -// `VmRSS` in KiB, the figure the KFD's resident-system-memory accounting is -// about. Returns 0 when it cannot be read, which a case treats as "cannot -// measure" rather than as a pass. -size_t VmRssKib() { +// `RssFile` in KiB: the FILE-backed half of this process's resident set, which +// is exactly what a GGUF mapping contributes and what the KFD's +// resident-system-memory accounting walks. +// +// `VmRSS` WOULD NOT WORK HERE, and the reason is worth stating because the +// wrong one reads as the obviously correct one. The staging branch allocates and +// fills a device buffer of the SAME size as the weight, which on this fake +// backend is a `malloc` and is ANONYMOUS residency. Total RSS therefore ends +// roughly where it started: the source went and an equal-sized copy arrived, and +// a case watching `VmRSS` would measure the difference of two large numbers and +// call it zero. Splitting the two makes the assertion say what it means. +size_t RssFileKib() { std::FILE* f = std::fopen("/proc/self/status", "r"); if (f == nullptr) return 0; char line[256]; size_t kib = 0; while (std::fgets(line, sizeof(line), f) != nullptr) { - if (std::strncmp(line, "VmRSS:", 6) == 0) { - kib = static_cast(std::strtoull(line + 6, nullptr, 10)); + if (std::strncmp(line, "RssFile:", 8) == 0) { + kib = static_cast(std::strtoull(line + 8, nullptr, 10)); break; } } @@ -872,7 +880,7 @@ TEST_CASE("a STAGED borrow's source pages are released, and the RSS says so") { REQUIRE(f.ok()); f.Prefault(); - const size_t rss_resident = VmRssKib(); + const size_t rss_resident = RssFileKib(); REQUIRE(rss_resident > 0); // unreadable /proc is "cannot measure", not a pass const OwnedTensor w = BorrowWeight(f, kBigVocab, kBigHidden); @@ -896,7 +904,7 @@ TEST_CASE("a STAGED borrow's source pages are released, and the RSS says so") { // span rather than all of it, because the staging copy itself allocated 64 MiB // of device (here: malloc'd) memory that is also resident and is supposed to // stay: what is asserted is that the SOURCE went, against that background. - const size_t rss_after = VmRssKib(); + const size_t rss_after = RssFileKib(); CHECK(rss_resident > rss_after); CHECK(rss_resident - rss_after >= (f.size() / 2) / 1024); From 8bccb0286099577b1774acfc27b1457aa0cf452c Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 12 Sep 2026 23:57:42 +0000 Subject: [PATCH 05/10] test(MODEL-MM-QWEN4-EXP): cover the platform guard a surviving mutation found Deleting the host-addressable guard from MaybeReleaseStagedBorrowSource left all six new cases green. The obvious host-addressable case never reaches the helper: ResidentWeight aliases an aligned borrow and returns before the staging arm, so the guard looked tested and was not. A MISALIGNED borrow declines the alias and falls through to staging on a platform whose kernels can read host storage, which is the path the guard is load-bearing on. The new case takes it. The anonymous-borrow case grows from 96 bytes to 1 MiB in the same change. DropResidentInteriorPages madvises whole pages only, so a 96-byte buffer had no interior page and its byte assertion held whether the mmap_fd discriminator existed or not. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../test_resident_weight_host_addressable.cpp | 66 +++++++++++++++++-- 1 file changed, 62 insertions(+), 4 deletions(-) diff --git a/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp b/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp index 7e79d6ebe..8ce612c97 100644 --- a/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp +++ b/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp @@ -971,7 +971,15 @@ TEST_CASE("an ANONYMOUS borrow is never released: MADV_DONTNEED would ZERO it") // those pages. Clear the field and this case is what goes red -- with the // wrong bytes, not with a count. const PlatformArm arm(false); - const size_t nb = static_cast(kN * kK) * 2; + // BIG ENOUGH TO HAVE INTERIOR PAGES. `DropResidentInteriorPages` madvises + // whole pages only, so a 96-byte buffer has nothing to drop and the byte + // assertion below would hold whether the discriminator existed or not -- a + // tautology wearing the shape of a guarantee. 1 MiB is also over glibc's mmap + // threshold, so the block is its own page-aligned anonymous mapping, which is + // precisely the memory MADV_DONTNEED zeroes. + constexpr int64_t kAnonRows = 512; + constexpr int64_t kAnonCols = 1024; + const size_t nb = static_cast(kAnonRows * kAnonCols) * 2; auto* block = new uint8_t[nb]; for (size_t i = 0; i < nb; ++i) block[i] = static_cast(i & 0xFF); const std::vector expect(block, block + nb); @@ -982,8 +990,8 @@ TEST_CASE("an ANONYMOUS borrow is never released: MADV_DONTNEED would ZERO it") OwnedTensor w; w.dtype = DType::kBF16; w.rank = 2; - w.shape[0] = kN; - w.shape[1] = kK; + w.shape[0] = kAnonRows; + w.shape[1] = kAnonCols; w.nk = false; w.bytes = vllm::OwnedBytes::Borrow(block, nb, std::move(keep)); // mmap_fd deliberately LEFT AT -1: this is anonymous memory. @@ -991,10 +999,60 @@ TEST_CASE("an ANONYMOUS borrow is never released: MADV_DONTNEED would ZERO it") const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); Queue q = XpuQueue(); - (void)vllm::Qwen3_5EmbeddingTable(Fake(), q, w, kN, kK); + (void)vllm::Qwen3_5EmbeddingTable(Fake(), q, w, kAnonRows, kAnonCols); CHECK(vllm::BorrowReleaseSnapshot().calls == before.calls); CHECK(std::memcmp(w.bytes.data(), expect.data(), nb) == 0); } + +TEST_CASE("a host-addressable device that STAGES anyway still releases nothing") { + // THE CASE A SURVIVING MUTATION ASKED FOR, and it is worth saying which one. + // Deleting `if (backend.DeviceMemoryIsHostAddressable()) return false;` from + // `MaybeReleaseStagedBorrowSource` left every other case in this file green. + // The reason is that the obvious host-addressable case never reaches the + // helper at all: `ResidentWeight` ALIASES an aligned borrow and returns before + // the staging arm. So the guard looked tested and was not. + // + // It is load-bearing on the path this case takes. A MISALIGNED borrow declines + // the alias (`kDeclinedBorrow`) and falls through to staging on a platform + // whose kernels CAN read host storage. Releasing there would drop pages the + // next kernel reads directly, for no gain: the device copy is not the only + // copy that matters when both are the same RAM. Without the guard this case + // counts a release; with it, none. + const PlatformArm arm(true); // kernels CAN dereference host storage + MappedFile f(1u << 20); + REQUIRE(f.ok()); + f.Prefault(); + // Offset into the mapping so the borrow is NOT 256-byte aligned. `mmap` always + // returns a page boundary, so an unshifted borrow would be aliased in place and + // this case would measure the same nothing the aligned one does. + const size_t skew = 16; + REQUIRE(reinterpret_cast(f.data() + skew) % vllm::kDeviceAliasAlignment != 0); + const int64_t vocab = 64; + const int64_t hidden = ((1 << 20) - 4096) / (64 * 2); + const size_t nb = static_cast(vocab * hidden) * 2; + OwnedTensor w; + w.dtype = DType::kBF16; + w.rank = 2; + w.shape[0] = vocab; + w.shape[1] = hidden; + w.nk = false; + std::shared_ptr keep(static_cast(f.data()), + [](const void*) {}); + w.bytes = vllm::OwnedBytes::Borrow(f.data() + skew, nb, std::move(keep)); + w.mmap_fd = f.fd(); // file-backed, so ONLY the platform guard can refuse it + + const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); + Queue q = XpuQueue(); + const Tensor t = vllm::Qwen3_5EmbeddingTable(Fake(), q, w, vocab, hidden); + + // It really did stage: a declined alias is what puts this weight on the arm + // the guard sits on. Without this the case could pass by never getting there. + REQUIRE(w.d_dev != nullptr); + REQUIRE(t.data == w.d_dev.get()); + CHECK(vllm::BorrowReleaseSnapshot().calls == before.calls); + // ...and the host bytes the kernels may still follow are intact and unchanged. + CHECK(std::memcmp(w.bytes.data(), t.data, nb) == 0); +} #endif // __linux__ From 60b05ccabdae6a16c46592cb9bb4dfe3569d4ab5 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 13 Sep 2026 01:04:28 +0000 Subject: [PATCH 06/10] fix(MODEL-MM-QWEN4-EXP): guard the release on the predicate that SELECTS the arm The release asked `vt::Backend::DeviceMemoryIsHostAddressable()` while `ResidentWeight` selects its alias arm on the platform's `host_memory_is_device_addressable()`. Those answer different questions and can disagree: the platform one is "may a kernel follow a host pointer" (#125, #1299), the backend one is "is a DEVICE allocation host-dereferenceable", which GB10 answers false while being physically unified. A weight reaches the staging arm on a platform that answers YES whenever a misaligned borrow declines the alias, and the release then dropped pages the kernels may still read directly. The new guard case found it: one release counted where none was allowed. The caller now passes its already-computed platform answer in and the helper refuses on either, so there is one predicate computed once. That is the move #2406 made for QuantRepackForDevice, and for the same reason: a second spelling is how a refusal and its route predicate come to disagree about one weight. gfx1151 answers false to both since #2511, so the ROCm behaviour this row measures is unchanged. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../model_executor/models/qwen3_5_weights.h | 23 +++++++++++++++---- src/vllm/model_executor/models/qwen3_5.cpp | 5 +++- .../model_executor/models/qwen3_5_weights.cpp | 8 +++++-- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/include/vllm/model_executor/models/qwen3_5_weights.h b/include/vllm/model_executor/models/qwen3_5_weights.h index 50686ff33..a5211a69b 100644 --- a/include/vllm/model_executor/models/qwen3_5_weights.h +++ b/include/vllm/model_executor/models/qwen3_5_weights.h @@ -250,9 +250,21 @@ void AdoptDeviceBytesAsHost(vt::Backend& backend, const OwnedTensor& w); // // THREE PRECONDITIONS, ALL CHECKED HERE. // -// 1. The device cannot dereference host storage -// (`vt::Backend::DeviceMemoryIsHostAddressable()` false). Where it can, the -// bytes ARE the weight and `AdoptDeviceBytesAsHost` handles it instead. +// 1. The device cannot dereference host storage. TWO predicates answer that, +// they are not the same question, and asking only one is a defect this +// file's own gate caught. `vllm::platforms::Platform:: +// host_memory_is_device_addressable()` is what `ResidentWeight` selects the +// ALIAS arm on (issues #125, #1299): may a kernel follow a host pointer. +// `vt::Backend::DeviceMemoryIsHostAddressable()` is the converse: is a +// DEVICE allocation host-dereferenceable (GB10 answers false while being +// physically unified). A weight can reach the staging arm on a platform +// whose answer is YES -- a misaligned borrow declines the alias and falls +// through -- and releasing its pages there would drop memory the kernels +// may still read directly. So the CALLER passes its already-computed +// platform answer in as `host_addressable` and the helper refuses on +// either. One predicate, computed once, exactly as #2406 made +// `QuantRepackForDevice` take its `dev`: a second spelling is how a +// refusal and its route predicate come to disagree about one weight. // 2. `bytes` is BORROWED. An owned buffer is `ReleaseHost`'s business. // 3. `mmap_fd >= 0`. This is the discriminator that makes the call SAFE, and it // is not a convenience. `MADV_DONTNEED` on a file-backed private mapping @@ -277,9 +289,10 @@ void AdoptDeviceBytesAsHost(vt::Backend& backend, const OwnedTensor& w); // entirely when the preconditions do not hold, so a backend this does not apply // to pays nothing. // -// Returns true when pages were released. +// `host_addressable` is the caller's `host_memory_is_device_addressable()` +// answer for the queue's device. Returns true when pages were released. bool MaybeReleaseStagedBorrowSource(vt::Backend& backend, vt::Queue& queue, - const OwnedTensor& w); + const OwnedTensor& w, bool host_addressable); // What `MaybeReleaseStagedBorrowSource` has done in this process. `calls` counts // the releases that HAPPENED, not the invocations that declined, for the same diff --git a/src/vllm/model_executor/models/qwen3_5.cpp b/src/vllm/model_executor/models/qwen3_5.cpp index 257f5a352..10bab8420 100644 --- a/src/vllm/model_executor/models/qwen3_5.cpp +++ b/src/vllm/model_executor/models/qwen3_5.cpp @@ -1316,7 +1316,10 @@ Tensor ResidentWeight(Dev d, const OwnedTensor& w, std::vector shape = // re-tested its condition each time would madvise away the pages the GPU is // about to read, every step. The helper states its other two preconditions // and synchronizes the queue before it touches anything. - vllm::MaybeReleaseStagedBorrowSource(d.b, d.q, w); + vllm::MaybeReleaseStagedBorrowSource( + d.b, d.q, w, + vllm::platforms::GetPlatform(d.q.device.type) + .host_memory_is_device_addressable()); // Same adoption as the dense block's ResidentWeight: on a host-addressable // device the uploaded buffer IS the host buffer, so keeping the mirror // costs a second full copy of the model out of the same unified RAM. diff --git a/src/vllm/model_executor/models/qwen3_5_weights.cpp b/src/vllm/model_executor/models/qwen3_5_weights.cpp index 29b4f7c75..54a4585ff 100644 --- a/src/vllm/model_executor/models/qwen3_5_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_5_weights.cpp @@ -391,11 +391,15 @@ BorrowReleaseStats BorrowReleaseSnapshot() { } bool MaybeReleaseStagedBorrowSource(vt::Backend& backend, vt::Queue& queue, - const OwnedTensor& w) { + const OwnedTensor& w, bool host_addressable) { // See the header for each of the three. The ORDER matters only in that the // cheap, allocation-free tests come before the synchronize: a backend this // does not apply to must not pay a stream sync per weight to find that out. - if (backend.DeviceMemoryIsHostAddressable()) return false; + // + // BOTH host-addressability predicates, because they answer different + // questions and a weight can reach this arm with them disagreeing. The + // header says which is which. + if (host_addressable || backend.DeviceMemoryIsHostAddressable()) return false; if (w.bytes.empty() || !w.bytes.borrowed()) return false; if (w.mmap_fd < 0) return false; // THE STAGING COPY IS ASYNCHRONOUS. `RocmBackend::Copy` is `hipMemcpyAsync` on From 8a2a2d4588e89f60b89d2dcce2374e86c7ae0738 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 13 Sep 2026 02:24:14 +0000 Subject: [PATCH 07/10] record(MODEL-MM-QWEN4-EXP): re-point the Fp8Weight anchor after the header grew again The predicate fix added lines to qwen3_5_weights.h, so the symbol QUANT-QWEN38-27B-NVFP4-ARM cites moved from 719 to 732. Both halves of the markdown link carry the number and both are updated; repairing only the visible label is what left the checker red the first time. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .agents/quantization-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md index a35886ce0..6dc227a34 100644 --- a/.agents/quantization-matrix.md +++ b/.agents/quantization-matrix.md @@ -35,7 +35,7 @@ otherwise it remains `PARTIAL` or `INVENTORIED` even if parsing works. | `QUANT-GGUF-CIQ-GEMM` | Compute-in-quant GEMM: activation quant (Q8_0/Q8_K) + per-type vec_dot dispatch for Q8_0/Q4_K/Q5_K/Q6_K/Q3_K/Q4_0; portable C++ tier, then x86/Arm SIMD + repack tiers. **G1-G4 landed** — the portable tier-0 path is complete, gated at the OP level, and **ROUTED end to end**: `vt::MatmulBT` dispatches a block-dtype weight to `kMatmulBTQuant`, keep-quant is the production DEFAULT wherever that op is registered, and the six routed encodings compute in quant with **no token movement**. **G6 (2026-07-23)** added the Arm **i8mm mmla `nrc==2` tier** for q8_0/q4_0/q4_K/q6_K (q3_K/q5_K have no upstream mmla → stay portable), 2x2-tiled into `kMatmulBTQuant` at even M,N: op-level q4_K **7–8.4×** / q6_K **3.8–4.5×** / q8_0 ~1.2× over portable, e2e prefill +8.4 % on the q8_0-dominant bench file (1.44× behind llama.cpp), tokens byte-identical. **G7 (2026-07-23)** added q8_0 **repack-at-load** (the `q8_0_4x8` tier `ggml_repack_get_optimal_repack_type` picks on NEON+i8mm): the loader repacks each q8_0 weight once into the `block_q8_0x4` interleave and `kMatmulBTQuant` dispatches a pre-shuffled i8mm gemm/gemv with no per-block register shuffles — op-level q8_0 **3.7–5.9×** over the mmla tier, **E2E prefill 1.92× same-binary → 223.8 t/s vs llama.cpp pp128 177.3 = at/beyond parity** (was ~1.5× behind), decode at parity, tokens byte-identical. **CPU prefill parity reached; the prefill-lever search is closed** (remaining gap = peak RSS 1.39×, loader-bound). G5 (x86) + G8 open. **The FRESH op-dispatch profile this row owed is DONE (2026-08-06, dgx aarch64, `main` @`dfd29060`, same bench file; see `.agents/benchmark-record.md` 'FRESH op-dispatch profile'), and it does NOT support starting G5 next:** `QuantRepackMatmul` is 5.06 % of prefill and 15.99 % of decode on aarch64 where the i8mm tier already landed. The profile re-ranks the CPU levers to (1) threadpool synchronisation at 47 % of decode (`ThreadReady`+`PollForWork`+`Barrier`; M=1 cannot amortise the barrier) and (2) CPU paged attention at ~39 % of prefill, of which 20.68 % is a per-ELEMENT dtype switch in the attention dot loop (`cpu_paged_attn.cpp:29` called from `:143`), the same defect class E1 already removed from the elementwise GEMM. G5 stays a real x86 gap worth closing for x86 users, but it is not the top lever, and the x86 box is VOID for timing so it cannot be speed-gated here | llama.cpp `ggml/src/ggml-cpu/ggml-cpu.c:211-406` traits table, `ggml-cpu/quants.c:174-860` generic vec_dot, `arch/{x86,arm}/quants.c`, `ggml-cpu/repack.cpp:4153-4830` at `237ad9b96` | G1: [block dtypes + geometry](../src/vt/dtype.cpp#L32), [quant traits table](../src/vt/cpu/cpu_quant_traits.cpp#L1), [shared block decoders](../src/vt/cpu/cpu_quant_dequant.cpp#L1), [op surface](../include/vt/quant.h#L1). G2: [activation quant + scratch sizing](../src/vt/cpu/cpu_quant_act.cpp#L1) (`quantize_row_q8_0/q8_K`). G3: [the six generic vec_dot](../src/vt/cpu/cpu_quant_dot.cpp#L1), [block-struct mirror](../src/vt/cpu/cpu_quant_blocks.h#L1), [`kMatmulBTQuant` quantized path + composite fallback](../src/vt/cpu/cpu_quant_gemm.cpp#L1). G4: [the routing point](../src/vt/ops.cpp#L158) — `vt::MatmulBT` sends a block-dtype `b` to `MatmulBTQuant` and is otherwise unchanged, which is sufficient because every model matmul helper already routes an `nk=true` weight there ([qwen3_5.cpp:1067](../src/vllm/model_executor/models/qwen3_5.cpp#L1067)); plus [the default flip + `expand_nk`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) and [the untransposed expand path](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L194). G6: [Arm i8mm mmla tier](../src/vt/cpu/cpu_quant_dot_arm.cpp#L1) (q8_0/q4_0/q4_K/q6_K `vmmlaq_s32`, HWCAP2_I8MM-probed, `VT_CPU_QUANT_MMLA` defeat) + [2x2 tile in kMatmulBTQuant](../src/vt/cpu/cpu_quant_gemm.cpp#L85), per-file `+i8mm` in CMakeLists | [G1 traits cross-check + fallback units](../tests/vt/test_ops_quant_traits.cpp#L1) — 8 cases / 5,615 assertions green (was 5,694; its composite case now covers Q8_K alone because the six weight types legitimately no longer take that path): vt geometry vs the reader's `GgmlTraits` vs ggml-common.h arithmetic all agree, and the composite equals the loader dequant byte-for-byte. [G2/G3 units](../tests/vt/test_ops_quant_dot.cpp#L1) — 16 cases / 78,052 assertions green: every `vec_dot` gated against an INDEPENDENT f64 dequantize-then-dot reference (tolerance relative to the dot's L1 magnitude, actual agreement ~1e-6) over nblocks {1,2,3,5,7,16} incl. single-block and odd multiples; ragged K throws at every layer; upstream thresholds ported unwidened (test-quantize-fns:17-28, test-backend-ops:4277 NMSE ≤ 5e-4 at M {1,4,32,512} × N {1,7,16}); bit-exact run-to-run and across threads 1/2/4; byte-exact encoder gate pins the rounding rules; 14-mutant battery, 13 caught, the 1 uncaught mutant provably unreachable. [dequant units](../tests/vllm/test_gguf_dequant.cpp#L25) still green after the decoder move. DGX (G2/G3 re-confirmed, each gate STANDALONE, goldens md5 identical before/after): clean CUDA `-Werror` build 0 warnings + full regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16 on both 0.6B and 4B, OPT 6/6, DeepSeek-V2 8/8) + `test_qwen36_gguf_engine` 28/28 with 16/16 tokens on both APEX files + the new CPU units green on aarch64 with identical counts. **G4 (2026-07-22):** `test_qwen36_gguf_engine` PASSES STANDALONE on a CPU-only dgx build (where keep-quant is live) — 2/2 cases, 16/16 greedy tokens on APEX-Compact AND APEX-Balanced vs the same-file llama.cpp oracle, exercising 5 of the 6 routed encodings end to end; the CUDA regression set is UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, gguf 28/28 incl. `VT_CPU_REF=1`), goldens md5 identical. **Binding CPU A/B** (idle dgx aarch64, one flock, same binary, 3 reps, `Qwen3.5-2B-UD-Q8_K_XL`): decode 2.216 -> 7.650 t/s (**3.45x**), prefill 5.149 -> 21.44 t/s (**4.16x**), peak RSS 7.428 -> 6.401 GiB, output tokens byte-identical across the pre-G4, post-G4 and `VT_CPU_REF=1` arms. Still **3.38x / 8.20x / 2.29x behind llama.cpp** — the projected 9-17x did NOT hold because 60 % of that file's weight bytes are `f16`, which no block encoding covers. **That gap is now CLOSED by `KERNEL-GEMM-CPU-ELEM`** (2026-07-22, same box/recipe/binary discipline): the elementwise kernel went 18-24 -> 69-351 GFLOP/s bit-exactly, taking the CPU position to **decode 1.03x behind (parity within 3.1 %) and prefill 2.34x behind**, tokens unchanged (same md5). Its measured NEGATIVE re-ranks G5-G8 once more: M-blocking the elementwise GEMM bought 1.63x op-level and **0.0 % end-to-end**, so the 95.37 % `kMatmul` attribution these G-rows were ranked against is STALE and a FRESH op-dispatch profile is owed before G5/G6/G7 are started. **G6 (2026-07-23):** [Arm i8mm mmla tier](../src/vt/cpu/cpu_quant_dot_arm.cpp#L1) landed against the refreshed profile (kMatmulBTQuant 50 % + kMatmul 16 % + kMatmulBT 14 % = 80 % of prefill). [test_ops_quant_dot G6 cross-check](../tests/vt/test_ops_quant_dot.cpp#L1) — 19 cases / **78,162** assertions on dgx aarch64: q8_0/q4_0 mmla **BIT-IDENTICAL** to the portable/scalar tier (`vmlaq_f32` non-fused under `-ffp-contract=off`), q4_K/q6_K within NMSE ≤ 5e-4, mmla GEMM bit-identical across threads 1/2/4/20. `test_qwen36_gguf_engine` 2/2 · 16/16 on both APEX files with mmla live (q8_0/q4_K/q6_K at prefill), bench-file token md5 `d235db12f2cd304007530286a1755c95` byte-identical across mmla-OFF/ON/`VT_CPU_REF=1`. Op-level portable→i8mm: q8_0 ~1.2×, q6_K 3.8–4.5×, q4_K 7–8.4×; e2e prefill same-binary 1.084× (1.56×→1.44× behind llama.cpp pp128). CUDA `-Werror` 0-warn, regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138, Qwen3-dense 184, OPT, DeepSeek-V2 223), goldens untouched. **G7 (2026-07-23):** [q8_0 repack transform](../src/vt/cpu/cpu_quant_repack.cpp#L1) + [i8mm repack gemm/gemv](../src/vt/cpu/cpu_quant_repack_arm.cpp#L1) dispatched from [`kMatmulBTQuant`](../src/vt/cpu/cpu_quant_gemm.cpp#L151) on `b.repacked`; loader repacks via [`OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) under `GgufLoadPolicy::quant_repack`, flag carried to the kernel through [`ResidentWeight`](../src/vllm/model_executor/models/qwen3_5.cpp#L702). [test_ops_quant_repack](../tests/vt/test_ops_quant_repack.cpp#L1) — 305 assertions on dgx aarch64: repacked gemm/gemv `memcmp`-equal to plain `kMatmulBTQuant` across decode/leftover/prefill, f32+bf16 out, strided activations, threads 1/2/4/20; interleave matches `make_block_q8_0x4` byte-for-byte (110 on x86, numeric skip). `test_qwen36_gguf_engine` STANDALONE 2/2·16/16 on APEX Compact+Balanced (repack live), token md5 `d235db12f2cd304007530286a1755c95` byte-identical across repack-ON/OFF/`VT_CPU_REF=1`. Binding dgx aarch64 (idle, one flock, 6 interleaved reps): op-level q8_0 3.7–5.9× (518→2401/583→3456/514→1902 GFLOP/s); E2E prefill **1.92×** (1096→572 ms), **223.8 t/s vs llama.cpp pp128 177.3 = 1.26× at/beyond parity**, decode at parity, RSS unchanged; fresh profile q8_0 GEMM 55%→~21%, prefill-lever search CLOSED. CUDA `-Werror` 0-warn, regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, Llama 16/16), goldens content-hash identical . **P0 REGRESSION FOUND + FIXED (2026-08-06, `CLAIM-QUANT-GGUF-CIQ-GROUPED-DTYPE`):** the GROUPED provider `MatmulBTQuantGroupedKernel` was f32-ONLY — it advanced a `float*` by `act.stride[0]` and declared the row `kF32` whatever `act.dtype` said, so a bf16/f16 activation was mis-strode 2x AND mis-decoded. Every prior caller/test passed f32; qwen3_5 W3b `KqGrouped` (bf16 act, `b4f5610a`) was the first non-f32 caller, so CPU-only GGUF 35B decode became all-token-0 while the CUDA gate stayed byte-exact (CUDA always honoured `act.dtype`). Fixed at [`cpu_quant_gemm.cpp:220-268`](../src/vt/cpu/cpu_quant_gemm.cpp) (rows addressed by `SizeOf(act.dtype)`/`SizeOf(out.dtype)`; `repacked`/`q8_0_aligned` now propagate onto the per-expert slice — the CIQ-G7 all-zero mode). Gated per activation dtype + bf16-out by 2 NEW cases in [`test_ops_quant_dot.cpp`](../tests/vt/test_ops_quant_dot.cpp) (RED pre-fix on f16+bf16 for all 12 weight encodings, GREEN after; f32 unaffected either way) | [CIQ GEMM leaf](specs/gguf-compute-in-quant-gemm.md) | `ANCHOR-BACKFILL` | `CLAIM-QUANT-GGUF-CIQ-G7-1` | | `QUANT-GGUF-KEEPQ-LOADER` | Keep-quantized GGUF loader: block-resident 2-D matmul weights ([N,K], no transpose), per-tensor routing, `VT_CPU_REF` dequant-oracle switch, bench-branch `7c91a42` merge. **L1+L2+L3 landed** — block residency, the TOTAL per-tensor routing policy and the `VT_CPU_REF` oracle switch all exist and are gated. **Keep-quant is DEFAULT ON since CIQ G4** wherever the running device has a registered `kMatmulBTQuant` (CPU, and since 2026-07-29 also **CUDA** for the Q8_K family via the `KERNEL-QUANT-CIQ-GEMM-CUDA` kCUDA provider — a CUDA runner now keeps k-quant/i-quant blocks COMPRESSED instead of expanding), with `VT_GGUF_KEEP_QUANT=0` as the opt-out. L4 measured; **L5 LANDED** (mmap in-place residency + tied-head sharing + read-once page release) — peak RSS 6.401 -> **3.884 GiB**, 2.29x -> **1.39x** llama.cpp, byte-identical | llama.cpp `src/llama-model-loader.cpp:1047,1385` (file-typed residency), `:1676` + `ggml/src/llama-mmap.cpp:490` (`unmap_fragment`), `ggml/src/ggml-cpu/repack.cpp:4727` (repack-at-load hook) at `237ad9b96` | L1: dense-arch (`qwen35`) GGUF path on main via the registry — [dense GGUF load](../src/vllm/model_executor/models/qwen3_5_dense.cpp#L60), [arch->registered-ID map](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L212), [F16/BF16 row dequant](../src/vllm/model_executor/model_loader/gguf_dequant.cpp#L61). L2: [block residency `OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) — raw ggml blocks into an `OwnedTensor` with a block `vt::DType`, file `[N,K]` orientation, `nk=true`, no transpose; stacked experts split by byte range. L3: [routing policy + `VT_CPU_REF`/`VT_GGUF_KEEP_QUANT`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L1) (6 roles, no `default:` label so an unrouted role is a `-Werror=switch` build failure) wired at every loader call site via [`OwnMatmulWeight`/`RequireExpand`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L185). **Default now device-derived** (CIQ G4): [`GgufQuantComputeAvailable`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) gates it on `vt::OpRegistered(kMatmulBTQuant, CurrentPlatform().device_type())`, and the same condition drives `expand_nk`, which stops transposing a weight that must expand. **`expand_nk` now also covers the GDN split projections** (2026-07-23, `CLAIM-CPU-GDN-ORIENT-1`): a fresh op-dispatch profile found `LoadGdnGguf`'s `in_proj_qkv/z/b/a` + `out_proj` were the ONE expanded weight family still transposed to [K,N] (nk=false → slow `kMatmul`, 17.9 % of prefill); the new [`gdn_expand_nk` field](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) + [`MakeGdnProj`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L131) keep them [N,K] nk=true (V-head reorder applied first, orthogonal to orientation; `VT_GGUF_GDN_NK=0` A/B opt-out) → M-blocked `kMatmulBT`, same-binary prefill **1.090×** / decode 1.09×, byte-identical (`kMatmul` 72→0 calls in prefill) | [L2/L3 units](../tests/vllm/test_gguf_keep_quant.cpp#L1) — 17 cases / 5,574 assertions green. **Gate 1 (losslessness) proven PER ENCODING**, one case each for Q4_0/Q8_0/Q3_K/Q4_K/Q5_K/Q6_K: resident bytes `memcmp`-equal to the file span and resident-block dequant BYTE-IDENTICAL to the direct-from-file expansion (f32 and bf16), over pseudo-random block bytes constrained only to finite f16 scales; at loader level the kept weight rehydrates to the expanded `[K,N]` bf16 tensor byte for byte, per weight and per expert, on dense and MoE fixtures. **Totality**: the audit hook proves `routed == the file's complete tensor list` on both fixtures, plus 6 roles × 12 encodings × 6 shapes against a LONGHAND expectation (12 keep / 420 expand, so neither outcome is vacuous). **Gate 2 (oracle stability)**: `VT_CPU_REF=1` keeps nothing quantized and every weight is bit-identical to the historical load; on dgx [`test_qwen36_gguf_engine`](../tests/parity/test_qwen36_gguf_engine.cpp#L143) under `VT_CPU_REF=1` is 28/28 assertions, 16/16 tokens on both APEX files — same as without. 10-mutant battery, 10 caught (the expert-slice-offset mutant survived the first pass, exposed a real coverage hole, and drove the MoE fixture). DGX (each gate STANDALONE, production flags, goldens md5 identical before/after `2965ef5772b556d3f3f86fedf4221b2f`): clean CUDA `-Werror` 0 warnings + regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16 on both, OPT 6/6, DeepSeek-V2 8/8) + gguf units green on aarch64 with identical counts; full CPU ctest 154/154. **RSS at G4 was 6.401 GiB (2.29x); L5 took it to 3.884 GiB (1.39x)** — binding, idle dgx aarch64, same-binary 3-rep A/B: mmap in-place residency (borrow kept q8_0 blocks out of the mapping, refcounted, -0.998 GiB), tied-head sharing (one bf16 vocab matrix for embed+lm_head, -0.946 GiB), read-once page release (MADV_DONTNEED the expanded tensors' file pages, port of llama.cpp `unmap_fragment`, -0.573 GiB). Decode TPOT 41.7 ms UNCHANGED, prefill TTFT +4% (first-touch faults move into the timed window), output md5 `d235db12f2cd304007530286a1755c95` identical across BEFORE/AFTER/ORACLE. Lifetime safety tested explicitly (borrow outlives the GgufFile AND the on-disk file; shared head freed once either order). **L6 (2026-07-23, `CLAIM-QUANT-GGUF-KEEPF16-L6-1`) implemented keep-f16 residency and REFUTED the "remaining gap is the f16 expansion" attribution above.** New `kKeepF16` residency + [`OwnGgufF16`/`OwnGgufKeptSlice`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L79) keep F16 matmul weights (+ F16 embed/tied head, one shared f16 vocab matrix via [`OwnedBytes::KeepAlive`](../include/vllm/model_executor/models/owned_bytes.h)) resident as F16, consumed by the elementwise f16 GEMM. Binding A/B: peak RSS 3.884 → **3.832 GiB (−52 MB, RSS-NEUTRAL)** — L5's page-release ALREADY dropped the f16 file pages, so keep-f16 only swaps an anonymous bf16 buffer for equal-size file-backed f16 pages. smaps attribution: keep-f16 file-backed **2.634 GiB ≈ llama.cpp's 2.68 file** (weight residency AT PARITY), anon 1.20 GiB; the **remaining ~1.08 GiB gap is the engine's ANONYMOUS activation/KV workspace, NOT weights** — the real, separate CPU RSS lever. Also regresses prefill (TTFT 577 → ~1000 ms, first-touch faults into the timed window; decode at parity). Tokens byte-identical (md5 `d235db1…`). Ships DEFAULT OFF at L6. **L7 (2026-07-23, `CLAIM-QUANT-GGUF-RSS-L7-1`) REVERSED L6's refutation and CLOSED the CPU RSS gap to 1.01× llama.cpp.** The profile disproved the "workspace" attribution — DevicePool 20 MiB, whole KV 115 MiB, both ≤ llama.cpp. The 1 GiB residual was a q8_0 repack-source DOUBLE-COUNT: on aarch64 the G7 repack COPIES q8_0 into an anonymous buffer while the f16 borrows keep the mapping alive, so the DEAD source blocks stay file-backed. [`OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) now `DropSpanResidency`es the repack source (port of llama.cpp `unmap_fragment`), and [`PrefaultBorrowedSpan`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L21) faults borrowed weights at load (port of llama.cpp mmap prefetch), removing L6's prefill regression — so [keep-f16 flips DEFAULT ON](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L168) (`VT_GGUF_KEEP_F16=0` opt-out). Binding A/B (idle dgx aarch64, base-vs-L7 same-binary): peak RSS **3.884 → 2.832 GiB = 1.39× → 1.01× llama.cpp** (File 2.632 → 1.629, the released q8_0 source; anon 1.200 unchanged), prefill **1.18× AHEAD** (204 vs pp128 173.2, denominator SUPERSEDED by #1003), decode ~parity (24.4 vs 25.09), tokens BYTE-IDENTICAL (md5 `809f2d0…` base/L7/oracle). **Against our own keep-f16-off arm the default costs about 9% of prefill (224 → 204 t/s) and about 1.4% of decode (TPOT 40.4 → 40.95 ms) for 1.05 GiB, settled 2026-08-17 as a product decision, NOT by the competitor floor.** Anon 1.200 GiB is IRREDUCIBLE (repacked q8_0 1.06 + KV 0.115 + pool 0.02). Regressions UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, Llama 16/16, GGUF engine 28/28); `test_gguf_keep_quant` 36/36 (+1 L7 prefault byte-transparency case, x86+aarch64) | [keep-quant loader leaf](specs/gguf-keep-quant-loader.md) | `ANCHOR-BACKFILL` | `CLAIM-QUANT-GGUF-RSS-L7-1` | | `QUANT-QWEN38-27B-GGUF-ARM` | The `Qwen3.8-27B-Q4_K_M.gguf` arm end to end: tensor accounting, text decode, the multimodal legs, and this ARTIFACT's own tokenizer and chat template. The standing GGUF k-quant requirement for a model whose bf16 arm is already gated ([#915](https://github.com/mudler/vllm.cpp/issues/915)), and the arm `BACKEND-GATE-CUDA-LLAMACPP` in the [backend matrix](backend-matrix.md) is already recorded as blocked on. **Header-verified 2026-08-18** at `unsloth/Qwen3.8-27B-GGUF`@`fe1e2a23d973adb629709749dc4f6756df66ef10`: GGUF v3, arch `qwen35`, 866 tensors, F32 456 / Q4_K 294 / Q6_K 67 / Q5_K 48 / Q8_0 1, data end == file size 17,106,775,008. **Two facts [#821](https://github.com/mudler/vllm.cpp/issues/821) did not record and which change the scope:** `qwen35.block_count = 65` with `qwen35.nextn_predict_layers = 1`, so block 64 is the MTP/`nextn` DRAFTER (`blk.64.nextn.{eh_proj,enorm,hnorm,shared_head_norm}` plus a full-attention block and an FFN) — exactly the 15-tensor difference from the same model's 851-tensor BF16 GGUF, and a loader that reads `block_count` as decoder depth builds a 65-layer model out of a 64-layer checkpoint plus a drafter; and `tokenizer.ggml.padding_token_id = 248055` against 248044 in the BF16 GGUF and `null` in the official HF config, which is why the tokenizer gate belongs to the ARM. NOT blocked on kernels: every dtype this file carries is already computed natively on BOTH tiers. The CUDA tier really has no prefill/decode split (`LaunchGemm` [cuda_quant_dot.cu:1609](../src/vt/cuda/cuda_quant_dot.cu#L1609) sizes its grid `m*n` and the encoding switch at [:1864](../src/vt/cuda/cuda_quant_dot.cu#L1864) never sees `M`); the CPU tier DOES branch on `M` at [cpu_quant_gemm.cpp:190](../src/vt/cpu/cpu_quant_gemm.cpp#L190), which takes the Arm i8mm `mmla` 2x2 tile only for even `M` and `N` and sends decode (`M=1`) to the portable `nrc==1` path. That is a kernel-TIER split, NOT a coverage split -- no dtype gains or loses support at any `M`, both arms end in the same `BlockVecDot` table -- so the conclusion stands and it is a W3 speed fact rather than a W2 gap | llama.cpp `b10451` = `10bf611e5` ([pin](oracles/llama-cpp.md), **`gateable = yes`** since [#857](https://github.com/mudler/vllm.cpp/issues/857) landed 2026-08-22) is the arm's ORACLE and its only comparator — at the vLLM pin `555967922` there is no in-tree GGUF reader (`6635279d8` moved it out of tree) and SGLang's alias table does not reach `qwen3_5` ([#979](https://github.com/mudler/vllm.cpp/issues/979)). llama.cpp is never the MIRROR | the single-file GGUF entry [qwen3_5_gguf_weights.cpp:1474](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L1474); the CUDA quant enum [cuda_quant_dot.cu:700](../src/vt/cuda/cuda_quant_dot.cu#L700) and CPU [cpu_quant_dot.cpp:787](../src/vt/cpu/cpu_quant_dot.cpp#L787) already cover Q4_K/Q5_K/Q6_K, and Q8_0 has its own path [cuda_quant_dot.cu:1659](../src/vt/cuda/cuda_quant_dot.cu#L1659) | **W2 LANDED the accounting**, modelled on [muse_glimmer_gguf_manifest.inc](../tests/vllm/models/muse_glimmer_gguf_manifest.inc): committed header-only manifests [qwen38_27b_q4km_gguf_manifest.inc](../tests/vllm/models/qwen38_27b_q4km_gguf_manifest.inc) (866 names, 51 kv) and [qwen38_27b_mmproj_gguf_manifest.inc](../tests/vllm/models/qwen38_27b_mmproj_gguf_manifest.inc) (334 names, 35 kv), generated by [gen-qwen38-27b-gguf-manifest.py](../scripts/gen-qwen38-27b-gguf-manifest.py) from the mirrored bytes; the accounting gate [test_qwen38_27b_gguf_manifest.cpp:223](../tests/vllm/models/test_qwen38_27b_gguf_manifest.cpp#L223) (6 cases, 464 assertions hermetic, 4745 over the shipped bytes under `VLLM_CPP_QWEN38_27B_{GGUF,MMPROJ}`, ZERO unaccounted in BOTH directions on both files); and the reachability gate [test_gguf_accounting_reach.cpp:184](../tests/vllm/entrypoints/test_gguf_accounting_reach.cpp#L184) (6 cases, 22 assertions), which enters through `LoadedEngine::FromModelDir` and reds 3/6 when either refusal call site in `model_loader.cpp` is deleted while the manifest target stays green at 6/6. The `nextn` correction was a gap that DID NOT EXIST: [qwen3_5_gguf_weights.cpp:889](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L889) has taken `block_count - nextn_predict_layers` since `1a4db5c3c`, and `mtp_num_hidden_layers` has been republished since `493327b4e`; what was missing was a gate, because [test_qwen3_5_gguf_mtp.cpp:36](../tests/vllm/models/test_qwen3_5_gguf_mtp.cpp#L36) is asset-gated on `VLLM_MTP_GGUF_MODEL`, skips SILENTLY when it is unset, and checks only `num_hidden_layers > 0` rather than the arithmetic its own comment claims. **W3 RAN THE TOKEN GATE ON 2026-08-23 AND IT FAILED.** Two `rc run` jobs on `thor:gpu0` (`64f66cda`, `8e0d8e54`), same GGUF file both sides, greedy, 48 tokens, concurrency 1, MTP OFF so both engines decode the same 851 tensors and the same 64-layer trunk (llama.cpp ignores all 15 of `blk.64`, re-observed as exactly 15 `unused tensor` warnings). **Tokenizer EXACT 6/6** through three of our paths (`examples/tokenize`, `vllm-cli` prompt counts, and the agreeing generation prefixes), so the #1355 prompt-token undercount is absent here. **Generation DIVERGES 5/6**, first differing index 7/34/20/-/14/32 with prompt 3 token-exact 48/48. Teacher-forcing the oracle along OUR ids over all 288 steps puts our token at the oracle's **rank 1 on 282 and rank 2 on 6, never rank 3 or worse**, losing by 0.027-0.178 logits against absolute logits of 15.9-22.6 - a PRECISION difference in the quantized compute path, not a wiring defect. The near-tie band was NOT reached for: the oracle's greedy decode reproduced #857's text byte for byte from a different build, so it is deterministic and the band's premise fails. No speed or memory number is admissible from this arm; resident bytes were measured only to refuse a dequant hypothesis (ours 24.997 GiB vs the oracle's 30.917 GiB on the same box and file, so NO dequant-to-bf16 blow-up). **2026-09-02, the cause is FOUND and PARTLY FIXED and the gate still FAILS:** our final logits carried only bf16 RESOLUTION (288 of 288 top-1 logits exactly on the bf16 grid, ULP 0.125 at magnitude 16-32, against contested gaps of 0.027-0.178), because a GGUF keep-quant head reached the bf16-output logits helper on the `nk` LAYOUT flag. Routing a block-quant head to the f32-output GEMM takes the arm from **5 of 6 to 3 of 6** divergent prompts, measured as one tree built twice on `thor:gpu0` (`c0b3fc6d`) whose bf16 arm reproduces 2026-08-23 index for index. The two SMALLEST margins (0.027185, 0.058050) resolved; the three largest (0.085434, 0.115482, 0.178236) did not, at the same indices with the same agreeing prefixes, so the residual is a MAGNITUDE term rather than a resolution one. `TOKEN_GATE` stays `FAIL` and no speed or memory axis becomes admissible. [Evidence](../docs/bench-evidence/qwen38-27b-q4km-logits-f32-20260902.md), and the superseded 5-of-6 run [Evidence](../docs/bench-evidence/qwen38-27b-q4km-token-gate-20260823.md) | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | -| `QUANT-QWEN38-27B-NVFP4-ARM` | The `unsloth/Qwen3.8-27B-NVFP4` artifact, which is **not what its name says**. **Its pinned revision is GONE:** `a767244d27bd76589a3e3b2ab4e64032c4ebc7af`, the revision [#821](https://github.com/mudler/vllm.cpp/issues/821) names, answers HTTP 404 and `git ls-remote` reports one ref, `refs/heads/main` = `7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108` — the second in-place re-quantization this publisher has done in this family, after `unsloth/Qwen3.6-27B-NVFP4`. So the user-reported load failure on #821 is CORROBORATED at a different revision, never reproduced. At the live revision (header-verified 2026-08-18, 1953 + 15 tensors, `8 + header_len + max(data_offsets[1])` == file size 22,568,192,096) `quantization_config.format` is `mixed-precision`: `group_0` is FP8 W8A8 with **per-CHANNEL** weight scales and **DYNAMIC per-token** activations over `self_attn.(q\|k\|v\|o)_proj`, `linear_attn.(in_proj_qkv\|in_proj_z\|out_proj)`, `lm_head` and `layers.(56..63).mlp.*`; `group_1` is `nvfp4-pack-quantized` W4A4 over the remaining `mlp.*`; plus an 8-bit static `kv_cache_scheme` and an `ignore` list of **303 entries** -- not just the vision tower: 48 x `linear_attn`, `linear_attn.norm`, `linear_attn.in_proj_b` and `linear_attn.in_proj_a` (the GDN layer count), 27 x 4 vision blocks, 2 mergers, and `re:^mtp.*`. That list is what makes the predicate claim provable rather than asserted: `in_proj_a`/`in_proj_b` are IGNORED while `in_proj_qkv`/`in_proj_z`/`out_proj` are `group_0` TARGETS, so a resolver that reads the groups but not the `ignore` list gets the GDN block wrong in both directions. **`*.input_scale` appears ZERO times in the checkpoint.** Four independent blockers, each anchored in the spec: the unconditional `.input_scale` read, a per-channel BF16 `weight_scale` that `ReadF32Scalar` refuses on BOTH count and dtype, no representation for a dynamic per-token activation scheme, and a scheme that is never read from the config at all. The NVFP4 half is the half CLOSEST to working; the FP8 tower is the blocker. **A SECOND artifact of the same model is now in scope and it is a DIFFERENT FORMAT:** `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121`@`36f717a22990e82c54c1d48ee77c491b87825680`, needed by campaign [#1574](https://github.com/mudler/vllm.cpp/issues/1574), declares `quant_method: "modelopt"` and `quant_algo: "MIXED_PRECISION"` with a `quantized_layers` map of 401 EXACT module names, an EMPTY `ignore`, per-TENSOR STATIC FP8 (`weight_scale` and `input_scale` both `F32 []`, 208 modules) and ModelOpt-spelled W4A16_NVFP4 g16 weight-only (`weight` U8 + `weight_scale` F8_E4M3 + `weight_scale_2` `F32 []`, 193 modules including `lm_head`) over ALL 64 layers' MLP -- no layer-56 boundary. Header-verified 2026-08-21 over all four shards (970/976/40/15 = 2001 names; `8 + header_len + max(data_offsets[1])` == the size the hub reports for each; the four sizes exceed the index's `metadata.total_size` 21,921,427,300 by exactly the four headers plus their 8-byte prefixes). **NONE of W4's four blockers applies to it** -- they were properties of the unsloth artifact, not of the format -- so both halves LOAD. The fifth blocker W5 found instead: `ct::Config` stops at `quant_method != "compressed-tensors"`, so nothing in this tree read this config at all and every routing decision fell to the per-projection tensor-NAME probe, which is wrong in BOTH directions silently. `nvidia/Qwen3.6-27B-NVFP4`@`0893e160`, the #466 gate model, is the same ModelOpt shape (2194 names, same 208/193 split, `exclude_modules` `["mtp*","mtp.layers.0*"]`, a `kv_cache_scheme` with ZERO scales shipped) and reaches the same call site, which is why W5 refuses a DISAGREEMENT rather than routing by the declaration ([#1597](https://github.com/mudler/vllm.cpp/issues/1597) owes that) | vLLM `555967922` is the MIRROR and the primary oracle — it runs this format, so nothing here may diverge from its compressed-tensors semantics. A generic mixed-precision resolver already exists at [modelopt_mixed_precision.h](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h). Through W4 **no production file included it** -- the only two includes in the tree were its own two tests, and `nemotron_h_weights.cpp` (which this row previously named) includes `nemotron_h.h`, `nemotron_h_loader.h`, `nvfp4_dequant.h` and `vt/unaligned.h` and reads its quant config inline; a 33,575-byte header reachable only from tests is an `AGENTS.md` §"Nothing lands dead" item. W4 did NOT adopt it and argued the exception: the two headers resolve two DIFFERENT upstream formats that share only the English word "mixed", and its artifact is the compressed-tensors one. **W5 is the FIRST production wiring**, for the artifact that really is ModelOpt: [qwen3_5_dense_weights.cpp](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp) now includes it and `LoadQwen3_5Dense` calls [RefusalForQuantizationConfig](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h) once per checkpoint. So the tree ends with ONE resolver per format, which is upstream's own structure | the failing read [qwen3_5_weights.cpp:642](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L642) (`:457` was WRONG and is `OwnedBytes::Borrow`; re-derived by W4), the resolution [compressed_tensors_config.h:354](../src/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_config.h#L354) (`Resolve`, ignore-first then first-matching target) and its production call site [qwen3_5_dense_weights.cpp:875](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L875), its refusal [dense_weight_loaders.h:164](../include/vllm/model_executor/models/dense_weight_loaders.h#L164) (the count check at `:168`, the dtype check at `:172`; #1258 moved `ReadF32Scalar` down 63 lines and `:101` is now an unrelated `try`/`catch` probe), the scalar-only `Fp8Weight` [qwen3_5_weights.h:719](../include/vllm/model_executor/models/qwen3_5_weights.h#L719), the static activation quant [qwen3_5.cpp:3629](../src/vllm/model_executor/models/qwen3_5.cpp#L3629), and the stale predicate [qwen3_5_dense_weights.cpp:699](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L699) (the `.linear_attn.in_proj_` early-false at `:702`) that declares the GDN input projections never quantized — true for the 3.6 unsloth artifact ([hf_snapshot.h:287](../tests/parity/hf_snapshot.h#L287)) and false for this one | W4: [test_qwen38_27b_nvfp4_arm.cpp:208](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L208) (per-scheme composition), [:582](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L582) (the FP8 refusal through `LoadQwen3_5Dense`), [:666](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L666) (the env-gated live re-read); 8 cases / 190 assertions hermetic and 204 with `VLLM_CPP_QWEN38_27B_NVFP4_DIR` set, over the committed [qwen38_27b_nvfp4_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_manifest.inc) (1953) and [qwen38_27b_nvfp4_mtp_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_mtp_manifest.inc) (15), summing to the index's 1968; per-scheme composition 466/672/475/323/32 tensors over 233/168/317/267/16 modules, zero unclassified. Manifest-capture precedent [minimax_h3_nvfp4_manifest.inc](../tests/vllm/models/minimax_h3_nvfp4_manifest.inc), captured the same way this row's numbers were: an HTTP range read of the file's own header. W5: [test_qwen38_27b_modelopt_mtp_arm.cpp](../tests/vllm/models/test_qwen38_27b_modelopt_mtp_arm.cpp), 22 cases / 1687 assertions hermetic, over four committed header-only manifests ([s1](../tests/vllm/models/qwen38_27b_modelopt_mtp_s1_manifest.inc) 970, [s2](../tests/vllm/models/qwen38_27b_modelopt_mtp_s2_manifest.inc) 976, [s3](../tests/vllm/models/qwen38_27b_modelopt_mtp_s3_manifest.inc) 40, [s4](../tests/vllm/models/qwen38_27b_modelopt_mtp_s4_manifest.inc) 15) summing to the index's 2001; per-scheme composition 720/579/702 tensors over 256/193/536 modules with zero KV scales and zero unclassified, and the 256 split as 208 `kDirect` Linears plus 48 `kPrefix` `linear_attn` CONTAINERS so the count cannot be right for the wrong reason; 937 weight-bearing modules split 208/193/536. RED before wiring was 5 cases failing with an EMPTY refusal -- the loader accepted every config/tensor disagreement and both unloadable algorithms. Twelve negative mutations, every one detected, including the deleted production call site (8 cases red), a suffix list without `.weight_scale_2` (7), one manifest row removed with its count literal left behind (5, and all 22 cases still RUN -- deriving the row count with `std::size` turned what was an out-of-bounds read into an ordinary red), the NVFP4 refusal branch disabled (2) and an unseen operand family skipped rather than refused (2). The fresh review found the last two: the NVFP4 branch is the whole cross-check for 193 of the 401 declared modules and `if (false && ...)` on it left the suite fully green, and `Refusal` skipped a tensor name whose family `SplitOperand` has never seen, which its own contract forbids reading as unquantized. A separate case pins that the refusal is SILENT on the `nvidia/Qwen3.6-27B-NVFP4` shape -- wildcard `exclude_modules`, an `input_scale` on every NVFP4 module, and a `kv_cache_scheme` with zero scales -- and its own mutation reds it while every other case stays green. W4's gate is untouched and re-ran 9 cases / 194 assertions green on the same tree. The token gate is PENDING on [#1632](https://github.com/mudler/vllm.cpp/issues/1632): the pinned oracle DOES run a model inside an `rc` lease (2026-08-18), at `max_num_batched_tokens` 512 against a recorded denominator of 8192, and this artifact's ~20.4 GiB are not staged where a lease can read them. It supersedes #1185, closed 2026-08-18 as local-only | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | +| `QUANT-QWEN38-27B-NVFP4-ARM` | The `unsloth/Qwen3.8-27B-NVFP4` artifact, which is **not what its name says**. **Its pinned revision is GONE:** `a767244d27bd76589a3e3b2ab4e64032c4ebc7af`, the revision [#821](https://github.com/mudler/vllm.cpp/issues/821) names, answers HTTP 404 and `git ls-remote` reports one ref, `refs/heads/main` = `7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108` — the second in-place re-quantization this publisher has done in this family, after `unsloth/Qwen3.6-27B-NVFP4`. So the user-reported load failure on #821 is CORROBORATED at a different revision, never reproduced. At the live revision (header-verified 2026-08-18, 1953 + 15 tensors, `8 + header_len + max(data_offsets[1])` == file size 22,568,192,096) `quantization_config.format` is `mixed-precision`: `group_0` is FP8 W8A8 with **per-CHANNEL** weight scales and **DYNAMIC per-token** activations over `self_attn.(q\|k\|v\|o)_proj`, `linear_attn.(in_proj_qkv\|in_proj_z\|out_proj)`, `lm_head` and `layers.(56..63).mlp.*`; `group_1` is `nvfp4-pack-quantized` W4A4 over the remaining `mlp.*`; plus an 8-bit static `kv_cache_scheme` and an `ignore` list of **303 entries** -- not just the vision tower: 48 x `linear_attn`, `linear_attn.norm`, `linear_attn.in_proj_b` and `linear_attn.in_proj_a` (the GDN layer count), 27 x 4 vision blocks, 2 mergers, and `re:^mtp.*`. That list is what makes the predicate claim provable rather than asserted: `in_proj_a`/`in_proj_b` are IGNORED while `in_proj_qkv`/`in_proj_z`/`out_proj` are `group_0` TARGETS, so a resolver that reads the groups but not the `ignore` list gets the GDN block wrong in both directions. **`*.input_scale` appears ZERO times in the checkpoint.** Four independent blockers, each anchored in the spec: the unconditional `.input_scale` read, a per-channel BF16 `weight_scale` that `ReadF32Scalar` refuses on BOTH count and dtype, no representation for a dynamic per-token activation scheme, and a scheme that is never read from the config at all. The NVFP4 half is the half CLOSEST to working; the FP8 tower is the blocker. **A SECOND artifact of the same model is now in scope and it is a DIFFERENT FORMAT:** `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121`@`36f717a22990e82c54c1d48ee77c491b87825680`, needed by campaign [#1574](https://github.com/mudler/vllm.cpp/issues/1574), declares `quant_method: "modelopt"` and `quant_algo: "MIXED_PRECISION"` with a `quantized_layers` map of 401 EXACT module names, an EMPTY `ignore`, per-TENSOR STATIC FP8 (`weight_scale` and `input_scale` both `F32 []`, 208 modules) and ModelOpt-spelled W4A16_NVFP4 g16 weight-only (`weight` U8 + `weight_scale` F8_E4M3 + `weight_scale_2` `F32 []`, 193 modules including `lm_head`) over ALL 64 layers' MLP -- no layer-56 boundary. Header-verified 2026-08-21 over all four shards (970/976/40/15 = 2001 names; `8 + header_len + max(data_offsets[1])` == the size the hub reports for each; the four sizes exceed the index's `metadata.total_size` 21,921,427,300 by exactly the four headers plus their 8-byte prefixes). **NONE of W4's four blockers applies to it** -- they were properties of the unsloth artifact, not of the format -- so both halves LOAD. The fifth blocker W5 found instead: `ct::Config` stops at `quant_method != "compressed-tensors"`, so nothing in this tree read this config at all and every routing decision fell to the per-projection tensor-NAME probe, which is wrong in BOTH directions silently. `nvidia/Qwen3.6-27B-NVFP4`@`0893e160`, the #466 gate model, is the same ModelOpt shape (2194 names, same 208/193 split, `exclude_modules` `["mtp*","mtp.layers.0*"]`, a `kv_cache_scheme` with ZERO scales shipped) and reaches the same call site, which is why W5 refuses a DISAGREEMENT rather than routing by the declaration ([#1597](https://github.com/mudler/vllm.cpp/issues/1597) owes that) | vLLM `555967922` is the MIRROR and the primary oracle — it runs this format, so nothing here may diverge from its compressed-tensors semantics. A generic mixed-precision resolver already exists at [modelopt_mixed_precision.h](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h). Through W4 **no production file included it** -- the only two includes in the tree were its own two tests, and `nemotron_h_weights.cpp` (which this row previously named) includes `nemotron_h.h`, `nemotron_h_loader.h`, `nvfp4_dequant.h` and `vt/unaligned.h` and reads its quant config inline; a 33,575-byte header reachable only from tests is an `AGENTS.md` §"Nothing lands dead" item. W4 did NOT adopt it and argued the exception: the two headers resolve two DIFFERENT upstream formats that share only the English word "mixed", and its artifact is the compressed-tensors one. **W5 is the FIRST production wiring**, for the artifact that really is ModelOpt: [qwen3_5_dense_weights.cpp](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp) now includes it and `LoadQwen3_5Dense` calls [RefusalForQuantizationConfig](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h) once per checkpoint. So the tree ends with ONE resolver per format, which is upstream's own structure | the failing read [qwen3_5_weights.cpp:642](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L642) (`:457` was WRONG and is `OwnedBytes::Borrow`; re-derived by W4), the resolution [compressed_tensors_config.h:354](../src/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_config.h#L354) (`Resolve`, ignore-first then first-matching target) and its production call site [qwen3_5_dense_weights.cpp:875](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L875), its refusal [dense_weight_loaders.h:164](../include/vllm/model_executor/models/dense_weight_loaders.h#L164) (the count check at `:168`, the dtype check at `:172`; #1258 moved `ReadF32Scalar` down 63 lines and `:101` is now an unrelated `try`/`catch` probe), the scalar-only `Fp8Weight` [qwen3_5_weights.h:732](../include/vllm/model_executor/models/qwen3_5_weights.h#L732), the static activation quant [qwen3_5.cpp:3629](../src/vllm/model_executor/models/qwen3_5.cpp#L3629), and the stale predicate [qwen3_5_dense_weights.cpp:699](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L699) (the `.linear_attn.in_proj_` early-false at `:702`) that declares the GDN input projections never quantized — true for the 3.6 unsloth artifact ([hf_snapshot.h:287](../tests/parity/hf_snapshot.h#L287)) and false for this one | W4: [test_qwen38_27b_nvfp4_arm.cpp:208](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L208) (per-scheme composition), [:582](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L582) (the FP8 refusal through `LoadQwen3_5Dense`), [:666](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L666) (the env-gated live re-read); 8 cases / 190 assertions hermetic and 204 with `VLLM_CPP_QWEN38_27B_NVFP4_DIR` set, over the committed [qwen38_27b_nvfp4_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_manifest.inc) (1953) and [qwen38_27b_nvfp4_mtp_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_mtp_manifest.inc) (15), summing to the index's 1968; per-scheme composition 466/672/475/323/32 tensors over 233/168/317/267/16 modules, zero unclassified. Manifest-capture precedent [minimax_h3_nvfp4_manifest.inc](../tests/vllm/models/minimax_h3_nvfp4_manifest.inc), captured the same way this row's numbers were: an HTTP range read of the file's own header. W5: [test_qwen38_27b_modelopt_mtp_arm.cpp](../tests/vllm/models/test_qwen38_27b_modelopt_mtp_arm.cpp), 22 cases / 1687 assertions hermetic, over four committed header-only manifests ([s1](../tests/vllm/models/qwen38_27b_modelopt_mtp_s1_manifest.inc) 970, [s2](../tests/vllm/models/qwen38_27b_modelopt_mtp_s2_manifest.inc) 976, [s3](../tests/vllm/models/qwen38_27b_modelopt_mtp_s3_manifest.inc) 40, [s4](../tests/vllm/models/qwen38_27b_modelopt_mtp_s4_manifest.inc) 15) summing to the index's 2001; per-scheme composition 720/579/702 tensors over 256/193/536 modules with zero KV scales and zero unclassified, and the 256 split as 208 `kDirect` Linears plus 48 `kPrefix` `linear_attn` CONTAINERS so the count cannot be right for the wrong reason; 937 weight-bearing modules split 208/193/536. RED before wiring was 5 cases failing with an EMPTY refusal -- the loader accepted every config/tensor disagreement and both unloadable algorithms. Twelve negative mutations, every one detected, including the deleted production call site (8 cases red), a suffix list without `.weight_scale_2` (7), one manifest row removed with its count literal left behind (5, and all 22 cases still RUN -- deriving the row count with `std::size` turned what was an out-of-bounds read into an ordinary red), the NVFP4 refusal branch disabled (2) and an unseen operand family skipped rather than refused (2). The fresh review found the last two: the NVFP4 branch is the whole cross-check for 193 of the 401 declared modules and `if (false && ...)` on it left the suite fully green, and `Refusal` skipped a tensor name whose family `SplitOperand` has never seen, which its own contract forbids reading as unquantized. A separate case pins that the refusal is SILENT on the `nvidia/Qwen3.6-27B-NVFP4` shape -- wildcard `exclude_modules`, an `input_scale` on every NVFP4 module, and a `kv_cache_scheme` with zero scales -- and its own mutation reds it while every other case stays green. W4's gate is untouched and re-ran 9 cases / 194 assertions green on the same tree. The token gate is PENDING on [#1632](https://github.com/mudler/vllm.cpp/issues/1632): the pinned oracle DOES run a model inside an `rc` lease (2026-08-18), at `max_num_batched_tokens` 512 against a recorded denominator of 8192, and this artifact's ~20.4 GiB are not staged where a lease can read them. It supersedes #1185, closed 2026-08-18 as local-only | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | | `QUANT-GGUF-PRESETS` | Representative mixed-file gates for every llama.cpp output preset family | llama.cpp `tools/quantize/quantize.cpp:34-74` | only custom APEX mixed files are executable; no general preset dispatch | [APEX gates](../tests/parity/test_qwen36_gguf_engine.cpp#L143) do not prove llama.cpp preset breadth | [coverage spike](specs/quantization-coverage.md); split exact preset IDs before `READY` | `INVENTORIED` | - | ## 1. llama.cpp / GGUF encodings From 5c8f6de999db2b79c2b5642596971a4ecd8dfe59 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 13 Sep 2026 03:26:02 +0000 Subject: [PATCH 08/10] fix(MODEL-MM-QWEN4-EXP): release the source pages from the arm qwen4_exp actually takes There are two ResidentWeights and this row's model uses the other one. The release landed in the one in the unnamed namespace of qwen3_5.cpp, which shadows the header one inside that translation unit. That covers the Qwen3.5 dense weights and, through KqResidentSlice and KqGrouped, the shared MoE seam's keep-quant expert towers, which qwen4_exp does reach. It covers nothing else in qwen4_exp: every attention, norm, hyper-connection, PLE and lm_head weight stages through dense_attn::ResidentWeight instead, at 12 call sites in qwen4_exp_forward.cpp, 10 in qwen4_exp_qsa_block.cpp, 7 in qwen4_exp_ple_block.cpp and 1 in qwen4_exp_registry.cpp, with no reference to the qwen3_5.cpp function anywhere in that model. Measured with one arm wired: the load peak fell from 27.67 GB to 8.18 GB and host RssFile still climbed to 21.08 GB during the forward and stayed there. The same call now sits in the same place in dense_attn_block.h: inside the d_dev memo, immediately before AdoptDeviceBytesAsHost, with the platform's own host_memory_is_device_addressable() answer passed in rather than re-derived. Three cases repeat the release, the memo and the platform term against that seam. Deleting the new call site turns the first of them red on both its counter and its RssFile assertion: 0 releases counted, and resident file pages 138100 kB before against 138228 kB after. The replacement justification in RocmPlatform::residency_policy() was itself false and is corrected. It claimed the flag has two readers, ShouldReleaseHostWeights and ShouldInterleaveLoadStream, that both also require marlin_committed, so flipping it would change no behaviour. DirectDeviceLoadEligible (qwen3_5_dense_weights.cpp:146-180) is a third reader, requires no marlin_committed, and gates StageAndReleaseLoadedDense for every Qwen3.5-dense safetensors load. Flipping the flag WOULD change behaviour on ROCm. The comment now carries the full enumeration. Spec section 6 declared a focused selector, -tc=*resident*, that selects zero cases: no case name in the file contains that substring, so doctest reported 0 assertions and SUCCESS. It is -tc=*release* now, which selects 9 cases and 52 assertions, and the counts are printed with the evidence. Two preconditions of the helper are NOT convicted by this harness and are recorded rather than chased: deleting the backend's DeviceMemoryIsHostAddressable() term leaves the suite green because the fake backend answers false unconditionally, and deleting the synchronize leaves it green because HostBackend::Copy is a memcpy that cannot express a live DMA. ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W owns both. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W.md | 19 +++ .agents/quantization-matrix.md | 2 +- .../specs/rocm-host-residency-after-upload.md | 85 +++++++++-- .../model_executor/models/dense_attn_block.h | 34 ++++- .../model_executor/models/qwen3_5_weights.h | 19 ++- src/vllm/platforms/rocm.cpp | 39 +++-- .../test_resident_weight_host_addressable.cpp | 133 ++++++++++++++++++ 7 files changed, 305 insertions(+), 26 deletions(-) create mode 100644 .agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W.md diff --git a/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W.md b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W.md new file mode 100644 index 000000000..8d4ca9b00 --- /dev/null +++ b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W +Title: OWED: the DeviceMemoryIsHostAddressable and Synchronize terms of MaybeReleaseStagedBorrowSource are UNGATED +Row: MODEL-MM-QWEN4-EXP +State: OPEN +Kind: task +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-13 +Updated: 2026-09-13 +Closed: - + +## Problem + +Deleting backend.DeviceMemoryIsHostAddressable() from MaybeReleaseStagedBorrowSource (qwen3_5_weights.cpp) leaves the focused suite 25/25 green with the binary proven changed, and so does deleting backend.Synchronize(queue). Both survive because of the harness, not because the code is wrong. The fake backend in tests/vllm/model_executor/test_resident_weight_host_addressable.cpp answers DeviceMemoryIsHostAddressable() false unconditionally, so only the platform half of the host-addressability pair is ever exercised; and HostBackend::Copy is a synchronous memcpy that does not override Synchronize, so no case in this tree can express a DMA still reading the source pages when they are dropped. Convicting the first needs a backend answering true while the platform answers false, a combination no fleet device presents and that the file's process-global registrar cannot hold beside the existing one. Convicting the second needs a backend whose Copy defers. Recorded as owed rather than repaired by weakening or contorting a passing test. See .agents/specs/rocm-host-residency-after-upload.md, its Ungated-guarantees section. + +## Resolution + +- diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md index 6dc227a34..d7373b947 100644 --- a/.agents/quantization-matrix.md +++ b/.agents/quantization-matrix.md @@ -35,7 +35,7 @@ otherwise it remains `PARTIAL` or `INVENTORIED` even if parsing works. | `QUANT-GGUF-CIQ-GEMM` | Compute-in-quant GEMM: activation quant (Q8_0/Q8_K) + per-type vec_dot dispatch for Q8_0/Q4_K/Q5_K/Q6_K/Q3_K/Q4_0; portable C++ tier, then x86/Arm SIMD + repack tiers. **G1-G4 landed** — the portable tier-0 path is complete, gated at the OP level, and **ROUTED end to end**: `vt::MatmulBT` dispatches a block-dtype weight to `kMatmulBTQuant`, keep-quant is the production DEFAULT wherever that op is registered, and the six routed encodings compute in quant with **no token movement**. **G6 (2026-07-23)** added the Arm **i8mm mmla `nrc==2` tier** for q8_0/q4_0/q4_K/q6_K (q3_K/q5_K have no upstream mmla → stay portable), 2x2-tiled into `kMatmulBTQuant` at even M,N: op-level q4_K **7–8.4×** / q6_K **3.8–4.5×** / q8_0 ~1.2× over portable, e2e prefill +8.4 % on the q8_0-dominant bench file (1.44× behind llama.cpp), tokens byte-identical. **G7 (2026-07-23)** added q8_0 **repack-at-load** (the `q8_0_4x8` tier `ggml_repack_get_optimal_repack_type` picks on NEON+i8mm): the loader repacks each q8_0 weight once into the `block_q8_0x4` interleave and `kMatmulBTQuant` dispatches a pre-shuffled i8mm gemm/gemv with no per-block register shuffles — op-level q8_0 **3.7–5.9×** over the mmla tier, **E2E prefill 1.92× same-binary → 223.8 t/s vs llama.cpp pp128 177.3 = at/beyond parity** (was ~1.5× behind), decode at parity, tokens byte-identical. **CPU prefill parity reached; the prefill-lever search is closed** (remaining gap = peak RSS 1.39×, loader-bound). G5 (x86) + G8 open. **The FRESH op-dispatch profile this row owed is DONE (2026-08-06, dgx aarch64, `main` @`dfd29060`, same bench file; see `.agents/benchmark-record.md` 'FRESH op-dispatch profile'), and it does NOT support starting G5 next:** `QuantRepackMatmul` is 5.06 % of prefill and 15.99 % of decode on aarch64 where the i8mm tier already landed. The profile re-ranks the CPU levers to (1) threadpool synchronisation at 47 % of decode (`ThreadReady`+`PollForWork`+`Barrier`; M=1 cannot amortise the barrier) and (2) CPU paged attention at ~39 % of prefill, of which 20.68 % is a per-ELEMENT dtype switch in the attention dot loop (`cpu_paged_attn.cpp:29` called from `:143`), the same defect class E1 already removed from the elementwise GEMM. G5 stays a real x86 gap worth closing for x86 users, but it is not the top lever, and the x86 box is VOID for timing so it cannot be speed-gated here | llama.cpp `ggml/src/ggml-cpu/ggml-cpu.c:211-406` traits table, `ggml-cpu/quants.c:174-860` generic vec_dot, `arch/{x86,arm}/quants.c`, `ggml-cpu/repack.cpp:4153-4830` at `237ad9b96` | G1: [block dtypes + geometry](../src/vt/dtype.cpp#L32), [quant traits table](../src/vt/cpu/cpu_quant_traits.cpp#L1), [shared block decoders](../src/vt/cpu/cpu_quant_dequant.cpp#L1), [op surface](../include/vt/quant.h#L1). G2: [activation quant + scratch sizing](../src/vt/cpu/cpu_quant_act.cpp#L1) (`quantize_row_q8_0/q8_K`). G3: [the six generic vec_dot](../src/vt/cpu/cpu_quant_dot.cpp#L1), [block-struct mirror](../src/vt/cpu/cpu_quant_blocks.h#L1), [`kMatmulBTQuant` quantized path + composite fallback](../src/vt/cpu/cpu_quant_gemm.cpp#L1). G4: [the routing point](../src/vt/ops.cpp#L158) — `vt::MatmulBT` sends a block-dtype `b` to `MatmulBTQuant` and is otherwise unchanged, which is sufficient because every model matmul helper already routes an `nk=true` weight there ([qwen3_5.cpp:1067](../src/vllm/model_executor/models/qwen3_5.cpp#L1067)); plus [the default flip + `expand_nk`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) and [the untransposed expand path](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L194). G6: [Arm i8mm mmla tier](../src/vt/cpu/cpu_quant_dot_arm.cpp#L1) (q8_0/q4_0/q4_K/q6_K `vmmlaq_s32`, HWCAP2_I8MM-probed, `VT_CPU_QUANT_MMLA` defeat) + [2x2 tile in kMatmulBTQuant](../src/vt/cpu/cpu_quant_gemm.cpp#L85), per-file `+i8mm` in CMakeLists | [G1 traits cross-check + fallback units](../tests/vt/test_ops_quant_traits.cpp#L1) — 8 cases / 5,615 assertions green (was 5,694; its composite case now covers Q8_K alone because the six weight types legitimately no longer take that path): vt geometry vs the reader's `GgmlTraits` vs ggml-common.h arithmetic all agree, and the composite equals the loader dequant byte-for-byte. [G2/G3 units](../tests/vt/test_ops_quant_dot.cpp#L1) — 16 cases / 78,052 assertions green: every `vec_dot` gated against an INDEPENDENT f64 dequantize-then-dot reference (tolerance relative to the dot's L1 magnitude, actual agreement ~1e-6) over nblocks {1,2,3,5,7,16} incl. single-block and odd multiples; ragged K throws at every layer; upstream thresholds ported unwidened (test-quantize-fns:17-28, test-backend-ops:4277 NMSE ≤ 5e-4 at M {1,4,32,512} × N {1,7,16}); bit-exact run-to-run and across threads 1/2/4; byte-exact encoder gate pins the rounding rules; 14-mutant battery, 13 caught, the 1 uncaught mutant provably unreachable. [dequant units](../tests/vllm/test_gguf_dequant.cpp#L25) still green after the decoder move. DGX (G2/G3 re-confirmed, each gate STANDALONE, goldens md5 identical before/after): clean CUDA `-Werror` build 0 warnings + full regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16 on both 0.6B and 4B, OPT 6/6, DeepSeek-V2 8/8) + `test_qwen36_gguf_engine` 28/28 with 16/16 tokens on both APEX files + the new CPU units green on aarch64 with identical counts. **G4 (2026-07-22):** `test_qwen36_gguf_engine` PASSES STANDALONE on a CPU-only dgx build (where keep-quant is live) — 2/2 cases, 16/16 greedy tokens on APEX-Compact AND APEX-Balanced vs the same-file llama.cpp oracle, exercising 5 of the 6 routed encodings end to end; the CUDA regression set is UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, gguf 28/28 incl. `VT_CPU_REF=1`), goldens md5 identical. **Binding CPU A/B** (idle dgx aarch64, one flock, same binary, 3 reps, `Qwen3.5-2B-UD-Q8_K_XL`): decode 2.216 -> 7.650 t/s (**3.45x**), prefill 5.149 -> 21.44 t/s (**4.16x**), peak RSS 7.428 -> 6.401 GiB, output tokens byte-identical across the pre-G4, post-G4 and `VT_CPU_REF=1` arms. Still **3.38x / 8.20x / 2.29x behind llama.cpp** — the projected 9-17x did NOT hold because 60 % of that file's weight bytes are `f16`, which no block encoding covers. **That gap is now CLOSED by `KERNEL-GEMM-CPU-ELEM`** (2026-07-22, same box/recipe/binary discipline): the elementwise kernel went 18-24 -> 69-351 GFLOP/s bit-exactly, taking the CPU position to **decode 1.03x behind (parity within 3.1 %) and prefill 2.34x behind**, tokens unchanged (same md5). Its measured NEGATIVE re-ranks G5-G8 once more: M-blocking the elementwise GEMM bought 1.63x op-level and **0.0 % end-to-end**, so the 95.37 % `kMatmul` attribution these G-rows were ranked against is STALE and a FRESH op-dispatch profile is owed before G5/G6/G7 are started. **G6 (2026-07-23):** [Arm i8mm mmla tier](../src/vt/cpu/cpu_quant_dot_arm.cpp#L1) landed against the refreshed profile (kMatmulBTQuant 50 % + kMatmul 16 % + kMatmulBT 14 % = 80 % of prefill). [test_ops_quant_dot G6 cross-check](../tests/vt/test_ops_quant_dot.cpp#L1) — 19 cases / **78,162** assertions on dgx aarch64: q8_0/q4_0 mmla **BIT-IDENTICAL** to the portable/scalar tier (`vmlaq_f32` non-fused under `-ffp-contract=off`), q4_K/q6_K within NMSE ≤ 5e-4, mmla GEMM bit-identical across threads 1/2/4/20. `test_qwen36_gguf_engine` 2/2 · 16/16 on both APEX files with mmla live (q8_0/q4_K/q6_K at prefill), bench-file token md5 `d235db12f2cd304007530286a1755c95` byte-identical across mmla-OFF/ON/`VT_CPU_REF=1`. Op-level portable→i8mm: q8_0 ~1.2×, q6_K 3.8–4.5×, q4_K 7–8.4×; e2e prefill same-binary 1.084× (1.56×→1.44× behind llama.cpp pp128). CUDA `-Werror` 0-warn, regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138, Qwen3-dense 184, OPT, DeepSeek-V2 223), goldens untouched. **G7 (2026-07-23):** [q8_0 repack transform](../src/vt/cpu/cpu_quant_repack.cpp#L1) + [i8mm repack gemm/gemv](../src/vt/cpu/cpu_quant_repack_arm.cpp#L1) dispatched from [`kMatmulBTQuant`](../src/vt/cpu/cpu_quant_gemm.cpp#L151) on `b.repacked`; loader repacks via [`OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) under `GgufLoadPolicy::quant_repack`, flag carried to the kernel through [`ResidentWeight`](../src/vllm/model_executor/models/qwen3_5.cpp#L702). [test_ops_quant_repack](../tests/vt/test_ops_quant_repack.cpp#L1) — 305 assertions on dgx aarch64: repacked gemm/gemv `memcmp`-equal to plain `kMatmulBTQuant` across decode/leftover/prefill, f32+bf16 out, strided activations, threads 1/2/4/20; interleave matches `make_block_q8_0x4` byte-for-byte (110 on x86, numeric skip). `test_qwen36_gguf_engine` STANDALONE 2/2·16/16 on APEX Compact+Balanced (repack live), token md5 `d235db12f2cd304007530286a1755c95` byte-identical across repack-ON/OFF/`VT_CPU_REF=1`. Binding dgx aarch64 (idle, one flock, 6 interleaved reps): op-level q8_0 3.7–5.9× (518→2401/583→3456/514→1902 GFLOP/s); E2E prefill **1.92×** (1096→572 ms), **223.8 t/s vs llama.cpp pp128 177.3 = 1.26× at/beyond parity**, decode at parity, RSS unchanged; fresh profile q8_0 GEMM 55%→~21%, prefill-lever search CLOSED. CUDA `-Werror` 0-warn, regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, Llama 16/16), goldens content-hash identical . **P0 REGRESSION FOUND + FIXED (2026-08-06, `CLAIM-QUANT-GGUF-CIQ-GROUPED-DTYPE`):** the GROUPED provider `MatmulBTQuantGroupedKernel` was f32-ONLY — it advanced a `float*` by `act.stride[0]` and declared the row `kF32` whatever `act.dtype` said, so a bf16/f16 activation was mis-strode 2x AND mis-decoded. Every prior caller/test passed f32; qwen3_5 W3b `KqGrouped` (bf16 act, `b4f5610a`) was the first non-f32 caller, so CPU-only GGUF 35B decode became all-token-0 while the CUDA gate stayed byte-exact (CUDA always honoured `act.dtype`). Fixed at [`cpu_quant_gemm.cpp:220-268`](../src/vt/cpu/cpu_quant_gemm.cpp) (rows addressed by `SizeOf(act.dtype)`/`SizeOf(out.dtype)`; `repacked`/`q8_0_aligned` now propagate onto the per-expert slice — the CIQ-G7 all-zero mode). Gated per activation dtype + bf16-out by 2 NEW cases in [`test_ops_quant_dot.cpp`](../tests/vt/test_ops_quant_dot.cpp) (RED pre-fix on f16+bf16 for all 12 weight encodings, GREEN after; f32 unaffected either way) | [CIQ GEMM leaf](specs/gguf-compute-in-quant-gemm.md) | `ANCHOR-BACKFILL` | `CLAIM-QUANT-GGUF-CIQ-G7-1` | | `QUANT-GGUF-KEEPQ-LOADER` | Keep-quantized GGUF loader: block-resident 2-D matmul weights ([N,K], no transpose), per-tensor routing, `VT_CPU_REF` dequant-oracle switch, bench-branch `7c91a42` merge. **L1+L2+L3 landed** — block residency, the TOTAL per-tensor routing policy and the `VT_CPU_REF` oracle switch all exist and are gated. **Keep-quant is DEFAULT ON since CIQ G4** wherever the running device has a registered `kMatmulBTQuant` (CPU, and since 2026-07-29 also **CUDA** for the Q8_K family via the `KERNEL-QUANT-CIQ-GEMM-CUDA` kCUDA provider — a CUDA runner now keeps k-quant/i-quant blocks COMPRESSED instead of expanding), with `VT_GGUF_KEEP_QUANT=0` as the opt-out. L4 measured; **L5 LANDED** (mmap in-place residency + tied-head sharing + read-once page release) — peak RSS 6.401 -> **3.884 GiB**, 2.29x -> **1.39x** llama.cpp, byte-identical | llama.cpp `src/llama-model-loader.cpp:1047,1385` (file-typed residency), `:1676` + `ggml/src/llama-mmap.cpp:490` (`unmap_fragment`), `ggml/src/ggml-cpu/repack.cpp:4727` (repack-at-load hook) at `237ad9b96` | L1: dense-arch (`qwen35`) GGUF path on main via the registry — [dense GGUF load](../src/vllm/model_executor/models/qwen3_5_dense.cpp#L60), [arch->registered-ID map](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L212), [F16/BF16 row dequant](../src/vllm/model_executor/model_loader/gguf_dequant.cpp#L61). L2: [block residency `OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) — raw ggml blocks into an `OwnedTensor` with a block `vt::DType`, file `[N,K]` orientation, `nk=true`, no transpose; stacked experts split by byte range. L3: [routing policy + `VT_CPU_REF`/`VT_GGUF_KEEP_QUANT`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L1) (6 roles, no `default:` label so an unrouted role is a `-Werror=switch` build failure) wired at every loader call site via [`OwnMatmulWeight`/`RequireExpand`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L185). **Default now device-derived** (CIQ G4): [`GgufQuantComputeAvailable`](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) gates it on `vt::OpRegistered(kMatmulBTQuant, CurrentPlatform().device_type())`, and the same condition drives `expand_nk`, which stops transposing a weight that must expand. **`expand_nk` now also covers the GDN split projections** (2026-07-23, `CLAIM-CPU-GDN-ORIENT-1`): a fresh op-dispatch profile found `LoadGdnGguf`'s `in_proj_qkv/z/b/a` + `out_proj` were the ONE expanded weight family still transposed to [K,N] (nk=false → slow `kMatmul`, 17.9 % of prefill); the new [`gdn_expand_nk` field](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L95) + [`MakeGdnProj`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L131) keep them [N,K] nk=true (V-head reorder applied first, orthogonal to orientation; `VT_GGUF_GDN_NK=0` A/B opt-out) → M-blocked `kMatmulBT`, same-binary prefill **1.090×** / decode 1.09×, byte-identical (`kMatmul` 72→0 calls in prefill) | [L2/L3 units](../tests/vllm/test_gguf_keep_quant.cpp#L1) — 17 cases / 5,574 assertions green. **Gate 1 (losslessness) proven PER ENCODING**, one case each for Q4_0/Q8_0/Q3_K/Q4_K/Q5_K/Q6_K: resident bytes `memcmp`-equal to the file span and resident-block dequant BYTE-IDENTICAL to the direct-from-file expansion (f32 and bf16), over pseudo-random block bytes constrained only to finite f16 scales; at loader level the kept weight rehydrates to the expanded `[K,N]` bf16 tensor byte for byte, per weight and per expert, on dense and MoE fixtures. **Totality**: the audit hook proves `routed == the file's complete tensor list` on both fixtures, plus 6 roles × 12 encodings × 6 shapes against a LONGHAND expectation (12 keep / 420 expand, so neither outcome is vacuous). **Gate 2 (oracle stability)**: `VT_CPU_REF=1` keeps nothing quantized and every weight is bit-identical to the historical load; on dgx [`test_qwen36_gguf_engine`](../tests/parity/test_qwen36_gguf_engine.cpp#L143) under `VT_CPU_REF=1` is 28/28 assertions, 16/16 tokens on both APEX files — same as without. 10-mutant battery, 10 caught (the expert-slice-offset mutant survived the first pass, exposed a real coverage hole, and drove the MoE fixture). DGX (each gate STANDALONE, production flags, goldens md5 identical before/after `2965ef5772b556d3f3f86fedf4221b2f`): clean CUDA `-Werror` 0 warnings + regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16 on both, OPT 6/6, DeepSeek-V2 8/8) + gguf units green on aarch64 with identical counts; full CPU ctest 154/154. **RSS at G4 was 6.401 GiB (2.29x); L5 took it to 3.884 GiB (1.39x)** — binding, idle dgx aarch64, same-binary 3-rep A/B: mmap in-place residency (borrow kept q8_0 blocks out of the mapping, refcounted, -0.998 GiB), tied-head sharing (one bf16 vocab matrix for embed+lm_head, -0.946 GiB), read-once page release (MADV_DONTNEED the expanded tensors' file pages, port of llama.cpp `unmap_fragment`, -0.573 GiB). Decode TPOT 41.7 ms UNCHANGED, prefill TTFT +4% (first-touch faults move into the timed window), output md5 `d235db12f2cd304007530286a1755c95` identical across BEFORE/AFTER/ORACLE. Lifetime safety tested explicitly (borrow outlives the GgufFile AND the on-disk file; shared head freed once either order). **L6 (2026-07-23, `CLAIM-QUANT-GGUF-KEEPF16-L6-1`) implemented keep-f16 residency and REFUTED the "remaining gap is the f16 expansion" attribution above.** New `kKeepF16` residency + [`OwnGgufF16`/`OwnGgufKeptSlice`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L79) keep F16 matmul weights (+ F16 embed/tied head, one shared f16 vocab matrix via [`OwnedBytes::KeepAlive`](../include/vllm/model_executor/models/owned_bytes.h)) resident as F16, consumed by the elementwise f16 GEMM. Binding A/B: peak RSS 3.884 → **3.832 GiB (−52 MB, RSS-NEUTRAL)** — L5's page-release ALREADY dropped the f16 file pages, so keep-f16 only swaps an anonymous bf16 buffer for equal-size file-backed f16 pages. smaps attribution: keep-f16 file-backed **2.634 GiB ≈ llama.cpp's 2.68 file** (weight residency AT PARITY), anon 1.20 GiB; the **remaining ~1.08 GiB gap is the engine's ANONYMOUS activation/KV workspace, NOT weights** — the real, separate CPU RSS lever. Also regresses prefill (TTFT 577 → ~1000 ms, first-touch faults into the timed window; decode at parity). Tokens byte-identical (md5 `d235db1…`). Ships DEFAULT OFF at L6. **L7 (2026-07-23, `CLAIM-QUANT-GGUF-RSS-L7-1`) REVERSED L6's refutation and CLOSED the CPU RSS gap to 1.01× llama.cpp.** The profile disproved the "workspace" attribution — DevicePool 20 MiB, whole KV 115 MiB, both ≤ llama.cpp. The 1 GiB residual was a q8_0 repack-source DOUBLE-COUNT: on aarch64 the G7 repack COPIES q8_0 into an anonymous buffer while the f16 borrows keep the mapping alive, so the DEAD source blocks stay file-backed. [`OwnGgufQuantBlocks`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L20) now `DropSpanResidency`es the repack source (port of llama.cpp `unmap_fragment`), and [`PrefaultBorrowedSpan`](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L21) faults borrowed weights at load (port of llama.cpp mmap prefetch), removing L6's prefill regression — so [keep-f16 flips DEFAULT ON](../src/vllm/model_executor/model_loader/gguf_keep_quant.cpp#L168) (`VT_GGUF_KEEP_F16=0` opt-out). Binding A/B (idle dgx aarch64, base-vs-L7 same-binary): peak RSS **3.884 → 2.832 GiB = 1.39× → 1.01× llama.cpp** (File 2.632 → 1.629, the released q8_0 source; anon 1.200 unchanged), prefill **1.18× AHEAD** (204 vs pp128 173.2, denominator SUPERSEDED by #1003), decode ~parity (24.4 vs 25.09), tokens BYTE-IDENTICAL (md5 `809f2d0…` base/L7/oracle). **Against our own keep-f16-off arm the default costs about 9% of prefill (224 → 204 t/s) and about 1.4% of decode (TPOT 40.4 → 40.95 ms) for 1.05 GiB, settled 2026-08-17 as a product decision, NOT by the competitor floor.** Anon 1.200 GiB is IRREDUCIBLE (repacked q8_0 1.06 + KV 0.115 + pool 0.02). Regressions UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6, DeepSeek-V2 8/8, Llama 16/16, GGUF engine 28/28); `test_gguf_keep_quant` 36/36 (+1 L7 prefault byte-transparency case, x86+aarch64) | [keep-quant loader leaf](specs/gguf-keep-quant-loader.md) | `ANCHOR-BACKFILL` | `CLAIM-QUANT-GGUF-RSS-L7-1` | | `QUANT-QWEN38-27B-GGUF-ARM` | The `Qwen3.8-27B-Q4_K_M.gguf` arm end to end: tensor accounting, text decode, the multimodal legs, and this ARTIFACT's own tokenizer and chat template. The standing GGUF k-quant requirement for a model whose bf16 arm is already gated ([#915](https://github.com/mudler/vllm.cpp/issues/915)), and the arm `BACKEND-GATE-CUDA-LLAMACPP` in the [backend matrix](backend-matrix.md) is already recorded as blocked on. **Header-verified 2026-08-18** at `unsloth/Qwen3.8-27B-GGUF`@`fe1e2a23d973adb629709749dc4f6756df66ef10`: GGUF v3, arch `qwen35`, 866 tensors, F32 456 / Q4_K 294 / Q6_K 67 / Q5_K 48 / Q8_0 1, data end == file size 17,106,775,008. **Two facts [#821](https://github.com/mudler/vllm.cpp/issues/821) did not record and which change the scope:** `qwen35.block_count = 65` with `qwen35.nextn_predict_layers = 1`, so block 64 is the MTP/`nextn` DRAFTER (`blk.64.nextn.{eh_proj,enorm,hnorm,shared_head_norm}` plus a full-attention block and an FFN) — exactly the 15-tensor difference from the same model's 851-tensor BF16 GGUF, and a loader that reads `block_count` as decoder depth builds a 65-layer model out of a 64-layer checkpoint plus a drafter; and `tokenizer.ggml.padding_token_id = 248055` against 248044 in the BF16 GGUF and `null` in the official HF config, which is why the tokenizer gate belongs to the ARM. NOT blocked on kernels: every dtype this file carries is already computed natively on BOTH tiers. The CUDA tier really has no prefill/decode split (`LaunchGemm` [cuda_quant_dot.cu:1609](../src/vt/cuda/cuda_quant_dot.cu#L1609) sizes its grid `m*n` and the encoding switch at [:1864](../src/vt/cuda/cuda_quant_dot.cu#L1864) never sees `M`); the CPU tier DOES branch on `M` at [cpu_quant_gemm.cpp:190](../src/vt/cpu/cpu_quant_gemm.cpp#L190), which takes the Arm i8mm `mmla` 2x2 tile only for even `M` and `N` and sends decode (`M=1`) to the portable `nrc==1` path. That is a kernel-TIER split, NOT a coverage split -- no dtype gains or loses support at any `M`, both arms end in the same `BlockVecDot` table -- so the conclusion stands and it is a W3 speed fact rather than a W2 gap | llama.cpp `b10451` = `10bf611e5` ([pin](oracles/llama-cpp.md), **`gateable = yes`** since [#857](https://github.com/mudler/vllm.cpp/issues/857) landed 2026-08-22) is the arm's ORACLE and its only comparator — at the vLLM pin `555967922` there is no in-tree GGUF reader (`6635279d8` moved it out of tree) and SGLang's alias table does not reach `qwen3_5` ([#979](https://github.com/mudler/vllm.cpp/issues/979)). llama.cpp is never the MIRROR | the single-file GGUF entry [qwen3_5_gguf_weights.cpp:1474](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L1474); the CUDA quant enum [cuda_quant_dot.cu:700](../src/vt/cuda/cuda_quant_dot.cu#L700) and CPU [cpu_quant_dot.cpp:787](../src/vt/cpu/cpu_quant_dot.cpp#L787) already cover Q4_K/Q5_K/Q6_K, and Q8_0 has its own path [cuda_quant_dot.cu:1659](../src/vt/cuda/cuda_quant_dot.cu#L1659) | **W2 LANDED the accounting**, modelled on [muse_glimmer_gguf_manifest.inc](../tests/vllm/models/muse_glimmer_gguf_manifest.inc): committed header-only manifests [qwen38_27b_q4km_gguf_manifest.inc](../tests/vllm/models/qwen38_27b_q4km_gguf_manifest.inc) (866 names, 51 kv) and [qwen38_27b_mmproj_gguf_manifest.inc](../tests/vllm/models/qwen38_27b_mmproj_gguf_manifest.inc) (334 names, 35 kv), generated by [gen-qwen38-27b-gguf-manifest.py](../scripts/gen-qwen38-27b-gguf-manifest.py) from the mirrored bytes; the accounting gate [test_qwen38_27b_gguf_manifest.cpp:223](../tests/vllm/models/test_qwen38_27b_gguf_manifest.cpp#L223) (6 cases, 464 assertions hermetic, 4745 over the shipped bytes under `VLLM_CPP_QWEN38_27B_{GGUF,MMPROJ}`, ZERO unaccounted in BOTH directions on both files); and the reachability gate [test_gguf_accounting_reach.cpp:184](../tests/vllm/entrypoints/test_gguf_accounting_reach.cpp#L184) (6 cases, 22 assertions), which enters through `LoadedEngine::FromModelDir` and reds 3/6 when either refusal call site in `model_loader.cpp` is deleted while the manifest target stays green at 6/6. The `nextn` correction was a gap that DID NOT EXIST: [qwen3_5_gguf_weights.cpp:889](../src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp#L889) has taken `block_count - nextn_predict_layers` since `1a4db5c3c`, and `mtp_num_hidden_layers` has been republished since `493327b4e`; what was missing was a gate, because [test_qwen3_5_gguf_mtp.cpp:36](../tests/vllm/models/test_qwen3_5_gguf_mtp.cpp#L36) is asset-gated on `VLLM_MTP_GGUF_MODEL`, skips SILENTLY when it is unset, and checks only `num_hidden_layers > 0` rather than the arithmetic its own comment claims. **W3 RAN THE TOKEN GATE ON 2026-08-23 AND IT FAILED.** Two `rc run` jobs on `thor:gpu0` (`64f66cda`, `8e0d8e54`), same GGUF file both sides, greedy, 48 tokens, concurrency 1, MTP OFF so both engines decode the same 851 tensors and the same 64-layer trunk (llama.cpp ignores all 15 of `blk.64`, re-observed as exactly 15 `unused tensor` warnings). **Tokenizer EXACT 6/6** through three of our paths (`examples/tokenize`, `vllm-cli` prompt counts, and the agreeing generation prefixes), so the #1355 prompt-token undercount is absent here. **Generation DIVERGES 5/6**, first differing index 7/34/20/-/14/32 with prompt 3 token-exact 48/48. Teacher-forcing the oracle along OUR ids over all 288 steps puts our token at the oracle's **rank 1 on 282 and rank 2 on 6, never rank 3 or worse**, losing by 0.027-0.178 logits against absolute logits of 15.9-22.6 - a PRECISION difference in the quantized compute path, not a wiring defect. The near-tie band was NOT reached for: the oracle's greedy decode reproduced #857's text byte for byte from a different build, so it is deterministic and the band's premise fails. No speed or memory number is admissible from this arm; resident bytes were measured only to refuse a dequant hypothesis (ours 24.997 GiB vs the oracle's 30.917 GiB on the same box and file, so NO dequant-to-bf16 blow-up). **2026-09-02, the cause is FOUND and PARTLY FIXED and the gate still FAILS:** our final logits carried only bf16 RESOLUTION (288 of 288 top-1 logits exactly on the bf16 grid, ULP 0.125 at magnitude 16-32, against contested gaps of 0.027-0.178), because a GGUF keep-quant head reached the bf16-output logits helper on the `nk` LAYOUT flag. Routing a block-quant head to the f32-output GEMM takes the arm from **5 of 6 to 3 of 6** divergent prompts, measured as one tree built twice on `thor:gpu0` (`c0b3fc6d`) whose bf16 arm reproduces 2026-08-23 index for index. The two SMALLEST margins (0.027185, 0.058050) resolved; the three largest (0.085434, 0.115482, 0.178236) did not, at the same indices with the same agreeing prefixes, so the residual is a MAGNITUDE term rather than a resolution one. `TOKEN_GATE` stays `FAIL` and no speed or memory axis becomes admissible. [Evidence](../docs/bench-evidence/qwen38-27b-q4km-logits-f32-20260902.md), and the superseded 5-of-6 run [Evidence](../docs/bench-evidence/qwen38-27b-q4km-token-gate-20260823.md) | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | -| `QUANT-QWEN38-27B-NVFP4-ARM` | The `unsloth/Qwen3.8-27B-NVFP4` artifact, which is **not what its name says**. **Its pinned revision is GONE:** `a767244d27bd76589a3e3b2ab4e64032c4ebc7af`, the revision [#821](https://github.com/mudler/vllm.cpp/issues/821) names, answers HTTP 404 and `git ls-remote` reports one ref, `refs/heads/main` = `7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108` — the second in-place re-quantization this publisher has done in this family, after `unsloth/Qwen3.6-27B-NVFP4`. So the user-reported load failure on #821 is CORROBORATED at a different revision, never reproduced. At the live revision (header-verified 2026-08-18, 1953 + 15 tensors, `8 + header_len + max(data_offsets[1])` == file size 22,568,192,096) `quantization_config.format` is `mixed-precision`: `group_0` is FP8 W8A8 with **per-CHANNEL** weight scales and **DYNAMIC per-token** activations over `self_attn.(q\|k\|v\|o)_proj`, `linear_attn.(in_proj_qkv\|in_proj_z\|out_proj)`, `lm_head` and `layers.(56..63).mlp.*`; `group_1` is `nvfp4-pack-quantized` W4A4 over the remaining `mlp.*`; plus an 8-bit static `kv_cache_scheme` and an `ignore` list of **303 entries** -- not just the vision tower: 48 x `linear_attn`, `linear_attn.norm`, `linear_attn.in_proj_b` and `linear_attn.in_proj_a` (the GDN layer count), 27 x 4 vision blocks, 2 mergers, and `re:^mtp.*`. That list is what makes the predicate claim provable rather than asserted: `in_proj_a`/`in_proj_b` are IGNORED while `in_proj_qkv`/`in_proj_z`/`out_proj` are `group_0` TARGETS, so a resolver that reads the groups but not the `ignore` list gets the GDN block wrong in both directions. **`*.input_scale` appears ZERO times in the checkpoint.** Four independent blockers, each anchored in the spec: the unconditional `.input_scale` read, a per-channel BF16 `weight_scale` that `ReadF32Scalar` refuses on BOTH count and dtype, no representation for a dynamic per-token activation scheme, and a scheme that is never read from the config at all. The NVFP4 half is the half CLOSEST to working; the FP8 tower is the blocker. **A SECOND artifact of the same model is now in scope and it is a DIFFERENT FORMAT:** `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121`@`36f717a22990e82c54c1d48ee77c491b87825680`, needed by campaign [#1574](https://github.com/mudler/vllm.cpp/issues/1574), declares `quant_method: "modelopt"` and `quant_algo: "MIXED_PRECISION"` with a `quantized_layers` map of 401 EXACT module names, an EMPTY `ignore`, per-TENSOR STATIC FP8 (`weight_scale` and `input_scale` both `F32 []`, 208 modules) and ModelOpt-spelled W4A16_NVFP4 g16 weight-only (`weight` U8 + `weight_scale` F8_E4M3 + `weight_scale_2` `F32 []`, 193 modules including `lm_head`) over ALL 64 layers' MLP -- no layer-56 boundary. Header-verified 2026-08-21 over all four shards (970/976/40/15 = 2001 names; `8 + header_len + max(data_offsets[1])` == the size the hub reports for each; the four sizes exceed the index's `metadata.total_size` 21,921,427,300 by exactly the four headers plus their 8-byte prefixes). **NONE of W4's four blockers applies to it** -- they were properties of the unsloth artifact, not of the format -- so both halves LOAD. The fifth blocker W5 found instead: `ct::Config` stops at `quant_method != "compressed-tensors"`, so nothing in this tree read this config at all and every routing decision fell to the per-projection tensor-NAME probe, which is wrong in BOTH directions silently. `nvidia/Qwen3.6-27B-NVFP4`@`0893e160`, the #466 gate model, is the same ModelOpt shape (2194 names, same 208/193 split, `exclude_modules` `["mtp*","mtp.layers.0*"]`, a `kv_cache_scheme` with ZERO scales shipped) and reaches the same call site, which is why W5 refuses a DISAGREEMENT rather than routing by the declaration ([#1597](https://github.com/mudler/vllm.cpp/issues/1597) owes that) | vLLM `555967922` is the MIRROR and the primary oracle — it runs this format, so nothing here may diverge from its compressed-tensors semantics. A generic mixed-precision resolver already exists at [modelopt_mixed_precision.h](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h). Through W4 **no production file included it** -- the only two includes in the tree were its own two tests, and `nemotron_h_weights.cpp` (which this row previously named) includes `nemotron_h.h`, `nemotron_h_loader.h`, `nvfp4_dequant.h` and `vt/unaligned.h` and reads its quant config inline; a 33,575-byte header reachable only from tests is an `AGENTS.md` §"Nothing lands dead" item. W4 did NOT adopt it and argued the exception: the two headers resolve two DIFFERENT upstream formats that share only the English word "mixed", and its artifact is the compressed-tensors one. **W5 is the FIRST production wiring**, for the artifact that really is ModelOpt: [qwen3_5_dense_weights.cpp](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp) now includes it and `LoadQwen3_5Dense` calls [RefusalForQuantizationConfig](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h) once per checkpoint. So the tree ends with ONE resolver per format, which is upstream's own structure | the failing read [qwen3_5_weights.cpp:642](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L642) (`:457` was WRONG and is `OwnedBytes::Borrow`; re-derived by W4), the resolution [compressed_tensors_config.h:354](../src/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_config.h#L354) (`Resolve`, ignore-first then first-matching target) and its production call site [qwen3_5_dense_weights.cpp:875](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L875), its refusal [dense_weight_loaders.h:164](../include/vllm/model_executor/models/dense_weight_loaders.h#L164) (the count check at `:168`, the dtype check at `:172`; #1258 moved `ReadF32Scalar` down 63 lines and `:101` is now an unrelated `try`/`catch` probe), the scalar-only `Fp8Weight` [qwen3_5_weights.h:732](../include/vllm/model_executor/models/qwen3_5_weights.h#L732), the static activation quant [qwen3_5.cpp:3629](../src/vllm/model_executor/models/qwen3_5.cpp#L3629), and the stale predicate [qwen3_5_dense_weights.cpp:699](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L699) (the `.linear_attn.in_proj_` early-false at `:702`) that declares the GDN input projections never quantized — true for the 3.6 unsloth artifact ([hf_snapshot.h:287](../tests/parity/hf_snapshot.h#L287)) and false for this one | W4: [test_qwen38_27b_nvfp4_arm.cpp:208](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L208) (per-scheme composition), [:582](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L582) (the FP8 refusal through `LoadQwen3_5Dense`), [:666](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L666) (the env-gated live re-read); 8 cases / 190 assertions hermetic and 204 with `VLLM_CPP_QWEN38_27B_NVFP4_DIR` set, over the committed [qwen38_27b_nvfp4_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_manifest.inc) (1953) and [qwen38_27b_nvfp4_mtp_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_mtp_manifest.inc) (15), summing to the index's 1968; per-scheme composition 466/672/475/323/32 tensors over 233/168/317/267/16 modules, zero unclassified. Manifest-capture precedent [minimax_h3_nvfp4_manifest.inc](../tests/vllm/models/minimax_h3_nvfp4_manifest.inc), captured the same way this row's numbers were: an HTTP range read of the file's own header. W5: [test_qwen38_27b_modelopt_mtp_arm.cpp](../tests/vllm/models/test_qwen38_27b_modelopt_mtp_arm.cpp), 22 cases / 1687 assertions hermetic, over four committed header-only manifests ([s1](../tests/vllm/models/qwen38_27b_modelopt_mtp_s1_manifest.inc) 970, [s2](../tests/vllm/models/qwen38_27b_modelopt_mtp_s2_manifest.inc) 976, [s3](../tests/vllm/models/qwen38_27b_modelopt_mtp_s3_manifest.inc) 40, [s4](../tests/vllm/models/qwen38_27b_modelopt_mtp_s4_manifest.inc) 15) summing to the index's 2001; per-scheme composition 720/579/702 tensors over 256/193/536 modules with zero KV scales and zero unclassified, and the 256 split as 208 `kDirect` Linears plus 48 `kPrefix` `linear_attn` CONTAINERS so the count cannot be right for the wrong reason; 937 weight-bearing modules split 208/193/536. RED before wiring was 5 cases failing with an EMPTY refusal -- the loader accepted every config/tensor disagreement and both unloadable algorithms. Twelve negative mutations, every one detected, including the deleted production call site (8 cases red), a suffix list without `.weight_scale_2` (7), one manifest row removed with its count literal left behind (5, and all 22 cases still RUN -- deriving the row count with `std::size` turned what was an out-of-bounds read into an ordinary red), the NVFP4 refusal branch disabled (2) and an unseen operand family skipped rather than refused (2). The fresh review found the last two: the NVFP4 branch is the whole cross-check for 193 of the 401 declared modules and `if (false && ...)` on it left the suite fully green, and `Refusal` skipped a tensor name whose family `SplitOperand` has never seen, which its own contract forbids reading as unquantized. A separate case pins that the refusal is SILENT on the `nvidia/Qwen3.6-27B-NVFP4` shape -- wildcard `exclude_modules`, an `input_scale` on every NVFP4 module, and a `kv_cache_scheme` with zero scales -- and its own mutation reds it while every other case stays green. W4's gate is untouched and re-ran 9 cases / 194 assertions green on the same tree. The token gate is PENDING on [#1632](https://github.com/mudler/vllm.cpp/issues/1632): the pinned oracle DOES run a model inside an `rc` lease (2026-08-18), at `max_num_batched_tokens` 512 against a recorded denominator of 8192, and this artifact's ~20.4 GiB are not staged where a lease can read them. It supersedes #1185, closed 2026-08-18 as local-only | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | +| `QUANT-QWEN38-27B-NVFP4-ARM` | The `unsloth/Qwen3.8-27B-NVFP4` artifact, which is **not what its name says**. **Its pinned revision is GONE:** `a767244d27bd76589a3e3b2ab4e64032c4ebc7af`, the revision [#821](https://github.com/mudler/vllm.cpp/issues/821) names, answers HTTP 404 and `git ls-remote` reports one ref, `refs/heads/main` = `7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108` — the second in-place re-quantization this publisher has done in this family, after `unsloth/Qwen3.6-27B-NVFP4`. So the user-reported load failure on #821 is CORROBORATED at a different revision, never reproduced. At the live revision (header-verified 2026-08-18, 1953 + 15 tensors, `8 + header_len + max(data_offsets[1])` == file size 22,568,192,096) `quantization_config.format` is `mixed-precision`: `group_0` is FP8 W8A8 with **per-CHANNEL** weight scales and **DYNAMIC per-token** activations over `self_attn.(q\|k\|v\|o)_proj`, `linear_attn.(in_proj_qkv\|in_proj_z\|out_proj)`, `lm_head` and `layers.(56..63).mlp.*`; `group_1` is `nvfp4-pack-quantized` W4A4 over the remaining `mlp.*`; plus an 8-bit static `kv_cache_scheme` and an `ignore` list of **303 entries** -- not just the vision tower: 48 x `linear_attn`, `linear_attn.norm`, `linear_attn.in_proj_b` and `linear_attn.in_proj_a` (the GDN layer count), 27 x 4 vision blocks, 2 mergers, and `re:^mtp.*`. That list is what makes the predicate claim provable rather than asserted: `in_proj_a`/`in_proj_b` are IGNORED while `in_proj_qkv`/`in_proj_z`/`out_proj` are `group_0` TARGETS, so a resolver that reads the groups but not the `ignore` list gets the GDN block wrong in both directions. **`*.input_scale` appears ZERO times in the checkpoint.** Four independent blockers, each anchored in the spec: the unconditional `.input_scale` read, a per-channel BF16 `weight_scale` that `ReadF32Scalar` refuses on BOTH count and dtype, no representation for a dynamic per-token activation scheme, and a scheme that is never read from the config at all. The NVFP4 half is the half CLOSEST to working; the FP8 tower is the blocker. **A SECOND artifact of the same model is now in scope and it is a DIFFERENT FORMAT:** `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121`@`36f717a22990e82c54c1d48ee77c491b87825680`, needed by campaign [#1574](https://github.com/mudler/vllm.cpp/issues/1574), declares `quant_method: "modelopt"` and `quant_algo: "MIXED_PRECISION"` with a `quantized_layers` map of 401 EXACT module names, an EMPTY `ignore`, per-TENSOR STATIC FP8 (`weight_scale` and `input_scale` both `F32 []`, 208 modules) and ModelOpt-spelled W4A16_NVFP4 g16 weight-only (`weight` U8 + `weight_scale` F8_E4M3 + `weight_scale_2` `F32 []`, 193 modules including `lm_head`) over ALL 64 layers' MLP -- no layer-56 boundary. Header-verified 2026-08-21 over all four shards (970/976/40/15 = 2001 names; `8 + header_len + max(data_offsets[1])` == the size the hub reports for each; the four sizes exceed the index's `metadata.total_size` 21,921,427,300 by exactly the four headers plus their 8-byte prefixes). **NONE of W4's four blockers applies to it** -- they were properties of the unsloth artifact, not of the format -- so both halves LOAD. The fifth blocker W5 found instead: `ct::Config` stops at `quant_method != "compressed-tensors"`, so nothing in this tree read this config at all and every routing decision fell to the per-projection tensor-NAME probe, which is wrong in BOTH directions silently. `nvidia/Qwen3.6-27B-NVFP4`@`0893e160`, the #466 gate model, is the same ModelOpt shape (2194 names, same 208/193 split, `exclude_modules` `["mtp*","mtp.layers.0*"]`, a `kv_cache_scheme` with ZERO scales shipped) and reaches the same call site, which is why W5 refuses a DISAGREEMENT rather than routing by the declaration ([#1597](https://github.com/mudler/vllm.cpp/issues/1597) owes that) | vLLM `555967922` is the MIRROR and the primary oracle — it runs this format, so nothing here may diverge from its compressed-tensors semantics. A generic mixed-precision resolver already exists at [modelopt_mixed_precision.h](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h). Through W4 **no production file included it** -- the only two includes in the tree were its own two tests, and `nemotron_h_weights.cpp` (which this row previously named) includes `nemotron_h.h`, `nemotron_h_loader.h`, `nvfp4_dequant.h` and `vt/unaligned.h` and reads its quant config inline; a 33,575-byte header reachable only from tests is an `AGENTS.md` §"Nothing lands dead" item. W4 did NOT adopt it and argued the exception: the two headers resolve two DIFFERENT upstream formats that share only the English word "mixed", and its artifact is the compressed-tensors one. **W5 is the FIRST production wiring**, for the artifact that really is ModelOpt: [qwen3_5_dense_weights.cpp](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp) now includes it and `LoadQwen3_5Dense` calls [RefusalForQuantizationConfig](../src/vllm/model_executor/layers/quantization/modelopt_mixed_precision.h) once per checkpoint. So the tree ends with ONE resolver per format, which is upstream's own structure | the failing read [qwen3_5_weights.cpp:642](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L642) (`:457` was WRONG and is `OwnedBytes::Borrow`; re-derived by W4), the resolution [compressed_tensors_config.h:354](../src/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_config.h#L354) (`Resolve`, ignore-first then first-matching target) and its production call site [qwen3_5_dense_weights.cpp:875](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L875), its refusal [dense_weight_loaders.h:164](../include/vllm/model_executor/models/dense_weight_loaders.h#L164) (the count check at `:168`, the dtype check at `:172`; #1258 moved `ReadF32Scalar` down 63 lines and `:101` is now an unrelated `try`/`catch` probe), the scalar-only `Fp8Weight` [qwen3_5_weights.h:747](../include/vllm/model_executor/models/qwen3_5_weights.h#L747), the static activation quant [qwen3_5.cpp:3629](../src/vllm/model_executor/models/qwen3_5.cpp#L3629), and the stale predicate [qwen3_5_dense_weights.cpp:699](../src/vllm/model_executor/models/qwen3_5_dense_weights.cpp#L699) (the `.linear_attn.in_proj_` early-false at `:702`) that declares the GDN input projections never quantized — true for the 3.6 unsloth artifact ([hf_snapshot.h:287](../tests/parity/hf_snapshot.h#L287)) and false for this one | W4: [test_qwen38_27b_nvfp4_arm.cpp:208](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L208) (per-scheme composition), [:582](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L582) (the FP8 refusal through `LoadQwen3_5Dense`), [:666](../tests/vllm/models/test_qwen38_27b_nvfp4_arm.cpp#L666) (the env-gated live re-read); 8 cases / 190 assertions hermetic and 204 with `VLLM_CPP_QWEN38_27B_NVFP4_DIR` set, over the committed [qwen38_27b_nvfp4_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_manifest.inc) (1953) and [qwen38_27b_nvfp4_mtp_manifest.inc](../tests/vllm/models/qwen38_27b_nvfp4_mtp_manifest.inc) (15), summing to the index's 1968; per-scheme composition 466/672/475/323/32 tensors over 233/168/317/267/16 modules, zero unclassified. Manifest-capture precedent [minimax_h3_nvfp4_manifest.inc](../tests/vllm/models/minimax_h3_nvfp4_manifest.inc), captured the same way this row's numbers were: an HTTP range read of the file's own header. W5: [test_qwen38_27b_modelopt_mtp_arm.cpp](../tests/vllm/models/test_qwen38_27b_modelopt_mtp_arm.cpp), 22 cases / 1687 assertions hermetic, over four committed header-only manifests ([s1](../tests/vllm/models/qwen38_27b_modelopt_mtp_s1_manifest.inc) 970, [s2](../tests/vllm/models/qwen38_27b_modelopt_mtp_s2_manifest.inc) 976, [s3](../tests/vllm/models/qwen38_27b_modelopt_mtp_s3_manifest.inc) 40, [s4](../tests/vllm/models/qwen38_27b_modelopt_mtp_s4_manifest.inc) 15) summing to the index's 2001; per-scheme composition 720/579/702 tensors over 256/193/536 modules with zero KV scales and zero unclassified, and the 256 split as 208 `kDirect` Linears plus 48 `kPrefix` `linear_attn` CONTAINERS so the count cannot be right for the wrong reason; 937 weight-bearing modules split 208/193/536. RED before wiring was 5 cases failing with an EMPTY refusal -- the loader accepted every config/tensor disagreement and both unloadable algorithms. Twelve negative mutations, every one detected, including the deleted production call site (8 cases red), a suffix list without `.weight_scale_2` (7), one manifest row removed with its count literal left behind (5, and all 22 cases still RUN -- deriving the row count with `std::size` turned what was an out-of-bounds read into an ordinary red), the NVFP4 refusal branch disabled (2) and an unseen operand family skipped rather than refused (2). The fresh review found the last two: the NVFP4 branch is the whole cross-check for 193 of the 401 declared modules and `if (false && ...)` on it left the suite fully green, and `Refusal` skipped a tensor name whose family `SplitOperand` has never seen, which its own contract forbids reading as unquantized. A separate case pins that the refusal is SILENT on the `nvidia/Qwen3.6-27B-NVFP4` shape -- wildcard `exclude_modules`, an `input_scale` on every NVFP4 module, and a `kv_cache_scheme` with zero scales -- and its own mutation reds it while every other case stays green. W4's gate is untouched and re-ran 9 cases / 194 assertions green on the same tree. The token gate is PENDING on [#1632](https://github.com/mudler/vllm.cpp/issues/1632): the pinned oracle DOES run a model inside an `rc` lease (2026-08-18), at `max_num_batched_tokens` 512 against a recorded denominator of 8192, and this artifact's ~20.4 GiB are not staged where a lease can read them. It supersedes #1185, closed 2026-08-18 as local-only | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | | `QUANT-GGUF-PRESETS` | Representative mixed-file gates for every llama.cpp output preset family | llama.cpp `tools/quantize/quantize.cpp:34-74` | only custom APEX mixed files are executable; no general preset dispatch | [APEX gates](../tests/parity/test_qwen36_gguf_engine.cpp#L143) do not prove llama.cpp preset breadth | [coverage spike](specs/quantization-coverage.md); split exact preset IDs before `READY` | `INVENTORIED` | - | ## 1. llama.cpp / GGUF encodings diff --git a/.agents/specs/rocm-host-residency-after-upload.md b/.agents/specs/rocm-host-residency-after-upload.md index 3ebabe89a..64d078066 100644 --- a/.agents/specs/rocm-host-residency-after-upload.md +++ b/.agents/specs/rocm-host-residency-after-upload.md @@ -28,7 +28,25 @@ weight is copied to the device once at load and the host pages are never read again. There is nothing to keep warm. **Two. Nothing releases the source pages after the device upload.** -`ResidentWeight` (`src/vllm/model_executor/models/qwen3_5.cpp`) stages the weight + +THERE ARE TWO `ResidentWeight`s AND THIS ROW'S MODEL USES THE OTHER ONE. The +first draft of this spec named only the `ResidentWeight` in the unnamed namespace +of `src/vllm/model_executor/models/qwen3_5.cpp`, which is true of Qwen3.5 and +false of `qwen4_exp`. That function shadows the header one inside its own +translation unit, so it serves the Qwen3.5 dense weights and — through +`KqResidentSlice` and `KqGrouped` — the shared MoE seam's keep-quant expert +towers, which `qwen4_exp` does reach via `RunQwen4ExpMoeBlock` -> +`RunMoeBlock` -> `MoeBlock`. Everything else in `qwen4_exp` stages through +`dense_attn::ResidentWeight` +(`include/vllm/model_executor/models/dense_attn_block.h`): 12 call sites in +`qwen4_exp_forward.cpp`, 10 in `qwen4_exp_qsa_block.cpp`, 7 in +`qwen4_exp_ple_block.cpp`, 1 in `qwen4_exp_registry.cpp`, and not one reference +to the `qwen3_5.cpp` function anywhere in that model. Fixing one arm and not the +other was measured, not argued: with the release in `qwen3_5.cpp` alone, the load +peak on `strix:gpu0` fell from 27.67 GB to 8.18 GB and host `RssFile` still +climbed to 21.08 GB during the forward and stayed there. + +BOTH arms have the same body and the same defect. Each stages the weight with `Alloc` + `Copy` and then calls `AdoptDeviceBytesAsHost`, which returns immediately for a GGUF borrow. `OwnedTensor::ReleaseHost()` likewise refuses to `madvise` a BORROWED buffer, arguing that the pages are clean and file-backed and @@ -91,7 +109,9 @@ during load and zero after it. Ours is O(whole model) for the process lifetime. ### Fix 1 — release the source pages after a staging upload -In `ResidentWeight`'s staging arm, once the device copy exists and has completed, +In BOTH `ResidentWeight` staging arms — the one in `qwen3_5.cpp`'s unnamed +namespace and `dense_attn::ResidentWeight` in `dense_attn_block.h`, which is the +one `qwen4_exp` takes — once the device copy exists and has completed, drop the resident interior pages of a BORROWED, file-backed source span when the platform is NOT host-addressable. The borrow itself is untouched and stays a valid, re-faultable `PROT_READ MAP_PRIVATE` view, so a later read re-faults from @@ -106,10 +126,12 @@ times per forward step on this checkpoint. A release that re-tests its condition on every call would `MADV_DONTNEED` the very pages the GPU is about to read, on every step, and the kernel would fault them straight back in. Correctness survives that; throughput does not. The release therefore goes inside -`AdoptDeviceBytesAsHost`, which is reached only from behind `if (!w.d_dev)` — the -same memo that made the aligned-borrow branch of `MakeHostBytesDeviceAliasable` -a repeat hazard when it had none. The red test covers the REPEAT call, not only -the first. +`if (!w.d_dev)`, immediately before the `AdoptDeviceBytesAsHost` call and NOT +inside it — the same memo that made the aligned-borrow branch of +`MakeHostBytesDeviceAliasable` a repeat hazard when it had none. It is a separate +helper because `AdoptDeviceBytesAsHost` returns early for a GGUF borrow by +design, which is exactly the case this release exists for. The red test covers +the REPEAT call, not only the first, on both arms. **The copy must have completed.** `RocmBackend::Copy` is `hipMemcpyAsync` on a stream. Dropping the source pages while the copy may still be reading them is a @@ -156,6 +178,17 @@ cases there: memo away must fail this case. 3. **A host-addressable platform is unchanged.** The alias arm never reaches the release, and the existing cases in this file must stay green. +4. **The SECOND arm, which is the one this row's model takes.** Three more cases + repeat 1-3 against `dense_attn::ResidentWeight`. They call that seam directly, + as `test_resident_weight_f32_copy_retires.cpp:263` already does for + `dense_attn::ResidentWeightF32` and for the same reason: the fake backend + implements memory operations only and registers no `Embedding`, `MatmulBT` or + `RmsNorm` for `kXPU`, so every `qwen4_exp` entry point above that seam refuses + on a missing op before residency is asked about. The seam is production code + in a production header, not a test hook; reachability is carried by the 31 + production call sites named in §1, and the mutation that convicts the wiring + is deleting the `MaybeReleaseStagedBorrowSource` call from + `dense_attn_block.h`. For fix 2 the instrument already exists: `NoteGgufPrefaultedSpan` / `GgufPrefaultSnapshot` in `include/vllm/config/weight_residency.h` count spans @@ -163,13 +196,21 @@ actually prefaulted, and were added precisely because a prefault changes no byte and a byte-transparency case cannot see it. A truth-table case over the new device helper pins the default per device and pins that an explicit knob wins. -Every added assertion is mutation-proven: delete the release, delete the memo, -delete the synchronize, delete the device term, and the focused suite must go -red for each. +Every added assertion is mutation-proven for the release, the memo and the +platform term, on both arms. The synchronize and the BACKEND half of the +host-addressability pair are NOT convicted by this harness and are recorded +as an ungated guarantee rather than chased with a contorted test. ## 6. Gates -- Focused: `-tc=*resident*`, `-tc=*prefault*`, `-tc=*DSA*` (273 assertions). +- Focused: `-tc=*release*`, `-tc=*prefault*`, `-tc=*DSA*`. + `-tc=*resident*` was the first draft of this line and it SELECTED NOTHING: + no case name in `test_resident_weight_host_addressable.cpp` contains the + substring `resident` (`residency` does not, the `t` is missing), so doctest + reported 0 cases, 0 assertions and `Status: SUCCESS!` — a third of the declared + focused gate passing without measuring anything. Every declared selector's case + and assertion counts are printed with the evidence, because a selector that + matches nothing is indistinguishable from one that passes. - Full ROCm cross-device suite: 60 cases / 84833 assertions, unchanged. - `scripts/check-agent-record.py`, `scripts/check-commit-style.py`, `scripts/check-commit-trailers.py`, `scripts/check-pr-size.py`. @@ -199,6 +240,30 @@ red for each. legitimate result and is reported with `wchan` evidence rather than papered over. +## Ungated guarantees + +Two preconditions of `MaybeReleaseStagedBorrowSource` that this tree's harness +cannot convict. This section is deliberately NOT spelled `## Owed`: that heading +is the ownership surface for a ROWLESS issue under `.agents/issues/_owed`, and +`scripts/issue_records.py` refuses a row-owned issue that appears under it. +Nothing here is unreached code; both call sites are production and both are +gated. What is owed is an INSTRUMENT. The code is correct and the gap is in the instrument, so neither +test is contorted to manufacture a conviction. `ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W` +owns both. + +- **The `backend.DeviceMemoryIsHostAddressable()` term is UNGATED.** Deleting it + leaves 25/25 green with the binary proven changed. The fake backend in + `test_resident_weight_host_addressable.cpp` answers `false` unconditionally, so + only the platform half of the pair is ever exercised. The helper does refuse on + either — that claim is about the code and is true — but only one arm is + measured. Convicting the other needs a fake backend that answers `true` while + the platform answers `false`, which is a combination no device in this fleet + presents and which the file's registrar cannot hold alongside the existing one. +- **The `backend.Synchronize(queue)` call is UNGATED.** Deleting it also leaves + 25/25 green. `HostBackend::Copy` is a synchronous `memcpy` and the class does + not override `Synchronize`, so no case in this tree can express a DMA still + reading the source pages. Convicting it needs a backend whose `Copy` defers. + ## Deferred, with an issue - `ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW` (row-owned, not started) covers chunked H2D through a pinned bounce diff --git a/include/vllm/model_executor/models/dense_attn_block.h b/include/vllm/model_executor/models/dense_attn_block.h index 14cde1226..7b0f8c58b 100644 --- a/include/vllm/model_executor/models/dense_attn_block.h +++ b/include/vllm/model_executor/models/dense_attn_block.h @@ -203,7 +203,9 @@ inline Tensor ResidentWeight(Dev d, const OwnedTensor& w, std::vector s // what serves it. Each arm therefore asserts the precondition IT needs, which // is also why the staging assert sits inside `if (!w.d_dev)` — a resident // weight re-read on the hot path pays nothing for this. - if (vllm::platforms::GetPlatform(d.q.device.type).is_cpu()) { + const vllm::platforms::Platform& plat = + vllm::platforms::GetPlatform(d.q.device.type); + if (plat.is_cpu()) { VT_CHECK(!w.bytes.empty(), std::string("resident weight: EMPTY tensor has no host bytes to " "alias (host-alias arm, dtype ") + @@ -241,6 +243,36 @@ inline Tensor ResidentWeight(Dev d, const OwnedTensor& w, std::vector s d.b.Copy(d.q, p, w.bytes.data(), nb); Backend* bk = &d.b; w.d_dev = std::shared_ptr(p, [bk](void* q) { bk->Free(q); }); + // THE SOURCE PAGES ARE SPENT, AND THIS IS THE ARM QWEN4-EXP ACTUALLY TAKES. + // The identical release landed first in `qwen3_5.cpp`'s own `ResidentWeight` + // (the TU-local one, which shadows this function inside that file), and that + // covered the Qwen3.5 dense weights and — through `KqResidentSlice` and + // `KqGrouped` — the shared MoE seam's keep-quant expert towers. It did NOT + // cover Qwen4-Exp's attention, norm, hyper-connection, PLE and lm_head + // weights, because every one of those stages HERE: 12 call sites in + // `qwen4_exp_forward.cpp`, 10 in `qwen4_exp_qsa_block.cpp`, 7 in + // `qwen4_exp_ple_block.cpp` and 1 in `qwen4_exp_registry.cpp`, and not one + // reference to the `qwen3_5.cpp` function anywhere in that model. Measured + // with the release in the other function only: host `RssFile` on + // `strix:gpu0` still climbed to 21.08 GB during the forward and stayed + // there (.agents/specs/rocm-host-residency-after-upload.md §4). + // + // INSIDE THE `d_dev` MEMO ON PURPOSE, which is #1299's lesson and the same + // placement the other arm was reviewed on: this function runs about 1,361 + // times per forward step on the target checkpoint, so a release that + // re-tested its condition on every call would `MADV_DONTNEED` the pages the + // GPU is about to read, every step. The helper states its other two + // preconditions (a BORROWED span, and `mmap_fd >= 0` so the pages are + // re-faultable from a file rather than anonymous) and synchronizes the + // queue before it touches anything. + // + // The platform answer is the one COMPUTED ABOVE, passed in. It is + // `host_memory_is_device_addressable()` and NOT the backend's + // `DeviceMemoryIsHostAddressable()`: those answer different questions and + // GB10 answers them differently, which is what the guard repair on this + // branch existed for. The helper asks the backend's separately. + vllm::MaybeReleaseStagedBorrowSource(d.b, d.q, w, + plat.host_memory_is_device_addressable()); // The host mirror is now redundant wherever device memory is host- // addressable (Vulkan). See AdoptDeviceBytesAsHost — this is what keeps a // unified-memory box from holding the whole model twice. diff --git a/include/vllm/model_executor/models/qwen3_5_weights.h b/include/vllm/model_executor/models/qwen3_5_weights.h index a5211a69b..8650c26be 100644 --- a/include/vllm/model_executor/models/qwen3_5_weights.h +++ b/include/vllm/model_executor/models/qwen3_5_weights.h @@ -265,6 +265,13 @@ void AdoptDeviceBytesAsHost(vt::Backend& backend, const OwnedTensor& w); // either. One predicate, computed once, exactly as #2406 made // `QuantRepackForDevice` take its `dev`: a second spelling is how a // refusal and its route predicate come to disagree about one weight. +// +// ONLY THE PLATFORM HALF IS GATED, and that is recorded rather than +// implied. Deleting `backend.DeviceMemoryIsHostAddressable()` here leaves +// the focused suite green: the fake backend in +// `test_resident_weight_host_addressable.cpp` answers false +// unconditionally, so no case ever presents the disagreeing pair. Owed in +// `ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W`. // 2. `bytes` is BORROWED. An owned buffer is `ReleaseHost`'s business. // 3. `mmap_fd >= 0`. This is the discriminator that makes the call SAFE, and it // is not a convenience. `MADV_DONTNEED` on a file-backed private mapping @@ -281,13 +288,21 @@ void AdoptDeviceBytesAsHost(vt::Backend& backend, const OwnedTensor& w); // kernel would fault them straight back in. Correctness survives that; // throughput does not. The one production call site is inside // `if (!w.d_dev)` and `BorrowReleaseSnapshot().calls` is what makes that -// checkable rather than asserted. +// checkable rather than asserted. THERE ARE TWO SUCH CALL SITES AND BOTH ARE +// PRODUCTION: the `ResidentWeight` in the unnamed namespace of `qwen3_5.cpp` +// (Qwen3.5's dense weights, and the shared MoE seam's keep-quant expert towers +// through `KqResidentSlice` and `KqGrouped`), and `dense_attn::ResidentWeight` +// in `dense_attn_block.h`, which is the one every `qwen4_exp` attention, norm, +// hyper-connection, PLE and lm_head weight takes. Wiring only the first left +// host `RssFile` on `strix:gpu0` at 21.08 GB through the forward. // // It SYNCHRONIZES `queue` before releasing anything, because the staging copy is // `hipMemcpyAsync` on a stream and dropping the source pages under a live DMA is // a correctness bug rather than a residency one. The synchronize is skipped // entirely when the preconditions do not hold, so a backend this does not apply -// to pays nothing. +// to pays nothing. THE SYNCHRONIZE IS ALSO UNGATED, for a harness reason: +// `HostBackend::Copy` is a synchronous `memcpy` and the class does not override +// `Synchronize`, so no case in this tree can express a live DMA. Same issue. // // `host_addressable` is the caller's `host_memory_is_device_addressable()` // answer for the queue's device. Returns true when pages were released. diff --git a/src/vllm/platforms/rocm.cpp b/src/vllm/platforms/rocm.cpp index 78faf7b20..e7089b219 100644 --- a/src/vllm/platforms/rocm.cpp +++ b/src/vllm/platforms/rocm.cpp @@ -140,18 +140,33 @@ class RocmPlatform final : public Platform { // on a 31 GiB host until the load wedged in `svm_range_set_attr` // (.agents/specs/rocm-host-residency-after-upload.md). // - // THE FLAG NONETHELESS STAYS FALSE, for a different and checkable reason. - // Nothing reads it except `ShouldReleaseHostWeights` and - // `ShouldInterleaveLoadStream` (`platforms/interface.h`), and BOTH also - // require `marlin_committed`, which no ROCm path sets. Flipping it here would - // therefore change no behaviour while asserting a release/pool measurement - // nobody has taken on this board. The host-residency release that defect - // needed is gated instead on the property that is load-bearing and checkable - // at the call site — the device cannot dereference host memory and the source - // is a re-faultable read-only file mapping — in - // `MaybeReleaseStagedBorrowSource` (qwen3_5_weights.h). Flip these two when a - // board's release/pool behavior is actually measured, not as a side effect of - // making the budget check reachable. + // THE FLAG NONETHELESS STAYS FALSE, and the reason is that flipping it WOULD + // change behaviour on this board, not that it would change none. + // + // A first draft of this comment said the flag had two readers, + // `ShouldReleaseHostWeights` and `ShouldInterleaveLoadStream` + // (`platforms/interface.h:88-96`), that both also require `marlin_committed`, + // which no ROCm path sets, and that flipping it here was therefore inert. + // That is false and a fresh review measured it. There is a THIRD reader and it + // requires no `marlin_committed`: `DirectDeviceLoadEligible` + // (`qwen3_5_dense_weights.cpp:146-180`) returns + // `platform.residency_policy().release_host_weights_after_upload` as its last + // term, and it is what gates `StageAndReleaseLoadedDense` at + // `qwen3_5_dense_weights.cpp:1205-1212` for every Qwen3.5-dense safetensors + // load. `needs_weight_staging()` is true on ROCm, so flipping this bit would + // arm a whole staging-and-release load path on gfx1151 that nobody has + // measured there. Correcting a false comment and installing a new one is the + // failure this file has now produced twice; the enumeration above is the whole + // of it, from `grep -rn release_host_weights_after_upload src include tests`. + // + // The host-residency release that the defect needed is gated instead on the + // property that is load-bearing and checkable at the call site — the device + // cannot dereference host memory and the source is a re-faultable read-only + // file mapping — in `MaybeReleaseStagedBorrowSource` (qwen3_5_weights.h), + // called from BOTH staging arms (`qwen3_5.cpp`'s `ResidentWeight` and + // `dense_attn_block.h`'s). Flip these two when a board's release/pool + // behavior is actually measured, not as a side effect of making the budget + // check reachable. ResidencyPolicy residency_policy() const override { ResidencyPolicy p; p.device_memory_total_bytes = device_memory_total_bytes_; diff --git a/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp b/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp index 8ce612c97..06b565292 100644 --- a/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp +++ b/tests/vllm/model_executor/test_resident_weight_host_addressable.cpp @@ -61,6 +61,8 @@ #include "vllm/config/weight_residency.h" #include "vllm/model_executor/model_loader/gguf_keep_quant.h" +#include "vllm/model_executor/models/dense_attn_block.h" +#include "vllm/model_executor/models/dense_device_glue.h" #include "vllm/model_executor/models/qwen3_5.h" #include "vllm/model_executor/models/qwen3_5_dense.h" #include "vllm/model_executor/models/qwen3_5_internal.h" @@ -1055,6 +1057,137 @@ TEST_CASE("a host-addressable device that STAGES anyway still releases nothing") } #endif // __linux__ +// ═══════════════════════════════════════════════════════════════════════════ +// THE SECOND STAGING ARM — `dense_attn::ResidentWeight`, which is the one +// QWEN4-EXP TAKES. +// +// THE CASES ABOVE PROVE THE WRONG FUNCTION FOR THIS ROW, AND THAT IS MEASURED. +// `Qwen3_5EmbeddingTable` bridges the `ResidentWeight` defined in the unnamed +// namespace of `qwen3_5.cpp`, which SHADOWS the header one inside that +// translation unit. That function serves the Qwen3.5 dense weights and, through +// `KqResidentSlice` and `KqGrouped`, the shared MoE seam's keep-quant expert +// towers -- so the MoE half of the target checkpoint does reach it. What never +// reaches it is Qwen4-Exp's attention, norm, hyper-connection, PLE and lm_head +// weights: every one of those stages through `dense_attn::ResidentWeight` +// instead (12 call sites in `qwen4_exp_forward.cpp`, 10 in +// `qwen4_exp_qsa_block.cpp`, 7 in `qwen4_exp_ple_block.cpp`, 1 in +// `qwen4_exp_registry.cpp`, and zero references to the `qwen3_5.cpp` function +// anywhere in that model). With the release in the other function only, host +// `RssFile` on `strix:gpu0` still climbed to 21.08 GB during the forward and +// stayed there. +// +// WHY THESE CALL THE SEAM DIRECTLY. `dense_attn::ResidentWeight` is production +// code in a production header, not a test hook, and this file's fake backend +// implements memory operations only -- it registers no `Embedding`, `MatmulBT` +// or `RmsNorm` for `kXPU`, so every Qwen4-Exp entry point ABOVE this seam +// (`RunQwen4ExpPleBlock`, `RunQwen4ExpMoeBlock`, `ModelRegistry::Forward`) +// refuses on a missing op before residency is even asked about. This is the +// same shape and the same argument as +// `tests/vllm/model_executor/test_resident_weight_f32_copy_retires.cpp:263`, +// which gates `dense_attn::ResidentWeightF32` by a direct call for exactly this +// reason and carries its reachability separately. The reachability evidence +// here is the 30 production call sites named above; the mutation that convicts +// this wiring is deleting the `MaybeReleaseStagedBorrowSource` call from +// `dense_attn_block.h`, which turns case one below red. +#if defined(__linux__) +TEST_CASE("dense_attn: a STAGED borrow's source pages are released too") { + const PlatformArm arm(false); // a device that cannot read host memory + MappedFile f(static_cast(kBigVocab * kBigHidden) * 2); + REQUIRE(f.ok()); + f.Prefault(); + + const size_t rss_resident = RssFileKib(); + REQUIRE(rss_resident > 0); // unreadable /proc is "cannot measure", not a pass + + const OwnedTensor w = BorrowWeight(f, kBigVocab, kBigHidden); + const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); + + Queue q = XpuQueue(); + vllm::dense_attn::Dev d{Fake(), q}; + const Tensor t = vllm::dense_attn::ResidentWeight(d, w, {kBigVocab, kBigHidden}); + + REQUIRE(w.d_dev != nullptr); // it really staged + REQUIRE(t.data == w.d_dev.get()); + + const vllm::BorrowReleaseStats after = vllm::BorrowReleaseSnapshot(); + CHECK(after.calls == before.calls + 1); + CHECK(after.bytes == before.bytes + f.size()); + + // THE ASSERTION THE COUNTER CANNOT MAKE, exactly as in the `qwen3_5.cpp` arm: + // the file pages this case faulted in are gone from the resident set, against + // the background of the same-sized staging allocation that is supposed to + // stay. Half the span is the bar for that reason. + const size_t rss_after = RssFileKib(); + CHECK(rss_resident > rss_after); + CHECK(rss_resident - rss_after >= (f.size() / 2) / 1024); + + // ...AND THE BORROW IS STILL A VALID VIEW. A private file mapping re-faults + // the identical bytes; if this ever differs, something anonymous was touched. + CHECK(w.bytes.data() == f.data()); + CHECK(std::memcmp(w.bytes.data(), t.data, f.size()) == 0); +} + +TEST_CASE("dense_attn: the source release happens ONCE, not on every step") { + // #1299's shape on the second arm. `dense_attn::ResidentWeight` is the one + // Qwen4-Exp calls about 1,361 times per forward step, so the memo placement + // matters MORE here, not less. Move the call outside `if (!w.d_dev)` and the + // count below becomes 8. + const PlatformArm arm(false); + MappedFile f(1u << 20); + REQUIRE(f.ok()); + f.Prefault(); + const int64_t vocab = 64; + const int64_t hidden = (1 << 20) / (64 * 2); + const OwnedTensor w = BorrowWeight(f, vocab, hidden); + + const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); + Queue q = XpuQueue(); + vllm::dense_attn::Dev d{Fake(), q}; + for (int i = 0; i < 8; ++i) + (void)vllm::dense_attn::ResidentWeight(d, w, {vocab, hidden}); + const vllm::BorrowReleaseStats after = vllm::BorrowReleaseSnapshot(); + + CHECK(after.calls == before.calls + 1); + CHECK(after.bytes == before.bytes + f.size()); +} + +TEST_CASE("dense_attn: a HOST-ADDRESSABLE device releases nothing here either") { + // The platform term, passed in from THIS call site. Delete it -- hand the + // helper a literal `false` -- and this case counts a release where the + // kernels can still follow the host pointer. + const PlatformArm arm(true); + MappedFile f(1u << 20); + REQUIRE(f.ok()); + f.Prefault(); + // MISALIGNED on purpose, so the weight DECLINES the alias and reaches the + // staging arm the guard sits on. An aligned borrow returns before it. + const size_t skew = 16; + REQUIRE(reinterpret_cast(f.data() + skew) % vllm::kDeviceAliasAlignment != 0); + const int64_t vocab = 64; + const int64_t hidden = ((1 << 20) - 4096) / (64 * 2); + const size_t nb = static_cast(vocab * hidden) * 2; + OwnedTensor w; + w.dtype = DType::kBF16; + w.rank = 2; + w.shape[0] = vocab; + w.shape[1] = hidden; + w.nk = false; + std::shared_ptr keep(static_cast(f.data()), + [](const void*) {}); + w.bytes = vllm::OwnedBytes::Borrow(f.data() + skew, nb, std::move(keep)); + w.mmap_fd = f.fd(); // file-backed, so ONLY the platform guard can refuse it + + const vllm::BorrowReleaseStats before = vllm::BorrowReleaseSnapshot(); + Queue q = XpuQueue(); + vllm::dense_attn::Dev d{Fake(), q}; + const Tensor t = vllm::dense_attn::ResidentWeight(d, w, {vocab, hidden}); + REQUIRE(w.d_dev != nullptr); + REQUIRE(t.data == w.d_dev.get()); + CHECK(vllm::BorrowReleaseSnapshot().calls == before.calls); + CHECK(std::memcmp(w.bytes.data(), t.data, nb) == 0); +} +#endif // __linux__ + // --------------------------------------------------------------------------- // FIX 2: THE LOAD-TIME PREFAULT'S DEVICE TERM (`GgufPrefaultForDevice`). From a6e243eea4f0c8394d90a54608cfe14193fff146 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 13 Sep 2026 05:02:45 +0000 Subject: [PATCH 09/10] record(MODEL-MM-QWEN4-EXP): the model gate is measured, and the stall survives both fixes The 67.56 GiB UD-IQ1_S produces no token on strix:gpu0 with the release wired into both ResidentWeight staging arms. Two runs, each killed at a 1200 s deadline, peak VmHWM 25.99 and 26.05 GiB, peak RssFile 20.22 and 20.35 GiB, peak RssAnon 5.55 GiB both, peak device memory 29.69 GiB both. Host RssFile during the forward is essentially unchanged from the 21.08 GB the one-arm build showed, so releasing the spent source pages is not by itself sufficient on this board. Device memory is no longer UNVERIFIED. rocm-smi is not on PATH in the leased container, so the figure is read from /sys/class/drm/card0/device/mem_info_vram_used: 154 MB at rest, 31.88 GB of the board's 33.27 GB total at the deadline. The earlier "717 MB of 103 GB" figure does not describe this board and is withdrawn. Sampling every thread's stat and wchan every 6 s, the uninterruptible thread is in svm_range_set_attr for 153 of 196 samples in run 1 and 148 of 198 in run 2, in folio_wait_bit_common for 41 and 47, and in lock_mm_and_find_vma for 3. The KFD SVM path is still the dominant blocker; the page-cache share is a CIFS confound that is named rather than separated, because the checkpoint lives on the share. That is section 7's fourth risk realised, and it makes ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW required rather than optional. No throughput, latency or prefill number is recorded, because no token was produced. The artifact was asserted HIP-linked before it was timed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../specs/rocm-host-residency-after-upload.md | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/.agents/specs/rocm-host-residency-after-upload.md b/.agents/specs/rocm-host-residency-after-upload.md index 64d078066..3c5a085e4 100644 --- a/.agents/specs/rocm-host-residency-after-upload.md +++ b/.agents/specs/rocm-host-residency-after-upload.md @@ -225,6 +225,58 @@ as an ungated guarantee rather than chased with a contorted test. is recorded and the thread's `/proc//stat` state and `wchan` are reported instead. +## 6a. The model gate: MEASURED, AND THE STALL SURVIVES BOTH FIXES + +Measured 2026-09-13 on `strix:gpu0` (gfx1151, Radeon 8060S, ROCm 7.2.4 / HIP +7.2.53211, 30 GiB host, `mem_total_bytes` 33,270,497,280), under `rc` job +`cd38c438-e5a7-487f-9aab-27324f04f2d7`, box idle and exclusively leased. Built on +the worker from a clean clone of `969dd6f` with +`cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_HIP=ON` +`-DVLLM_CPP_HIP_ARCHITECTURES=gfx1151 -DROCM_PATH=/opt/rocm`, `ninja -j 6 +vllm-cli`, 210 s. The artifact was ASSERTED HIP-linked before it was timed: +`ldd` shows `libamdhip64.so.7`, `libhsa-runtime64.so.1`, `libhipblaslt.so.1` and +`librocblas.so.5`, all from `/opt/rocm-7.2.4/lib`. Run through +`examples/vllm-cli --device auto --max-tokens 8 --temperature 0 +--max-num-seqs 1` over +`/workspace/ckpt/qwen4exp-flash-next-iq1s/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf` +(shard 1 of 3; 10,946,624 + 49,990,818,368 + 22,544,696,352 bytes). +`VT_ROCM_MANAGED_ALLOC` unset. + +**NO TOKEN, TWICE.** Each run was killed at a 1200 s deadline (`SIGKILL`, exit +137) having produced no output past the auto-fit line. Two runs, not three: the +result is the same failure in both and the axis is not a number. + +| | run 1 | run 2 | +|---|---|---| +| peak `VmHWM` | 27,250,548 kB (25.99 GiB) | 27,320,420 kB (26.05 GiB) | +| peak `RssFile` | 21,207,224 kB (20.22 GiB) | 21,338,160 kB (20.35 GiB) | +| peak `RssAnon` | 5,821,836 kB (5.55 GiB) | 5,821,172 kB (5.55 GiB) | +| peak device memory | 31,878,860,800 B (29.69 GiB) | 31,880,183,808 B (29.69 GiB) | +| token | none, killed at 1200 s | none, killed at 1200 s | + +**Device memory is no longer UNVERIFIED.** `rocm-smi` is not on `PATH` in the +leased container, so this is read from +`/sys/class/drm/card0/device/mem_info_vram_used`: 154,816,512 B at rest, climbing +to 31.88 GB of the board's 33.27 GB total. The earlier "717 MB of 103 GB" figure +does not describe this board and is withdrawn rather than carried forward. + +**Where it blocks.** Sampling `/proc//{stat,wchan}` every 6 s over both +runs, the uninterruptible thread is in `svm_range_set_attr` for 153 of 196 +samples in run 1 and 148 of 198 in run 2, in `folio_wait_bit_common` for 41 and +47, and in `lock_mm_and_find_vma` for 3. So the KFD SVM path is still the +dominant blocker, and a second, smaller share is plain page-cache read wait — +the checkpoint is on a CIFS mount (`//192.168.68.102/Data`), which is its own +confound and is NOT separated here. + +**THIS IS §7's FOURTH RISK, REALISED.** The release is wired into both staging +arms and gated, and the checkpoint still does not forward. Host `RssFile` during +the forward is 20.2-20.4 GiB, essentially unchanged from the 21.08 GB the +one-arm build showed, so releasing the spent source pages is not by itself +sufficient on this board. The pageable, file-backed source is therefore the next +suspect and `ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW` (chunked H2D through a +pinned bounce buffer) is required, not optional. No throughput, latency or +prefill number is recorded, because no token was produced. + ## 7. Risks - **A released page that something still reads.** The borrow stays valid, so a From 98e2cd7da25f475b09e96ba3f928adc5491df00f Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 13 Sep 2026 05:34:06 +0000 Subject: [PATCH 10/10] record(MODEL-MM-QWEN4-EXP): name the four other families that execute this seam, and make every gate line reproducible MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh review returned FAIL on two counts, and both are about what the record CLAIMS rather than about what the code does. The release logic is unchanged. THE SHARED SEAM'S REACH WAS STATED AS THE REACH WITHIN ONE MODEL. `dense_attn::ResidentWeight` is a production header, and of the 42 files that call it only four are qwen4_exp's. Every sentence in the spec, the pull request body and the eight commit bodies framed the reach as "12 call sites in `qwen4_exp_forward.cpp`, 10 in `qwen4_exp_qsa_block.cpp`, 7 in `qwen4_exp_ple_block.cpp`, 1 in `qwen4_exp_registry.cpp`" -- 30 sites, which §5 also miscounted as 31. Spec §4a now carries the enumeration, built by listing every caller of the two GGUF keep-quant borrow producers `OwnGgufQuantBlocks` and `OwnGgufF16` (the only writers of `mmap_fd`, which is the field the release requires) and tracing each one forward to its consumption site. Four other families execute the release on discrete CUDA and non-unified ROCm, so on `dgx`, `thor`, `orin` and `strix`: GLM-MoE-DSA at 24 sites in `glm_moe_dsa_forward.cpp` (its expert towers are refused one frame earlier by `GlmResidentExpertSlice`, its MLA, router, norms, rope cache, embedding and head are not), GLM5-Next at `glm5_next_moe.cpp:243-245`, Muse-Glimmer at twelve sites in `muse_glimmer.cpp` plus `muse_glimmer_mm.cpp:264`, and the Qwen3.5 DFlash draft head, whose `EmbedTable()` is the target's kept-F16 GGUF table after the rebind in `model_loader.cpp`. DeepSeek-V4 and Laguna have GGUF loaders and were checked and EXCLUDED, with the reason recorded so nobody re-derives them. NO TEST COVERS ANY FAMILY BUT QWEN4-EXP, and that is recorded as owed on ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X rather than answered with a case that would prove nothing. The focused harness's fake backend implements memory operations only, so another family's entry point refuses on a missing op before residency is asked about, and calling the seam directly with another family's tensor shape would add a family's name and not a family's call site. The same reasoning is why the one cheap-looking repair was NOT made: giving the header `ResidentWeight` the `VT_CHECK(!w.expert_streamed)` its translation-unit-local twin has would make the shared seam refuse a weight it accepts today, on paths no test covers. That hazard, `ResidentWeightF32`'s independent `d_dev_f32` memo and GLM5-Next's host-fallback re-read are owed on ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W. §6 SHIPPED A SECOND SELECTOR THAT MATCHES NOTHING, on the line edited to repair the first. `-tc=*DSA*` against `test_resident_weight_host_addressable` reports 0 cases, 0 assertions and `Status: SUCCESS!`; `DSA` selects two cases in `test_backend_cross_device`, which is a separate line, and the section named no binary at all. Every selector now names its binary and carries the case and assertion counts it actually produced, measured on `strix:gpu0` under rc job f8a6ceec-d8bf-4e99-b6e4-3b2cbf71f8b9: focused binary 28/139 whole, `*release*` 9/52, `*prefault*` 2/11, `*dense_attn*` 3/19, and cross-device 60/84833 whole with `*DSA*` 2/273. The rest of the section was then swept for a third dud and the sweep is reported rather than asserted: the four checkers are the only other gate lines, none accepts or defaults to a selector, and each was run with no arguments to confirm it refuses rather than passing empty. Each now carries the invocation that measures the branch instead of the one commit at HEAD. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../BACKEND-GATE-ROCM-VLLM/ISSUE-GH-3048.md | 30 +++ .agents/issues/BACKEND-ROCM/ISSUE-GH-3009.md | 82 ++++++++ .agents/issues/BACKEND-ROCM/ISSUE-GH-3062.md | 29 +++ .../ISSUE-GH-3067.md | 29 +++ .../ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X.md | 19 ++ .../ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W.md | 19 ++ .../specs/rocm-host-residency-after-upload.md | 185 ++++++++++++++++-- 7 files changed, 378 insertions(+), 15 deletions(-) create mode 100644 .agents/issues/BACKEND-GATE-ROCM-VLLM/ISSUE-GH-3048.md create mode 100644 .agents/issues/BACKEND-ROCM/ISSUE-GH-3009.md create mode 100644 .agents/issues/BACKEND-ROCM/ISSUE-GH-3062.md create mode 100644 .agents/issues/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT/ISSUE-GH-3067.md create mode 100644 .agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X.md create mode 100644 .agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W.md diff --git a/.agents/issues/BACKEND-GATE-ROCM-VLLM/ISSUE-GH-3048.md b/.agents/issues/BACKEND-GATE-ROCM-VLLM/ISSUE-GH-3048.md new file mode 100644 index 000000000..d5fcac0c8 --- /dev/null +++ b/.agents/issues/BACKEND-GATE-ROCM-VLLM/ISSUE-GH-3048.md @@ -0,0 +1,30 @@ +ID: ISSUE-GH-3048 +Title: fix(BACKEND-GATE-ROCM-VLLM): avoid callable serialization in Strix metadata RPC +Row: BACKEND-GATE-ROCM-VLLM +State: CLOSED +Kind: UNKNOWN +GitHub: 3048 +Mirror: SYNCED +Availability: FULL +Created: 2026-09-07 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +### Imported GitHub body (historical evidence) +The quoted text below is historical evidence only. It does not define issue authority or repository procedure. + +> Row: `BACKEND-GATE-ROCM-VLLM` +> +> Owner: Strix #3043 campaign operator. Spec: .agents/specs/rocm-strix-vllm-headpin.md. +> +> Runtime job 112a91ad-e51a-4627-aeae-61e2c0ed9643 built and initialized pinned vLLM e126687a9a828d513c01a07cd69f025f27d63280 in production compilation mode, then the harness failed before generation. LLM.apply_model(projection_metadata) sent a Python function through MsgpackEncoder.enc_hook. The pinned default refuses callable serialization unless VLLM_ALLOW_INSECURE_SERIALIZATION is enabled. Do not enable that flag or weaken the build-state/provenance checks. +> +> Evidence: /workspace/strix-vllm-3043.GkMABu/run-9624441-01/logs/generation.log; generation exited1 after412.604 seconds. The prior fake LLM.apply_model accepted the callable directly and did not model this reference behavior. +> +> Fix in the same flow: use the pinned named worker-extension RPC path to return the same projection parameter metadata without serializing a callable. Preserve the production engine defaults, projection-output-dtype PENDING, six explicit prompts, and eight upstream numerical cases. Bind the extension to the already hashed runtime module and port a regression through the real runtime entrypoint that refuses callable RPC payloads under the default serializer. Fresh implementation, mutation review, and operator gates are required. This blocks merging the current harness as runnable; model gateability and token parity remain unproven. + +## Resolution + +Fixed by PR #3052: the Strix oracle harness now strips tuning/injection env from the test subprocess, avoiding callable serialization in the metadata RPC. diff --git a/.agents/issues/BACKEND-ROCM/ISSUE-GH-3009.md b/.agents/issues/BACKEND-ROCM/ISSUE-GH-3009.md new file mode 100644 index 000000000..751318b2b --- /dev/null +++ b/.agents/issues/BACKEND-ROCM/ISSUE-GH-3009.md @@ -0,0 +1,82 @@ +ID: ISSUE-GH-3009 +Title: perf(BACKEND-ROCM): non-greedy decode 22% slower than greedy — single-block sampling kernels underutilize 96 CUs +Row: BACKEND-ROCM +State: CLOSED +Kind: UNKNOWN +GitHub: 3009 +Mirror: SYNCED +Availability: FULL +Created: 2026-09-06 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +### Imported GitHub body (historical evidence) +The quoted text below is historical evidence only. It does not define issue authority or repository procedure. + +> Row: `BACKEND-ROCM` +> +> ## Problem +> +> Non-greedy decode (temperature + top-p + softmax + Gumbel-max sample) on ROCm runs **22% slower** than greedy decode at single-request batch size. On a Qwen3.5-4B Q4_K_M model (vocab ~152K) with fp8 KV cache on an RX 7900 XTX (gfx1100, 96 CUs): +> +> | Path | tok/s | ms/tok | +> |---|---|---| +> | Greedy (temp=0) | 97.8 | 10.2 | +> | Non-greedy (temp=0.7, top_p=0.9) | 75.6 | 13.2 | +> | **Sampling overhead** | — | **3.0 ms/tok** | +> +> ## Root cause +> +> Three per-row sampling kernels in `src/vt/rocm/rocm_sample.hip` launch as `<<<1, 256>>>` — one block of 256 threads (4 wavefronts) on **1 CU out of 96**, scanning the full ~152K vocab: +> +> | Kernel | Avg µs/call | Bottleneck | +> |---|---|---| +> | `ApplyTopKTopPRowK` | 1270 | Ternary search, up to 64 iterations over full vocab | +> | `RandomSampleK` | 1665 | `GumbelScore` → `ExpNoise` computes `log(u)` in double precision per element — compute-bound, 148 serial evaluations per thread | +> | `SoftmaxK` | 406 | 3-pass reduce (max, sum, normalize) | +> | **Total** | **3341** | 3.3 ms/tok on 1 CU | +> +> The greedy `ArgmaxK` has the same single-block shape but is cheaper per element (one compare vs one `log`), so the gap is specific to the non-greedy path. +> +> ## Evidence +> +> rocprofv3 kernel trace, 32 decode tokens, Qwen3.5-4B Q4_K_M, fp8 KV, gfx1100: +> +> ``` +> ApplyTemperatureK 32 calls avg= 3.8 us +> ApplyTopKTopPRowK 32 calls avg= 1270.0 us +> SoftmaxK 32 calls avg= 406.0 us +> RandomSampleK 32 calls avg= 1665.0 us +> ``` +> +> The CUDA backend (`src/vt/cuda/cuda_sample.cu`) already has a multi-block split-phase greedy argmax (`ArgBlocksPerRow`, `ArgmaxPartialKernel` + `ArgmaxFinalKernel`), but the ROCm random sample has no equivalent. +> +> ## Proposed fix +> +> Two changes, both in `src/vt/rocm/rocm_sample.hip`: +> +> 1. **Widen per-row sampling block from 256 to 1024 threads** (`kVocabBlock = 1024`). 16 wavefronts vs 4 gives 4× more latency hiding on the same CU. The `BlockRed*` helpers use `blockDim.x` instead of the compile-time constant. Top-p drops 5.5×, softmax 3.8×. +> +> 2. **Split-phase random sample** (`VT_SAMPLE_SPLIT=1`, default ON). Spread the Gumbel-score argmax across `kSampleSplitBlocks = 128` blocks (all 96 CUs), each thread handling 1–2 elements instead of 148. Phase A writes per-block `(score, index)` partials to scratch; Phase B reduces them. `ArgReduce` is associative and order-independent (same property the greedy split relies on), so the result is **bit-identical** to the single-block kernel. +> +> ## Measured result +> +> | Path | Before | After | Change | +> |---|---|---|---| +> | Greedy | 97.2 tok/s | 97.8 tok/s | unchanged | +> | Non-greedy | 75.6 tok/s | 96.5 tok/s | **+27.6%** | +> | Gap | 22% | 3.4% | | +> +> ## Correctness +> +> - `test_ops_sample`: 29/29 pass (2 CUDA-only skipped), 278K assertions +> - `test_sampler`: 21/21 pass +> - A/B test: 20 seed/temperature combinations (seeds 1–12345, temps 0.3–1.5) produce **bit-identical** output with `VT_SAMPLE_SPLIT=0` vs `VT_SAMPLE_SPLIT=1` +> - Determinism: 3 consecutive runs with same seed produce identical output +> - Full test suite: 225/225 pass, 3 consecutive runs, zero failures + +## Resolution + +Fixed by PR #3010: sampling block widen and split-phase implementation. diff --git a/.agents/issues/BACKEND-ROCM/ISSUE-GH-3062.md b/.agents/issues/BACKEND-ROCM/ISSUE-GH-3062.md new file mode 100644 index 000000000..60d3e469b --- /dev/null +++ b/.agents/issues/BACKEND-ROCM/ISSUE-GH-3062.md @@ -0,0 +1,29 @@ +ID: ISSUE-GH-3062 +Title: fix(BACKEND-ROCM): isolate split-sampling scratch by queue and device +Row: BACKEND-ROCM +State: CLOSED +Kind: UNKNOWN +GitHub: 3062 +Mirror: SYNCED +Availability: FULL +Created: 2026-09-08 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +### Imported GitHub body (historical evidence) +The quoted text below is historical evidence only. It does not define issue authority or repository procedure. + +> Row: `BACKEND-ROCM` +> +> PR #3010 stores split-sampling partial buffers and capacity in process-wide statics. Concurrent queues can overwrite another queue's partials between the two kernels. Calls on another device can reuse allocations from the first device. Host pointer and capacity updates also race. +> +> The repair belongs in #3010. Use the existing device-and-stream scratch ownership mechanism and preserve captured graph lifetimes. Add regression coverage for overlapping queues and allocation ownership. Retaining allocations for captured graphs must follow the existing backend lifetime contract. +> +> Acceptance requires focused tests, an independent mutation review, and the operator's ROCm gate. CPU-only skips do not satisfy the device gate. +> + +## Resolution + +Fixed by PR #3010: sampling block widen and split-phase implementation. diff --git a/.agents/issues/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT/ISSUE-GH-3067.md b/.agents/issues/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT/ISSUE-GH-3067.md new file mode 100644 index 000000000..feeaee975 --- /dev/null +++ b/.agents/issues/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT/ISSUE-GH-3067.md @@ -0,0 +1,29 @@ +ID: ISSUE-GH-3067 +Title: test(KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT): seal every device table entry +Row: KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT +State: CLOSED +Kind: UNKNOWN +GitHub: 3067 +Mirror: SYNCED +Availability: FULL +Created: 2026-09-08 +Updated: 2026-09-12 +Closed: 2026-09-12 + +## Problem + +### Imported GitHub body (historical evidence) +The quoted text below is historical evidence only. It does not define issue authority or repository procedure. + +> Row: `KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT` +> +> PR #3029 copies I-quant lookup tables into ROCm constant memory. Its owning spec, `.agents/specs/kernel-quant-ciq-gemm-rocm-iquant.md`, requires a complete table seal. The new header also explicitly requires a device snapshot and byte comparison. The changed tests do not implement that gate. +> +> Fresh review of integration head `b2ee9d8389caf974f3178613a6313e788dd93c4b` found that all four host-parsed arrays currently match their CPU references. This does not verify the executing device bytes or pin every entry in a committed test. +> +> Add the ROCm equivalent of the existing CUDA device-table snapshot and compare every entry against the pinned oracle reference. Preserve the current table values, kernel arithmetic, and dispatch. Mutating one entry in each table must make the focused device seal fail. Run independent review and the operator's leased ROCm gate before merging #3029. Keep the other formats owed by #1940 open. +> + +## Resolution + +Fixed by PR #3029: IQ4_XS and IQ3_XXS now have ROCm DotIQ4XS/DotIQ3XXS kernels in rocm_grouped_gemm.hip, admitted in DeviceKeepQuantSupported. diff --git a/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X.md b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X.md new file mode 100644 index 000000000..ab7d5c428 --- /dev/null +++ b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X +Title: OWED: dense_attn::ResidentWeight's staged-borrow release is UNCOVERED on the four other GGUF model families that execute it +Row: MODEL-MM-QWEN4-EXP +State: OPEN +Kind: task +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-13 +Updated: 2026-09-13 +Closed: - + +## Problem + +Fix 1 of .agents/specs/rocm-host-residency-after-upload.md put MaybeReleaseStagedBorrowSource inside dense_attn::ResidentWeight (include/vllm/model_executor/models/dense_attn_block.h:246-274), which is a SHARED seam. Four production model families besides qwen4_exp reach it with weights that satisfy both of the release's predicates (a BORROWED span with mmap_fd >= 0, which only the GGUF keep-quant borrow producers set, and a platform plus backend that cannot dereference host memory): GLM-MoE-DSA (glm_moe_dsa_forward.cpp:162-186, :269, :334), GLM5-Next (glm5_next_moe.cpp:243-245), Muse-Glimmer (muse_glimmer.cpp:156, 251, 259, 273, 295, 309, 315, 326, 395, 421, 434, 435 and muse_glimmer_mm.cpp:264) and the Qwen3.5 DFlash draft head sharing a GGUF target's kept-F16 embedding table (qwen3_dflash.cpp:597, 906, 1851, 1956). The release therefore fires on dgx, thor, orin and strix for all five families, and NO test covers any family but qwen4_exp. The focused harness cannot reach the others today: its fake backend registers memory operations only, so every entry point above the seam refuses on a missing op before residency is asked about, and section 4a of the spec records this rather than contorting a case. What is owed is a harness that can drive a second family's production entry point on a non-host-addressable fake platform, and one case per family that convicts the release at that family's own call site. See .agents/specs/rocm-host-residency-after-upload.md section 4a for the verified enumeration and for the two families checked and excluded (DeepSeek-V4, Laguna). + +## Resolution + +- diff --git a/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W.md b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W.md new file mode 100644 index 000000000..67a4ea86a --- /dev/null +++ b/.agents/issues/MODEL-MM-QWEN4-EXP/ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W +Title: OWED: three latent second-reader hazards around the staged-borrow release, including the header ResidentWeight's missing expert_streamed refusal +Row: MODEL-MM-QWEN4-EXP +State: OPEN +Kind: task +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-09-13 +Updated: 2026-09-13 +Closed: - + +## Problem + +A fresh review of the staged-borrow release found three hazards that are latent today and that this row deliberately did NOT re-engineer, because each repair is a change to a shared seam with a blast radius nothing measures (see ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X). ONE. dense_attn::ResidentWeightF32 (dense_attn_block.h:389) memoizes on d_dev_f32 INDEPENDENTLY of d_dev and reads w.bytes to upcast, so a weight that ResidentWeight has staged and released and that is then upcast would read dropped pages. It is unreachable only because every ResidentWeightF32 target today is a LoadNormBf16/ExpandBf16 owned tensor whose mmap_fd is -1, which is an accident of the current loaders and not an invariant either function states. TWO. The header ResidentWeight carries no VT_CHECK(!w.expert_streamed); its translation-unit-local twin at qwen3_5.cpp:1181 does, so nothing stops a streamed tower being staged and released on the header path. This was NOT closed in the same flow on purpose: adding the refusal would make the header seam REFUSE a weight it accepts today, and GLM5-Next stages three expert banks through it at glm5_next_moe.cpp:243-245 with no test on any staging device, so a one-line VT_CHECK could remove a working path rather than close a hazard. It needs the cross-family harness first. THREE. glm5_next_moe.cpp:292 and :300 are a host-fallback arm that reads the same OwnedTensor bytes, reachable in a partial-stage window because :243-245 are three sequential, un-transactional uploads. It re-faults, so it is correct, but the cost is a full re-read of an expert bank off the file. The window is narrower than it first reads: that arm VT_CHECKs its queue is CPU (glm5_next_moe.cpp:271) while the release only fires on a non-CPU device, so reaching it needs a CPU queue paired with a separate device pointer whose banks have already staged. Recorded with that qualification rather than dropped, because the re-read is real whenever the pairing occurs. + +## Resolution + +- diff --git a/.agents/specs/rocm-host-residency-after-upload.md b/.agents/specs/rocm-host-residency-after-upload.md index 3c5a085e4..b62ea2766 100644 --- a/.agents/specs/rocm-host-residency-after-upload.md +++ b/.agents/specs/rocm-host-residency-after-upload.md @@ -2,7 +2,12 @@ Row: `MODEL-MM-QWEN4-EXP` Issues: `ISSUE-LOCAL-01M2BZ5DZ2710201WMH9TNSVH3` (the defect), -`ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW` (owed: chunked H2D) +`ISSUE-LOCAL-01M2BZ5QK4XRETK48CXKSHKRDW` (owed: chunked H2D), +`ISSUE-LOCAL-01M2CCNA0S74WT5WBV50B3VD0W` (owed: two ungated helper terms), +`ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X` (owed: the four other families that +execute this seam are uncovered), +`ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W` (owed: three latent second-reader +hazards) ## 1. The defect @@ -159,6 +164,100 @@ NOT built here. See `## Deferred, with an issue`. The stale comment at `src/vllm/platforms/rocm.cpp` (§2). +## 4a. The shared seam's blast radius — WHO ELSE EXECUTES FIX 1 + +Fix 1's release sits in `dense_attn::ResidentWeight` +(`include/vllm/model_executor/models/dense_attn_block.h:246-274`), which is a +SHARED seam in a production header. §1 and §5 above state the reach as "12 call +sites in `qwen4_exp_forward.cpp`, 10 in `qwen4_exp_qsa_block.cpp`, 7 in +`qwen4_exp_ple_block.cpp`, 1 in `qwen4_exp_registry.cpp`" (30, not the 31 §5 +previously said — the four numbers sum to 30 and each was recounted by +occurrence, not by line). That is the reach WITHIN THIS ROW'S MODEL, and reading +it as the reach of the change is the mistake this section exists to correct. +`grep -rn 'ResidentWeight(' src include`, with `ResidentWeightF32` and the two +definitions removed, names 42 calling files. Four of them are qwen4_exp's. The +other 38 are 36 production translation units plus two shared headers, +`layers/linear.h` and `qwen3_5_weights.h`, and four other model families reach +the function with weights that satisfy the release's predicates. + +**The release fires only where BOTH hold.** Anything else returns at +`qwen3_5_weights.cpp:402-404` before it touches a page. + +1. The weight is a BORROWED span with `mmap_fd >= 0`. Only the GGUF keep-quant + borrow producers set that field: `OwnGgufQuantBlocks` + (`qwen3_5_gguf_weights.cpp:174`) and `OwnGgufF16` (`:244`), with + `qwen3_5_weights.cpp:363` and `qwen4_exp_moe.cpp:69` propagating it to a + slice view. The safetensors borrow sets `mmap_src` / `mmap_src_bytes` and + never `mmap_fd`, so every safetensors-only model is refused. +2. Neither the platform (`host_memory_is_device_addressable()`) nor the backend + (`DeviceMemoryIsHostAddressable()`) can dereference host memory. CPU returns + before the staging arm entirely; Vulkan (`vulkan_backend.cpp:135`), Metal and + a unified ROCm part (`rocm_backend.hip:488`) refuse on one of the two. + +**So it fires on discrete CUDA and on non-unified ROCm — `dgx`, `thor`, `orin` +and `strix`, which is the whole measurement fleet.** + +**The families, enumerated rather than sampled.** Every caller of +`OwnGgufQuantBlocks` / `OwnGgufF16` was listed, and each was traced forward to +whether its borrowed tensors reach this function. + +| family | loader that sets `mmap_fd` | consumed at | +|---|---|---| +| Qwen4-Exp (this row) | `qwen4_exp_weights.cpp:140`, `:734` and five `OwnGgufQuantBlocks` sites; `qwen4_exp_moe.cpp:69` propagates | `qwen4_exp_forward.cpp` (12), `qwen4_exp_qsa_block.cpp` (10), `qwen4_exp_ple_block.cpp` (7), `qwen4_exp_registry.cpp` (1) | +| GLM-MoE-DSA | `glm_moe_dsa_loader.cpp:308`, `:356`, `:417`, `:442` and five `OwnGgufQuantBlocks` sites | `glm_moe_dsa_forward.cpp:162-186` (the MLA block: `q_a_proj`, `q_a_layernorm`, `q_b_proj`, `kv_a_proj_with_mqa`, `kv_a_layernorm`, `kv_b_proj`, `w_uk_t`, `w_uv`, `o_proj`, and the indexer's `wq_b` / `wk` / `weights_proj` / `k_norm_weight` / `k_norm_bias`), `:269` (`down_proj`), `:334` (the router), `:359` (`e_score_correction_bias`), `:443`, `:462` (the layer norms), `:548` (the rope cache), `:598` (`final_norm`), `:606-607` (`lm_head`, tied or not), `:672` (`embed_tokens`) — 24 sites in all | +| GLM5-Next | `glm5_next_loader.cpp:190` and five `OwnGgufQuantBlocks` sites; `glm5_next_bridge.cpp` two more | `glm5_next_moe.cpp:243-245` (`gate_exps`, `up_exps`, `down_exps`) | +| Muse-Glimmer | `muse_glimmer_gguf_weights.cpp:190` and two `OwnGgufQuantBlocks` sites | `muse_glimmer.cpp:156`, `:251`, `:259`, `:273`, `:295`, `:309`, `:315`, `:326`, `:395`, `:421`, `:434`, `:435`; `muse_glimmer_mm.cpp:264` | +| Qwen3.5 DFlash draft head | the TARGET's kept-F16 embedding table, `qwen3_5_gguf_weights.cpp:826`, rebound onto the draft by `model_loader.cpp:2010-2060` | `qwen3_dflash.cpp:597`, `:906`, `:1851`, `:1956`, all four through `Qwen3DflashWeights::EmbedTable()` | + +**One exclusion inside a listed family is load-bearing.** GLM-MoE-DSA's 228 +routed-expert towers do NOT reach the release: `GlmResidentExpertSlice` +(`glm_moe_dsa_forward.cpp:112-122`) refuses a discrete device by name before any +`ResidentWeight` call, because the expert-stream lane serves those towers out of +host slot storage. Its MLA, MLP `down_proj`, router, norms, rope cache, +embedding and head weights all reach it; its expert mass does not. + +**Checked and EXCLUDED, with the reason, so a later reader does not re-derive +them.** + +- **DeepSeek-V4** has a GGUF loader (`deepseek_v4_weights.cpp`, four + `OwnGgufQuantBlocks` sites), but every one of its header-`ResidentWeight` call + sites — `deepseek_v4.cpp:1330-1332` and `:1476`, + `deepseek_v4_exl3_device.cpp:70-72` — takes an EXL3 trellis field + (`d_trellis` / `d_suh` / `d_svh`) that the EXL3 loader owns and that never + carries `mmap_fd >= 0`. A future GGUF trellis arm would make it reachable. +- **Laguna** has a GGUF loader (`laguna_weights.cpp`, five sites) and calls + `dense_attn::ResidentWeight` nowhere. +- **`layers/linear.h`**, the shared LinearMethod seam, calls the function seven + times (`:63`, `:97`, `:142`, `:143`, `:207`, `:213`, `:243`), so it was + checked separately. Of its consumers only `glm_moe_dsa_forward.cpp` and + `muse_glimmer.cpp` are in the GGUF-borrow set, so it adds no family that this + table does not already name. +- **Every other model** — gemma, phi, minicpm, olmo2, stablelm, commandr, + deepseek_v2, dots3, nemotron_h, qwen3_vl and the rest — loads from + safetensors, so `mmap_fd` is -1 and the helper returns at `:404`. + +**NO TEST COVERS ANY FAMILY BUT QWEN4-EXP, AND THAT IS RECORDED AS OWED RATHER +THAN CLOSED HERE.** `ISSUE-LOCAL-01M2CKN5516AKE7W2JVDV86Z8X` owns it. The reason +is the same wall §5 names: the focused harness's fake backend implements memory +operations only and registers no `Embedding`, `MatmulBT` or `RmsNorm` for +`kXPU`, so a GLM5-Next or Muse-Glimmer entry point refuses on a missing op long +before residency is asked about, and its process-global registrar cannot hold a +second fake backend beside the existing one. A case that called +`dense_attn::ResidentWeight` directly with a GLM5-Next-shaped tensor would add a +family's NAME and not a family's CALL SITE, which is the shape of green this +protocol exists to refuse. What is owed is a harness that can drive a second +family's production entry point, and that is a row of its own. + +**The one repair that looked cheap was NOT taken, for the same reason.** The +header `ResidentWeight` has no `VT_CHECK(!w.expert_streamed)` although its +translation-unit-local twin at `qwen3_5.cpp:1181` does. Adding it would make the +shared seam REFUSE a weight it accepts today, and GLM5-Next stages three expert +banks through it at `glm5_next_moe.cpp:243-245` with no test on any staging +device — so the one-liner could remove a working path instead of closing a +hazard, and there is no gate here that would say which. +`ISSUE-LOCAL-01M2CKNHVKJXT29XN8WM11KF6W` owns it together with two other latent +second-reader hazards. + ## 5. Tests — red first `tests/vllm/model_executor/test_resident_weight_host_addressable.cpp` already @@ -185,8 +284,9 @@ cases there: implements memory operations only and registers no `Embedding`, `MatmulBT` or `RmsNorm` for `kXPU`, so every `qwen4_exp` entry point above that seam refuses on a missing op before residency is asked about. The seam is production code - in a production header, not a test hook; reachability is carried by the 31 - production call sites named in §1, and the mutation that convicts the wiring + in a production header, not a test hook; reachability is carried by the 30 + production call sites named in §1 (and §4a names the OTHER families that + execute the same seam), and the mutation that convicts the wiring is deleting the `MaybeReleaseStagedBorrowSource` call from `dense_attn_block.h`. @@ -203,17 +303,72 @@ as an ungated guarantee rather than chased with a contorted test. ## 6. Gates -- Focused: `-tc=*release*`, `-tc=*prefault*`, `-tc=*DSA*`. - `-tc=*resident*` was the first draft of this line and it SELECTED NOTHING: - no case name in `test_resident_weight_host_addressable.cpp` contains the - substring `resident` (`residency` does not, the `t` is missing), so doctest - reported 0 cases, 0 assertions and `Status: SUCCESS!` — a third of the declared - focused gate passing without measuring anything. Every declared selector's case - and assertion counts are printed with the evidence, because a selector that - matches nothing is indistinguishable from one that passes. -- Full ROCm cross-device suite: 60 cases / 84833 assertions, unchanged. -- `scripts/check-agent-record.py`, `scripts/check-commit-style.py`, - `scripts/check-commit-trailers.py`, `scripts/check-pr-size.py`. +**EVERY SELECTOR NAMES ITS BINARY AND PRINTS ITS COUNTS, AND THE REASON IS THAT +THIS SECTION HAS NOW SHIPPED THE SAME DUD TWICE.** `-tc=*resident*` was the +first draft of the focused line and it SELECTED NOTHING: no case name in +`tests/vllm/model_executor/test_resident_weight_host_addressable.cpp` contains +the substring `resident` (`residency` does not — the `t` is missing), so doctest +reported 0 cases, 0 assertions and `Status: SUCCESS!`. The line that replaced it +carried `-tc=*DSA*` beside `-tc=*release*` and `-tc=*prefault*`, and `DSA` +selects nothing in that binary EITHER — it matches two cases in +`test_backend_cross_device`, which is a SEPARATE line below, and this section +named no binary at all, so nothing in the text said which of the two a reader +should run it against. A selector that matches nothing is indistinguishable from +one that passes. So each line below states its binary, its selector and the +case and assertion counts that selector actually produced. + +Measured on `strix:gpu0` (gfx1151, ROCm 7.2.4) under `rc` job +`f8a6ceec-d8bf-4e99-b6e4-3b2cbf71f8b9`, from a clean clone at +`a6e243eea4f0c8394d90a54608cfe14193fff146` built in a private `/tmp` tree with +`cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_HIP=ON` +`-DVLLM_CPP_HIP_ARCHITECTURES=gfx1151 -DROCM_PATH=/opt/rocm` +`-DVLLM_CPP_BUILD_TESTS=ON`, `ninja -j 4`, and run with +`LD_LIBRARY_PATH=/opt/rocm-7.2.4/lib` — without which the binary exits 127 +having measured nothing. Every later commit on this branch touches `.agents/` +markdown only, so `git diff --stat a6e243ee..HEAD -- src include tests` is empty +and these binaries are the branch's. + +| binary | selector | cases | assertions | +|---|---|---|---| +| `test_resident_weight_host_addressable` | (none — whole binary) | 28 | 139 | +| `test_resident_weight_host_addressable` | `-tc=*release*` | 9 | 52 | +| `test_resident_weight_host_addressable` | `-tc=*prefault*` | 2 | 11 | +| `test_resident_weight_host_addressable` | `-tc=*dense_attn*` | 3 | 19 | +| `test_backend_cross_device` | (none — whole binary) | 60 | 84833 | +| `test_backend_cross_device` | `-tc=*DSA*` | 2 | 273 | + +`-tc=*DSA*` against `test_resident_weight_host_addressable` is recorded here as +the DUD it is rather than deleted: 0 cases, 0 assertions, +`Status: SUCCESS!`. It is not a gate line. It is the measurement that proves the +previous revision of this section declared one that measured nothing. + +**THE REST OF THIS SECTION WAS SWEPT FOR A THIRD DUD, AND THE SWEEP IS REPORTED +RATHER THAN ASSERTED.** Every remaining gate line in this spec is a checker +invocation, and a checker cannot produce the silent-success shape unless it +accepts a selector or defaults to one. Each was run with no arguments to see +what it does: `check-commit-style.py` refuses (`the following arguments are +required: --range`), `check-commit-trailers.py` refuses (`pass exactly one of +--range or --message-file`), `check-pr-size.py` refuses (`the following +arguments are required: --base, --head`) and `check-agent-record.py` takes no +selector and validates the whole tree. None of the four can pass while measuring +nothing, so there is no third dud — but all four were previously named in this +section WITHOUT an invocation, and a bare `check-commit-style.py` in a gate line +is a line nobody can reproduce. Each now carries the exact invocation, and each +measures the BRANCH rather than the one commit at `HEAD`: + +```sh +python3 scripts/check-agent-record.py +python3 scripts/check-commit-style.py --range origin/main..HEAD +python3 scripts/check-commit-trailers.py --range origin/main..HEAD +python3 scripts/check-pr-size.py --base origin/main --head HEAD \ + --branch row/MODEL-MM-QWEN4-EXP-ROCM-RESIDENCY +python3 scripts/agent-pr-body.py --pr 3173 +``` + +`agent-pr-body.py` exits 0 when the body will land clean, 1 when the body fails +the contract and 3 when it could not be read. A 3 is `REMOTE_UNVERIFIED` and is +never a pass. + - The model gate: does the 67.56 GiB UD-IQ1_S at `/workspace/ckpt/qwen4exp-flash-next-iq1s` forward on `strix:gpu0` and produce a token? If it does, that is this row's G3, and load time, peak host and device @@ -223,7 +378,7 @@ as an ungated guarantee rather than chased with a contorted test. (`ISSUE-LOCAL-01M2BY2M2ATNVR3XQKV2DB1BJD`), so every measurement is repeated at least three times and reported as a spread. If it does not forward, NO number is recorded and the thread's `/proc//stat` state and `wchan` are reported - instead. + instead. §6a is that measurement. ## 6a. The model gate: MEASURED, AND THE STALL SURVIVES BOTH FIXES