feat(BACKEND-ROCM-QUANT-GATHER): gather packed embeddings on ROCm

[VikashLoomba]

Row

BACKEND-ROCM-QUANT-GATHER. Refs #3093.

What changed

ROCm now keeps every admitted quantized embedding table packed and decodes selected rows directly into the caller's F32 or BF16 output. The provider covers all 19 codecs, preserves full-width ID checks, and binds the queue device before allocation and launch. The Qwen3.5 GGUF reader admits the existing Q8_K block geometry; its embedding materializer accepts decoder-only formats while ordinary matrix weights retain their dot-kernel requirement. Public load/completion tests reach every format and compare exact tokens and logits with same-binary dense controls.

Before starting

Issue #3093 tracks the missing ROCm quantized embedding registration. The committed spec and its reader/materializer amendments precede implementation. This row uses one pull request. Its spec owns the child inventory; the parent backend lifecycle remains unchanged.

The branch includes the separately reviewed full-attention prerequisite from #3101. Its later runner test correction is imported byte-for-byte from 694f04525174d0c2dac8030ba8f1398f2683ec82. The gather change does not modify that prerequisite's product implementation.

Evidence

The gfx1100 operator executed all 15 final CTests without a skip. The primitive suite passes nine cases and 11,784 assertions; the public suite passes 2,850 assertions. All 456 completion files equal the full trace capture byte-for-byte. All 76 operation outputs meet the pinned primary/secondary gates; 68 are byte-identical, with the remaining eight within the original primary tolerance. The applicable secondary GET_ROWS geometry passes six exact F32/BF16-RNE comparisons. The required two-device queue, bounds, and recovery witness passes.

Matching rocprofv3 1.3.5 traces preserve each arm's untraced output bytes. Native operation allocations exactly match packed tables, I64 IDs, requested outputs, and constant 16-byte error buffers. The primary runs packed GPU selections followed by native decoders; its tensor memory counters are labeled separately from its profiler-visible allocator pool. A complete public trace passes 228 engines and 2,850 assertions, with BF16 KV reads/writes and exact token/logit repeats.

The commands and their receipts are retained in native-final-v4-commands.json and native-final-v4-operator-results.json. The operator ran:

ctest --test-dir build-rocm-quant-gather --output-on-failure --verbose -R '^(test_rocm_embedding_quant|test_capi_rocm_embedding_quant|test_gguf_keep_quant|test_ops_embedding_quant|test_gguf_dequant|test_backend_cross_device|test_gguf|test_rocm_arch|test_rocm_backend|test_rocm_fp8_kv_cache|test_ops_rocm_silu_rounding|test_exl3_rocm|test_gemma4_rocm_fp8_seams|test_backend_multidevice|test_runner)$'

The implementation head is c5bb0f1f71f1db2bac5f436189a6a018fb62cbc3. Exact-SHA path classification, commit style, and trailer checks pass. All 4,233 committed source, test, and build files match the frozen review manifest.

The complete library builds cleanly with HIP and CPU-only configurations. The CPU runner suite passes 41 cases and 1,914 assertions. Exact recipes, hashes, source anchors, oracle outputs, raw traces, and operator receipts are in docs/bench-evidence/rocm-quant-gather/.

• scripts/agent-preflight.sh --staged --quiet exits 0 with no failures and five argument-dependent skips. CPU ISA passes separately; ARM, CUDA code-generation, and CUDA Triton artifact checks do not apply to this HIP build.
• Fresh review of the immutable implementation passed with no findings. The completed mutation runs detect all 44 injected faults, including all 19 codec changes, removal of production call sites, bounds handling, BF16 rounding, device selection, malformed metadata, and empty-table behavior. Four positive GPU controls pass. A separate reviewer build passes five CPU regressions; three accelerator-only controls explicitly skip in that CPU build. The reviewer and operator each ran the complete preflight on this head: all 31 affected translation units compiled. CPU ISA and PR classification checks cover the applicable generic skips. The operator then reran all 15 native CTests and the required two-device witness; all passed. All 456 completion captures and 81 generated corpus files match the previous frozen run byte-for-byte.
• The changed feature statement is scoped to the existing gather row in docs/FEATURES.md.

Speed claims

• This pull request makes no speed claim.

Honest gaps

The bounded oracle comparison is qualified at this head: the comparator now rejects any primary record that carries an exception after generation and verifies the sealed model, configuration, prompt, repeat, dtype, and runtime identities plus the complete cache-view bounds before accepting a report. A focused validator test keeps the successful Q4_0 captures as its positive control and reddens when a post-generation exception is injected. A second review round then pinned every qualification guard with its own negative case on both the token and memory report paths (status-only refusal, exception and traceback shapes, wrong pins and runtime identities, wrong workload seals, FP32 dtype, an FP8 request, non-greedy sampling, a changed capacity contract, and invalid storage geometry), and registered the suite in the preflight SUITES array and the CI python lane so a guard regression fails a standing gate. The operator comparison of the 96-run primary matrix records 93 of 96 token-identical cases.

The three IQ3_S first-prompt mismatches are an ORACLE-SIDE codebook difference, not a native defect: native IQ3_S decode is bit-exact with the format-defining llama.cpp grid, while the pinned vLLM gguf plugin decodes 150 of 2048 codebook byte positions at 62 where the grid has 15 (effective magnitude 15.5 vs 15, +3.333%). A CPU substitution experiment (plugin-codebook rows placed into an otherwise identical dense model) reproduces the primary's exact divergent tokens, and stock llama.cpp reproduces the native tokens byte-for-byte. The finding, receipts, and two-sided citations are recorded with the row evidence, and it is reported upstream as vllm-gguf-plugin#129; that grid correction decides whether the three IQ3_S-p0 cases then pass or stay recorded oracle-blocked.

The original-fixture coverage is satisfied at this head. All 32 upstream fixtures were downloaded with the developer's authority and verified at 289,655,872 bytes; their 160 exported tensors over the 16 primary codecs were compared native-versus-plugin and pass with 320 outputs, 140 byte-exact, and every remaining element inside the unchanged upstream tolerance (atol=0.01, rtol=0.04). IQ1_M and the F16 output case remain the documented exclusions, and Q8_K, MXFP4, and IQ1_XXXS stay with their separate secondary evidence. The comparison report now labels its own scope from the run rather than a fixed string; the label defect found during this round is filed and fixed in flow as #3126.

This is ready for review. The exact Q4_0 model reaches the pinned primary renderer but it refuses Qwen3_5TextConfig where Qwen3_5Config is required, before tokens. The secondary llama.cpp oracles run through their committed one-line s_copy overlay and match native on all 12 bounded requests; the pristine binaries abort on their first decode, and the upstream runtime pads the requested 64-cell context to its own 256-cell minimum, so the matched-memory claim is scoped to the native/primary pair (both 65,536 bytes) and the secondary runs claim tokens and logits only.

The original 32 upstream fixtures await authority for their 289,655,872-byte download. Generated operation tables do not replace those fixtures. The fork's real 2.4T model qualification remains separately owed under #933. The row stays ACTIVE until the remaining gates pass. No CUDA/HIP combined-device execution is claimed.

Repairs after the review above, verified at this head. d896102f1 re-homes three evidence paths pr-size could not classify: the run's capture archive moves up beside its seven sibling archives with its sha256 unchanged (9f700062…a1bc7, still the value capture-manifest.json seals), and the two minimal-diff .patch artifacts become documents that reproduce each diff verbatim, record the original file's sha256, and point at the sealed full-context archive members. 74baf8c08 replaces the stale assertion that Q8_K never appears in a GGUF file — the reader's case 15 is correct and mirrors llama.cpp, which carries a real Q8_K traits row and validates its row data on load — with assertions that pin the reader geometry against vt::BlockElems/BlockBytes, the id round-trip, and the properties that actually distinguish Q8_K: no weight-side vec_dot, not keep-quant capable, and not taken by the generic composite. 79176b72e, 873807215 and fb4d78049 remove the same false claim from the four remaining comments that carried it, including the CUDA gather header that this branch is what made false, and repair the three line anchors this branch's own edits had made stale. A fresh reviewer mutated each guarantee in a scratch copy and returned PASS, including the classifier mutation that reproduces CI's unclassified-path error verbatim and the geometry, HasQuantDotKernel and restored-CHECK_THROWS mutations that each red the intended assertion; the operator independently reran the two CPU suites (10/10 cases and 6226/6226 assertions; 55/55 and 10350/10350).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Codex:gpt-6-astra [Codex CLI]

[VikashLoomba]

CI triage after the repair push — head fb4d78049.

Two branch-local failures were repaired here, and the corrections they implied were carried through the rest of the tree:

• pr-size rejected three paths in docs/bench-evidence/rocm-quant-gather/oracle-qualification/: the two minimal-diff .patch files, which have no admitted evidence class, and the run's capture archive, which sat one directory too deep. d896102f1 moves qualification-captures.tar.gz up beside the row's seven sibling archives (its sha256 is unchanged, 9f700062…a1bc7, still the value capture-manifest.json seals) and re-homes the two .patch artifacts as documents that reproduce each diff verbatim, record the original file's sha256, and point at the sealed full-context archive members. The checker now reports OK: every explicit path class is within its review budget. (rc=0); all 131 changed paths classify under origin/main's own copy of the checker as well.
• build-test-cpu (and both sanitize-cpu lanes) failed test_ops_quant_traits at tests/vt/test_ops_quant_traits.cpp:136, where CHECK_THROWS( vllm::GgmlTraits(15) ) asserted that Q8_K never appears in a GGUF file. The reader's case 15 on this branch is correct and mirrors llama.cpp, which carries a real Q8_K traits row (block geometry 256/292, ggml.c:879-884 at the pinned b10451) and validates Q8_K row data as it loads a tensor (llama-model-loader.cpp:1416 into ggml-quants.c:5590-5598). 74baf8c08 replaces the false absence with stronger assertions: the reader geometry agrees with vt::BlockElems/BlockBytes, the id round-trips, and Q8_K has no weight-side vec_dot, is not keep-quant capable, and is not taken by the generic composite. The operator independently reran it: test_ops_quant_traits 10/10 cases and 6226/6226 assertions, test_gguf_keep_quant 55/55 and 10350/10350.
• 79176b72e, 873807215 and fb4d78049 then removed the same falsified claim from the four remaining comments that carried it (two in the test file, plus include/vt/dtype.h, src/vt/cpu/cpu_quant_dequant.cpp, and src/vt/cuda/cuda_quant_dequant.cuh, where this branch is what made the claim false) and repaired the three line anchors this branch's own edits had made stale (quantization-matrix.md rows 72, 76 and 78). A fresh reviewer mutated each guarantee in a scratch copy and returned PASS: the classifier mutation reproduces CI's unclassified-path error verbatim, and the test mutations (wrong geometry, forced HasQuantDotKernel, the restored CHECK_THROWS shape) each red the intended assertion.

The remaining red checks are red in the same way on origin/main itself, in main's own run 34538060205:

Check	Failure	Main evidence
sanitize-cpu (address,undefined)	test_dots3_note_vision, test_dots3_note_audio, test_openai_api_server_dots3_mm_forward	job 103074166410 fails the same three tests
sanitize-cpu (thread)	test_gemma4_moe_fp8_native_arm_guard	job 103074166517 fails the same test (numbered 162 there, 163 here)
windows-msvc-cpu	test_openai_api_server.exe exits -1073740791 (0xC0000409) at localiser cases 58/59/61	job 103074166371 is the same crash and exit code
windows-msvc-vulkan	the same crash and exit code	job 103074166311

Both sanitize-cpu lanes also carried this branch's own test_ops_quant_traits failure, which 74baf8c08 removes. pr-size passes. mergeStateStatus is UNSTABLE because of the four pre-existing checks above.