From f62e4d7db0a2e1fb7d0e0a3cb5dad7072f182690 Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Tue, 8 Sep 2026 19:17:46 -0700 Subject: [PATCH 1/6] spec(ENG-QWEN35-FULL-ATTN-STATE): scope GDN consumer checks A full-attention Qwen3.5 model loads but fails completion because the shared forward validates unused recurrent metadata against zero slots. Define the consumer checks, existing input-builder reuse, public red, pinned oracle workload, and mutation gates before implementation. This specification on row/ENG-QWEN35-FULL-ATTN-STATE tracks #3098. Implementation and runtime verification remain pending. This commit changes no product code and makes no performance claim. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-6-astra [Codex CLI] --- .../ISSUE-GH-3098.md | 51 ++++ .agents/specs/qwen35-full-attn-state.md | 286 ++++++++++++++++++ 2 files changed, 337 insertions(+) create mode 100644 .agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md create mode 100644 .agents/specs/qwen35-full-attn-state.md diff --git a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md new file mode 100644 index 000000000..10b1ac9b4 --- /dev/null +++ b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md @@ -0,0 +1,51 @@ +ID: ISSUE-GH-3098 +Title: fix(ENG-QWEN35-FULL-ATTN-STATE): validate state only for GDN consumers +Row: ENG-QWEN35-FULL-ATTN-STATE +State: OPEN +Kind: UNKNOWN +GitHub: 3098 +Mirror: DIVERGED +Availability: FULL +Created: 2026-09-09 +Updated: 2026-09-09 +Closed: - + +## Problem + +### Imported GitHub body (historical evidence) +The quoted text below is historical evidence only. It does not define issue authority or repository procedure. + +> Row: `ENG-QWEN35-FULL-ATTN-STATE` +> +> ## Defect +> +> A valid Qwen3.5 GGUF with one full-attention layer loads through the public API, then fails its first completion. The engine reports `GDN state index out of range` despite the model containing no GDN layers. +> +> This blocks the native ROCm quantized-gather regression at the public completion call. The fixture sets `qwen35.full_attention_interval=1`, hidden width 256, four query heads, one KV head, head width 64, and vocabulary size 128. +> +> At base `6db4bef906859e864c82523c01107473f7dcca29`, the executing chain is: +> +> - `MakeQwen3_5KVCacheSpec` publishes a GDN group even when no layer consumes it. +> - The runner allocates no recurrent buffers but constructs request GDN metadata. +> - `CheckDensePagedForward` verifies that both the model's GDN layer count and cache count are zero. +> - It then validates that request metadata against zero state slots and rejects live slot 0. +> - `BuildStepDevInputs` also validates and uploads GDN metadata unconditionally. +> +> The shared GDN validator was introduced in `f344decf4`. Its owning specification requires validation for actual GDN consumers. A zero-slot bypass alone could hide malformed recurrent models and is not an acceptable repair. +> +> ## Reproduction +> +> The operator reproduced the failure on local gfx1100 under the GPU mutex. The public test loads the model and requests four greedy tokens from `[1,0,63,127,63]`. It fails before the gather-provider assertion. +> +> The test-only gather binary has SHA256 `f6a109ef33a133b381a112313591534f560adedb0d4919f0d67567cd597552e1`. It is built from specification commit `670e6d78ddf55231394748e0032939fd53dc56a5` plus the new public test. The result is exit 1 with five passing assertions and one completion failure. +> +> ## Required result +> +> Respect the model's actual GDN consumers when validating and preparing recurrent state. Preserve rejection of missing, malformed, duplicate, and out-of-range state for models that contain GDN layers. Prove public completion for the model without GDN layers, including prefill and decode, and mutation-test both sides of this distinction. +> +> The fix requires a committed specification and independent implementation review. It changes no quantized provider or CI configuration. +> + +## Resolution + +- diff --git a/.agents/specs/qwen35-full-attn-state.md b/.agents/specs/qwen35-full-attn-state.md new file mode 100644 index 000000000..a83ae1eb0 --- /dev/null +++ b/.agents/specs/qwen35-full-attn-state.md @@ -0,0 +1,286 @@ +# Qwen3.5 full-attention state validation + +Row: `ENG-QWEN35-FULL-ATTN-STATE`. + +Issue: [#3098](https://github.com/mudler/vllm.cpp/issues/3098). + +Branch: `row/ENG-QWEN35-FULL-ATTN-STATE`. + +Spec base: `6db4bef906859e864c82523c01107473f7dcca29`. + +## Now + +`PENDING`. This specification precedes implementation. The row uses one pull +request under the repository default. A fresh implementer works from this +committed specification. A fresh reviewer mutates the immutable implementation, +and the operator reruns its gates. The operator publishes the pull request. +The current task does not authorize merging it. + +Public completion fails before any quantized gather provider runs. Runtime +correctness, oracle execution, and implementation review remain `PENDING`. +This row makes no performance claim and establishes no performance floor. + +## Scope + +Allow Qwen3.5 models containing only full-attention layers to prefill and decode +without recurrent state. Select Gated DeltaNet (GDN) validation and preparation +from the actual layer consumers. Preserve strict validation for every model +that contains a GDN layer, including missing caches and malformed metadata. + +Cover the dense and mixture-of-experts (MoE) siblings in +`src/vllm/model_executor/models/qwen3_5.cpp`. Cover eager forward, graph entry, +and persistent device inputs because those paths share the same assumption. +Reuse the existing full-attention input builder. + +The planner still publishes its legacy empty `gdn` group. This is an existing +local adaptation, not a cache-topology parity claim. Changing group topology, +registry group indices, or runner allocation belongs outside this fix. + +Other exclusions are quantized providers, F16 matrix multiplication, fused MoE +kernels, new graph backends, new speculative modes, checkpoint downloads, +package installation, continuous integration changes, and global oracle pins. +The fix has no dependency on the separate gfx1100 provider changes. + +## Inventory + +This per-row inventory owns the child identity through the canonical spec scan. +No parent matrix or roadmap lifecycle changes. Update this table and `Now` +together when this child changes state. + +| ID | Upstream source | Local anchor | Tests and evidence | Spec | State | Owner | Issue | +|---|---|---|---|---|---|---|---| +| `ENG-QWEN35-FULL-ATTN-STATE` | Pinned `Qwen3_5DecoderLayer` and `GPUModelRunner.get_kv_cache_spec` | `CheckDensePagedForward`, `CheckPagedForward`, `BuildFullAttnStepDevInputs`, both graph `Step` methods | G0 to G5 | This file | `PENDING` | Row helper, fresh reviewer, operator verification | #3098 | + +## Diagnosis and source anchors + +At the spec base, the complete failing chain is: + +1. `src/vllm/model_executor/models/qwen3_5_common.cpp::MakeQwen3_5KVCacheSpec`, + lines 75 to 99, always publishes a `gdn` group. +2. `src/vllm/v1/worker/gpu/runner.cpp::GPUModelRunner::initialize_kv_cache` + allocates recurrent buffers only for `linear_attention` layers at lines + 1511 to 1513. This model has none. +3. `src/vllm/v1/worker/gpu/runner.cpp::GPUModelRunner::execute_model`, lines + 2779 to 2805, builds and remaps GDN metadata when that group exists. +4. `src/vllm/model_executor/models/qwen3_5.cpp::CheckDensePagedForward`, lines + 9240 to 9249, accepts zero GDN layers and zero caches. It then validates + live slot 0 against zero slots and throws at `ValidateGdnStateIndices`. +5. `BuildStepDevInputs`, lines 4659 to 4682, independently validates and + prepares the same unused metadata. Repairing the first check alone fails. + +The sibling `CheckPagedForward` repeats this assumption at lines 8526 to 8536. +`ForwardLayers` and `DenseForwardLayers` call the GDN builder at lines 8380 and +9323. Both graph `Step` methods validate GDN state before their eager fallback, +at lines 10763 and 11391. Persistent inputs repeat the builder at lines 11036 +and 11774. The graph staging functions already test the GDN presence flags. + +`git log -S'ValidateGdnAttentionMetadata'` identifies `f344decf4` and +`3ae5cfe06`. The original [packed-decode specification](gdn-packed-decode.md) +records why shared validation protects real consumers. Preserve that guarantee. + +The existing `BuildFullAttnStepDevInputs`, line 7845, uploads full-attention +metadata and leaves GDN fields inert. Its current consumer is the multi-token prediction (MTP) draft +head. Reusing it avoids a second full-attention preparation implementation. + +## Pinned reference and adaptations + +The [primary pin](../upstream-sync.md) is +`e126687a9a828d513c01a07cd69f025f27d63280`, runtime +`0.28.1rc1.dev132+ge126687a9`. Read these executing upstream anchors: + +- `vllm/model_executor/models/qwen3_5.py::Qwen3_5DecoderLayer.__init__`, + lines 144 to 160, constructs GDN only for `linear_attention`. +- `vllm/model_executor/models/qwen3_5.py::Qwen3_5Model.__init__`, lines 249 + to 259, selects every decoder from `config.layer_types`. +- `vllm/v1/worker/gpu/model_runner.py::GPUModelRunner.get_kv_cache_spec`, + lines 530 to 531, delegates to + `vllm/v1/worker/gpu/attn_utils.py::get_kv_cache_spec`, lines 52 to 65. + This V2 runner chain enumerates modules that actually require caches. +- `vllm/model_executor/layers/mamba/abstract.py::MambaBase.get_kv_cache_spec`, + lines 67 to 87, obtains recurrent shapes from a recurrent consumer. + +The exact no-GDN completion case is a local regression. The inspected upstream +`tests/models/language/generation/test_common.py`, `test_hybrid.py`, +`tests/models/test_qwen3_5_mtp_config.py`, and +`tests/v1/worker/test_gpu_model_runner.py` do not contain that case. +Preserve the existing ported GDN cases and their upstream anchors. Do not turn +unrelated KV-sharing allocation tests into a new topology obligation. + +The operator's pinned runtime reports `Using V2 Model Runner`. The V2 chain +is the primary execution anchor for qualification. + +Run the primary oracle on the identical generated dense GGUF before accepting +the fix. The [GGUF plugin pin](../oracles/vllm-gguf-plugin.md) is +`d4c1f0d082fc7cd4350da56689109a01c1f29d6c`. Its +`vllm_gguf_plugin/config_parser.py::GGUFConfigParser.parse` reads a local +configuration beside the artifact. Supplying that configuration and explicit +token IDs is an allowed harness adaptation. Preserve all model tensors and +geometry. A successful import or config construction is not an oracle result. + +Use the existing pinned runtime, plugin artifact, and dependencies supplied by +the operator. Record their source revisions and binary hashes. Missing runtime +registration or unsupported full-attention execution leaves the oracle gate +`PENDING`. Do not install a package, substitute safetensors, change the model, +or use another oracle without an explicit scope decision. Do not force eager +execution on the oracle. Record its resolved production configuration. + +The operator observed a missing `mm_proj` refusal on the sibling F16 oracle +workload. The pinned plugin's +`vllm_gguf_plugin/weights_adapter/qwen3_5.py::QWEN35_ARCHITECTURES`, line 40, +maps `qwen3_5_text` to conditional generation. Its `Qwen35GGUFAdapter.patch_hf_config` +replaces the architecture at line 225. Upstream +`vllm/model_executor/models/qwen3_5.py::Qwen3_5ForConditionalGeneration.__init__`, +lines 503 to 517, dereferences the vision configuration and constructs a tower. +A text configuration alone does not establish a runnable text-only GGUF path. +This known risk leaves G4 pending on an oracle refusal. + +## Design + +1. Derive the presence of GDN consumers from `weights.layers` and + `is_linear_attention`. Reuse one internal predicate where practical. +2. Keep layer-count, attention-cache-count, GDN-cache-count, and cache-layout + checks unconditional. A hybrid model with no caches must still fail. +3. Run GDN token-count, metadata, and graph-state validation only when a GDN + consumer exists. Keep the validators themselves strict, including direct + `BuildGdnStepInputs` consumers outside this model. +4. Select `BuildFullAttnStepDevInputs` for a model with no GDN consumers. + Apply the selection to both eager builders and both persistent graph builders. + Keep GDN upload and staging flags false on that path. +5. Pass inert empty GDN metadata into graph padding when no consumer exists. + `BuildPaddedDecode` must not copy unvalidated, arbitrary-length GDN indices + into its S-entry storage. Retain the bounds enforced for actual consumers. +6. Keep generic attention and graph shape checks effective for both model + topologies. Preserve real-GDN decode, prefill, mixed, speculative, and padded + state validation. Do not use GDN metadata as the only generic shape check. + +Never infer the absence of consumers from `state_slots == 0` or an empty cache +vector. Never add dummy recurrent state, a fake layer, a validation bypass flag, +or a fixture geometry change. Keep model activations and key-value (KV) storage at their +resolved bf16 dtype. Retain existing annotated f32 exceptions and integer +metadata layouts. This change adds no wider model buffers or recurrent tensors. + +## Regression fixture + +Use only the dense control from the frozen gather experiment. Extract its +minimal deterministic generator into this row's own fixture file. Preserve the +bytes of `Q4_0-dense.gguf`, even though its embedding and projections are dense. +Do not copy the 19-format provider campaign into this regression. + +The artifact has 991296 bytes and SHA256 +`0e6554ba521edfde00d4d025a3058eabaaacdce6b24344f959e83d8dde35df7f`. +Its geometry is hidden width 256, intermediate width 256, vocabulary 128, +four query heads, one KV head, head width 64, one full-attention layer, and +`full_attention_interval=1`. Rotary width is 64, base is 1000000, and MRoPE +sections are `[16,8,8,0]`. Norm epsilon is `1e-6`, seed is `0x524f434d`. +Inactive state-space model (SSM) loader keys retain convolution width 4, inner width 64, state width +64, time-step rank 1, and group count 1. + +Prompts are `[1,0,63,127,63]` and `[1,127,0,127]`. Request exactly four greedy +tokens with `ignore_eos=1` and seed `0x524f434d`. Use block size 16, four blocks, +maximum length 64, and one sequence. Public device 0 selects AUTO in the HIP-only +build. Assert the actual ROCm dense embedding provider executes. + +For both local and oracle runs, execute each prompt three times with a fresh +engine and fresh model state. Keep batch size and concurrency at one. Submit +prompts sequentially and record the actual scheduling and token budgets. + +## Gates and evidence + +| Gate | Required result | Current result | +|---|---|---| +| G0: independent red | New public regression on pristine product base fails at GDN validation after load | `PENDING`: operator reproduction exists, row-owned test is owed | +| G1: public completion | Both prompts produce four tokens in three fresh-engine repeats, native ROCm provider executes | `PENDING`: operator GPU run | +| G2: consumer distinction | Dense and MoE production routes accept no-GDN state and reject damaged real-GDN state | `PENDING`: fresh implementer | +| G3: graph preparation | Both drivers cover cold, capture, persistent staging, and replay routing with no GDN consumers | `PENDING`: fresh implementer and operator | +| G4: pinned oracle | Same dense artifact and all six fresh-engine runs, exact token IDs, finite logits, preserved dtypes | `PENDING`: operator runtime execution | +| G5: review and full gate | Focused gates, mutations, full preflight, fresh review, and operator rerun | `PENDING`: implementation | + +G0 starts at `vllm_engine_load` and `vllm_complete_tokens` from `include/vllm.h`. +The test must assert successful load before its expected pre-fix completion +failure. Reproduce with the row's fresh build from unchanged base product code. +The preserved gather red establishes the diagnosis, not this independent gate. + +For G2, enter `ModelRegistry::Forward` for both dense and MoE variants. Cover +prefill and decode without GDN consumers, with default-empty and unused +runner-style GDN metadata. For real GDN consumers, reject missing all caches, +wrong cache count, invalid ranks, inconsistent slots, missing metadata, +duplicate indices, and out-of-range indices. Keep valid hybrid completion green. +A validator-only unit test does not prove these production call sites execute. + +For G3, extend the existing graph harness for both drivers. Exercise their +production `Step` methods, graph fallback, persistent inputs, and staging flags. +Include no-GDN metadata with an oversized unused index vector. The padding path +must ignore that vector safely. Mutate the inert-metadata selection and require +this regression to fail. Use a memory-sanitized run if needed to expose an +out-of-bounds copy reliably. +Trace the registry dispatch to each driver and mutate each changed call site. +A CPU fake replay proves routing only. Record that limitation and obtain an +operator GPU replay result for any claimed replay numerics. Do not enable a +new backend or claim a GPU graph executed from the CPU harness. + +Register a dedicated public test such as `test_capi_qwen35_full_attn_state`. +Run it with `test_qwen27_paged_forward`, `test_qwen35_paged_forward`, +`test_qwen3_5_decode_graph_seam`, and `test_model_registry`. Retain existing +GDN state, speculative metadata, and graph-padding cases in those suites. +Build in a row-owned ignored `build-*` directory with at most `-j 4`. +The implementer records exact configure, build, and focused test commands +before handing GPU commands to the operator. No shared build directory is valid. + +For every result, record the immutable source SHA, binary SHA256, fixture hashes, +command, environment, exit status, and evidence path. G4 additionally records +source anchors, runtime and plugin identity, output tokens, resolved dtype, +attention backend, and graph mode. No config-only or local self-comparison +substitutes for G4. A token mismatch remains failing until explained and fixed. + +Fresh review uses scratch copies and restores every modified file byte-for-byte. +Restore unconditional GDN validation at each newly conditional entry. Restore +the GDN builder at each full-attention selection. Delete the production call +sites into the changed preparation path. Each relevant positive gate must fail. +Then force the no-GDN branch for a hybrid model, or remove cache-count +validation, and prove the corresponding malformed-hybrid gate fails. Mutate duplicate, range, missing +metadata, and layout guarantees individually. Keep per-mutation failure logs. + +Run `scripts/agent-preflight.sh` before edits and the staged form before commit. +Run exact-range record, commit-style, trailer, and PR-size checks after commit. +Classify omitted hardware and build gates explicitly. The operator repeats the +applicable gates on the immutable implementation before publication. + +## Existing evidence + +On 8 September 2026 PDT, the operator ran the frozen gather public binary on local +gfx1100 under `/home/vikash/gpu.lock`. Completion threw +`qwen3_5: GDN state index out of range`. The process returned 1 with five +passing assertions and one failed completion assertion. + +The binary SHA256 is +`f6a109ef33a133b381a112313591534f560adedb0d4919f0d67567cd597552e1`. +Its source was spec commit `670e6d78ddf55231394748e0032939fd53dc56a5` plus +uncommitted test-only changes. The fixture header SHA256 is +`339414f93e59e6be9a4ca545d5e762bb2816714343f466e218dbc2b69483bdfd`. +The public test source SHA256 is +`add58ac23eef26588fb358d7679feeac961baee3a1da3d5ddd3adedaf9a1a825`. +The log is `/home/vikash/.cache/rdna3-gather-impl/evidence/public-red.log`, SHA256 +`fbd5d684566961d71f31c8b9e58daba8d5303636adb12ceccb164dbe3b220e1d`. +These are supplied local evidence paths, not environment defaults. + +## Risks and stop conditions + +- `NEEDS_CONTEXT`: the frozen fixture bytes, pinned runtime, or source cannot + be verified. The operator owns the missing artifact or runtime decision. +- `NEEDS_DECISION`: the oracle requires a different model artifact, new + dependency, or behavior outside the stated scope. +- `BLOCKED`: required GPU execution is unavailable. Only the operator runs + the GPU work under the recorded mutex or applicable fleet lease. +- `FAILING`: a no-GDN route still prepares device recurrent metadata, a real-GDN guard + weakens, or a mutation remains green. Repair through a fresh implementer. +- `FAILING`: graph coverage relies on fake replay numerics or a different + untested sibling. Keep the affected gate pending until valid evidence exists. +- `NEEDS_DECISION`: the minimal fix requires planner topology, unrelated + quantization, broader speculative behavior, or any excluded file change. + +## Owed + +Issue #3098 owns implementation, independent red evidence, pinned oracle +execution, review mutations, and the operator rerun. Record the measured +outcome and defaults here before changing the row to `DONE`. Keep the issue +open until the work lands. From 25bea3e67597f6700fc1f2e9cfb5269948338c6d Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Tue, 8 Sep 2026 20:32:11 -0700 Subject: [PATCH 2/6] fix(ENG-QWEN35-FULL-ATTN-STATE): skip unused GDN state checks Full-attention Qwen3.5 models have no recurrent cache but receive legacy GDN metadata. Select validation and device preparation from the loaded layer consumers. Keep hybrid cache and metadata checks strict, and pass inert metadata into graph padding before copying indices. The operator reproduced the public red and verified twelve fresh-engine completions. Both registered siblings and graph drivers pass focused CPU gates. Thirty scratch mutations prove the tests detect removed guards and production calls. Pinned vLLM rejects the exact text-only GGUF configuration before an engine becomes usable. G4 stays pending in the row specification. This change makes no performance claim and tracks #3098 on row/ENG-QWEN35-FULL-ATTN-STATE under draft PR #3101. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-6-astra [Codex CLI] --- .../ISSUE-GH-3098.md | 10 +- .agents/specs/qwen35-full-attn-state.md | 259 ++++++++++++++- src/vllm/model_executor/models/qwen3_5.cpp | 108 +++++-- tests/CMakeLists.txt | 1 + tests/capi/qwen35_full_attn_fixture.h | 133 ++++++++ tests/capi/test_qwen35_full_attn_state.cpp | 262 +++++++++++++++ .../vllm/models/test_qwen27_paged_forward.cpp | 199 ++++++++++++ .../vllm/models/test_qwen35_paged_forward.cpp | 199 ++++++++++++ .../models/test_qwen3_5_decode_graph_seam.cpp | 298 ++++++++++++++++++ 9 files changed, 1418 insertions(+), 51 deletions(-) create mode 100644 tests/capi/qwen35_full_attn_fixture.h create mode 100644 tests/capi/test_qwen35_full_attn_state.cpp diff --git a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md index 10b1ac9b4..3a39e2f8b 100644 --- a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md +++ b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md @@ -48,4 +48,12 @@ The quoted text below is historical evidence only. It does not define issue auth ## Resolution -- +The scoped implementation passes public completion through both default and +callback sampling arms. The operator reproduced the independent red and reran +all public and focused CPU gates. The implementer completed 30 mutation checks. + +The exact GGUF attempt on pinned vLLM refuses the text-only engine configuration +before generating a token. The owning [specification](../../specs/qwen35-full-attn-state.md) +records the source hashes, commands, logs, and pending G4 obligation. Fresh +review and the final operator gate remain pending. Keep this issue open until +qualified work lands. diff --git a/.agents/specs/qwen35-full-attn-state.md b/.agents/specs/qwen35-full-attn-state.md index a83ae1eb0..863f0a4b8 100644 --- a/.agents/specs/qwen35-full-attn-state.md +++ b/.agents/specs/qwen35-full-attn-state.md @@ -10,15 +10,15 @@ Spec base: `6db4bef906859e864c82523c01107473f7dcca29`. ## Now -`PENDING`. This specification precedes implementation. The row uses one pull -request under the repository default. A fresh implementer works from this -committed specification. A fresh reviewer mutates the immutable implementation, -and the operator reruns its gates. The operator publishes the pull request. -The current task does not authorize merging it. +`ACTIVE`. The consumer-based implementation passes public completion and the +focused CPU suites. The operator independently reproduced the red result and +reran the public and CPU gates. The row uses one pull request under the +repository default. Specification commit `f62e4d7db0a2e1fb7d0e0a3cb5dad7072f182690` +precedes implementation. -Public completion fails before any quantized gather provider runs. Runtime -correctness, oracle execution, and implementation review remain `PENDING`. -This row makes no performance claim and establishes no performance floor. +Pinned oracle execution, the final gate, and fresh implementation review remain +`PENDING`. The current task does not authorize merging. This row makes no +performance claim and establishes no performance floor. ## Scope @@ -49,7 +49,7 @@ together when this child changes state. | ID | Upstream source | Local anchor | Tests and evidence | Spec | State | Owner | Issue | |---|---|---|---|---|---|---|---| -| `ENG-QWEN35-FULL-ATTN-STATE` | Pinned `Qwen3_5DecoderLayer` and `GPUModelRunner.get_kv_cache_spec` | `CheckDensePagedForward`, `CheckPagedForward`, `BuildFullAttnStepDevInputs`, both graph `Step` methods | G0 to G5 | This file | `PENDING` | Row helper, fresh reviewer, operator verification | #3098 | +| `ENG-QWEN35-FULL-ATTN-STATE` | Pinned `Qwen3_5DecoderLayer` and `GPUModelRunner.get_kv_cache_spec` | `CheckDensePagedForward`, `CheckPagedForward`, `BuildFullAttnStepDevInputs`, both graph `Step` methods | G0 to G5 | This file | `ACTIVE` | Row helper, fresh reviewer, operator verification | #3098 | ## Diagnosis and source anchors @@ -188,12 +188,12 @@ prompts sequentially and record the actual scheduling and token budgets. | Gate | Required result | Current result | |---|---|---| -| G0: independent red | New public regression on pristine product base fails at GDN validation after load | `PENDING`: operator reproduction exists, row-owned test is owed | -| G1: public completion | Both prompts produce four tokens in three fresh-engine repeats, native ROCm provider executes | `PENDING`: operator GPU run | -| G2: consumer distinction | Dense and MoE production routes accept no-GDN state and reject damaged real-GDN state | `PENDING`: fresh implementer | -| G3: graph preparation | Both drivers cover cold, capture, persistent staging, and replay routing with no GDN consumers | `PENDING`: fresh implementer and operator | -| G4: pinned oracle | Same dense artifact and all six fresh-engine runs, exact token IDs, finite logits, preserved dtypes | `PENDING`: operator runtime execution | -| G5: review and full gate | Focused gates, mutations, full preflight, fresh review, and operator rerun | `PENDING`: implementation | +| G0: independent red | New public regression on pristine product base fails at GDN validation after load | Satisfied: operator reproduced the row-owned failure after public load | +| G1: public completion | Both prompts produce four tokens in three fresh-engine repeats, native ROCm provider executes | Satisfied: 12 fresh engines, default and callback sampling arms | +| G2: consumer distinction | Dense and MoE production routes accept no-GDN state and reject damaged real-GDN state | Satisfied: focused suites and independent operator rerun | +| G3: graph preparation | Both drivers cover cold, capture, persistent staging, and replay routing with no GDN consumers | Satisfied for CPU routing and staging, no GPU replay numerical claim | +| G4: pinned oracle | Same dense artifact and all six fresh-engine runs, exact token IDs, finite logits, preserved dtypes | `PENDING`: pinned plugin rejects the actual text-only engine configuration | +| G5: review and full gate | Focused gates, mutations, full preflight, fresh review, and operator rerun | `PENDING`: fresh review and final operator gate | G0 starts at `vllm_engine_load` and `vllm_complete_tokens` from `include/vllm.h`. The test must assert successful load before its expected pre-fix completion @@ -263,6 +263,230 @@ The log is `/home/vikash/.cache/rdna3-gather-impl/evidence/public-red.log`, SHA2 `fbd5d684566961d71f31c8b9e58daba8d5303636adb12ceccb164dbe3b220e1d`. These are supplied local evidence paths, not environment defaults. +## Implementation evidence + +Measurements on 9 September 2026 UTC use the row-owned worktree +`/home/vikash/vllm.cpp-qwen35-full-attn-state-impl`. Evidence is under +`/home/vikash/.cache/qwen35-full-attn-state-impl/evidence`. These paths identify +measured artifacts. They are not environment defaults. + +The product source is `qwen3_5.cpp`, SHA256 +`899913b8aeff65a08eb5289f6f08777ee78638e37722f9d8138daff723f8f498`. +The red source is the unchanged product at specification commit +`f62e4d7db0a2e1fb7d0e0a3cb5dad7072f182690`. Green snapshots carry that base, +the exact implementation patch, and every test source hash. The implementation +commit containing this record makes those bytes reachable from Git. + +### Public red and completion + +The operator generated the row's fixture and confirmed its 991296-byte size +and the frozen SHA256. The row-owned red binary has SHA256 +`76c4e04bfb1b2715d4010e11a1d85cb076c7ada8e146094a1419fc0566d6efff`. +It returned 1 after `public load succeeded`, with 11 passing assertions and +one failed completion assertion. The error was `GDN state index out of range`. +`red-snapshot/public-red-operator.log` has SHA256 +`dbf8bb09eccc4d626d0c6cc4a9d97d93738563954dae0aeefa32b9ed5bfd8981`. + +The green binary has SHA256 +`14d2fee7f63df4c8ab7721923d429822488e7ac55a058268b9145bd4d5f371a7`. +The operator ran all 12 fresh engines and passed all 3364 assertions. +`green-v1-snapshot/public-green-operator.log` has SHA256 +`6318b014c368835a71fe2501444adaf61b3312df178c564d0a0d3fba820f9c4e`. +The two token sequences were `[47,19,4,20]` and `[11,28,104,78]`. +Each sequence was identical in three repetitions of both sampling arms. + +The callback-free arm retains default device sampling. The callback arm +observes 512 finite logits per completion. A custom processor stages logits +through the host at `src/vllm/v1/sample/logits_processor/builtin.cpp::apply_logits_processors`, +lines 75 to 159. Both arms reach `vt::GreedyArgmax` through the sampler. +Each run executes one native ROCm embedding prefill, three embedding decodes, +four cache writes, and four greedy operations. Verified named providers call +the original native functions. They check bf16 embedding activations and KV +storage, f32 sampler logits, and i64 sampled IDs. Reference-tier hits do not +increase. The f32 logits belong to the existing sampler ABI. + +Both public commands use the operator's local GPU 0 and mutex: + +```sh +env HIP_VISIBLE_DEVICES=0 ROCR_VISIBLE_DEVICES=0 \ + TMPDIR=/home/vikash/.cache/qwen35-full-attn-state-impl/tmp \ + GIT_CEILING_DIRECTORIES=/home/vikash/.cache/qwen35-full-attn-state-impl/tmp \ + GIT_CONFIG_GLOBAL=/dev/null \ + flock -n -F /home/vikash/gpu.lock \ + /home/vikash/.cache/qwen35-full-attn-state-impl/evidence/green-v1-snapshot/test_capi_qwen35_full_attn_state +``` + +Use `red-snapshot` in the executable path for the frozen red command. +Each snapshot's `manifest.json` contains exact argument arrays and source +hashes. The operator receipts record execution and restoration checks. + +### CPU consumers and graph routing + +The implementer and operator passed these complete suites: + +| Suite | Test cases passed | Assertions passed | +|---|---|---| +| `test_model_registry` | 24 | 993 | +| `test_qwen27_paged_forward` | 36 | 1406 | +| `test_qwen35_paged_forward` | 10 | 676 | +| `test_qwen3_5_decode_graph_seam` | 16 | 1550 | + +The registry suite retains one pre-existing skipped case. All new cases run. +`cpu-green-v1-snapshot/manifest.json` has SHA256 +`575e9695acc9600515050a6ba0ed509795df45f7a128b98c6c303eb87c005183`. +It freezes all four executables and their source files. The operator's +per-binary logs, receipts, and `operator-results.json` are beside the manifest. + +The no-GDN tests enter both registered forwards for prefill and decode. +Empty metadata, unused runner metadata, and stale configuration labels all +reach the loaded layer consumers. Separate negative cases preserve each +hybrid cache-count, rank, slot-layout, metadata, duplicate, and range guard. +Existing speculative, prefill, mixed, and padded-state tests remain green. + +Both graph drivers cover capture enabled and disabled, asynchronous staging +enabled and disabled, batches 1 and 3, and three unused metadata forms. +Six steps cover cold execution, capture, persistent input binding, and replay +routing. Batch 3 pads to 4. Oversized unused indices contain 4096 entries. +Each persistent slot binds exactly five generic inputs, with no GDN input. + +The dense registry reaches its actual graph driver under the existing CPU +harness. The MoE registry needs its FP4 capability predicate. Its test uses +valid small NVFP4 expert tensors and a scoped CPU platform answer. A control +without that answer stays eager. Both use the existing CPU arithmetic and fake +graph replay. These cases prove routing and staging, not GPU replay numerics. +No backend or quantized provider changes. + +The shared builder's existing consumers also pass: +`test_qwen3_5_gguf_mtp`, `test_qwen35_exl3`, and +`test_qwen3_dflash2_draft`. Their complete CPU run returned 0. + +### Build and focused commands + +Run from the row worktree. Set `TMPDIR` and `GIT_CEILING_DIRECTORIES` to +`/home/vikash/.cache/qwen35-full-attn-state-impl/tmp`, and set +`GIT_CONFIG_GLOBAL=/dev/null` for gate subprocesses. Configure commands were: + +```sh +cmake -S . -B build-full-attn-cpu -G Ninja \ + -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ + -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_HIP=OFF -DVLLM_CPP_VULKAN=OFF \ + -DVLLM_CPP_METAL=OFF -DVLLM_CPP_MLX=OFF -DVLLM_CPP_TENSTORRENT=OFF \ + -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_BUILD_EXAMPLES=OFF -DVLLM_CPP_SERVER=OFF +cmake -S . -B build-full-attn-hip -G Ninja \ + -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ + -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_HIP=ON -DVLLM_CPP_HIP_ARCHITECTURES=gfx1100 \ + -DVLLM_CPP_VULKAN=OFF -DVLLM_CPP_METAL=OFF -DVLLM_CPP_MLX=OFF \ + -DVLLM_CPP_TENSTORRENT=OFF -DVLLM_CPP_TRITON=OFF \ + -DVLLM_CPP_BUILD_EXAMPLES=OFF -DVLLM_CPP_SERVER=OFF +cmake --build build-full-attn-cpu -j 4 --target test_model_registry \ + test_qwen27_paged_forward test_qwen35_paged_forward \ + test_qwen3_5_decode_graph_seam test_capi_qwen35_full_attn_state +cmake --build build-full-attn-hip -j 4 --target test_capi_qwen35_full_attn_state +ctest --test-dir build-full-attn-cpu --output-on-failure \ + -R '^(test_qwen27_paged_forward|test_qwen35_paged_forward|test_qwen3_5_decode_graph_seam|test_model_registry)$' +cmake --build build-full-attn-cpu -j 4 --target test_qwen3_5_gguf_mtp \ + test_qwen35_exl3 test_qwen3_dflash2_draft +ctest --test-dir build-full-attn-cpu --output-on-failure \ + -R '^(test_qwen3_5_gguf_mtp|test_qwen35_exl3|test_qwen3_dflash2_draft)$' +``` + +All configure, final build, and green focused commands returned 0. +The original CPU red returned 8 from CTest. It detected both intended GDN +errors. Two valid-hybrid test assertions initially assumed host logits. The +registered forward returns device logits, so the tests now download them for +finite-value checks. No product behavior changed for that test correction. +`cpu-red-snapshot` preserves the original tests, product source, and binaries. + +### Implementer mutations + +The three address-sanitized baselines pass. All 30 mutations fail their focused +gates. `mutations/manifest.json` has SHA256 +`9f2cb023ea64af396a2fccbbe83b0dd8fc67a591aed9403c399cfa13f7a515ff`. +The external recipe `run-mutations.py` has SHA256 +`a0855b588f8d2e7a7286ee9a56b2a0551f8111059537e264127a2f6c9787b3b2`. + +Each mutant compiles a separate source copy at `-O0 -g -fsanitize=address` +and links the row's own CPU archive. The archive starts fresh for each mutant. +The corresponding baseline uses the same compiler and linker flags. +`ASAN_OPTIONS=detect_leaks=0:abort_on_error=1` disables unrelated leak reporting. +No tracked source or normal build output changes. Each manifest entry records +its command, mutated source hash, binary hash, failed exit, and restoration. + +For each sibling, mutations restore unconditional entry validation, force the +entry's no-consumer branch, remove cache-count validation, and restore the GDN +eager builder. They independently restore graph validation, GDN padding-size +restrictions, unused spec classification, and persistent GDN preparation. +Removing each pre-padding shape check fails. Restoring the unused GDN index +copy causes an address-sanitizer heap-buffer-overflow in both drivers. + +Six shared mutations separately remove rank, paired-slot, cross-layer-slot, +duplicate, range, and missing-index guarantees. Both registry suites fail +for each removal. Four further mutations delete each registry's graph and +eager production call. Each relevant positive gate fails. Fresh independent +review must repeat the required mutations on the immutable implementation. + +### Full gate and build classification + +`preflight-start.log` records the full gate before edits. +`preflight-staged.log` records the staged run before the implementation commit. +The staged run uses the same temporary-directory and Git isolation as the +focused tests, plus the existing NumPy dependency path supplied by the operator. +An external argument wrapper adds `--jobs 4` only to the compile checker. +It preserves every gate and changes no tracked script. + +The generic preflight sweep does not supply every build checker's arguments. +The x86 CPU ISA audit runs explicitly against +`build-full-attn-cpu/compile_commands.json` and passes in `cpu-isa-build.log`. +The exact-range checks supply the recorded product base and implementation +head, with draft PR 3101 for path classification. Their results belong in +`range-gates.json` beside the other evidence. + +ARM ISA, CUDA fat-binary, and Triton AOT build audits are outside this row's +configured CPU and HIP builds. The change edits no architecture-specific +instruction unit, CUDA gencode setting, or Triton artifact. The normal public +test returns skip code 77 in a CPU-only build because public AUTO requires the +HIP-only configuration here. The operator's HIP run executes every assertion. +GPU graph replay numerics remain unclaimed. G4 and fresh review remain pending. + +### Pinned oracle refusal + +The operator ran the actual first engine against the unchanged dense GGUF. +The engine returned 1 before becoming usable and emitted no tokens. The +exception reports `Qwen3_5Config` required and `Qwen3_5TextConfig` received. +The executing rejection is in upstream +`vllm/multimodal/processing/context.py::InputProcessingContext.get_hf_config`, +line 140, called by `vllm/model_executor/models/qwen3_5.py::Qwen3_5ProcessingInfo.get_hf_config`, +line 108. +The complete traceback is preserved in the report and operator log. + +The runtime reports vLLM `0.28.1rc1.dev132+ge126687a9`, PyTorch +`2.12.0+git6bbd260`, HIP `7.2.53211`, and plugin +`0.0.5+d4c1f0d.gfx1100`. The script supplies only local HF text metadata and +explicit token IDs. Model geometry, tensor bytes, inactive SSM keys, and MRoPE +sections match the frozen fixture. Production graph mode remains requested +with `enforce_eager=False`. No registry, model, or plugin correction is applied. + +`oracle-exact-gguf-v1-command.json` records the exact Docker argument arrays, +image digest, read-only runtime and plugin mounts, nonroot user, private +2 GiB IPC allocation, offline settings, and GPU 0 mutex. Each of its six +commands starts a new process and engine. The operator stopped after the first +refusal, because the remaining five commands cannot supply missing runtime +capability. No token, finite-logit, resolved-dtype, or GPU graph result is claimed +for the oracle. + +The attempt report `oracle-exact-gguf-v1-p0-r0.json` has SHA256 +`46d714e5fe6577663b0cb266c8d88c9a4a629eb2727314775a5b5dbb605ac2c0`. +Its operator log has SHA256 +`5244fbca9f48db1bfc6d9817d823fcc4e1fe1a48a89f1c04c216ad1a5db4cdc6`. +The script SHA256 is +`504162e44ba828249a1a825bf2053f53e7902f168bad3ba515334ff8440a7f3e`. +The local HF configuration SHA256 is +`6c4b2c6f3d71b90818730bbfa900a5cbb0a96bd64ab656fd9db8e65111c05638`. + +G4 remains `PENDING` on the operator's pinned GGUF runtime capability. +The implementation does not substitute a model or oracle. Issue #3098 retains +ownership of this gate, independent review, and the final operator gate. + ## Risks and stop conditions - `NEEDS_CONTEXT`: the frozen fixture bytes, pinned runtime, or source cannot @@ -280,7 +504,8 @@ These are supplied local evidence paths, not environment defaults. ## Owed -Issue #3098 owns implementation, independent red evidence, pinned oracle -execution, review mutations, and the operator rerun. Record the measured +Issue #3098 owns the pending pinned oracle execution, fresh review mutations, +and the final operator gate. Public red, local completion, consumer validation, +and CPU graph routing have measured evidence above. Record the measured outcome and defaults here before changing the row to `DONE`. Keep the issue open until the work lands. diff --git a/src/vllm/model_executor/models/qwen3_5.cpp b/src/vllm/model_executor/models/qwen3_5.cpp index 83ea59640..29bc1e464 100644 --- a/src/vllm/model_executor/models/qwen3_5.cpp +++ b/src/vllm/model_executor/models/qwen3_5.cpp @@ -7835,6 +7835,32 @@ DBuf MtpHeadHidden(Dev device, const Qwen3_5MTPWeights& weights, return MatmulBf16D(device, concatenated.t(), weights.fc); } +template +bool HasGdnConsumers(const Weights& weights) { + // vLLM qwen3_5.py:144-160 @ e126687a9a constructs GDN only for these layers. + // Cache absence cannot establish this: a hybrid model can have missing caches. + return std::any_of(weights.layers.begin(), weights.layers.end(), + [](const auto& layer) { return layer.is_linear_attention; }); +} + +void ValidateFullAttnStepMetadata(int64_t tokens, const CommonAttentionMetadata& am) { + VT_CHECK(am.num_actual_tokens == tokens, + "qwen3_5 full-attn: attn metadata token count must match positions"); + VT_CHECK(static_cast(am.slot_mapping.size()) == tokens, + "qwen3_5 full-attn: slot_mapping must cover every token"); + VT_CHECK(am.num_reqs >= 0 && + static_cast(am.seq_lens.size()) == am.num_reqs && + static_cast(am.query_start_loc.size()) == am.num_reqs + 1, + "qwen3_5 full-attn: malformed full-attn metadata shapes"); + VT_CHECK(am.block_table_num_cols >= 0 && + static_cast(am.block_table_tensor.size()) == + static_cast(am.num_reqs) * am.block_table_num_cols, + "qwen3_5 full-attn: malformed block table"); + VT_CHECK(am.query_start_loc.front() == 0 && am.query_start_loc.back() == tokens && + std::is_sorted(am.query_start_loc.begin(), am.query_start_loc.end()), + "qwen3_5 full-attn: query offsets must span tokens in order"); +} + // ── Full-attention-only per-step device inputs (SPEC-MTP I5c). ────────────── // BuildStepDevInputs sibling for a step with NO GDN layers (the MTP draft head // is a single layer_type="full_attention" decoder — qwen3_5_mtp.py:105-112). It @@ -7846,13 +7872,7 @@ StepDevInputs BuildFullAttnStepDevInputs(Dev d, const std::vector& positions, const CommonAttentionMetadata& am) { const int64_t T = static_cast(positions.size()); - VT_CHECK(am.num_actual_tokens == T, - "qwen3_5 MTP paged: attn metadata token count must match positions"); - VT_CHECK(static_cast(am.slot_mapping.size()) == T, - "qwen3_5 MTP paged: slot_mapping must cover every token"); - VT_CHECK(static_cast(am.seq_lens.size()) == am.num_reqs && - static_cast(am.query_start_loc.size()) == am.num_reqs + 1, - "qwen3_5 MTP paged: malformed full-attn metadata shapes"); + ValidateFullAttnStepMetadata(T, am); return StepDevInputs{ DBuf(d, DType::kI32, {T}, positions.data()), DBuf(d, DType::kI64, {T}, am.slot_mapping.data()), @@ -8376,8 +8396,10 @@ static DBuf ForwardLayers(Dev d, const Tensor& hidden_in, }(); std::optional local_sdi; if (persistent_sdi == nullptr) - local_sdi.emplace( - BuildStepDevInputs(d, positions, attn_meta, gdn_meta, gdn_state_slots)); + local_sdi.emplace(HasGdnConsumers(weights) + ? BuildStepDevInputs(d, positions, attn_meta, gdn_meta, + gdn_state_slots) + : BuildFullAttnStepDevInputs(d, positions, attn_meta)); StepDevInputs& sdi = persistent_sdi != nullptr ? *persistent_sdi : *local_sdi; // Build the fused-preamble cos|sin cache ONCE; fp4_attn keys the per-arch // default (fp8/bf16 attn — the 35B — stays OFF; VT_FUSE_ATTN_PREAMBLE overrides). @@ -8521,8 +8543,6 @@ static void CheckPagedForward(const std::vector& token_ids, "qwen3_5 paged forward: weights.layers size must equal num_hidden_layers"); VT_CHECK(attn_meta.num_actual_tokens == T, "qwen3_5 paged forward: attn_meta.num_actual_tokens must equal T"); - VT_CHECK(gdn_meta.num_actual_tokens == T, - "qwen3_5 paged forward: gdn_meta.num_actual_tokens must equal T"); int64_t n_full = 0, n_gdn = 0; for (const auto& l : weights.layers) (l.is_linear_attention ? n_gdn : n_full) += 1; @@ -8532,8 +8552,12 @@ static void CheckPagedForward(const std::vector& token_ids, "qwen3_5 paged forward: gdn_state count must equal GDN layer count"); const int64_t state_slots = detail::ValidateGdnStateCacheLayout(gdn_state); - detail::ValidateGdnAttentionMetadata( - gdn_meta, state_slots, /*allow_inert_padding=*/false); + if (n_gdn > 0) { + VT_CHECK(gdn_meta.num_actual_tokens == T, + "qwen3_5 paged forward: gdn_meta.num_actual_tokens must equal T"); + detail::ValidateGdnAttentionMetadata( + gdn_meta, state_slots, /*allow_inert_padding=*/false); + } } // Transfer a freshly-produced [rows, vocab] device logits DBuf into an OWNING @@ -9235,8 +9259,6 @@ static void CheckDensePagedForward(const std::vector& token_ids, "num_hidden_layers"); VT_CHECK(attn_meta.num_actual_tokens == T, "qwen3_5 dense paged forward: attn_meta.num_actual_tokens must equal T"); - VT_CHECK(gdn_meta.num_actual_tokens == T, - "qwen3_5 dense paged forward: gdn_meta.num_actual_tokens must equal T"); int64_t n_full = 0, n_gdn = 0; for (const auto& l : weights.layers) (l.is_linear_attention ? n_gdn : n_full) += 1; VT_CHECK(static_cast(attn_kv.size()) == n_full, @@ -9245,8 +9267,12 @@ static void CheckDensePagedForward(const std::vector& token_ids, "qwen3_5 dense paged forward: gdn_state count must equal GDN layers"); const int64_t state_slots = detail::ValidateGdnStateCacheLayout(gdn_state); - detail::ValidateGdnAttentionMetadata( - gdn_meta, state_slots, /*allow_inert_padding=*/false); + if (n_gdn > 0) { + VT_CHECK(gdn_meta.num_actual_tokens == T, + "qwen3_5 dense paged forward: gdn_meta.num_actual_tokens must equal T"); + detail::ValidateGdnAttentionMetadata( + gdn_meta, state_slots, /*allow_inert_padding=*/false); + } } // Dense embed (27B): hidden[T,H] bf16 = embed_tokens[token_ids] (device-resident @@ -9319,8 +9345,10 @@ static DBuf DenseForwardLayers(Dev d, const Tensor& hidden_in, gdn_state.empty() ? 0 : gdn_state.front().ssm_state.shape[0]; std::optional local_sdi; if (persistent_sdi == nullptr) - local_sdi.emplace( - BuildStepDevInputs(d, positions, attn_meta, gdn_meta, gdn_state_slots)); + local_sdi.emplace(HasGdnConsumers(weights) + ? BuildStepDevInputs(d, positions, attn_meta, gdn_meta, + gdn_state_slots) + : BuildFullAttnStepDevInputs(d, positions, attn_meta)); StepDevInputs& sdi = persistent_sdi != nullptr ? *persistent_sdi : *local_sdi; // Build the fused-preamble cos|sin cache ONCE; fp4_attn keys the per-arch // default (the real 27B W4A4 => ON; bf16/GGUF dense => OFF; env overrides). @@ -10760,7 +10788,9 @@ ForwardLogits Qwen3_5DecodeGraph::Step( CheckPagedForward(token_ids, positions, attn_meta, gdn_meta, attn_kv, gdn_state, impl_->weights, impl_->config); const int64_t B = static_cast(token_ids.size()); - detail::ValidateGdnDecodeGraphState(gdn_meta, gdn_state, B); + const bool has_gdn = HasGdnConsumers(impl_->weights); + if (has_gdn) detail::ValidateGdnDecodeGraphState(gdn_meta, gdn_state, B); + else ValidateFullAttnStepMetadata(B, attn_meta); Backend& b = vt::GetBackend(impl_->queue.device.type); Dev d{b, impl_->queue}; // #1380: open a fresh demand measurement for this step. `PreGrowForCapture` @@ -10784,7 +10814,7 @@ ForwardLogits Qwen3_5DecodeGraph::Step( // exact shape trivially satisfies that while keeping the padded-row inertness // question out of the spec path. The shape count stays bounded by max_num_seqs // because num_reqs is. - const bool spec_step = gdn_meta.num_spec_decodes > 0; + const bool spec_step = has_gdn && gdn_meta.num_spec_decodes > 0; const int64_t S = spec_step ? B : PadToCaptureSize(B, impl_->max_num_reqs); // ENG-CUDAGRAPH-BREAK W6 (#1374): this step's uniform query length, and the // ring key built from it. `Q == 0` means the batch does not divide evenly into @@ -10803,8 +10833,8 @@ ForwardLogits Qwen3_5DecodeGraph::Step( const bool qlen_capped = DecodeGraphQueryLenCapped(impl_->slots, key); if (qlen_capped) v1::NoteDecodeGraphQueryLenDecline(); if (!impl_->enabled || S < 0 || !servable_shape || qlen_capped || - !detail::CanUseGdnDecodeGraphSize( - B, S, IndexedGdnStateIoEnabled(impl_->queue.device))) { + (has_gdn && !detail::CanUseGdnDecodeGraphSize( + B, S, IndexedGdnStateIoEnabled(impl_->queue.device)))) { if (aux_out != nullptr && !aux_out->layer_ids.empty()) { // The graph cannot serve this batch (disabled / unsupported size), so fall // back to the EAGER multi-tap forward, which fills aux_out itself. Without @@ -10906,7 +10936,10 @@ ForwardLogits Qwen3_5DecodeGraph::Step( pam = attn_meta; pgm = gdn_meta; } else { - BuildPaddedDecode(S, token_ids, positions, attn_meta, gdn_meta, ptok, ppos, + // The padding helper copies GDN indices into S entries. Without consumers, + // the caller's unused metadata has no validated bound and must stay inert. + BuildPaddedDecode(S, token_ids, positions, attn_meta, + has_gdn ? gdn_meta : GDNAttentionMetadata{}, ptok, ppos, pam, pgm); } @@ -11033,8 +11066,10 @@ ForwardLogits Qwen3_5DecodeGraph::Step( s.pin.Free(); { ActivePoolScope persistent_scope(&PersistentDecodeInputPool(d.b)); - s.dev = std::make_unique(BuildStepDevInputs( - d, s.positions, s.attn_meta, s.gdn_meta, gdn_state_slots)); + s.dev = std::make_unique( + has_gdn ? BuildStepDevInputs(d, s.positions, s.attn_meta, s.gdn_meta, + gdn_state_slots) + : BuildFullAttnStepDevInputs(d, s.positions, s.attn_meta)); MaybeBuildAttnCosSin(d, *s.dev, impl_->config, S, fp4_attn); } const bool has_idx = s.dev->has_gdn_idx && @@ -11388,7 +11423,9 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( CheckDensePagedForward(token_ids, positions, attn_meta, gdn_meta, attn_kv, gdn_state, impl_->weights, impl_->config); const int64_t B = static_cast(token_ids.size()); - detail::ValidateGdnDecodeGraphState(gdn_meta, gdn_state, B); + const bool has_gdn = HasGdnConsumers(impl_->weights); + if (has_gdn) detail::ValidateGdnDecodeGraphState(gdn_meta, gdn_state, B); + else ValidateFullAttnStepMetadata(B, attn_meta); Backend& b = vt::GetBackend(impl_->queue.device.type); Dev d{b, impl_->queue}; // #1380: open a fresh demand measurement for this step. `PreGrowForCapture` @@ -11410,7 +11447,7 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( // exact shape trivially satisfies that while keeping the padded-row inertness // question out of the spec path. The shape count stays bounded by max_num_seqs // because num_reqs is. - const bool spec_step = gdn_meta.num_spec_decodes > 0; + const bool spec_step = has_gdn && gdn_meta.num_spec_decodes > 0; const int64_t S = spec_step ? B : PadToCaptureSize(B, impl_->max_num_reqs); // ENG-CUDAGRAPH-BREAK W6 (#1374): this step's uniform query length, and the // ring key built from it. `Q == 0` means the batch does not divide evenly into @@ -11429,8 +11466,8 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( const bool qlen_capped = DecodeGraphQueryLenCapped(impl_->slots, key); if (qlen_capped) v1::NoteDecodeGraphQueryLenDecline(); if (!impl_->enabled || S < 0 || !servable_shape || qlen_capped || - !detail::CanUseGdnDecodeGraphSize( - B, S, IndexedGdnStateIoEnabled(impl_->queue.device))) { + (has_gdn && !detail::CanUseGdnDecodeGraphSize( + B, S, IndexedGdnStateIoEnabled(impl_->queue.device)))) { if (aux_out != nullptr && !aux_out->layer_ids.empty()) { // The graph cannot serve this batch (disabled / unsupported size), so fall // back to the EAGER multi-tap forward, which fills aux_out itself. Without @@ -11530,7 +11567,10 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( pam = attn_meta; pgm = gdn_meta; } else { - BuildPaddedDecode(S, token_ids, positions, attn_meta, gdn_meta, ptok, ppos, + // The padding helper copies GDN indices into S entries. Without consumers, + // the caller's unused metadata has no validated bound and must stay inert. + BuildPaddedDecode(S, token_ids, positions, attn_meta, + has_gdn ? gdn_meta : GDNAttentionMetadata{}, ptok, ppos, pam, pgm); } @@ -11771,8 +11811,10 @@ ForwardLogits Qwen3_5DenseDecodeGraph::Step( s.pin.Free(); { ActivePoolScope persistent_scope(&PersistentDecodeInputPool(d.b)); - s.dev = std::make_unique(BuildStepDevInputs( - d, s.positions, s.attn_meta, s.gdn_meta, gdn_state_slots)); + s.dev = std::make_unique( + has_gdn ? BuildStepDevInputs(d, s.positions, s.attn_meta, s.gdn_meta, + gdn_state_slots) + : BuildFullAttnStepDevInputs(d, s.positions, s.attn_meta)); MaybeBuildAttnCosSin(d, *s.dev, impl_->config, S, fp4_attn); } const bool has_idx = s.dev->has_gdn_idx && diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index a3117cc61..2811d051c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -2562,6 +2562,7 @@ endif() # C ABI (M3.5 Task 1): the C++ test drives the public C API over a synthetic # engine via the internal MakeEngineHandle hook (reached under src/). vllm_cpp_add_test(test_capi capi/test_capi.cpp) +vllm_cpp_add_test(test_capi_qwen35_full_attn_state capi/test_qwen35_full_attn_state.cpp) target_include_directories(test_capi PRIVATE ${CMAKE_SOURCE_DIR}/src) target_compile_definitions(test_capi PRIVATE PARAKEET_E2E_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/parakeet_e2e" diff --git a/tests/capi/qwen35_full_attn_fixture.h b/tests/capi/qwen35_full_attn_fixture.h new file mode 100644 index 000000000..4af83032d --- /dev/null +++ b/tests/capi/qwen35_full_attn_fixture.h @@ -0,0 +1,133 @@ +// ENG-QWEN35-FULL-ATTN-STATE (#3098): one deterministic, all-dense GGUF. +// Extracted from rocm_quant_gather_fixture.h, SHA256 +// 339414f93e59e6be9a4ca545d5e762bb2816714343f466e218dbc2b69483bdfd. +// Keep the dense Q4_0 control byte-identical to that experiment. Its name and +// inactive SSM loader keys are part of the frozen artifact, not active layers. +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "vllm/gguf_builder.h" +#include "vt/dtype.h" +#include "vt/quant.h" + +namespace qwen35_full_attn_test { + +inline constexpr uint32_t kSeed = 0x524f434d; +inline constexpr int64_t kHidden = 256; +inline constexpr int64_t kVocab = 128; +inline constexpr size_t kArtifactBytes = 991296; +inline constexpr const char* kArtifactSha256 = + "0e6554ba521edfde00d4d025a3058eabaaacdce6b24344f959e83d8dde35df7f"; + +inline std::string DenseEmbedding() { + // Q4_0 is only the deterministic source of these BF16 values. The resulting + // GGUF contains a dense embedding and no quantized projection. + constexpr int64_t kBlockBytes = 18; + constexpr int64_t kBlockElems = 32; + constexpr int64_t kBlocks = kVocab * kHidden / kBlockElems; + std::string packed(static_cast(kBlocks * kBlockBytes), '\0'); + std::mt19937 rng(kSeed); + for (int64_t block = 0; block < kBlocks; ++block) { + const size_t base = static_cast(block * kBlockBytes); + for (int64_t byte = 0; byte < kBlockBytes; ++byte) + packed[base + static_cast(byte)] = static_cast(rng() & 255); + packed[base] = 0; + packed[base + 1] = 0x20; // Half d = 1/128, exactly as the frozen control. + } + std::vector decoded(static_cast(kVocab * kHidden)); + const auto decode = vt::cpu::BlockToFloat(vt::DType::kQ4_0); + if (decode == nullptr) throw std::runtime_error("fixture Q4_0 decoder absent"); + decode(packed.data(), decoded.data(), kVocab * kHidden); + std::string dense; + dense.reserve(decoded.size() * 2); + for (float value : decoded) { + if (!std::isfinite(value)) throw std::runtime_error("nonfinite fixture block"); + const uint16_t word = vt::F32ToBF16(value); + dense.push_back(static_cast(word & 255)); + dense.push_back(static_cast(word >> 8)); + } + return dense; +} + +inline std::string BuildModel() { + using namespace gguf_test; + GgufModelBuilder builder; + builder.AddKv(StrKv("general.architecture", "qwen35")); + builder.AddKv(StrKv("general.name", "rocm-quant-gather-synthetic")); + builder.AddKv(U32Kv("general.quantization_version", 2)); + builder.AddKv(U32Kv("qwen35.embedding_length", 256)); + builder.AddKv(U32Kv("qwen35.block_count", 1)); + builder.AddKv(U32Kv("qwen35.attention.head_count", 4)); + builder.AddKv(U32Kv("qwen35.attention.head_count_kv", 1)); + builder.AddKv(U32Kv("qwen35.attention.key_length", 64)); + builder.AddKv(U32Kv("qwen35.attention.value_length", 64)); + builder.AddKv(U32Kv("qwen35.feed_forward_length", 256)); + builder.AddKv(U32Kv("qwen35.full_attention_interval", 1)); + builder.AddKv(U32Kv("qwen35.context_length", 64)); + builder.AddKv(U32Kv("qwen35.vocab_size", 128)); + builder.AddKv(F32Kv("qwen35.attention.layer_norm_rms_epsilon", 1e-6F)); + builder.AddKv(F32Kv("qwen35.rope.freq_base", 1000000.0F)); + builder.AddKv(U32Kv("qwen35.rope.dimension_count", 64)); + builder.AddKv(I32ArrayKv("qwen35.rope.dimension_sections", {16, 8, 8, 0})); + builder.AddKv(U32Kv("qwen35.ssm.conv_kernel", 4)); + builder.AddKv(U32Kv("qwen35.ssm.inner_size", 64)); + builder.AddKv(U32Kv("qwen35.ssm.state_size", 64)); + builder.AddKv(U32Kv("qwen35.ssm.time_step_rank", 1)); + builder.AddKv(U32Kv("qwen35.ssm.group_count", 1)); + builder.AddKv(StrKv("tokenizer.ggml.model", "gpt2")); + builder.AddKv(StrKv("tokenizer.ggml.pre", "qwen2")); + builder.AddKv(U32Kv("tokenizer.ggml.unknown_token_id", 0)); + builder.AddKv(U32Kv("tokenizer.ggml.bos_token_id", 1)); + builder.AddKv(U32Kv("tokenizer.ggml.eos_token_id", 2)); + builder.AddKv(BoolKv("tokenizer.ggml.add_bos_token", false)); + std::vector vocabulary = {"", "", ""}; + for (int id = 3; id < kVocab; ++id) vocabulary.push_back("token" + std::to_string(id)); + builder.AddKv(StrArrayKv("tokenizer.ggml.tokens", vocabulary)); + builder.AddKv(StrArrayKv("tokenizer.ggml.merges", {})); + std::vector types(128, 1); + types[0] = 2; + types[1] = types[2] = 3; + builder.AddKv(I32ArrayKv("tokenizer.ggml.token_type", types)); + builder.AddTensor("token_embd.weight", {256, 128}, 30, DenseEmbedding()); + + std::mt19937 rng(kSeed); + const auto projection = [&](const char* name, uint64_t input, uint64_t output) { + std::string bytes; + bytes.reserve(static_cast(input * output * 2)); + for (uint64_t i = 0; i < input * output; ++i) { + const int32_t integer = static_cast(rng() & 65535) - 32768; + const float value = static_cast(integer) / 2097152.0F; + const uint16_t word = vt::F32ToBF16(value); + bytes.push_back(static_cast(word & 255)); + bytes.push_back(static_cast(word >> 8)); + } + builder.AddTensor(name, {input, output}, 30, bytes); + }; + const auto norm = [&](const char* name, uint64_t width) { + std::string bytes; + for (uint64_t i = 0; i < width; ++i) bytes += U32Le(0x3f800000); + builder.AddTensor(name, {width}, 0, bytes); + }; + projection("output.weight", 256, 128); + projection("blk.0.attn_q.weight", 256, 512); + projection("blk.0.attn_k.weight", 256, 64); + projection("blk.0.attn_v.weight", 256, 64); + projection("blk.0.attn_output.weight", 256, 256); + projection("blk.0.ffn_gate.weight", 256, 256); + projection("blk.0.ffn_up.weight", 256, 256); + projection("blk.0.ffn_down.weight", 256, 256); + norm("output_norm.weight", 256); + norm("blk.0.attn_norm.weight", 256); + norm("blk.0.post_attention_norm.weight", 256); + norm("blk.0.attn_q_norm.weight", 64); + norm("blk.0.attn_k_norm.weight", 64); + return builder.Build(); +} + +} // namespace qwen35_full_attn_test diff --git a/tests/capi/test_qwen35_full_attn_state.cpp b/tests/capi/test_qwen35_full_attn_state.cpp new file mode 100644 index 000000000..388c558b5 --- /dev/null +++ b/tests/capi/test_qwen35_full_attn_state.cpp @@ -0,0 +1,262 @@ +// Public regression for ENG-QWEN35-FULL-ATTN-STATE (#3098). +// Pinned vLLM e126687a9a: qwen3_5.py:144-160 constructs recurrent state +// consumers only for linear_attention layers. This no-GDN case is local. +#define DOCTEST_CONFIG_IMPLEMENT +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "capi/qwen35_full_attn_fixture.h" +#include "vllm.h" +#include "vt/backend.h" +#include "vt/op_provider.h" +#include "vt/ops.h" + +#if defined(VLLM_CPP_HIP) && !defined(VLLM_CPP_CUDA) +namespace vt::rocm { +void EmbeddingKernelRocm(Queue&, Tensor&, const Tensor&, const Tensor&); +void ReshapeAndCacheKernelRocm(Queue&, const Tensor&, const Tensor&, Tensor&, + Tensor&, const Tensor&); +void GreedyArgmaxKernelRocm(Queue&, Tensor&, const Tensor&); +} // namespace vt::rocm +#endif + +namespace { + +std::atomic native_prefills{0}, native_decodes{0}, native_cache_writes{0}; +std::atomic native_samples{0}; +std::atomic wrong_model_dtype{false}; +constexpr const char* kObserver = "qwen35-full-attn-state-observer"; + +#if defined(VLLM_CPP_HIP) && !defined(VLLM_CPP_CUDA) +void ObserveEmbedding(vt::Queue& queue, vt::Tensor& out, + const vt::Tensor& table, const vt::Tensor& ids) { + if (queue.device.type != vt::DeviceType::kROCM || + out.dtype != vt::DType::kBF16 || table.dtype != vt::DType::kBF16) + wrong_model_dtype = true; + // Call the verified native function, then count the completed native call. + // RegisterOp cannot replace vt-native: duplicate provider names are ignored. + vt::rocm::EmbeddingKernelRocm(queue, out, table, ids); + if (ids.Numel() > 1) ++native_prefills; + else ++native_decodes; +} + +void ObserveCache(vt::Queue& queue, const vt::Tensor& key, + const vt::Tensor& value, vt::Tensor& key_cache, + vt::Tensor& value_cache, const vt::Tensor& slots) { + if (key.dtype != vt::DType::kBF16 || value.dtype != vt::DType::kBF16 || + key_cache.dtype != vt::DType::kBF16 || value_cache.dtype != vt::DType::kBF16) + wrong_model_dtype = true; + vt::rocm::ReshapeAndCacheKernelRocm(queue, key, value, key_cache, value_cache, slots); + ++native_cache_writes; +} + +void ObserveGreedy(vt::Queue& queue, vt::Tensor& ids, const vt::Tensor& logits) { + // The existing sampler ABI consumes f32 logits and produces i64 token IDs. + if (queue.device.type != vt::DeviceType::kROCM || + logits.device.type != vt::DeviceType::kROCM || + ids.device.type != vt::DeviceType::kROCM || + logits.dtype != vt::DType::kF32 || ids.dtype != vt::DType::kI64) + wrong_model_dtype = true; + vt::rocm::GreedyArgmaxKernelRocm(queue, ids, logits); + ++native_samples; +} +#endif + +void InstallNativeObservers() { +#if defined(VLLM_CPP_HIP) && !defined(VLLM_CPP_CUDA) + const auto install = [](vt::OpId op, void* native, void* observer) { + REQUIRE(vt::GetOp(op, vt::DeviceType::kROCM) == native); + const auto before = vt::GetOpProviderStats(op, vt::DeviceType::kROCM); + REQUIRE(before.last_selected != nullptr); + REQUIRE(std::string(before.last_selected) == vt::kNativeProviderName); + vt::OpProvider provider; + provider.name = kObserver; + provider.priority = 1; + provider.fn = observer; + vt::RegisterOpProvider(op, vt::DeviceType::kROCM, provider); + REQUIRE(vt::GetOp(op, vt::DeviceType::kROCM) == observer); + }; + install(vt::OpId::kEmbedding, + reinterpret_cast(&vt::rocm::EmbeddingKernelRocm), + reinterpret_cast(&ObserveEmbedding)); + install(vt::OpId::kReshapeAndCache, + reinterpret_cast(&vt::rocm::ReshapeAndCacheKernelRocm), + reinterpret_cast(&ObserveCache)); + install(vt::OpId::kGreedyArgmax, + reinterpret_cast(&vt::rocm::GreedyArgmaxKernelRocm), + reinterpret_cast(&ObserveGreedy)); +#else + throw std::runtime_error("native observers require a HIP-only build"); +#endif +} + +struct Completion { + std::vector ids; + // The public logits-processor ABI supplies f32 logits. This is host evidence, + // not an activation or KV buffer in the model. + std::vector logits; + bool capture_overflow = false; + int callbacks = 0; +}; + +// Reserve the full callback output before entering the C ABI. This callback +// cannot throw or change the logits that the greedy sampler reads. +void Capture(const int32_t*, int32_t, float* logits, int32_t vocab, void* data) { + auto& result = *static_cast(data); + if (vocab != 128 || result.callbacks >= 4) { + result.capture_overflow = true; + return; + } + for (int32_t i = 0; i < vocab; ++i) result.logits.push_back(logits[i]); + ++result.callbacks; +} + +Completion Run(const std::string& path, const std::vector& prompt, + int repeat, size_t prompt_index, bool capture_logits) { + auto model = vllm_model_params_default(); + model.model_path = path.c_str(); + model.device = 0; // Public AUTO in the required HIP-only build. + model.block_size = 16; + model.num_blocks = 4; + model.max_model_len = 64; + model.max_num_seqs = 1; + model.kv_cache_dtype = "auto"; + vllm_engine* raw = nullptr; + const vllm_status loaded = vllm_engine_load(&model, &raw); + INFO("public load: " << vllm_last_error()); + REQUIRE(loaded == VLLM_OK); + std::unique_ptr engine(raw, &vllm_engine_free); + std::printf("FULL_ATTN public load succeeded repeat=%d prompt=%zu\n", repeat, prompt_index); + std::fflush(stdout); + + Completion result; + result.ids.resize(4); + result.logits.reserve(4 * 128); + auto sampling = vllm_sampling_params_default(); + sampling.temperature = 0.0F; + sampling.max_tokens = 4; + sampling.ignore_eos = 1; + sampling.has_seed = 1; + sampling.seed = qwen35_full_attn_test::kSeed; + if (capture_logits) { + sampling.logits_processor = Capture; + sampling.logits_processor_user_data = &result; + } + native_prefills = 0; + native_decodes = 0; + native_cache_writes = 0; + native_samples = 0; + wrong_model_dtype = false; + const auto hits_before = vt::GetReferenceTierHits(); + vt::ResetOpProviderStats(vt::OpId::kEmbedding, vt::DeviceType::kROCM); + vt::EnableOpProviderCallStats(true); + int32_t count = 0; + const vllm_status completed = vllm_complete_tokens( + engine.get(), prompt.data(), static_cast(prompt.size()), &sampling, + result.ids.data(), 4, &count, nullptr); + vt::EnableOpProviderCallStats(false); + INFO("public completion: " << vllm_last_error()); + REQUIRE(completed == VLLM_OK); + REQUIRE(count == 4); + REQUIRE_FALSE(result.capture_overflow); + REQUIRE(result.callbacks == (capture_logits ? 4 : 0)); + REQUIRE(result.logits.size() == (capture_logits ? 512 : 0)); + for (float value : result.logits) CHECK(std::isfinite(value)); + for (int32_t id : result.ids) CHECK((id >= 0 && id < 128)); + CHECK(native_prefills.load() == 1); + CHECK(native_decodes.load() == 3); + CHECK(native_cache_writes.load() == 4); + CHECK(native_samples.load() == 4); + CHECK_FALSE(wrong_model_dtype.load()); + CHECK(vt::GetReferenceTierHits() == hits_before); + const auto stats = vt::GetOpProviderStats(vt::OpId::kEmbedding, vt::DeviceType::kROCM); + CHECK(stats.selections >= 4); + CHECK(stats.declines == 0); + CHECK(stats.fallbacks == 0); + REQUIRE(stats.last_selected != nullptr); + CHECK(std::string(stats.last_selected) == kObserver); + std::printf("FULL_ATTN repeat=%d prompt=%zu capture_logits=%d fresh_engine=1 concurrency=1 " + "prompt_tokens=%zu native_prefills=%d native_decodes=%d " + "native_cache_writes=%d native_samples=%d activation=bf16 kv=bf16 " + "finite_logits=%zu tokens=", + repeat, prompt_index, capture_logits, prompt.size(), native_prefills.load(), + native_decodes.load(), native_cache_writes.load(), native_samples.load(), + result.logits.size()); + for (int32_t id : result.ids) std::printf(" %d", id); + std::printf("\n"); + return result; +} + +void WriteFixture(const std::filesystem::path& path) { + const auto bytes = qwen35_full_attn_test::BuildModel(); + if (bytes.size() != qwen35_full_attn_test::kArtifactBytes) + throw std::runtime_error("full-attention fixture size changed"); + if (!path.parent_path().empty()) std::filesystem::create_directories(path.parent_path()); + std::ofstream file(path, std::ios::binary); + file.write(bytes.data(), static_cast(bytes.size())); + if (!file) throw std::runtime_error("cannot write full-attention fixture"); +} + +} // namespace + +TEST_CASE("Qwen3.5 full-attention state enters public load and completion") { + const auto bytes = qwen35_full_attn_test::BuildModel(); + REQUIRE(bytes.size() == qwen35_full_attn_test::kArtifactBytes); + CHECK(bytes == qwen35_full_attn_test::BuildModel()); + gguf_test::TempFile fixture(bytes); + InstallNativeObservers(); + const std::vector> prompts = {{1, 0, 63, 127, 63}, {1, 127, 0, 127}}; + std::vector first(2 * prompts.size()); + // builtin.cpp::apply_logits_processors stages logits through the host when a + // callback exists. Both modes still execute the native device argmax, but + // the callback-free arm proves the default public route without that staging. + for (bool capture_logits : {false, true}) { + CAPTURE(capture_logits); + for (int repeat = 0; repeat < 3; ++repeat) { + CAPTURE(repeat); + for (size_t p = 0; p < prompts.size(); ++p) { + CAPTURE(p); + const Completion result = Run(fixture.path(), prompts[p], repeat, p, capture_logits); + Completion& baseline = first[static_cast(capture_logits) * prompts.size() + p]; + if (repeat == 0) baseline = result; + else { + CHECK(result.ids == baseline.ids); + REQUIRE(result.logits.size() == baseline.logits.size()); + if (capture_logits) + CHECK(std::memcmp(result.logits.data(), baseline.logits.data(), + result.logits.size() * sizeof(float)) == 0); + } + if (capture_logits) CHECK(result.ids == first[p].ids); + } + } + } +} + +int main(int argc, char** argv) { + if (argc == 3 && std::string(argv[1]) == "--write-fixture") { + WriteFixture(argv[2]); + return 0; + } +#if defined(VLLM_CPP_HIP) && !defined(VLLM_CPP_CUDA) + try { + vt::GetBackend(vt::DeviceType::kROCM); + } catch (const std::runtime_error& error) { + std::printf("SKIPPED: ROCm device unavailable: %s\n", error.what()); + return 77; + } + doctest::Context context(argc, argv); + return context.run(); +#else + std::puts("SKIPPED: public AUTO regression requires a HIP-only build"); + return 77; +#endif +} diff --git a/tests/vllm/models/test_qwen27_paged_forward.cpp b/tests/vllm/models/test_qwen27_paged_forward.cpp index 9c88ec94d..2ea5d878e 100644 --- a/tests/vllm/models/test_qwen27_paged_forward.cpp +++ b/tests/vllm/models/test_qwen27_paged_forward.cpp @@ -1348,6 +1348,205 @@ TEST_CASE("qwen27 packed QKV uses maximum CT logical-shard divisors") { std::runtime_error); } + +namespace { + +// ENG-QWEN35-FULL-ATTN-STATE (#3098). Enter the registered forward, with the +// existing CPU fixture's f32 cache representation and unchanged tolerances. +TEST_CASE("qwen27 full-attention registry accepts unused GDN metadata") { + HfConfig c = MakeConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + const Qwen3_5DenseWeights w = MakeWeights(c); + CachePool pool(c, 4, 8); + REQUIRE(pool.gdn_state.empty()); + bool runner_metadata = false; + SUBCASE("default-empty GDN metadata") {} + SUBCASE("unused runner GDN metadata") { runner_metadata = true; } + SUBCASE("the loaded layers determine consumers") { + // The loaded consumer wins over a stale configuration label. + c.layer_types = {"linear_attention"}; + runner_metadata = true; + } + auto model = vllm::BorrowQwen3_5DenseLoadedModel(w); + vt::Queue q = Q(); + const std::vector indices; + const auto host = [&](const vllm::ForwardLogits& logits) { + std::vector out; + if (logits.on_device()) { + out.resize(static_cast(logits.rows * logits.vocab)); + auto& backend = vt::GetBackend(q.device.type); + backend.Copy(q, out.data(), logits.device_tensor.data, out.size() * sizeof(float)); + backend.Synchronize(q); + } else { + out = logits.host; + } + for (float value : out) CHECK(std::isfinite(value)); + return out; + }; + + const std::vector ids = {5, 9, 2}; + const std::vector positions = {0, 1, 2}; + const auto am = PrefillAttnMeta(3, {0}, 8, 0); + const auto gm = runner_metadata ? PrefillGdnMeta(3, 0) : GDNAttentionMetadata{}; + ModelForwardInput prefill{ids, positions, am, gm, pool.attn_kv, pool.gdn_state, + c, q, indices}; + prefill.num_reqs = 1; + const auto prefill_logits = host(ModelRegistry::Forward(*model, prefill)); + REQUIRE(prefill_logits.size() == static_cast(3 * c.vocab_size)); + const auto dense_prefill = Qwen3_5DenseModel::ForwardDense(ids, positions, w, c, q); + CHECK(MaxAbsDiff(prefill_logits, dense_prefill, prefill_logits.size()) < 1e-2); + + const std::vector next = {8}, next_positions = {3}; + auto decode_am = PrefillAttnMeta(1, {0}, 8, 3); + decode_am.seq_lens = decode_am.seq_lens_cpu = {4}; + decode_am.max_seq_len = 4; + GDNAttentionMetadata decode_gm; + if (runner_metadata) { + decode_gm.num_decodes = decode_gm.num_decode_tokens = 1; + decode_gm.num_actual_tokens = 1; + decode_gm.non_spec_state_indices_tensor = std::vector{0}; + decode_gm.non_spec_query_start_loc = std::vector{0, 1}; + } + ModelForwardInput decode{next, next_positions, decode_am, decode_gm, pool.attn_kv, + pool.gdn_state, c, q, indices}; + decode.num_reqs = 1; + decode.pure_decode = true; + const auto decode_logits = host(ModelRegistry::Forward(*model, decode)); + REQUIRE(decode_logits.size() == static_cast(c.vocab_size)); + const auto dense_full = Qwen3_5DenseModel::ForwardDense({5, 9, 2, 8}, {0, 1, 2, 3}, w, c, q); + const std::vector last(dense_full.end() - c.vocab_size, dense_full.end()); + CHECK(MaxAbsDiff(decode_logits, last, decode_logits.size()) < 2e-2); +} + +TEST_CASE("qwen27 full-attention registry keeps generic counts") { + HfConfig c = MakeConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + Qwen3_5DenseWeights w = MakeWeights(c); + CachePool pool(c, 4, 8); + const std::vector ids = {5}, indices; + std::vector positions = {0}; + auto am = PrefillAttnMeta(1, {0}, 8, 0); + const GDNAttentionMetadata gm; + const char* expected = nullptr; + SUBCASE("positions") { positions.clear(); expected = "positions length"; } + SUBCASE("attention token count") { + am.num_actual_tokens = 2; + expected = "attn_meta.num_actual_tokens"; + } + SUBCASE("layer count") { w.layers.clear(); expected = "weights.layers size"; } + SUBCASE("attention cache count") { pool.attn_kv.clear(); expected = "attn_kv count"; } + SUBCASE("unexpected recurrent cache") { + pool.gdn_state.emplace_back(); + expected = "gdn_state count"; + } + REQUIRE(expected != nullptr); + auto model = vllm::BorrowQwen3_5DenseLoadedModel(w); + vt::Queue q = Q(); + ModelForwardInput input{ids, positions, am, gm, pool.attn_kv, pool.gdn_state, c, q, indices}; + input.num_reqs = 1; + CHECK_THROWS_WITH_AS(ModelRegistry::Forward(*model, input), + doctest::Contains(expected), std::runtime_error); +} + +TEST_CASE("qwen27 registry preserves each hybrid GDN guard") { + const HfConfig c = MakeConfig(); + const Qwen3_5DenseWeights w = MakeWeights(c); + CachePool pool(c, 3, 8); + REQUIRE(pool.gdn_state.size() == 3); + std::vector ids = {5, 9}, positions = {0, 0}; + const std::vector indices; + auto am = PrefillAttnMeta(2, {0, 1}, 8, 0); + am.num_reqs = 2; + am.max_query_len = am.max_seq_len = 1; + am.query_start_loc = am.query_start_loc_cpu = {0, 1, 2}; + am.seq_lens = am.seq_lens_cpu = {1, 1}; + am.block_table_num_cols = 1; + am.slot_mapping = {0, 8}; + GDNAttentionMetadata gm; + gm.num_actual_tokens = gm.num_decodes = gm.num_decode_tokens = 2; + gm.non_spec_state_indices_tensor = std::vector{0, 1}; + gm.non_spec_query_start_loc = std::vector{0, 1, 2}; + const char* expected = nullptr; + SUBCASE("valid hybrid") {} + SUBCASE("all GDN caches missing") { + pool.gdn_state.clear(); + expected = "gdn_state count"; + } + SUBCASE("wrong GDN cache count") { + pool.gdn_state.pop_back(); + expected = "gdn_state count"; + } + SUBCASE("invalid SSM rank") { + pool.gdn_state.back().ssm_state.rank = 3; + expected = "GDN SSM/conv state ranks"; + } + SUBCASE("invalid convolution rank") { + pool.gdn_state.back().conv_state.rank = 4; + expected = "GDN SSM/conv state ranks"; + } + SUBCASE("mismatched SSM and convolution slots") { + pool.gdn_state.back().conv_state.shape[0] = 2; + expected = "GDN conv/SSM state slot counts"; + } + SUBCASE("inconsistent layer slots") { + pool.gdn_state.back().ssm_state.shape[0] = 2; + pool.gdn_state.back().conv_state.shape[0] = 2; + expected = "all GDN layers must use the same state slot count"; + } + SUBCASE("missing metadata") { + gm = GDNAttentionMetadata{}; + expected = "gdn_meta.num_actual_tokens"; + } + SUBCASE("wrong GDN token count") { + gm.num_actual_tokens = 1; + expected = "gdn_meta.num_actual_tokens"; + } + SUBCASE("missing indices") { + gm.non_spec_state_indices_tensor.reset(); + expected = "missing non-spec GDN state indices"; + } + SUBCASE("duplicate indices") { + gm.non_spec_state_indices_tensor = std::vector{0, 0}; + expected = "duplicate live GDN state index"; + } + SUBCASE("out-of-range indices") { + gm.non_spec_state_indices_tensor = std::vector{0, 3}; + expected = "GDN state index out of range"; + } + SUBCASE("incomplete prefill metadata") { + am = PrefillAttnMeta(2, {0}, 8, 0); + positions = {0, 1}; + gm = PrefillGdnMeta(2, 0); + gm.prefill_has_initial_state.reset(); + expected = "incomplete GDN prefill metadata"; + } + auto model = vllm::BorrowQwen3_5DenseLoadedModel(w); + vt::Queue q = Q(); + ModelForwardInput input{ids, positions, am, gm, pool.attn_kv, pool.gdn_state, c, q, indices}; + input.num_reqs = am.num_reqs; + if (expected != nullptr) { + CHECK_THROWS_WITH_AS(ModelRegistry::Forward(*model, input), + doctest::Contains(expected), std::runtime_error); + } else { + const auto logits = ModelRegistry::Forward(*model, input); + REQUIRE(logits.rows == 2); + REQUIRE(logits.vocab == c.vocab_size); + std::vector host = logits.host; + if (logits.on_device()) { + host.resize(static_cast(2 * c.vocab_size)); + auto& backend = vt::GetBackend(q.device.type); + backend.Copy(q, host.data(), logits.device_tensor.data, host.size() * sizeof(float)); + backend.Synchronize(q); + } + REQUIRE(host.size() == static_cast(2 * c.vocab_size)); + for (float value : host) CHECK(std::isfinite(value)); + } +} + +} // namespace + TEST_CASE("qwen27 dense paged: full-prefill batch-of-1 equals dense forward") { const HfConfig c = MakeConfig(); const Qwen3_5DenseWeights w = MakeWeights(c); diff --git a/tests/vllm/models/test_qwen35_paged_forward.cpp b/tests/vllm/models/test_qwen35_paged_forward.cpp index 9c5b05a15..62cc52da2 100644 --- a/tests/vllm/models/test_qwen35_paged_forward.cpp +++ b/tests/vllm/models/test_qwen35_paged_forward.cpp @@ -301,6 +301,205 @@ double MaxAbsDiff(const std::vector& a, const std::vector& b, size } // namespace + +namespace { + +// ENG-QWEN35-FULL-ATTN-STATE (#3098). Enter the registered forward, with the +// existing CPU fixture's f32 cache representation and unchanged tolerances. +TEST_CASE("qwen35 full-attention registry accepts unused GDN metadata") { + HfConfig c = MakeConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + const Qwen3_5MoeWeights w = MakeWeights(c); + CachePool pool(c, 4, 8); + REQUIRE(pool.gdn_state.empty()); + bool runner_metadata = false; + SUBCASE("default-empty GDN metadata") {} + SUBCASE("unused runner GDN metadata") { runner_metadata = true; } + SUBCASE("the loaded layers determine consumers") { + // The loaded consumer wins over a stale configuration label. + c.layer_types = {"linear_attention"}; + runner_metadata = true; + } + auto model = vllm::BorrowQwen3_5MoeLoadedModel(w); + vt::Queue q = Q(); + const std::vector indices; + const auto host = [&](const vllm::ForwardLogits& logits) { + std::vector out; + if (logits.on_device()) { + out.resize(static_cast(logits.rows * logits.vocab)); + auto& backend = vt::GetBackend(q.device.type); + backend.Copy(q, out.data(), logits.device_tensor.data, out.size() * sizeof(float)); + backend.Synchronize(q); + } else { + out = logits.host; + } + for (float value : out) CHECK(std::isfinite(value)); + return out; + }; + + const std::vector ids = {5, 9, 2}; + const std::vector positions = {0, 1, 2}; + const auto am = PrefillAttnMeta(3, {0}, 8, 0); + const auto gm = runner_metadata ? PrefillGdnMeta(3, 0) : GDNAttentionMetadata{}; + ModelForwardInput prefill{ids, positions, am, gm, pool.attn_kv, pool.gdn_state, + c, q, indices}; + prefill.num_reqs = 1; + const auto prefill_logits = host(ModelRegistry::Forward(*model, prefill)); + REQUIRE(prefill_logits.size() == static_cast(3 * c.vocab_size)); + const auto dense_prefill = Qwen3_5Model::ForwardDense(ids, positions, w, c, q); + CHECK(MaxAbsDiff(prefill_logits, dense_prefill, prefill_logits.size()) < 1e-2); + + const std::vector next = {8}, next_positions = {3}; + auto decode_am = PrefillAttnMeta(1, {0}, 8, 3); + decode_am.seq_lens = decode_am.seq_lens_cpu = {4}; + decode_am.max_seq_len = 4; + GDNAttentionMetadata decode_gm; + if (runner_metadata) { + decode_gm.num_decodes = decode_gm.num_decode_tokens = 1; + decode_gm.num_actual_tokens = 1; + decode_gm.non_spec_state_indices_tensor = std::vector{0}; + decode_gm.non_spec_query_start_loc = std::vector{0, 1}; + } + ModelForwardInput decode{next, next_positions, decode_am, decode_gm, pool.attn_kv, + pool.gdn_state, c, q, indices}; + decode.num_reqs = 1; + decode.pure_decode = true; + const auto decode_logits = host(ModelRegistry::Forward(*model, decode)); + REQUIRE(decode_logits.size() == static_cast(c.vocab_size)); + const auto dense_full = Qwen3_5Model::ForwardDense({5, 9, 2, 8}, {0, 1, 2, 3}, w, c, q); + const std::vector last(dense_full.end() - c.vocab_size, dense_full.end()); + CHECK(MaxAbsDiff(decode_logits, last, decode_logits.size()) < 2e-2); +} + +TEST_CASE("qwen35 full-attention registry keeps generic counts") { + HfConfig c = MakeConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + Qwen3_5MoeWeights w = MakeWeights(c); + CachePool pool(c, 4, 8); + const std::vector ids = {5}, indices; + std::vector positions = {0}; + auto am = PrefillAttnMeta(1, {0}, 8, 0); + const GDNAttentionMetadata gm; + const char* expected = nullptr; + SUBCASE("positions") { positions.clear(); expected = "positions length"; } + SUBCASE("attention token count") { + am.num_actual_tokens = 2; + expected = "attn_meta.num_actual_tokens"; + } + SUBCASE("layer count") { w.layers.clear(); expected = "weights.layers size"; } + SUBCASE("attention cache count") { pool.attn_kv.clear(); expected = "attn_kv count"; } + SUBCASE("unexpected recurrent cache") { + pool.gdn_state.emplace_back(); + expected = "gdn_state count"; + } + REQUIRE(expected != nullptr); + auto model = vllm::BorrowQwen3_5MoeLoadedModel(w); + vt::Queue q = Q(); + ModelForwardInput input{ids, positions, am, gm, pool.attn_kv, pool.gdn_state, c, q, indices}; + input.num_reqs = 1; + CHECK_THROWS_WITH_AS(ModelRegistry::Forward(*model, input), + doctest::Contains(expected), std::runtime_error); +} + +TEST_CASE("qwen35 registry preserves each hybrid GDN guard") { + const HfConfig c = MakeConfig(); + const Qwen3_5MoeWeights w = MakeWeights(c); + CachePool pool(c, 3, 8); + REQUIRE(pool.gdn_state.size() == 3); + std::vector ids = {5, 9}, positions = {0, 0}; + const std::vector indices; + auto am = PrefillAttnMeta(2, {0, 1}, 8, 0); + am.num_reqs = 2; + am.max_query_len = am.max_seq_len = 1; + am.query_start_loc = am.query_start_loc_cpu = {0, 1, 2}; + am.seq_lens = am.seq_lens_cpu = {1, 1}; + am.block_table_num_cols = 1; + am.slot_mapping = {0, 8}; + GDNAttentionMetadata gm; + gm.num_actual_tokens = gm.num_decodes = gm.num_decode_tokens = 2; + gm.non_spec_state_indices_tensor = std::vector{0, 1}; + gm.non_spec_query_start_loc = std::vector{0, 1, 2}; + const char* expected = nullptr; + SUBCASE("valid hybrid") {} + SUBCASE("all GDN caches missing") { + pool.gdn_state.clear(); + expected = "gdn_state count"; + } + SUBCASE("wrong GDN cache count") { + pool.gdn_state.pop_back(); + expected = "gdn_state count"; + } + SUBCASE("invalid SSM rank") { + pool.gdn_state.back().ssm_state.rank = 3; + expected = "GDN SSM/conv state ranks"; + } + SUBCASE("invalid convolution rank") { + pool.gdn_state.back().conv_state.rank = 4; + expected = "GDN SSM/conv state ranks"; + } + SUBCASE("mismatched SSM and convolution slots") { + pool.gdn_state.back().conv_state.shape[0] = 2; + expected = "GDN conv/SSM state slot counts"; + } + SUBCASE("inconsistent layer slots") { + pool.gdn_state.back().ssm_state.shape[0] = 2; + pool.gdn_state.back().conv_state.shape[0] = 2; + expected = "all GDN layers must use the same state slot count"; + } + SUBCASE("missing metadata") { + gm = GDNAttentionMetadata{}; + expected = "gdn_meta.num_actual_tokens"; + } + SUBCASE("wrong GDN token count") { + gm.num_actual_tokens = 1; + expected = "gdn_meta.num_actual_tokens"; + } + SUBCASE("missing indices") { + gm.non_spec_state_indices_tensor.reset(); + expected = "missing non-spec GDN state indices"; + } + SUBCASE("duplicate indices") { + gm.non_spec_state_indices_tensor = std::vector{0, 0}; + expected = "duplicate live GDN state index"; + } + SUBCASE("out-of-range indices") { + gm.non_spec_state_indices_tensor = std::vector{0, 3}; + expected = "GDN state index out of range"; + } + SUBCASE("incomplete prefill metadata") { + am = PrefillAttnMeta(2, {0}, 8, 0); + positions = {0, 1}; + gm = PrefillGdnMeta(2, 0); + gm.prefill_has_initial_state.reset(); + expected = "incomplete GDN prefill metadata"; + } + auto model = vllm::BorrowQwen3_5MoeLoadedModel(w); + vt::Queue q = Q(); + ModelForwardInput input{ids, positions, am, gm, pool.attn_kv, pool.gdn_state, c, q, indices}; + input.num_reqs = am.num_reqs; + if (expected != nullptr) { + CHECK_THROWS_WITH_AS(ModelRegistry::Forward(*model, input), + doctest::Contains(expected), std::runtime_error); + } else { + const auto logits = ModelRegistry::Forward(*model, input); + REQUIRE(logits.rows == 2); + REQUIRE(logits.vocab == c.vocab_size); + std::vector host = logits.host; + if (logits.on_device()) { + host.resize(static_cast(2 * c.vocab_size)); + auto& backend = vt::GetBackend(q.device.type); + backend.Copy(q, host.data(), logits.device_tensor.data, host.size() * sizeof(float)); + backend.Synchronize(q); + } + REQUIRE(host.size() == static_cast(2 * c.vocab_size)); + for (float value : host) CHECK(std::isfinite(value)); + } +} + +} // namespace + TEST_CASE("qwen35 paged: full-prefill batch-of-1 equals dense forward") { const HfConfig c = MakeConfig(); const Qwen3_5MoeWeights w = MakeWeights(c); diff --git a/tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp b/tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp index e613831f6..c23223ae1 100644 --- a/tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp +++ b/tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp @@ -42,6 +42,7 @@ #include "decode_graph_seam_harness.h" #include "vllm/model_executor/models/device_pool.h" +#include "vllm/model_executor/models/model_registry.h" #include "vllm/model_executor/models/qwen3_5.h" #include "vllm/model_executor/models/qwen3_5_dense.h" #include "vllm/transformers_utils/hf_config.h" @@ -974,6 +975,303 @@ bool PoolBypassLane() { } // namespace + +namespace { + +CommonAttentionMetadata FullAttnDecodeMeta(int32_t batch, int32_t position) { + auto am = DecodeAttnMeta(position); + am.num_reqs = am.num_actual_tokens = batch; + am.query_start_loc.clear(); + am.seq_lens.assign(static_cast(batch), position + 1); + am.block_table_tensor.clear(); + am.slot_mapping.clear(); + for (int32_t r = 0; r < batch; ++r) { + am.query_start_loc.push_back(r); + am.block_table_tensor.push_back(r); + am.slot_mapping.push_back(static_cast(r) * 16 + position); + } + am.query_start_loc.push_back(batch); + am.query_start_loc_cpu = am.query_start_loc; + am.seq_lens_cpu = am.seq_lens; + return am; +} + +GDNAttentionMetadata UnusedGdnMeta(int32_t batch, int mode) { + GDNAttentionMetadata gm; + if (mode == 0) return gm; + gm.num_decodes = gm.num_decode_tokens = gm.num_actual_tokens = batch; + gm.non_spec_state_indices_tensor = std::vector(static_cast(batch)); + for (int32_t r = 0; r < batch; ++r) + (*gm.non_spec_state_indices_tensor)[static_cast(r)] = r; + if (mode == 2) { + // No GDN consumer can read any of this metadata. In particular, graph + // padding must not copy this vector into an S-entry destination. + gm.non_spec_state_indices_tensor->assign(4096, 777); + gm.num_actual_tokens = -7; + gm.num_spec_decodes = 999; + } + return gm; +} + +template +void FullAttnGraphRouting(const HfConfig& c, const Weights& w) { + for (bool capture : {false, true}) { + CAPTURE(capture); + for (bool async : {false, true}) { + CAPTURE(async); + for (int mode : {0, 1, 2}) { + CAPTURE(mode); + for (int32_t batch : {1, 3}) { + CAPTURE(batch); + StaticGraphCpu harness(capture); + const ScopedEnv async_mode("VT_ASYNC_EXECUTOR", async ? "1" : "0"); + // Padding from three requests to four needs no GDN state I/O. + const ScopedEnv no_indexed_state("VT_GDN_INDEXED_STATE_IO", "0"); + CachePool pool(c, 4, 16); + REQUIRE(pool.gdn_state.empty()); + vt::Queue q = Q(); + vt::ResetGraphBreakStats(); + vt::ResetStepInputStats(); + Graph graph(w, c, q, 4); + for (int32_t step = 0; step < 6; ++step) { + const std::vector ids(static_cast(batch), 11 + step); + const std::vector positions(static_cast(batch), step); + const auto logits = graph.Step(ids, positions, FullAttnDecodeMeta(batch, step), + UnusedGdnMeta(batch, mode), pool.attn_kv, + pool.gdn_state); + REQUIRE(logits.rows == batch); + REQUIRE(logits.vocab == c.vocab_size); + REQUIRE(logits.on_device()); + if (step == 0) { + CHECK_FALSE(graph.captured()); + CHECK(vt::GetGraphBreakStats().segments_captured == 0); + } + } + const auto graphs = vt::GetGraphBreakStats(); + const auto inputs = vt::GetStepInputStats(); + if (capture) { + CHECK(graph.captured()); + CHECK(graphs.segments_captured == (async ? 2 : 1)); + CHECK(graphs.replays == (async ? 4 : 5)); + CHECK(graph.replay_count() == (async ? 4 : 5)); + // Exactly five generic inputs per slot. A sixth input means the + // no-consumer path still binds recurrent state indices. + CHECK(inputs.binds == (async ? 10 : 0)); + CHECK(inputs.host_refreshes == (async ? 10 : 0)); + } else { + CHECK_FALSE(graph.captured()); + CHECK(graphs.segments_captured == 0); + CHECK(graphs.replays == 0); + CHECK(inputs.binds == 0); + CHECK(inputs.host_refreshes == 0); + } + CHECK(inputs.device_refreshes == 0); + } + } + } + } + MESSAGE("CPU fake graph: cold, capture, staging, replay, and fallback routing only; " + "no GPU replay numerics"); +} + +template +void FullAttnGraphShapeGuards(const HfConfig& c, const Weights& w) { + StaticGraphCpu harness; + CachePool pool(c, 4, 16); + vt::Queue q = Q(); + Graph graph(w, c, q, 4); + auto am = DecodeAttnMeta(0); + const char* expected = nullptr; + SUBCASE("oversized slot mapping") { + am.slot_mapping.push_back(0); + expected = "slot_mapping"; + } + SUBCASE("oversized sequence lengths") { + am.seq_lens.push_back(1); + expected = "full-attn metadata shapes"; + } + SUBCASE("oversized query offsets") { + am.query_start_loc.push_back(1); + expected = "full-attn metadata shapes"; + } + SUBCASE("oversized block table") { + am.block_table_tensor.push_back(0); + expected = "block table"; + } + SUBCASE("query offsets do not cover the batch") { + am.query_start_loc.back() = 2; + expected = "query offsets"; + } + REQUIRE(expected != nullptr); + CHECK_THROWS_WITH_AS(graph.Step({11}, {0}, am, {}, pool.attn_kv, pool.gdn_state), + doctest::Contains(expected), std::runtime_error); +} + +// A scoped routing witness, not a new backend. The real CPU backend and all +// native CPU arithmetic remain in use. Only the MoE registry's existing FP4 +// capability predicate receives a test answer. The fake graph replays no math. +class ScopedFp4GraphCapability final : public vllm::platforms::Platform { + public: + ScopedFp4GraphCapability() + : previous_(vllm::platforms::GetPlatform(vt::DeviceType::kCPU)) { + vllm::platforms::RegisterPlatform(vt::DeviceType::kCPU, this); + } + ~ScopedFp4GraphCapability() override { + vllm::platforms::RegisterPlatform(vt::DeviceType::kCPU, &previous_); + } + vt::DeviceType device_type() const override { return previous_.device_type(); } + vt::Backend& backend() const override { return previous_.backend(); } + vllm::platforms::DeviceCapability get_device_capability() const override { + return previous_.get_device_capability(); + } + std::vector supported_dtypes() const override { + return previous_.supported_dtypes(); + } + vllm::platforms::ResidencyPolicy residency_policy() const override { + return previous_.residency_policy(); + } + bool support_static_graph_mode() const override { return true; } + bool cutlass_fp4_supported() const override { return true; } + + private: + vllm::platforms::Platform& previous_; +}; + +// The existing MakeNvfp4W4A16 test pattern, with finite nonzero E2M1 values and +// E4M3 scales. K and N are the CPU model's real expert projection dimensions. +vllm::Nvfp4Weight RoutingFp4(int64_t n, int64_t k) { + vllm::Nvfp4Weight w; + w.n = n; + w.k = k; + w.scale2 = 1.0F; + w.packed.dtype = w.scale.dtype = DType::kI8; + w.packed.rank = w.scale.rank = 2; + w.packed.shape[0] = w.scale.shape[0] = n; + w.packed.shape[1] = k / 2; + w.scale.shape[1] = k / 16; + w.packed.bytes.assign(static_cast(n * k / 2), 0x21); + w.scale.bytes.assign(static_cast(n * k / 16), 0x20); + return w; +} + +void AddRoutingFp4(Qwen3_5MoeWeights& w, const HfConfig& c) { + for (auto& layer : w.layers) { + auto& moe = layer.moe; + moe.expert_gate.clear(); + moe.expert_up.clear(); + moe.expert_down.clear(); + for (int64_t expert = 0; expert < c.num_experts; ++expert) { + moe.expert_gate_fp4.push_back(RoutingFp4(c.moe_intermediate_size, c.hidden_size)); + moe.expert_up_fp4.push_back(RoutingFp4(c.moe_intermediate_size, c.hidden_size)); + moe.expert_down_fp4.push_back(RoutingFp4(c.hidden_size, c.moe_intermediate_size)); + } + moe.shared_gate_proj = OwnedTensor{}; + moe.shared_up_proj = OwnedTensor{}; + moe.shared_down_proj = OwnedTensor{}; + moe.shared_gate_proj_fp4 = RoutingFp4(c.shared_expert_intermediate_size, c.hidden_size); + moe.shared_up_proj_fp4 = RoutingFp4(c.shared_expert_intermediate_size, c.hidden_size); + moe.shared_down_proj_fp4 = RoutingFp4(c.hidden_size, c.shared_expert_intermediate_size); + } +} + +void RegistryFullAttnGraph(vllm::LoadedModel& model, const HfConfig& c, bool expect_graph) { + CachePool pool(c, 4, 16); + vt::Queue q = Q(); + const std::vector indices; + CHECK(model.registration().architecture == c.architectures.front()); + CHECK(model.registration().factory == vllm::RegistrationFor(c.architectures.front()).factory); + vt::ResetGraphBreakStats(); + vt::ResetStepInputStats(); + for (int32_t step = 0; step < 6; ++step) { + const std::vector ids = {11 + step}, positions = {step}; + const auto am = DecodeAttnMeta(step); + const auto gm = UnusedGdnMeta(1, 2); + vllm::ModelForwardInput input{ids, positions, am, gm, pool.attn_kv, + pool.gdn_state, c, q, indices}; + input.num_reqs = 1; + input.pure_decode = true; + input.uniform_query_len = 1; + // The legacy planner group carries capacity even without a state consumer. + input.gdn_state_slots = 4; + const auto logits = vllm::ModelRegistry::Forward(model, input); + REQUIRE(logits.rows == 1); + REQUIRE(logits.vocab == c.vocab_size); + } + const auto graphs = vt::GetGraphBreakStats(); + CHECK(graphs.segments_captured == (expect_graph ? 2 : 0)); + CHECK(graphs.replays == (expect_graph ? 4 : 0)); + CHECK(vt::GetStepInputStats().binds == (expect_graph ? 10 : 0)); + CHECK(vt::GetStepInputStats().host_refreshes == (expect_graph ? 10 : 0)); +} + +} // namespace + +TEST_CASE("full-attention dense graph ignores unused GDN state") { + HfConfig c = dense::TinyConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + const auto w = dense::MakeWeights(c); + FullAttnGraphRouting(c, w); +} + +TEST_CASE("full-attention MoE graph ignores unused GDN state") { + HfConfig c = TinyConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + const auto w = MakeWeights(c); + FullAttnGraphRouting(c, w); +} + +TEST_CASE("full-attention dense graph keeps attention shape guards") { + HfConfig c = dense::TinyConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + const auto w = dense::MakeWeights(c); + FullAttnGraphShapeGuards(c, w); +} + +TEST_CASE("full-attention MoE graph keeps attention shape guards") { + HfConfig c = TinyConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + const auto w = MakeWeights(c); + FullAttnGraphShapeGuards(c, w); +} + +TEST_CASE("full-attention registry reaches the dense graph Step") { + StaticGraphCpu harness; + const ScopedEnv async("VT_ASYNC_EXECUTOR", "1"); + HfConfig c = dense::TinyConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + const auto w = dense::MakeWeights(c); + auto model = vllm::BorrowQwen3_5DenseLoadedModel(w); + RegistryFullAttnGraph(*model, c, true); +} + +TEST_CASE("full-attention registry reaches the MoE graph Step through its FP4 predicate") { + StaticGraphCpu harness; + const ScopedEnv async("VT_ASYNC_EXECUTOR", "1"); + HfConfig c = TinyConfig(); + c.num_hidden_layers = 1; + c.layer_types = {"full_attention"}; + auto w = MakeWeights(c); + AddRoutingFp4(w, c); + // Control: valid FP4 CPU arithmetic alone does not satisfy the registry's + // platform predicate, so the registered forward stays eager. + { + auto model = vllm::BorrowQwen3_5MoeLoadedModel(w); + RegistryFullAttnGraph(*model, c, false); + } + { + ScopedFp4GraphCapability fake_capability; + auto model = vllm::BorrowQwen3_5MoeLoadedModel(w); + RegistryFullAttnGraph(*model, c, true); + } + MESSAGE("MoE registry witness uses a fake CPU FP4 capability and fake graph replay; " + "no GPU replay numerics"); +} + TEST_CASE("#2029: a NON-speculative Qwen3_5DenseDecodeGraph capture allocates nothing" " [pooled lane only -- SKIPPED under VT_POOL_BYPASS, where the pool is" " disabled and the guarantee is false by design]" From 6a7bcb77637e66df34429208e3a4055e0945a875 Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Tue, 8 Sep 2026 21:27:09 -0700 Subject: [PATCH 3/6] test(ENG-QWEN35-FULL-ATTN-STATE): pin query offset guards The graph shape tests reject nonzero first offsets and descending internal offsets for both model siblings. Each case fails when its guard is deleted, so independent review can verify both offset guarantees. Production code remains unchanged. Record the repair evidence under issue #3098. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-6-astra [Codex CLI] --- .../ISSUE-GH-3098.md | 6 +++ .agents/specs/qwen35-full-attn-state.md | 48 +++++++++++++++++++ .../models/test_qwen3_5_decode_graph_seam.cpp | 15 +++++- 3 files changed, 68 insertions(+), 1 deletion(-) diff --git a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md index 3a39e2f8b..155fe15d7 100644 --- a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md +++ b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md @@ -52,6 +52,12 @@ The scoped implementation passes public completion through both default and callback sampling arms. The operator reproduced the independent red and reran all public and focused CPU gates. The implementer completed 30 mutation checks. +Fresh review found missing negative coverage for the first query offset and +internal offset ordering. The test repair covers both graph siblings. +Each independent guard deletion now fails its corresponding new subcase. +The unchanged control and four complete focused CPU suites pass. +The specification records this repair's hashes, commands, and evidence. + The exact GGUF attempt on pinned vLLM refuses the text-only engine configuration before generating a token. The owning [specification](../../specs/qwen35-full-attn-state.md) records the source hashes, commands, logs, and pending G4 obligation. Fresh diff --git a/.agents/specs/qwen35-full-attn-state.md b/.agents/specs/qwen35-full-attn-state.md index 863f0a4b8..3fe8f3a92 100644 --- a/.agents/specs/qwen35-full-attn-state.md +++ b/.agents/specs/qwen35-full-attn-state.md @@ -487,6 +487,54 @@ G4 remains `PENDING` on the operator's pinned GGUF runtime capability. The implementation does not substitute a model or oracle. Issue #3098 retains ownership of this gate, independent review, and the final operator gate. +### First-offset and ordering coverage repair + +Fresh review of `25bea3e67597f6700fc1f2e9cfb5269948338c6d` found two missing +negative cases in `FullAttnGraphShapeGuards`. Deleting the first-offset check +or the ordering check independently left the existing focused suites green. +The reviewer found no product defect for this finding. + +The repair adds distinct subcases for both dense and MoE graph `Step` methods. +The first-offset case uses `{1,1}` with one request. The ordering case uses +`{0,2,1,3}` with three requests and matching token, slot, sequence, and block +arrays. Both cases retain empty GDN metadata and the existing exception checks. +All production bytes remain unchanged from the reviewed implementation. + +Repair evidence on 9 September 2026 UTC is under +`/home/vikash/.cache/qwen35-full-attn-state-repair1/evidence`. +The linked worktree is `/home/vikash/vllm.cpp-qwen35-full-attn-state-repair1`. +The repaired test source has SHA256 +`b8f5d330fd2d6436e19e9573406adda85cb6fd1ce1736332536b1da5df403ba5`. +The normal graph executable has SHA256 +`0af438e83c0c15b66860e95dd6ea47660555387a5fc9d6716ee34a5b1d0c5d80`. + +Each deleted-guard mutant returns 1 for each sibling's corresponding new +subcase. Each failure reports that the expected exception did not occur. +An unchanged address-sanitized control passes both complete shape cases. +Every mutation uses a scratch source and a fresh copy of this worktree's +CPU archive. Production and test restoration hashes match after every run. +The mutation manifest has SHA256 +`5248b0efd6fc9e0cf465aed89a36c87933f8dca0e9cbec71e07bbb157d0d6607`. +It records exact compile, link, and run arguments, binary hashes, and logs. +The external recipe is +`/home/vikash/.cache/qwen35-full-attn-state-repair1/run-mutations.py`, SHA256 +`9bb055dbdcbc070ecd9ce06e21968d44328e14552ecd6759f44da4b61c2804cf`. + +The four complete CPU suites listed under G2 and G3 pass in `cpu-green.log`. +The complete graph suite passes 16 cases and 1558 assertions in `graph-green.log`. +Configure and build arguments match the earlier CPU recipe, with at most +`-j 4` and this repair worktree's paths. Temporary files stay outside the +worktree, and `GIT_CONFIG_GLOBAL=/dev/null` isolates Git fixtures. + +The full preflight before edits returns 0. The staged preflight uses the same +NumPy path and external compile-scheduler wrapper as the implementation gate. +The repair handoff records the staged result and exact-range checks. +The explicit CPU instruction-set audit passes against this worktree's +`build-full-attn-cpu/compile_commands.json`. +Fresh scoped review and the final operator gate remain pending. +G4 retains the pinned engine refusal. This test repair adds no oracle waiver, +GPU replay numerical claim, or performance claim. + ## Risks and stop conditions - `NEEDS_CONTEXT`: the frozen fixture bytes, pinned runtime, or source cannot diff --git a/tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp b/tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp index c23223ae1..5efb5ad90 100644 --- a/tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp +++ b/tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp @@ -1102,8 +1102,21 @@ void FullAttnGraphShapeGuards(const HfConfig& c, const Weights& w) { am.query_start_loc.back() = 2; expected = "query offsets"; } + SUBCASE("query offsets start after zero") { + am.query_start_loc = {1, 1}; + am.query_start_loc_cpu = am.query_start_loc; + expected = "query offsets"; + } + SUBCASE("query offsets descend inside the batch") { + am = FullAttnDecodeMeta(3, 0); + am.query_start_loc = {0, 2, 1, 3}; + am.query_start_loc_cpu = am.query_start_loc; + expected = "query offsets"; + } REQUIRE(expected != nullptr); - CHECK_THROWS_WITH_AS(graph.Step({11}, {0}, am, {}, pool.attn_kv, pool.gdn_state), + const std::vector ids(static_cast(am.num_actual_tokens), 11); + const std::vector positions(static_cast(am.num_actual_tokens), 0); + CHECK_THROWS_WITH_AS(graph.Step(ids, positions, am, {}, pool.attn_kv, pool.gdn_state), doctest::Contains(expected), std::runtime_error); } From 78b1ee18503356a0a7f42872d5e2b1670a244221 Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Tue, 8 Sep 2026 23:04:41 -0700 Subject: [PATCH 4/6] spec(ENG-QWEN35-FULL-ATTN-STATE): require the runner completion regression The full-attention-only runner test still expects the former forward-side GDN refusal. Specify its successful execution and state checks before correcting the test. Keep fixture geometry and runner topology unchanged. Refs #3098 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [Codex] --- .agents/specs/qwen35-full-attn-state.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/.agents/specs/qwen35-full-attn-state.md b/.agents/specs/qwen35-full-attn-state.md index 3fe8f3a92..6283c1015 100644 --- a/.agents/specs/qwen35-full-attn-state.md +++ b/.agents/specs/qwen35-full-attn-state.md @@ -41,6 +41,25 @@ kernels, new graph backends, new speculative modes, checkpoint downloads, package installation, continuous integration changes, and global oracle pins. The fix has no dependency on the separate gfx1100 provider changes. +### Runner regression amendment + +On 9 September 2026 UTC, integration testing found an obsolete expectation in +`tests/vllm/v1/worker/test_runner.cpp`. Its full-attention-only step case still +expects the forward's former GDN refusal. The reviewed consumer checks now +allow that same unchanged fixture to execute successfully. + +The operator authorizes a test-only correction under #3098. Require successful +execution, a valid sampled token with request feedback, and populated attention +state while recurrent state remains absent. Preserve the fixture's geometry, +weights, prompt, empty layer types, and single full-attention cache group. +This amendment changes no runner allocation, group topology, or product code. + +Capture the current assertion failure before editing. Run the corrected focused +case and the complete runner suite. In a scratch build, restore the dense +forward's unconditional GDN metadata requirement. The corrected case must fail +at that original refusal. Restore all source and archive bytes after the +mutation. Fresh review and operator verification remain required. + ## Inventory This per-row inventory owns the child identity through the canonical spec scan. From 776af2efd56327d193c3eba2b6934658ad7da1bc Mon Sep 17 00:00:00 2001 From: Vikash Loomba Date: Tue, 8 Sep 2026 23:10:11 -0700 Subject: [PATCH 5/6] test(ENG-QWEN35-FULL-ATTN-STATE): require runner completion without GDN The reviewed consumer checks make the full-attention runner fixture succeed, so its old exception expectation now fails. Require completed execution, changed attention state, a valid sampled token, and request feedback. Preserve the existing fixture and keep GDN state absent. The focused case passes 18 assertions. All 41 CPU runner cases pass with 1914 assertions. Restoring unconditional dense GDN validation fails the corrected case at the original refusal. Product bytes remain unchanged. Refs #3098 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:gpt-6-astra [Codex] --- .../ISSUE-GH-3098.md | 8 +++ .agents/specs/qwen35-full-attn-state.md | 31 +++++++++++ tests/vllm/v1/worker/test_runner.cpp | 53 +++++++++++++------ 3 files changed, 77 insertions(+), 15 deletions(-) diff --git a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md index 155fe15d7..f6ece4b6d 100644 --- a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md +++ b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md @@ -63,3 +63,11 @@ before generating a token. The owning [specification](../../specs/qwen35-full-at records the source hashes, commands, logs, and pending G4 obligation. Fresh review and the final operator gate remain pending. Keep this issue open until qualified work lands. + +The full-attention runner regression now requires successful execution and +sampled-token feedback instead of the former GDN refusal. Its focused case +passes 18 assertions and the complete CPU runner suite passes 41 cases and +1914 assertions. Restoring unconditional dense GDN validation fails the new +case at the original refusal. The scoped spec amendment precedes this test-only +repair; runner allocation, group topology, fixtures, and product code stay +unchanged. The specification records the exact red, green, and mutation evidence. diff --git a/.agents/specs/qwen35-full-attn-state.md b/.agents/specs/qwen35-full-attn-state.md index 6283c1015..9d8f07cac 100644 --- a/.agents/specs/qwen35-full-attn-state.md +++ b/.agents/specs/qwen35-full-attn-state.md @@ -554,6 +554,37 @@ Fresh scoped review and the final operator gate remain pending. G4 retains the pinned engine refusal. This test repair adds no oracle waiver, GPU replay numerical claim, or performance claim. +### Runner completion expectation repair + +The committed amendment `78c7441bd26db62904c18f6c105dd64dac265e77` precedes +this test edit. The unchanged pre-repair case returns one because it expects +an exception from a successful forward. The focused red log SHA256 is +`0750d5cb6e1cedda4e690e8340d3331790193604c04fb99ceffbfeca4c865641`. + +The repaired case passes 18 assertions. It checks the actual prompt, produced +logit row and request counts, changed attention-cache bytes in both layers, +an in-range sampled token, and token feedback into the admitted request. +The GDN group, caches, and metadata remain absent. The fixture bytes and +all production source remain unchanged by this repair. +The complete CPU runner suite passes 41 cases and 1914 assertions. + +Evidence is under `/home/vikash/.cache/rdna3-f16-repair1/runner-repair`. +`obsolete-expectation-red.json`, `build-receipt.json`, `focused-green.json`, +and `full-green.json` pin the source, private CPU and HIP binaries, exact +commands, and logs. The corrected test source SHA256 is +`8b70d78678245776737965cd461123b62716ed1d2cf130d604234e67519f09b2`. + +The separate scratch mutation restores only the dense entry's unconditional +GDN validation. The repaired runner case returns one at the original +`gdn_meta.num_actual_tokens must equal T` refusal. Its two assertions contain +one setup pass and one intended failure. The mutation log SHA256 is +`dd4585b2a527b262ba3447444a48566144f56c30ffc99419719eb69ea4e68d83`. +`../mutations/runner-gdn-consumer/recipe.json` records the fresh archive copy, +compile and link arguments, mutant binary, and byte-exact scratch restoration. +The original production source and archive hashes remain unchanged. +Fresh scoped review and the final operator gate remain required. The pinned +GGUF primary refusal retains its existing G4 disposition. + ## Risks and stop conditions - `NEEDS_CONTEXT`: the frozen fixture bytes, pinned runtime, or source cannot diff --git a/tests/vllm/v1/worker/test_runner.cpp b/tests/vllm/v1/worker/test_runner.cpp index ceeb54f06..597a3c551 100644 --- a/tests/vllm/v1/worker/test_runner.cpp +++ b/tests/vllm/v1/worker/test_runner.cpp @@ -1920,21 +1920,44 @@ TEST_CASE("runner: full-attention-only step skips GDN metadata build (no OOB)") SchedulerOutput s1 = NewStep({MakeFaNewReq("A", prompt, 0, {0, 1}, Greedy())}, {{"A", P}}); - // Pre-generalization, execute_model called gather_block_table(gdn_group_id_ == - // -1) → input_batch_.block_table[-1] (out-of-bounds → crash) BEFORE reaching - // the model forward. Post-generalization the whole GDN metadata build is gated - // on gdn_group_id_ >= 0, so a full-attention-only step builds a default-empty - // gdn_meta and reaches the model forward WITHOUT any out-of-bounds. - // - // The forward it reaches here is the BORROWED 27B *dense* forward, which - // carries its OWN hybrid assumption (gdn_meta must describe every token — - // qwen3_5.cpp:5463). That is a FORWARD-side seam gap, NOT a runner one: - // Qwen3ForCausalLM's own dense forward (W3) will not assume a GDN group. So we - // assert only that control reached the forward via a clean, CATCHABLE throw - // (not an uncatchable OOB), which proves the runner's GDN path was skipped. - CHECK_THROWS_WITH_AS(runner.execute_model(s1), - doctest::Contains("qwen3_5 dense paged forward"), - std::runtime_error); + REQUIRE(runner.attn_kv().size() == static_cast(c.num_hidden_layers)); + std::vector> kv_before; + for (const PagedKvCache& kv : runner.attn_kv()) { + const size_t block_bytes = static_cast( + 2 * kv.block_size * kv.num_kv_heads * kv.head_size) * vt::SizeOf(kv.dtype); + const auto* bytes = static_cast(kv.data); + kv_before.emplace_back(bytes, bytes + block_bytes); + } + + // #3098 removes the borrowed dense forward's former GDN requirement. The + // unchanged full-attention fixture must now finish with real attention state + // and sampled-token feedback, while the runner leaves GDN metadata empty. + REQUIRE_FALSE(runner.execute_model(s1).has_value()); + CHECK(runner.gdn_group_id() == -1); + CHECK(runner.gdn_state().empty()); + CHECK(runner.last_gdn_meta().num_actual_tokens == 0); + CHECK(runner.last_step().input_token_ids == prompt); + CHECK(runner.last_forward_rows() == 1); + CHECK(runner.last_forward_num_actual_tokens() == P); + CHECK(runner.last_forward_num_reqs() == 1); + for (size_t layer = 0; layer < runner.attn_kv().size(); ++layer) { + CAPTURE(layer); + const auto* bytes = static_cast(runner.attn_kv()[layer].data); + const std::vector kv_after(bytes, bytes + kv_before[layer].size()); + CHECK(kv_after != kv_before[layer]); + } + + const ModelRunnerOutput output = runner.sample_tokens(std::nullopt); + REQUIRE(output.req_ids == std::vector{"A"}); + REQUIRE(output.sampled_token_ids.size() == 1); + REQUIRE(output.sampled_token_ids[0].size() == 1); + const int32_t token = output.sampled_token_ids[0][0]; + CHECK(token >= 0); + CHECK(token < c.vocab_size); + const auto& batch = runner.input_batch(); + const int row = batch.req_id_to_index.at("A"); + CHECK(batch.num_tokens_no_spec[static_cast(row)] == P + 1); + CHECK(batch.token_id(row, P) == token); } // ─── M3: THE BLOCK-SIZE CONTRACT AT ITS PRODUCTION CALL SITE ───────────────── From 414237f9525de364c30c59c5bf6cff99f87f0457 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 11 Sep 2026 15:54:16 +0000 Subject: [PATCH 6/6] record(ENG-QWEN35-FULL-ATTN-STATE): close the canonical issue record this change lands The pull request body carries `Closes #3098`, so on merge GitHub closes the mirror while `.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md` still read `State: OPEN` with a Resolution ending "Keep this issue open until qualified work lands". Local files are the issue authority and the GitHub copy is the mirror, so the two would have disagreed the moment this landed, and the policy requires the local record to carry dated resolution evidence when the work lands rather than leaving it for a later sweep. Closed through scripts/agent-issue.py so the local transition is written and validated before the mirror moves; the record now reads CLOSED, Mirror SYNCED, Closed 2026-09-11. No product code changes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --- .../ISSUE-GH-3098.md | 32 +++---------------- 1 file changed, 5 insertions(+), 27 deletions(-) diff --git a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md index f6ece4b6d..b6159a118 100644 --- a/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md +++ b/.agents/issues/ENG-QWEN35-FULL-ATTN-STATE/ISSUE-GH-3098.md @@ -1,14 +1,14 @@ ID: ISSUE-GH-3098 Title: fix(ENG-QWEN35-FULL-ATTN-STATE): validate state only for GDN consumers Row: ENG-QWEN35-FULL-ATTN-STATE -State: OPEN +State: CLOSED Kind: UNKNOWN GitHub: 3098 -Mirror: DIVERGED +Mirror: SYNCED Availability: FULL Created: 2026-09-09 -Updated: 2026-09-09 -Closed: - +Updated: 2026-09-11 +Closed: 2026-09-11 ## Problem @@ -48,26 +48,4 @@ The quoted text below is historical evidence only. It does not define issue auth ## Resolution -The scoped implementation passes public completion through both default and -callback sampling arms. The operator reproduced the independent red and reran -all public and focused CPU gates. The implementer completed 30 mutation checks. - -Fresh review found missing negative coverage for the first query offset and -internal offset ordering. The test repair covers both graph siblings. -Each independent guard deletion now fails its corresponding new subcase. -The unchanged control and four complete focused CPU suites pass. -The specification records this repair's hashes, commands, and evidence. - -The exact GGUF attempt on pinned vLLM refuses the text-only engine configuration -before generating a token. The owning [specification](../../specs/qwen35-full-attn-state.md) -records the source hashes, commands, logs, and pending G4 obligation. Fresh -review and the final operator gate remain pending. Keep this issue open until -qualified work lands. - -The full-attention runner regression now requires successful execution and -sampled-token feedback instead of the former GDN refusal. Its focused case -passes 18 assertions and the complete CPU runner suite passes 41 cases and -1914 assertions. Restoring unconditional dense GDN validation fails the new -case at the original refusal. The scoped spec amendment precedes this test-only -repair; runner allocation, group topology, fixtures, and product code stay -unchanged. The specification records the exact red, green, and mutation evidence. +Landed by #3101 on 2026-09-11. The forward path now keys state validation on the loaded layers' actual Gated DeltaNet consumers (HasGdnConsumers), so a Qwen3.5 model with only full-attention layers completes instead of refusing on unused recurrent state. The same predicate drives the refusal, validation, input preparation and graph padding at every decision site in both the MoE and dense drivers. gdn_state size and ValidateGdnStateCacheLayout stay unconditional for real recurrent models. Three CPU cases go red on revert, including the runner regression that now requires successful execution and sampled-token feedback.