perf(GFX1100-TG200): T6b cooperative attn preamble arm - #2868
Conversation
f426500 to
e2205c6
Compare
|
Held back with the rest of the I reviewed this change on its own and have no objection to it. Once #2790's base Landing today from this set: #2782 (with the grouped-Q8_0 repair), #2777 and |
e2205c6 to
83cb1b2
Compare
|
Rebased onto the repaired stack tip 2ff6af4 (base b9f2ef4, the external-contributor landing branch). Gates green (check-env-doc, check-agent-record, check-rocm-dp4a-intrinsic); vllm-cli + test_rocm_quant_dot compile and link at -Werror in rocm-dev:10.0.0 on gfx1100; the default path (all arms off) is byte-identical to the staging baseline on Qwen3.5-4B Q4_K_M (canonical TG200 prompt, 256 tokens, greedy, seed 0). Engine A/B numbers measured today (one window, 5 reps, one load per arm): default 48.4 tok/s median; VT_GEMV_MMVQ=1 alone 53.5 (+10.0%); the byte-exact arm group (VT_NORM_QUANT_FUSED, VT_QUANT_Q8K_WARP) verified token-identical solo; VT_SKINNY_BF16 and the keep-quant arms (VT_GDN_ROWPERM_KEEP_QUANT / VT_GDN_COLPERM_KEEP_QUANT) and the association-change arms (VT_ATTN_DECODE_GQA4, VT_RMSNORM_ROW_COOP) move early near-tie tokens on the engine and are NOT engine-token-identical — flagged here so the acceptance terms name them; the full opt-in stack measured 72.5 tok/s median (+49.6%) with that documented divergence. This branch's head is now 83cb1b2. Ping for re-review. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true |
9904cd3 to
697e27b
Compare
|
Rebased onto FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true |
c199099 to
98d68ff
Compare
…mble arm AttnQkNormRopeGateK prices decode at ~88us/call: items = t*(hq+hkv) = 24 at batch 1, so only 24 of 256 threads run, each doing three serial dh- loops (gate copy, sumsq, rope/norm store) through private cache lines (8 layers x 88us = 0.70 ms/token). AttnQkNormRopeGateCoopK maps one WARP per item: lanes stride j for the gate copy and stores, sumsq reduces through a fixed shfl_down tree, inv broadcasts. Every element's math is the donor's given inv; only the ss summation order changes, so outputs are NMSE-equal: cross_device passes identically under the flag (24/25, the one failure pre-existing). Opt-in behind VT_ATTN_PREAMBLE_COOP=1. Acceptance A/B interleaved x5: median 76.595 vs 73.196 tok/s (+4.6%), ON wins all five pairs; near-tie adjudication in the evidence file. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:ox-alpha [omp] Assisted-by: AGENT:OMEN-ALPHA [OMP]
98d68ff to
da4683e
Compare
2 participants
Closes #2867.
Row:
GFX1100-TG200T6b adds a warp-per-item cooperative attention preamble arm in
rocm_gdn_fused.hip, opt-in behindVT_ATTN_PREAMBLE_COOP=1(default OFF).The cooperative preamble distributes the attention precomputation across
warps to reduce serial latency.
A/B benchmark (interleaved, 5 pairs, Qwen3.5-4B Q4_K_M, 256 tokens)
+3.4% over T25 parent. Consistent across all 5 pairs (stdev 0.22%).
Token-identical.
Depends on the full T25 chain: #2782, #2790, #2792, #2796, #2800, #2804, #2807.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:GLM-5-2 [OMP]