Improve SVE2 optimizations of SimdDescrIntCosineDistancesMxNa - #905
Merged
Conversation
Replace the 4x1 unpack GEMM and missing 4/8-bit direct tiles with unpredicated 4x4 svdot kernels, vectorized unpack of packed 4-7 bit data, and transposed B norms. Drop the previous Mx1-only fallback for depths 4 and 8, extend MxNa tests, and note the change in release 7.2.165. Co-authored-by: igor.ermolaev <igor.ermolaev@altezza.org>
ermig1979
marked this pull request as ready for review
August 24, 2026 08:40
UnpackNormB used vst1q_f32 with stride N, which is not 16-byte aligned when N=7. Decode used the same NEON 16-byte loads on that layout and on unaligned stack temps. Those accesses are new on the SVE2 unpack path (NEON keeps N<8 on the direct path) and can raise a segment violation. Copy headers with memcpy, decode 4-wide tiles with scalar loads, align svld1rq tables, and bind SVE2 function pointers explicitly. Co-authored-by: igor.ermolaev <igor.ermolaev@altezza.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SVE2 path of
SimdDescrIntCosineDistancesMxNaused a 4x1 unpack GEMM (_microNu = 1) with scalar 4–7 bit unpack, and the direct kernel skipped 4x4 tiles for depths 4 and 8. That is much weaker than the NEON 6x16 unpack / 4x4 direct layout and was slower than NEON.This change:
svdot_u32micro-kernels for every depth on the direct path (nibble split for 4-bit, load+dot for 8-bit, existingsvtblunpack for 5/6/7-bit)UnpackDatafor 4–8 bits (svzipfor nibbles,svtblfor 5/6/7, predicated copy for 8)svld4/svst4loads/storesDescrIntCosineDistancesMxNaAutoTestwith sizes 8x8x24, 4x12x256, and 5x7x128Follow-up fix for a SIGSEGV on
CosineDistancesMxNa-5-7-128-4:UnpackNormBno longer usesvst1q_f32with strideN(unaligned whenN=7)svld1rqunpack tables are 16-byte alignedSimdSve2DescrInt.cpp/SimdSve2DescrIntCdd.cpp/SimdSve2DescrIntCdu.cppwere already inprj/vs2022/Sve2.vcxprojandSve2.vcxproj.filters.Correctness:
aarch64-linux-gnu-g++(-march=armv9-a+sve+sve2, scalable VL)gap=17andgap=1024),sve-max-vq=1/2/4andneoverse-n2: all passed./Test -fi=DescrIntCosineDistancesMxNa(includes 5x7x128-4): success