feat: add per-lane popcount on batch - #1401
Conversation
0a717d5 to
9bb2833
Compare
|
With the appropriate credits, https://github.com/WojciechMula/sse-popcount seems to provide relevant implementation for arch-specifics |
9bb2833 to
5722efd
Compare
|
Good point. I also should have listed that the algorithm I implemented I found it on wikipedia. |
5722efd to
ca782d8
Compare
|
See also https://github.com/kimwalisch/libpopcnt/tree/master based on the paper above for an AVX2 implementation you could add to this PR (with proper credit of course) |
ca782d8 to
0bdf5f0
Compare
bc4f4e4 to
321099e
Compare
| return bitwise_cast<T>(batch<uint16_t, A>(vpaddlq_u8(counts))); | ||
| else | ||
| { | ||
| #if defined(__ARM_FEATURE_DOTPROD) |
There was a problem hiding this comment.
@serge-sans-paille you know about arm more than I do. Is this a good idea? should this be a new architecture entirely? If so I will purge it from here and add it in a separate PR.
There was a problem hiding this comment.
yeah, it should be a proper architecture :-/
There was a problem hiding this comment.
Okay I will do a follow up PR with the missing architectures.
serge-sans-paille
left a comment
There was a problem hiding this comment.
Could you please add (and test) a scalar version for this, that uses the version you provide in xsimd_common_bitwise.hpp?
|
|
||
| namespace detail | ||
| { | ||
| // bit i is set when i / s is even: 0x55.. for s == 1, 0x33.. for |
There was a problem hiding this comment.
I'm not a big fan of neither the name, nor the explanation, nor the implementation.
What about the more generic:
// Create an integer of type U with Pattern P repeated as much as possible within U
template<class U, class P>
constexpr U repeat_pattern() {
static_assert(P!=0);
constexpr size_t pattern_size = sizeof(U) * CHAR_BITS - std::count_lzero(P); // FIXME: need to provide our implementation
static_assert(sizeof(U) * CHAR_BIT % pattern_size == 0);
constexpr size_t rep_count = sizeof(U) * CHAR_BIT % pattern_size;
U result = P;
for(size_t i = 1; i < rep_count; ++i) {
result |= P << ( pattern_size * i );
}
}
There was a problem hiding this comment.
Good point. I was wondering how to simplify this
| x = (x + (x >> 16)) & b_type(detail::alternating_mask<U>(16)); | ||
| if constexpr (bits >= 64) | ||
| x = (x + (x >> 32)) & b_type(detail::alternating_mask<U>(32)); | ||
| return bitwise_cast<T>(x); |
There was a problem hiding this comment.
you could / should check if __builtin_popcount is available and if so, use it.
There was a problem hiding this comment.
We fallback on it in the scalar case. Am I missing something? This is the fallback in case that builtin is not available. I am thinking of msvc.
xsimd_common_bit.hpp gated the standard <bit> path on XSIMD_CPP_VERSION > 202002L, so exactly C++20 skipped std::popcount and compiled the backport instead. Every other XSIMD_CPP_VERSION test in the tree uses >=. The static_assert in test_bit.cpp is the check: std::popcount is constexpr and the backport is not, so a C++20 build that stops reaching the standard library fails to compile. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
Count the bits set in each element of an integer batch. The common kernel is the SWAR fold; x86 uses the PSHUFB nibble lookup from SSSE3 up, NEON uses CNT with pairwise widening adds, SVE uses svcnt_x, and WASM uses i8x16.popcnt with pairwise widening extends, folding 32-bit counts with a shift-and-add pair for 64-bit elements. VSX and VXE use vec_popcnt, which the compiler maps to a single VPOPCNTB/H/W/D or VPOPCT. The scalar overload forwards to detail::popcount and returns T, so it agrees with the batch overload. The common kernel runs the whole fold on 64-bit lanes whatever T is, and masks the surviving byte at the end. Every step of the fold confines its own carries, so a bit that crosses a T boundary is masked off again, and a target with no narrow shift does not pay for one to be emulated. This drops emulated<128> u8 from 356 instructions to 20, and measures 35.4x there, 12.6x on emulated<128> u16 and 2.60x on avx2 u8. repeat_pattern<U, P>() builds the fold's masks from the pattern and the type of the repeat unit, so 0x5555555555555555 reads as repeat_pattern<uint64_t, uint8_t(0x55)>() rather than as a literal. For 64-bit elements the two x86 nibble tables carry a +4 and a -4 bias, after libpopcnt, so PSADBW yields the byte count and the 8-byte sum in one instruction. This drops the VPADDB, and measures 1.09x on SSE and AVX2 and 1.05x on AVX-512. sse2 gets its own kernel, since the SWAR fold is what a default x86-64 build runs and it leaves PSADBW and PMADDWD unused. PSADBW sums the eight byte counts of a 64-bit element in one instruction, PMADDWD adds the two halves of a 32-bit one, and the byte shifts drop the mask that the fold already applies. This cuts 64-bit elements from 25 instructions to 16, 32-bit ones from 22 to 19 and 8-bit ones from 15 to 14, and measures 1.5x, 1.09x and 1.08x. 16-bit elements keep the fold's instruction count, since SSE2 has no PMADDUBSW to combine the two byte counts of an element. avx512vnni gains a 32-bit kernel: VPDPBUSD does in one uop what the VPMADDUBSW and VPMADDWD pair does in two, and the zero accumulator is free because the register copy is eliminated at rename. This measures 1.14x. The same substitution on 256-bit vectors is neutral, since three ports serve them, so avxvnni gets no kernel. The CI job labelled avx512vnni built for knm, which enables avx5124vnniw rather than avx512vnni and selected the avx512pf arch, so it covered neither kernel. It now builds for cascadelake. Assisted-by: Claude Opus 5 <noreply@anthropic.com>
321099e to
1521188
Compare
Count the bits set in each element of an integer batch. The common kernel is the SWAR fold; NEON uses CNT with pairwise widening adds, SVE uses svcnt_x, and WASM uses i8x16.popcnt with pairwise widening extends, falling back to the common kernel for 64-bit elements.
I will add kernels in the future too when I have more of this. I need the bitwise operation for a Morton transform library I am writing.
Creating a new file because I plan to add more functions in the future:
countl_zero/countr_zero, bit_reverse, multishift, bit_deposit/bit_extract, bit_permute + bit_permute_constant, bit_matmul, GFNI kernels.
Reviewed by: Claude Opus 5 noreply@anthropic.com