Skip to content

feat: add per-lane popcount on batch - #1401

Open
DiamonDinoia wants to merge 2 commits into
xtensor-stack:masterfrom
DiamonDinoia:feat/popcount
Open

feat: add per-lane popcount on batch#1401
DiamonDinoia wants to merge 2 commits into
xtensor-stack:masterfrom
DiamonDinoia:feat/popcount

Conversation

@DiamonDinoia

Copy link
Copy Markdown
Contributor

Count the bits set in each element of an integer batch. The common kernel is the SWAR fold; NEON uses CNT with pairwise widening adds, SVE uses svcnt_x, and WASM uses i8x16.popcnt with pairwise widening extends, falling back to the common kernel for 64-bit elements.

I will add kernels in the future too when I have more of this. I need the bitwise operation for a Morton transform library I am writing.

Creating a new file because I plan to add more functions in the future:
countl_zero/countr_zero, bit_reverse, multishift, bit_deposit/bit_extract, bit_permute + bit_permute_constant, bit_matmul, GFNI kernels.

Reviewed by: Claude Opus 5 noreply@anthropic.com

@serge-sans-paille

Copy link
Copy Markdown
Contributor

With the appropriate credits, https://github.com/WojciechMula/sse-popcount seems to provide relevant implementation for arch-specifics

@DiamonDinoia

Copy link
Copy Markdown
Contributor Author

Good point. I also should have listed that the algorithm I implemented I found it on wikipedia.

@serge-sans-paille

Copy link
Copy Markdown
Contributor

See also https://github.com/kimwalisch/libpopcnt/tree/master based on the paper above for an AVX2 implementation you could add to this PR (with proper credit of course)

@DiamonDinoia
DiamonDinoia force-pushed the feat/popcount branch 2 times, most recently from bc4f4e4 to 321099e Compare August 25, 2026 13:21
Comment thread include/xsimd/arch/xsimd_neon.hpp Outdated
return bitwise_cast<T>(batch<uint16_t, A>(vpaddlq_u8(counts)));
else
{
#if defined(__ARM_FEATURE_DOTPROD)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@serge-sans-paille you know about arm more than I do. Is this a good idea? should this be a new architecture entirely? If so I will purge it from here and add it in a separate PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, it should be a proper architecture :-/

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I will do a follow up PR with the missing architectures.

@serge-sans-paille serge-sans-paille left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add (and test) a scalar version for this, that uses the version you provide in xsimd_common_bitwise.hpp?


namespace detail
{
// bit i is set when i / s is even: 0x55.. for s == 1, 0x33.. for

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a big fan of neither the name, nor the explanation, nor the implementation.
What about the more generic:

// Create an integer of type U with Pattern P repeated as much as possible within U
template<class U, class P>
constexpr U repeat_pattern() {
    static_assert(P!=0);
    constexpr size_t pattern_size = sizeof(U) * CHAR_BITS - std::count_lzero(P); // FIXME: need to provide our implementation
    static_assert(sizeof(U) * CHAR_BIT % pattern_size == 0);
    constexpr size_t rep_count = sizeof(U) * CHAR_BIT % pattern_size;
    U result = P;
    for(size_t i = 1; i < rep_count; ++i) {
        result |= P << ( pattern_size * i );
    }
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I was wondering how to simplify this

x = (x + (x >> 16)) & b_type(detail::alternating_mask<U>(16));
if constexpr (bits >= 64)
x = (x + (x >> 32)) & b_type(detail::alternating_mask<U>(32));
return bitwise_cast<T>(x);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could / should check if __builtin_popcount is available and if so, use it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We fallback on it in the scalar case. Am I missing something? This is the fallback in case that builtin is not available. I am thinking of msvc.

xsimd_common_bit.hpp gated the standard <bit> path on
XSIMD_CPP_VERSION > 202002L, so exactly C++20 skipped std::popcount and
compiled the backport instead. Every other XSIMD_CPP_VERSION test in the
tree uses >=.

The static_assert in test_bit.cpp is the check: std::popcount is
constexpr and the backport is not, so a C++20 build that stops reaching
the standard library fails to compile.

Assisted-by: Claude Opus 5 <noreply@anthropic.com>
Count the bits set in each element of an integer batch. The common kernel is
the SWAR fold; x86 uses the PSHUFB nibble lookup from SSSE3 up, NEON uses CNT
with pairwise widening adds, SVE uses svcnt_x, and WASM uses i8x16.popcnt with
pairwise widening extends, folding 32-bit counts with a shift-and-add pair for
64-bit elements. VSX and VXE use vec_popcnt, which the compiler maps to a
single VPOPCNTB/H/W/D or VPOPCT. The scalar overload forwards to
detail::popcount and returns T, so it agrees with the batch overload.

The common kernel runs the whole fold on 64-bit lanes whatever T is, and masks
the surviving byte at the end. Every step of the fold confines its own carries,
so a bit that crosses a T boundary is masked off again, and a target with no
narrow shift does not pay for one to be emulated. This drops emulated<128> u8
from 356 instructions to 20, and measures 35.4x there, 12.6x on emulated<128>
u16 and 2.60x on avx2 u8.

repeat_pattern<U, P>() builds the fold's masks from the pattern and the type of
the repeat unit, so 0x5555555555555555 reads as repeat_pattern<uint64_t,
uint8_t(0x55)>() rather than as a literal.

For 64-bit elements the two x86 nibble tables carry a +4 and a -4 bias, after
libpopcnt, so PSADBW yields the byte count and the 8-byte sum in one
instruction. This drops the VPADDB, and measures 1.09x on SSE and AVX2 and
1.05x on AVX-512.

sse2 gets its own kernel, since the SWAR fold is what a default x86-64 build
runs and it leaves PSADBW and PMADDWD unused. PSADBW sums the eight byte counts
of a 64-bit element in one instruction, PMADDWD adds the two halves of a 32-bit
one, and the byte shifts drop the mask that the fold already applies. This cuts
64-bit elements from 25 instructions to 16, 32-bit ones from 22 to 19 and 8-bit
ones from 15 to 14, and measures 1.5x, 1.09x and 1.08x. 16-bit elements keep
the fold's instruction count, since SSE2 has no PMADDUBSW to combine the two
byte counts of an element.

avx512vnni gains a 32-bit kernel: VPDPBUSD does in one uop what the VPMADDUBSW
and VPMADDWD pair does in two, and the zero accumulator is free because the
register copy is eliminated at rename. This measures 1.14x. The same
substitution on 256-bit vectors is neutral, since three ports serve them, so
avxvnni gets no kernel.

The CI job labelled avx512vnni built for knm, which enables avx5124vnniw rather
than avx512vnni and selected the avx512pf arch, so it covered neither kernel.
It now builds for cascadelake.

Assisted-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants