feat: BMI for bit filtering#10136
Conversation
|
run benchmark arrow-select |
|
Hi @devanbenz, thanks for the request (#10136 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: Dandandan, Fokko, Jefffrey, Omega359, adriangb, alamb, asubiotto, brunal, buraksenn, cetra3, codephage2020, coderfender, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mkleen, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas. File an issue against this benchmark runner |
|
run benchmark arrow-select |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing db/10098/bmi-null-bool-filter (76c7efc) to 1ba5d48 (merge-base) diff File an issue against this benchmark runner |
|
Benchmark for this request failed. Last 20 lines of output: Click to expandFile an issue against this benchmark runner |
|
run benchmark arrow_select |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing db/10098/bmi-null-bool-filter (76c7efc) to 1ba5d48 (merge-base) diff File an issue against this benchmark runner |
|
Benchmark for this request failed. Last 20 lines of output: Click to expandFile an issue against this benchmark runner |
@alamb my apologies, the bench is called |
|
run benchmark filter_bits |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing db/10098/bmi-null-bool-filter (76c7efc) to 1ba5d48 (merge-base) diff File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)New benchmark — branch-only results (no baseline comparison) Details
Resource Usagebranch
File an issue against this benchmark runner |
|
@alamb Could you please run the benchmark again but setting |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
run benchmark filter_bits env: |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing db/10098/bmi-null-bool-filter (8e326b4) to 9f37683 (merge-base) diff File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)New benchmark — branch-only results (no baseline comparison) Details
Resource Usagebranch
File an issue against this benchmark runner |
|
@alamb Taking a look at the host machines. It doesn't look like the processor supports BMI2 intrinsics. So the benches are basically equal. |
|
Now that #9848 Is merged in, I'm going to merge this PR with it and move the |
|
nice |
DRAFT
This commit adds the ability for bit filtering to be done using the
_pext_u64BMI.Right now this PR is directly copying the
bit_util.rscode from: https://github.com/apache/arrow-rs/pull/9848/changes#diff-f2fe5b601534125d0ba033762b62db93ae2f81c71a543b6694df33773ebfa308R939-R976 -- I've just moved thecompressfunction in toarrow-buffer. This draft PR is mostly meant to allow for me to benchmark on my ARM machine and x86_64 machine.I noticed while bench-marking using the BMI path (
filter_bits_compress) leads to a regression when we have a very dense or a very sparse filter mask. Besides those two outliers we gain a large performance gain from BMI.