perf(onpair): speed up compression with onpair 0.2.1 and row training - #9806
perf(onpair): speed up compression with onpair 0.2.1 and row training#9806gargiulofrancesco wants to merge 5 commits into
Conversation
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Merging this PR will not alter performance
|
| #[inline] | ||
| fn row(&self, i: usize) -> &[u8] { | ||
| let view = &self.views[i]; | ||
| if self.lengths[i] == 0 { |
There was a problem hiding this comment.
if len is 0 the view should also be 0, I would personally skip this branch
There was a problem hiding this comment.
lengths[i] is also zero for nulls, but masking can leave their views nonempty
robert3005
left a comment
robert3005
left a comment
There was a problem hiding this comment.
just the single comment
| flat.extend_from_slice(bytes); | ||
| offsets.push(u64::try_from(flat.len()).vortex_expect("offset must fit in u64")); | ||
| uncompressed_lengths | ||
| .push(i32::try_from(view.len()).vortex_expect("must fit in i32")); |
There was a problem hiding this comment.
this cannot fail, use debug assert
| offsets.push(u64::try_from(flat.len()).vortex_expect("offset must fit in u64")); | ||
| uncompressed_lengths | ||
| .push(i32::try_from(view.len()).vortex_expect("must fit in i32")); | ||
| total_bytes += view.len() as usize; |
| struct ViewRows<'a> { | ||
| views: &'a [BinaryView], | ||
| buffers: &'a [&'a ByteBuffer], | ||
| lengths: &'a [i32], |
| @@ -61,32 +55,32 @@ pub fn onpair_compress( | |||
| match mask.bit_buffer() { | |||
| AllOr::All => { | |||
| for view in views { | |||
There was a problem hiding this comment.
zip views with uncompressed_lengths
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
Signed-off-by: Francesco Gargiulo <gargiulo.fr@gmail.com>
3 participants
Summary
Speed up OnPair compression by upgrading to onpair 0.2.1 and training directly from Vortex string views, avoiding the full input copy and temporary offsets buffer.
Changes