Archive build speedup

[Sanya239]

No description provided.

[codecov]

Codecov Report

❌ Patch coverage is 92.17252% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.35%. Comparing base (4e52a07) to head (68b691f).

Files with missing lines	Patch %	Lines
include/pixie/detail/huffman_build_table.h	89.42%	21 Missing and 3 partials ⚠️
include/pixie/wavelet_tree/index.h	88.62%	7 Missing and 12 partials ⚠️
include/pixie/detail/wavelet_partition.h	95.89%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main      #67    +/-   ##
========================================
  Coverage   91.34%   91.35%            
========================================
  Files          53       56     +3     
  Lines       13118    13722   +604     
  Branches     1830     1940   +110     
========================================
+ Hits        11983    12536   +553     
- Misses        523      555    +32     
- Partials      612      631    +19     

Flag	Coverage Δ
gcov	91.35% <92.17%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Malkovsky]

@codex Please summarize the main algorithmic changes

[chatgpt-codex-connector]

Main algorithmic changes

1. Dependency-reduced byte histograms

   • Byte-frequency collection now uses four independent, cache-line-aligned counting lanes. Consecutive bytes update different lanes, reducing the dependency chain between increments; the lanes are combined only after scanning the input. include/pixie/detail/byte_histogram.hL15-L48
   • This histogram is used both when constructing a byte wavelet tree directly and during the first archive pass. include/pixie/wavelet_tree/index.hL778-L795 include/pixie/file_archive.hL552-L594

2. Bulk stable partitioning replaces element-at-a-time wavelet construction

   • Construction now processes ranks in blocks, generates up to 64 left/right direction bits at once, writes those bits directly into each node’s packed bit stream, and stably partitions the ranks into child subsequences. include/pixie/detail/wavelet_partition.hL313-L345 include/pixie/detail/wavelet_partition.hL480-L526
   • The partitioner avoids producing child data when that child is already a leaf. Compile-time WriteLeft/WriteRight specializations remove unnecessary scatter and copy work. include/pixie/detail/wavelet_partition.hL490-L520 include/pixie/detail/wavelet_partition.hL856-L890
   • Byte construction uses a bounded block buffer and one scratch buffer instead of materializing and recursively repartitioning the entire input. Blocks are converted to in-order leaf ranks, recursively partitioned, and appended bit-contiguously to the persistent node streams. include/pixie/wavelet_tree/index.hL435-L464 include/pixie/wavelet_tree/index.hL466-L510

3. Precomputed and SIMD-assisted compaction

   • AVX2 construction uses compile-time shuffle tables indexed by direction masks. A shuffle packs left-bound values forward and right-bound values backward, after which a fixed reverse shuffle restores stable order for the right partition. include/pixie/detail/wavelet_partition.hL35-L54 include/pixie/detail/wavelet_partition.hL61-L108
   • The optimized byte kernel handles 32-byte groups with those table-driven shuffles and can use AVX-512 VBMI2 compress stores for 64-byte groups when available. include/pixie/detail/wavelet_partition.hL737-L773 include/pixie/detail/wavelet_partition.hL775-L833
   • An equivalent AArch64 NEON table-lookup path was added, while the scalar implementation remains the fallback. include/pixie/detail/wavelet_partition.hL646-L731 include/pixie/detail/wavelet_partition.hL873-L883

4. Specialized byte-to-rank mapping

   • Before partitioning, byte symbols are translated to the Huffman tree’s in-order leaf ranks in bulk. The implementation uses NEON table lookups on AArch64 or paired lookup arithmetic on little-endian platforms, followed by a scalar tail. include/pixie/detail/wavelet_partition.hL528-L603
   • This rank mapping lets every internal-node decision become a simple threshold comparison—rank >= first_right—rather than repeatedly consulting symbol-specific codes. include/pixie/detail/wavelet_partition.hL614-L641

5. A specialized, bounded-depth Huffman builder for byte alphabets

   • For byte Huffman trees, the generic priority-queue topology builder is bypassed in favor of fixed-size arrays and a two-queue Huffman construction after frequency sorting. include/pixie/detail/huffman_build_table.hL98-L154 include/pixie/wavelet_tree/index.hL656-L668
   • Huffman code lengths are capped at 11 using a DEFLATE-style Kraft-balance adjustment. This bounds construction recursion and scratch requirements even for extremely skewed distributions. include/pixie/detail/huffman_build_table.hL157-L215
   • The adjusted lengths are converted into a fused canonical tree. The builder assigns symbols contiguous in-order ranks, records each internal node’s split rank, and classifies leaf configurations so later construction can skip dead child writes. include/pixie/detail/huffman_build_table.hL248-L267 include/pixie/detail/huffman_build_table.hL269-L375
   • Empty and single-symbol alphabets are handled directly without invoking the full Huffman procedure. include/pixie/detail/huffman_build_table.hL384-L409

6. Archive construction becomes an explicit two-pass streaming build

   • The first archive pass gathers metadata, validates UTF-8, counts newlines, computes byte frequencies, and hashes each content stream. include/pixie/file_archive.hL546-L613
   • The second pass streams content directly into the wavelet-tree builder rather than assembling the complete archive payload in memory. Size and hash checks detect a source that changes between passes. include/pixie/file_archive.hL618-L650

Net effect

The PR primarily changes construction, not query semantics: it replaces per-symbol and whole-input intermediate work with dependency-reduced counting, bounded block streaming, bulk direction-mask generation, table/SIMD-based stable compaction, and a fixed-array length-limited Huffman topology builder. Rank, select, serialization, and segment access continue to operate on the same persistent wavelet-tree representation.

Commands consulted

• ✅ git diff --stat 4e52a070b1167ce5459ffa49da18c790708eef92..68b691f409001a5a05b103a673ef581326cb929c
• ✅ git diff --name-status 4e52a070b1167ce5459ffa49da18c790708eef92..68b691f409001a5a05b103a673ef581326cb929c
• ✅ git show --stat --oneline 445d7d8
• ✅ git show --stat --oneline 87e3dce
• ✅ git show --stat --oneline 68b691f
• ✅ git diff --unified=3 4e52a07..68b691f -- include/pixie/detail/byte_histogram.h include/pixie/detail/wavelet_partition.h include/pixie/detail/huffman_build_table.h include/pixie/wavelet_tree/index.h include/pixie/file_archive.h
• ✅ rg -n "partition|histogram|Huffman|build" include/pixie/detail/byte_histogram.h include/pixie/detail/wavelet_partition.h include/pixie/detail/huffman_build_table.h include/pixie/wavelet_tree/index.h include/pixie/file_archive.h

No code changes were needed for this summary, so no commit or follow-up pull request was created.

View task →

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 68b691f409

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Require one mapping for each nonempty terminal branch

For a Huffman byte artifact whose alphabet contains absent symbols, a corrupted leaf map can assign an absent rank to the same node as the branch's present rank. This loop merely sets has_mapped_symbol repeatedly, so full validation accepts the artifact; subsequent rank and select queries for that absent symbol return occurrences belonging to the present symbol. Count the mappings and reject a nonempty terminal branch unless exactly one rank maps to it.

Useful? React with 👍 / 👎.

[Malkovsky]

@codex please give a breakdown of cpu usage for WT construction after this PR changes

[chatgpt-codex-connector]

WT construction CPU breakdown

I profiled Huffman-shaped WaveletTree<std::uint8_t> construction at the PR head (68b691f) and compared it with the base (4e52a07).

Workload and environment

• Input: 64 MiB of bytes.
• Distribution: deterministic, Zipf-like workload with 16 frequent symbols plus occasional full-range bytes.
• Alphabet: 256 symbols.
• Iterations: 8 constructions, totaling 512 MiB processed.
• Compiler: g++ -std=c++20 -O3 -g -pg -march=native.
• CPU: Intel Xeon Platinum 8370C, with AVX2 and AVX-512 support.
• Execution: construction is currently single-threaded, so it uses at most one logical CPU.

The benchmark deliberately exercises the new byte-specialized path. The constructor first creates the byte histogram and then passes the data into build_from_counts. include/pixie/wavelet_tree/index.hL775-L808

Results

Measurement	Base	PR head	Change
Total wall time, including deterministic input generation	24.060 s	2.930 s	8.21× faster
Total user CPU time	23.719 s	2.546 s	9.32× less
Total system CPU time	0.327 s	0.372 s	Essentially unchanged
Profiled WT construction CPU per 64 MiB tree	~2.92 s	~0.223 s	~13.1× faster
Approximate construction throughput from sampled CPU time	~21.9 MiB/s	~288 MiB/s	~13.1× higher

The wall-time speedup is lower than the construction-only speedup because deterministic input generation is identical between runs and becomes a substantial fraction of the much shorter head run.

PR-head construction CPU attribution

The profile sampled about 1.78 seconds inside WT construction across eight builds:

Construction component	Sampled CPU	Share of WT construction
Byte block mapping, recursive partitioning, and packed direction-stream emission	1.54 s	86.5%
Rank/select support construction	0.16 s	9.0%
Final PreWaveletNode → WaveletNode materialization	0.05 s	2.8%
Histogram, Huffman topology, validation, and other setup	~0.03 s	~1.7%
Total	1.78 s	100%

Detail within the dominant 86.5%

The byte-stream phase divided approximately as follows:

• ~0.95 s, or 53.4% of total construction CPU: optimized partition/compaction kernels.
• ~0.59 s, or 33.1% of total construction CPU: surrounding block processing—rank mapping, block movement, packed-bit appends, histogram verification, and recursive traversal.
• Together these account for the 86.5% byte-stream construction share.

This aligns with the implementation:

• Construction keeps a bounded block and scratch buffer, maps symbols to in-order ranks, and encodes each completed block recursively. include/pixie/wavelet_tree/index.hL441-L498
• It also accumulates and verifies the emitted histogram during that pass. include/pixie/wavelet_tree/index.hL466-L509
• On this host, the partition dispatcher selects the x86 SIMD implementation. include/pixie/detail/wavelet_partition.hL862-L890
• That implementation processes 32-byte and 16-byte groups with direction masks and compacted writes before handling the scalar tail. include/pixie/detail/wavelet_partition.hL775-L851

What is no longer expensive

The specialized Huffman topology builder was below meaningful sampling resolution. That is expected because it operates over at most 256 alphabet entries rather than over every input byte; it handles empty/single-symbol cases directly and otherwise builds lengths and the fused tree from fixed-size structures. include/pixie/detail/huffman_build_table.hL384-L409

Likewise, initial frequency collection is now a simple byte-histogram pass. Its four independent counting lanes reduce increment dependencies. include/pixie/detail/byte_histogram.hL15-L48

Interpretation

1. The optimization moved WT construction out of topology/bookkeeping and into bulk byte processing. Almost nine-tenths of the remaining CPU is spent in rank mapping, packed direction generation, and stable partitioning.
2. Rank/select materialization is now visible at about 9%. As partitioning gets faster, this unchanged downstream phase becomes a more meaningful fraction of total construction.
3. Huffman tree generation is effectively negligible for large inputs. Further optimizing it would not materially improve construction throughput.
4. The best next optimization target is the byte-stream phase, especially the partition/compaction kernels, which alone represent roughly half of construction CPU.
5. CPU utilization is one-core-bound rather than parallel. The head run reported 2.546 seconds of user CPU over 2.930 seconds wall time for the entire benchmark, including allocation/page-fault system time and unprofiled setup. This is not a multi-core construction pipeline.

Profiling caveat

GNU gprof attributed the heavily inlined SIMD partition body to the synthetic symbol frame_dummy. The call graph places all of those samples beneath the byte-block encoding callback, so I classified them as the optimized partition/compaction kernel. Percentages should be treated as sampling-level estimates, particularly for the ~1.7% residual bucket.

No repository changes were required, so I did not create a commit or follow-up PR.

Checks and commands

• ✅ git worktree add --detach /tmp/pixie-base 4e52a070b1167ce5459ffa49da18c790708eef92
• ✅ g++ -std=c++20 -O3 -g -pg -march=native -I/tmp/pixie-base/include /tmp/wt_profile.cpp -o /tmp/wt_base
• ✅ g++ -std=c++20 -O3 -g -pg -march=native -I/workspace/pixie/include /tmp/wt_profile.cpp -o /tmp/wt_head
• ✅ ./wt_base 67108864 8
• ✅ gprof ./wt_base gmon.out
• ✅ ./wt_head 67108864 8
• ✅ gprof ./wt_head gmon.out
• ✅ git worktree remove /tmp/pixie-base --force
• ✅ git status --short --branch — working tree remained clean.

View task →