Skip to content

fix: scale 4-bit PQ scan lookup tables by the table range - #92

Open
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/pq4-lut-table-range
Open

fix: scale 4-bit PQ scan lookup tables by the table range#92
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/pq4-lut-table-range

Conversation

@jackylee-ch

Copy link
Copy Markdown

scan_4bit_simd builds the u8 quantization factor from qmax, the largest summed distance over the first 200 rows, then applies it to individual table entries. Entries are ~1/m of a sum, so the table collapses into the low 255/m of the u8 range and rows past 200 rank worse. fastscan.rs already scales by the table's own max; this reuses its quantize_distance_table.

Tests:

  • new test_scan_4bit_simd_large fails before this change (row 200 off by 4.04, tolerance 2.80), passes after
  • cargo test --release --workspace (503 passed)
  • cargo fmt --all -- --check
  • cargo +1.98.0 clippy --release --all-targets --workspace -- -D warnings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant