fix(storage): fix large-block index truncation and multibyte title panic#10
Merged
Conversation
Index by_content/by_lang entries used a u16 length prefix, silently truncating/wrapping for any block over 64KB. tui.rs's document-list title truncation sliced by byte offset, which panics on multibyte titles whose truncation point lands mid-character. Bumps the on-disk format version since the index byte layout changed.
Adds a hash-join fast path for equi-join ON clauses (documents JOIN blocks ON b.document_id = d.id), avoiding the full left*right cross product the SQL engine previously always materialised — a join over an 80k-document/600k-block store dropped from ~112s to ~3s. Falls back to the existing cross-join for non-equality joins. Also speeds up `mq-db index` on large file sets: the page-file header was rewritten (seek to page 0 + write) on every single page append; it's now deferred to one write per store save. Reading files for a fresh index now happens across a worker-thread pool first, since reading thousands of small files is I/O-latency bound, not CPU bound. Indexing 80k files dropped from ~18s to ~7s.
Adds docs/index.html (a single-page site covering the pipeline, the interval-index pitch, SQL/join examples, features, and install instructions) plus a workflow to deploy docs/ on change. Syncs README's mermaid diagram theming and license badge color to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.