Repository navigation
perf(file-search): probe each backfill page's revisions by primary key - #9445
Merged
Merged
Conversation
The backfill walk checked each file for a revision with a correlated EXISTS in its select list. PostgreSQL plans that as an alternative subplan and picks between a per-row probe and a hashed subplan from the scan's own row estimate, before the page limit applies. Sized against every file left in the walk, the hashed form wins, so every page sequentially scanned and hashed the whole revision table to answer a thousand lookups. The page now left-joins the revision on its primary key, the file id, and reads hasRevision as the joined key being non-null. Under the limit the planner nests that join over the ordered keyset scan, one index probe per file returned. The key is unique, so the join cannot repeat a file, and the keyset predicate, order and limit are unchanged. On a production-sized replica the page went from about 500ms with a full scan of the revision table to about 12ms warm with index probes, with the keyset index scan on workspace_files unchanged. The new integration case walks a second page over a settled table and asserts the revision table is only read through its primary key, probed per file; it fails on the previous query with the hashed sequential scan.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
[Medium impact] The PR appears safe to merge. SummaryReplaces the backfill page’s revision existence check with a primary-key left join.
Reviews (1) · Last reviewed commit: "perf(file-search): probe each backfill p..." · Reviewed by Greptile |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EXISTSin its select list. PostgreSQL plans that as an alternative subplan and chooses the hashed form from the scan's row estimate, before the pageLIMITapplies, so every page sequentially scanned and hashed the wholeworkspace_file_search_revisiontable to answer a thousand lookupsLEFT JOINs the revision on its primary key (file_id = workspace_files.id) and readshasRevisionas the joined key being non-null. Under the limit the planner nests the join over the existing ordered keyset scan: one primary-key probe per file returnedfile_idis the revision's primary key, so the join cannot duplicate a file; the keyset predicate,ORDER BYandLIMITare byte-for-byte unchanged, as are the lock-and-reread of unindexed files, the insert, the owner wakeups and the cursor updateEXPLAINand replicaEXPLAIN (ANALYZE, BUFFERS)of both forms with a real mid-walk cursor: old plan keeps the keyset index scan plus a hashedSubPlanover aSeq Scanof the revision table (~500ms per page); new plan isLimit -> Nested Loop Left Join -> keyset Index Scan + Index Only Scan on workspace_file_search_revision_pkey(~12ms warm), same plan for the first page and a cursor near the end of the walkType of Change
Testing
dispatcher.integration.tscase walks a second backfill page over a settled table (no planner overrides) and asserts the revision table is read only throughworkspace_file_search_revision_pkeywithIndex Cond: (file_id = workspace_files.id); it fails on the previous query with exactly the hashedSeq ScansubplanhasRevisionboth ways and the walk semantics: only files missing a revision are seeded and locked, every live file is walked exactly once across pages, and the cursor seeks the keyset indexlib/workspace-files/search/integration (109 passed) and unit (178 passed) suites against Postgres 17,bun run type-check(apps/sim),bun run lint,bun run check:audits,bun run check:test-types, block-registry checkChecklist
test-auditauthoring gate)