Skip to content

perf(file-search): probe each backfill page's revisions by primary key - #9445

Merged
waleedlatif1 merged 1 commit into
stagingfrom
perf/file-search-backfill-revision-join
Oct 29, 2026
Merged

waleedlatif1 merged 1 commit into
stagingfrom
perf/file-search-backfill-revision-join

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • The file-search backfill walk checked each file for a revision with a correlated EXISTS in its select list. PostgreSQL plans that as an alternative subplan and chooses the hashed form from the scan's row estimate, before the page LIMIT applies, so every page sequentially scanned and hashed the whole workspace_file_search_revision table to answer a thousand lookups
  • The page now LEFT JOINs the revision on its primary key (file_id = workspace_files.id) and reads hasRevision as the joined key being non-null. Under the limit the planner nests the join over the existing ordered keyset scan: one primary-key probe per file returned
  • file_id is the revision's primary key, so the join cannot duplicate a file; the keyset predicate, ORDER BY and LIMIT are byte-for-byte unchanged, as are the lock-and-reread of unindexed files, the insert, the owner wakeups and the cursor update
  • Plain EXPLAIN and replica EXPLAIN (ANALYZE, BUFFERS) of both forms with a real mid-walk cursor: old plan keeps the keyset index scan plus a hashed SubPlan over a Seq Scan of the revision table (~500ms per page); new plan is Limit -> Nested Loop Left Join -> keyset Index Scan + Index Only Scan on workspace_file_search_revision_pkey (~12ms warm), same plan for the first page and a cursor near the end of the walk

Type of Change

  • Other: Performance

Testing

  • New dispatcher.integration.ts case walks a second backfill page over a settled table (no planner overrides) and asserts the revision table is read only through workspace_file_search_revision_pkey with Index Cond: (file_id = workspace_files.id); it fails on the previous query with exactly the hashed Seq Scan subplan
  • Existing cases still cover hasRevision both ways and the walk semantics: only files missing a revision are seeded and locked, every live file is walked exactly once across pages, and the cursor seeks the keyset index
  • lib/workspace-files/search/ integration (109 passed) and unit (178 passed) suites against Postgres 17, bun run type-check (apps/sim), bun run lint, bun run check:audits, bun run check:test-types, block-registry check

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

The backfill walk checked each file for a revision with a correlated EXISTS
in its select list. PostgreSQL plans that as an alternative subplan and picks
between a per-row probe and a hashed subplan from the scan's own row estimate,
before the page limit applies. Sized against every file left in the walk, the
hashed form wins, so every page sequentially scanned and hashed the whole
revision table to answer a thousand lookups.

The page now left-joins the revision on its primary key, the file id, and
reads hasRevision as the joined key being non-null. Under the limit the
planner nests that join over the ordered keyset scan, one index probe per file
returned. The key is unique, so the join cannot repeat a file, and the keyset
predicate, order and limit are unchanged.

On a production-sized replica the page went from about 500ms with a full scan
of the revision table to about 12ms warm with index probes, with the keyset
index scan on workspace_files unchanged.

The new integration case walks a second page over a settled table and asserts
the revision table is only read through its primary key, probed per file; it
fails on the previous query with the hashed sequential scan.
@vercel

vercel Bot commented Oct 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Oct 29, 2026 6:12pm UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

Replaces the backfill page’s revision existence check with a primary-key left join.

  • File-search backfill pages find revisions with primary-key lookups.

Reviews (1) · Last reviewed commit: "perf(file-search): probe each backfill p..." · Reviewed by Greptile

@waleedlatif1
waleedlatif1 merged commit 8948b54 into staging Oct 29, 2026
45 checks passed
@waleedlatif1
waleedlatif1 deleted the perf/file-search-backfill-revision-join branch October 29, 2026 18:16

This branch was previously deployed

1 inactive deployment
Preview — 1b91e9b2 Deployed Oct 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant