Skip to content

Start a text search with a small newest slice and widen it (#9219) - #9230

Merged
erikdarlingdata merged 4 commits into
devfrom
fix/9219-text-search-first-span
Nov 21, 2026
Merged

erikdarlingdata merged 4 commits into
devfrom
fix/9219-text-search-first-span

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Fixes #9219

Release note: find_queries_by_text now starts a search with the newest hour of the raw tier and widens from there, and an all-servers search for a common pattern tests the newest two hours on their own texts instead of the whole text table, so a large store answers with the newest matches and the span they cover instead of stopping with nothing searched.

Cause, measured

Seeded store on a local PostgreSQL 18.7 and TimescaleDB 2.32 instance: 8 servers, 320,000 texts of about 700 characters, 15 million query_stats rows over 4 days, trigram and sample indexes built, tables analyzed. The first of the 16 equal spans read a sixteenth of the tier (6 hours, about 940,000 rows), and every search paid for it before it could return anything.

Read (same store, warm) Before After
All servers, common pattern (join), 4 s budget 2.6 to 3.6 s 0.5 to 1.0 s
One server, common pattern, plain read 1.9 to 2.1 s 0.4 to 0.65 s
One server, rare pattern, limits 1, 2, 3, default 20 to 60 ms 20 to 60 ms

The store here is 3 to 4 times smaller than the field stores, so the 4 s budget was not crossed. The same store at a 1.5 s budget shows the shape: the all-servers common search stops with nothing searched (searched_from null) in 3 of 3 runs. I could not make the one-server rare shape stop at any limit on this data, so I could not reproduce the limit dependence from the issue. The probe plan did not change between limits 1, 2, 3 and the default here. A cut after any span now keeps its rows, and a smaller first span gives a bad plan less to read.

What changed

  • RawTierSpans now returns spans of 1, 1, 2, 4 and 8 parts and then 12 parts each (a part is 1/96 of the window, an hour of the 4-day tier), the oldest cut short at the floor. The window is 12 spans, down from 16. Spans still tile the window newest first, so a finished search returns the same rows in the same order.
  • DescribeSearched and searched read the span's real floor, so a stopped search names exactly the part it covered. A first slice that cannot finish still says nothing was searched.
  • Lite: no twin. Lite has no find_queries_by_text.

All-servers join: the newest spans test only their own texts (second round)

The all-servers join (a pattern more than 1000 texts contain) matched the pattern against the whole text table in every span, about 29,000 heap blocks on the rig and most of the first span's time. The newest two spans (an hour each of the four-day tier) now take the digests their own samples carry and read each text by primary key, with LIMIT 1 in a lateral subquery so the planner cannot flatten it back into a join with the whole table. EXPLAIN (ANALYZE, BUFFERS) on the rig shows Index Scan using query_text_dim_pkey per digest and no scan of the text table. No temporary table, no carried verdicts: the repeated match is gone from the spans that carry it, and a finished search lists the same rows in the same order.

Rig (same store as above, 320,000 texts, 3.0 M samples over 4 days, warm; the machine was busy, so read the ratios, not the digits). First span (one hour, about 26,000 distinct texts):

Pattern Window join (before) Lookup (after)
every text contains it 2.3 to 2.5 s 0.6 to 0.7 s
about a tenth of the texts contain it 0.3 to 0.6 s 0.5 to 0.7 s

The second row is the cost of the change: a lookup costs more per text than the join's one pass, so a pattern that matches few texts and a span that carries many gets slower, not faster. That is why the lookup is used for the newest two spans only. Every wider span (the four-hour span and up) keeps the window join: the lookup took 1.3 to 2.7 s there for a pattern every text contains (join 2.4 s) and 6 to 8 s for the widest spans (join 0.7 to 5 s), so using it everywhere would have made finished searches slower. The ruling to test only a span's own digests in every span was therefore applied to the spans where it pays and not past them; the join's repeated whole-table match remains in spans 3 and later, and a search that gets that far still pays it. I did not add carried verdicts: past 1,000 digests the carry stops, and a span carries more than that.

Tests

New class FindQueriesByTextFirstSpanLiveTests (13 tests). New class FindQueriesByTextLookupSpanLiveTests (2 tests, second round): the statement the tool runs for the newest span examines 7 text rows on a store of 6,000 texts that all name the pattern (6,007 when the tool runs the window join there, so it fails without the change), and a finished all-servers common search over rows in the lookup spans and the later spans lists the rows of the whole-window join in its order. A budget cannot model the lookup without a sleep, because the cost of a whole-table match grows with the store and a test store is small, so the first test counts text rows read, which does not depend on time; the all-servers budget tests above still cover the cut. A store that cannot read a span wider than two hours is modelled without a sleep: the budget's statement hook waits on an advisory lock that a second connection holds, so the source's own timeout cancels the statement, as it does a statement that cannot finish.

  • All-servers common pattern (join) and rare pattern (probe): stopped, the newest matches returned, searched_from set, complete false, about 4 hours covered.
  • One server, rare pattern, no database, index ready and off, limits 1, 2, 3 and 20: none stops with nothing searched. Limits 1, 2 and 3 answer ok and complete. Limit 20 stops with the four newest matches.
  • The first statement reads a small slice and spans never narrow (also a pure test in FindQueriesByTextTests).
  • A finished search is complete over the whole window. A pattern that is not there answers empty and complete. These two pass on dev as well and are guards, not fixes.
  • Revert proof: with the service file and the three test files I changed put back to dev's, 11 of the 13 fail (all but the two guards).
  • Existing tests that assumed 16 equal spans (FindQueriesByTextAllServersSpansLiveTests, FindQueriesByTextBudgetKeepsRowsLiveTests, FindQueriesByTextTests) now take the span count and first-span width from the tool. Their newest seeds moved from 1 hour to 30 minutes ago so they stay inside the newest hour.
  • Every *FindQueriesByText* class: 102 tests, all green on the rig (Darling.Tests, PostgreSQL 18.7), 0 warnings.

Gate: not run in the second round (the 55-minute wall clock ran out); the first round's gate lines were [guard-darling] Total: 4980, Failed: 0, [guard-lite] Total: 1242, Failed: 0, GATE GREEN. CI decides the rest.

Considered, not added: widening by measured span time (doubling already bounds the waste to one span), a planner setting for small limits (no store showed a limit-dependent plan).

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review November 21, 2026 06:53
CROSS JOIN LATERAL (SELECT 1 AS hit
FROM query_text_dim AS d
WHERE d.digest = f.digest
AND d.query_text ILIKE '%' || $1 || '%' ESCAPE '\'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. This lateral matches on the bare column, d.query_text ILIKE …. The #7674 note earlier in this file says the bare column lets the planner bitmap-scan the trigram index over the whole table. With d.digest = f.digest and LIMIT 1, a primary-key index scan is almost certainly the cheaper plan, so I don't expect that to happen here. But there is a gap in the guard: if the planner ever picked a BitmapAnd of the pkey and the trigram index for each digest, FindQueriesByTextLookupSpanLiveTests.TextRowsRead would not see the cost. It only counts nodes with Relation Name = query_text_dim, and a Bitmap Index Scan node carries Index Name instead. Matching (d.query_text || '') ILIKE here, as PlanCacheSql does, would rule that plan out entirely. The trigram index can't help a single-row lookup anyway, so nothing is lost.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — RawTierSpans returns spans of 1, 1, 2, 4 and 8 parts, then 12 parts each, with the oldest span cut short at the floor; the window is 12 spans, down from 16 — DarlingMcpFindQueriesByTextTools.cs:436-452. I traced the loop by hand on a 96 h + 2 s window. It gives 12 spans covering 1, 2, 4, 8, 16, 28 … 96 h.
  • VERIFIED — the spans still tile the window newest first, so a finished search returns the same rows in the same order — :445 takes the floor on the last span and each span's floor becomes the next span's ceiling. Tested by FindQueriesByTextTests.RawTierSpans_TileTheWindowNewestFirst_HalfOpenAndContiguous.
  • VERIFIED — DescribeSearched and searched read the real floor of the last completed span — :463, :481.
  • VERIFIED — a first slice that cannot finish still says nothing was searched — :481 gives null when completedSpans is 0, and the completedSpans == 0 message arms are at :799 and :990.
  • VERIFIED — Lite has no twin — grep finds find_queries_by_text nowhere under Lite/. Only a doc comment in PerformanceMonitor.Common/Mcp/McpHelpers.cs:926 mentions it.
  • VERIFIED — the newest two spans use a per-digest lookup with LIMIT 1 inside a lateral subquery, and later spans keep the window join — :349-362, :396 (AllServersLookupSpans = 2), :398 and :979.
  • VERIFIED — no temporary table and no carried verdicts on the all-servers path — the lookup statement is a pure CTE read, and BindAllServersSpan is unchanged.
  • VERIFIED — FindQueriesByTextFirstSpanLiveTests has 13 tests — 2 Facts + 8 InlineData + 3 Facts.
  • VERIFIED — the existing tests now take the span count and first-span width from the tool, and their newest seeds moved from 1 hour to 30 minutes ago — see the hunks in FindQueriesByTextAllServersSpansLiveTests.cs, FindQueriesByTextBudgetKeepsRowsLiveTests.cs and FindQueriesByTextTests.cs.
  • UNVERIFIED — the before/after timing tables (2.6–3.6 s → 0.5–1.0 s, and so on) — these are rig measurements; checking them needs the seeded store.
  • UNVERIFIED — EXPLAIN (ANALYZE, BUFFERS) shows query_text_dim_pkey scanned per digest and no scan of the text table — this needs a live plan on the rig.
  • UNVERIFIED — the newest span's statement examines 7 text rows (6,007 with the join) — this needs the live test to run against DARLING_TEST_PG.
  • UNVERIFIED — reverting the change makes 11 of the 13 new tests fail — this needs a run. By reading, the 6-hour first span would block at once in the slow-store model, so the claim is plausible.
  • UNVERIFIED — 102 *FindQueriesByText* tests green and the first round's gate lines — Darling.Tests targets Windows and does not build on this Linux runner (NETSDK1100), and the live tests need Postgres.

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFindQueriesByTextTools.cs:445 — the oldest-span cutoff. An off-by-one here leaves a gap or overlap at the floor, or a sliver span — covered by FindQueriesByTextTests.RawTierSpans_TileTheWindowNewestFirst_HalfOpenAndContiguous and FindQueriesByTextTests.DescribeSearched_EverySpan_IsTheWholeRawTier.
  2. Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFindQueriesByTextTools.cs:362 — the lookup statement must list exactly the rows the window join lists for its span (dedup across spans, NULL database, digest-less samples, the same hash on another server) — covered by FindQueriesByTextLookupSpanLiveTests.AFinishedSearch_ListsWhatTheWholeWindowJoinLists_WithRowsInTheLookupSpansAndTheLaterOnes and FindQueriesByTextAllServersSpansLiveTests.ACommonPattern_JoinsTheWindowInSpans_AndListsWhatTheWholeWindowJoinLists (live, Postgres only).
  3. Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFindQueriesByTextTools.cs:481 — searched_from on a stopped search now indexes the span list, so a wrong index misnames the covered part — covered by FindQueriesByTextFirstSpanLiveTests.AllServers_ACommonPattern_OnAStoreThatCannotReadTheWholeWindow_ReturnsTheNewestMatches_AndSaysWhatItSearched (covered width must be ≤ 5 h; the old equal-span formula gives 18 h) and FindQueriesByTextAllServersSpansLiveTests.ABudgetThatRunsOutAfterTheNewestSpan_KeepsThatSpansRows_AndSaysWhatItSearched.

Traps near this diff

  • "Truncation inferred from a full page: truncated = rows.Count >= limit says "more" when the page is exactly limit long" — does not apply — the spans still ask for limit + 1 - collected.Count (:976), and the flag is rows.Count > limit (:1336), both unchanged.
  • "A test that reads the clock twice measures the CI scheduler" — does not apply — SpanCount() and SlowStore.Budget take their own UtcNow. The span count and widths depend only on the window length (RawRetentionSpan / 96), not on the instant, so a second clock reading cannot change them. The seeds sit 20 minutes or more inside their spans (FindQueriesByTextFirstSpanLiveTests.cs:85-95).
  • "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — the lookup statement is a second form of the span join. Its test compares it with the whole-window reference PlanCacheAllServersSql and also with a literal key list (FindQueriesByTextLookupSpanLiveTests.cs:184-185), so both forms drifting together would still fail.
  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply — Lite has no find_queries_by_text (grep under Lite/ finds nothing). The tool's description text does not name span widths, so it did not need to change.
  • "Ruling (#5833): tests are load-bearing … never answer a trap with a test that reads source text or asserts how code is spelled" — does not apply — the new tests drive the tool, or count rows read from a live EXPLAIN ANALYZE (FindQueriesByTextLookupSpanLiveTests.cs:149-160). None of them reads source text.

Divergence (blind read vs PR body)

None — the body's account matches the diff on every claim graded above. Not a divergence, but one non-blocking note is posted inline at :361. The lookup matches on the bare query_text column, and the rows-read guard in TextRowsRead would not see a trigram Bitmap Index Scan if the planner ever chose one there. Matching on (d.query_text || ''), as PlanCacheSql does, would rule that plan out.

Reviewer checklist

  • Run the *FindQueriesByText* live classes against DARLING_TEST_PG, in particular FindQueriesByTextFirstSpanLiveTests and FindQueriesByTextLookupSpanLiveTests. This runner cannot build Darling.Tests (Windows target) and has no Postgres.
  • Run the full gate (the second round was not gated).
  • On a field-sized store, check the first-span statement's EXPLAIN (ANALYZE, BUFFERS) for the all-servers common pattern: a pkey index scan per digest and no trigram bitmap scan.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — the follow-up adds the lookup-span test's real-clock seeds to the LiveClock source check — the diff is one allowlist entry for FindQueriesByTextLookupSpanLiveTests.cs at Darling/Darling.Tests/LiveClockCensusTests.cs:64. The file exists and has 3 real-clock reads.

Riskiest lines

  1. Darling/Darling.Tests/LiveClockCensusTests.cs:64 — a new allowlist entry that could hide a real clock dependency — covered by UNCOVERED (the entry is a census exemption with no behaviour to fail; its justification text is prose, not checked by a test)
  2. none — the range has no other changes — n/a
  3. none — n/a

Traps near this diff

  • none applied; the change is test-only, and I did not open .github/REVIEW-TRAPS.md for this follow-up — does not apply — the diff touches no product code.

Divergence (blind read vs PR body)

None — I did not read the PR body for this follow-up. The diff is a single test allowlist line and nothing else.

Reviewer checklist

  • Run the Darling test project and confirm LiveClockCensusTests passes and the lookup-span live test is stable at several clock offsets.

@erikdarlingdata
erikdarlingdata merged commit 0cb21d0 into dev Nov 21, 2026
15 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/9219-text-search-first-span branch November 21, 2026 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant