Repository navigation
Start a text search with a small newest slice and widen it (#9219) - #9230
Conversation
| CROSS JOIN LATERAL (SELECT 1 AS hit | ||
| FROM query_text_dim AS d | ||
| WHERE d.digest = f.digest | ||
| AND d.query_text ILIKE '%' || $1 || '%' ESCAPE '\' |
There was a problem hiding this comment.
Non-blocking. This lateral matches on the bare column, d.query_text ILIKE …. The #7674 note earlier in this file says the bare column lets the planner bitmap-scan the trigram index over the whole table. With d.digest = f.digest and LIMIT 1, a primary-key index scan is almost certainly the cheaper plan, so I don't expect that to happen here. But there is a gap in the guard: if the planner ever picked a BitmapAnd of the pkey and the trigram index for each digest, FindQueriesByTextLookupSpanLiveTests.TextRowsRead would not see the cost. It only counts nodes with Relation Name = query_text_dim, and a Bitmap Index Scan node carries Index Name instead. Matching (d.query_text || '') ILIKE here, as PlanCacheSql does, would rule that plan out entirely. The trigram index can't help a single-row lookup anyway, so nothing is lost.
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED —
RawTierSpansreturns spans of 1, 1, 2, 4 and 8 parts, then 12 parts each, with the oldest span cut short at the floor; the window is 12 spans, down from 16 —DarlingMcpFindQueriesByTextTools.cs:436-452. I traced the loop by hand on a 96 h + 2 s window. It gives 12 spans covering 1, 2, 4, 8, 16, 28 … 96 h. - VERIFIED — the spans still tile the window newest first, so a finished search returns the same rows in the same order —
:445takes the floor on the last span and each span's floor becomes the next span's ceiling. Tested byFindQueriesByTextTests.RawTierSpans_TileTheWindowNewestFirst_HalfOpenAndContiguous. - VERIFIED —
DescribeSearchedandsearchedread the real floor of the last completed span —:463,:481. - VERIFIED — a first slice that cannot finish still says nothing was searched —
:481gives null whencompletedSpansis 0, and thecompletedSpans == 0message arms are at:799and:990. - VERIFIED — Lite has no twin — grep finds
find_queries_by_textnowhere underLite/. Only a doc comment inPerformanceMonitor.Common/Mcp/McpHelpers.cs:926mentions it. - VERIFIED — the newest two spans use a per-digest lookup with
LIMIT 1inside a lateral subquery, and later spans keep the window join —:349-362,:396(AllServersLookupSpans = 2),:398and:979. - VERIFIED — no temporary table and no carried verdicts on the all-servers path — the lookup statement is a pure CTE read, and
BindAllServersSpanis unchanged. - VERIFIED —
FindQueriesByTextFirstSpanLiveTestshas 13 tests — 2 Facts + 8 InlineData + 3 Facts. - VERIFIED — the existing tests now take the span count and first-span width from the tool, and their newest seeds moved from 1 hour to 30 minutes ago — see the hunks in
FindQueriesByTextAllServersSpansLiveTests.cs,FindQueriesByTextBudgetKeepsRowsLiveTests.csandFindQueriesByTextTests.cs. - UNVERIFIED — the before/after timing tables (2.6–3.6 s → 0.5–1.0 s, and so on) — these are rig measurements; checking them needs the seeded store.
- UNVERIFIED — EXPLAIN (ANALYZE, BUFFERS) shows
query_text_dim_pkeyscanned per digest and no scan of the text table — this needs a live plan on the rig. - UNVERIFIED — the newest span's statement examines 7 text rows (6,007 with the join) — this needs the live test to run against
DARLING_TEST_PG. - UNVERIFIED — reverting the change makes 11 of the 13 new tests fail — this needs a run. By reading, the 6-hour first span would block at once in the slow-store model, so the claim is plausible.
- UNVERIFIED — 102
*FindQueriesByText*tests green and the first round's gate lines —Darling.Teststargets Windows and does not build on this Linux runner (NETSDK1100), and the live tests need Postgres.
Riskiest lines
Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFindQueriesByTextTools.cs:445— the oldest-span cutoff. An off-by-one here leaves a gap or overlap at the floor, or a sliver span — covered byFindQueriesByTextTests.RawTierSpans_TileTheWindowNewestFirst_HalfOpenAndContiguousandFindQueriesByTextTests.DescribeSearched_EverySpan_IsTheWholeRawTier.Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFindQueriesByTextTools.cs:362— the lookup statement must list exactly the rows the window join lists for its span (dedup across spans, NULL database, digest-less samples, the same hash on another server) — covered byFindQueriesByTextLookupSpanLiveTests.AFinishedSearch_ListsWhatTheWholeWindowJoinLists_WithRowsInTheLookupSpansAndTheLaterOnesandFindQueriesByTextAllServersSpansLiveTests.ACommonPattern_JoinsTheWindowInSpans_AndListsWhatTheWholeWindowJoinLists(live, Postgres only).Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFindQueriesByTextTools.cs:481—searched_fromon a stopped search now indexes the span list, so a wrong index misnames the covered part — covered byFindQueriesByTextFirstSpanLiveTests.AllServers_ACommonPattern_OnAStoreThatCannotReadTheWholeWindow_ReturnsTheNewestMatches_AndSaysWhatItSearched(covered width must be ≤ 5 h; the old equal-span formula gives 18 h) andFindQueriesByTextAllServersSpansLiveTests.ABudgetThatRunsOutAfterTheNewestSpan_KeepsThatSpansRows_AndSaysWhatItSearched.
Traps near this diff
- "Truncation inferred from a full page:
truncated = rows.Count >= limitsays "more" when the page is exactlylimitlong" — does not apply — the spans still ask forlimit + 1 - collected.Count(:976), and the flag isrows.Count > limit(:1336), both unchanged. - "A test that reads the clock twice measures the CI scheduler" — does not apply —
SpanCount()andSlowStore.Budgettake their ownUtcNow. The span count and widths depend only on the window length (RawRetentionSpan / 96), not on the instant, so a second clock reading cannot change them. The seeds sit 20 minutes or more inside their spans (FindQueriesByTextFirstSpanLiveTests.cs:85-95). - "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — the lookup statement is a second form of the span join. Its test compares it with the whole-window reference
PlanCacheAllServersSqland also with a literal key list (FindQueriesByTextLookupSpanLiveTests.cs:184-185), so both forms drifting together would still fail. - "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply — Lite has no
find_queries_by_text(grep underLite/finds nothing). The tool's description text does not name span widths, so it did not need to change. - "Ruling (#5833): tests are load-bearing … never answer a trap with a test that reads source text or asserts how code is spelled" — does not apply — the new tests drive the tool, or count rows read from a live
EXPLAIN ANALYZE(FindQueriesByTextLookupSpanLiveTests.cs:149-160). None of them reads source text.
Divergence (blind read vs PR body)
None — the body's account matches the diff on every claim graded above. Not a divergence, but one non-blocking note is posted inline at :361. The lookup matches on the bare query_text column, and the rows-read guard in TextRowsRead would not see a trigram Bitmap Index Scan if the planner ever chose one there. Matching on (d.query_text || ''), as PlanCacheSql does, would rule that plan out.
Reviewer checklist
- Run the
*FindQueriesByText*live classes againstDARLING_TEST_PG, in particularFindQueriesByTextFirstSpanLiveTestsandFindQueriesByTextLookupSpanLiveTests. This runner cannot buildDarling.Tests(Windows target) and has no Postgres. - Run the full gate (the second round was not gated).
- On a field-sized store, check the first-span statement's
EXPLAIN (ANALYZE, BUFFERS)for the all-servers common pattern: a pkey index scan per digest and no trigram bitmap scan.
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED — the follow-up adds the lookup-span test's real-clock seeds to the LiveClock source check — the diff is one allowlist entry for FindQueriesByTextLookupSpanLiveTests.cs at Darling/Darling.Tests/LiveClockCensusTests.cs:64. The file exists and has 3 real-clock reads.
Riskiest lines
- Darling/Darling.Tests/LiveClockCensusTests.cs:64 — a new allowlist entry that could hide a real clock dependency — covered by UNCOVERED (the entry is a census exemption with no behaviour to fail; its justification text is prose, not checked by a test)
- none — the range has no other changes — n/a
- none — n/a
Traps near this diff
- none applied; the change is test-only, and I did not open
.github/REVIEW-TRAPS.mdfor this follow-up — does not apply — the diff touches no product code.
Divergence (blind read vs PR body)
None — I did not read the PR body for this follow-up. The diff is a single test allowlist line and nothing else.
Reviewer checklist
- Run the Darling test project and confirm LiveClockCensusTests passes and the lookup-span live test is stable at several clock offsets.
Claude posting for Erik Darling.
Fixes #9219
Release note: find_queries_by_text now starts a search with the newest hour of the raw tier and widens from there, and an all-servers search for a common pattern tests the newest two hours on their own texts instead of the whole text table, so a large store answers with the newest matches and the span they cover instead of stopping with nothing searched.
Cause, measured
Seeded store on a local PostgreSQL 18.7 and TimescaleDB 2.32 instance: 8 servers, 320,000 texts of about 700 characters, 15 million query_stats rows over 4 days, trigram and sample indexes built, tables analyzed. The first of the 16 equal spans read a sixteenth of the tier (6 hours, about 940,000 rows), and every search paid for it before it could return anything.
The store here is 3 to 4 times smaller than the field stores, so the 4 s budget was not crossed. The same store at a 1.5 s budget shows the shape: the all-servers common search stops with nothing searched (
searched_fromnull) in 3 of 3 runs. I could not make the one-server rare shape stop at any limit on this data, so I could not reproduce the limit dependence from the issue. The probe plan did not change between limits 1, 2, 3 and the default here. A cut after any span now keeps its rows, and a smaller first span gives a bad plan less to read.What changed
RawTierSpansnow returns spans of 1, 1, 2, 4 and 8 parts and then 12 parts each (a part is 1/96 of the window, an hour of the 4-day tier), the oldest cut short at the floor. The window is 12 spans, down from 16. Spans still tile the window newest first, so a finished search returns the same rows in the same order.DescribeSearchedandsearchedread the span's real floor, so a stopped search names exactly the part it covered. A first slice that cannot finish still says nothing was searched.find_queries_by_text.All-servers join: the newest spans test only their own texts (second round)
The all-servers join (a pattern more than 1000 texts contain) matched the pattern against the whole text table in every span, about 29,000 heap blocks on the rig and most of the first span's time. The newest two spans (an hour each of the four-day tier) now take the digests their own samples carry and read each text by primary key, with
LIMIT 1in a lateral subquery so the planner cannot flatten it back into a join with the whole table. EXPLAIN (ANALYZE, BUFFERS) on the rig showsIndex Scan using query_text_dim_pkeyper digest and no scan of the text table. No temporary table, no carried verdicts: the repeated match is gone from the spans that carry it, and a finished search lists the same rows in the same order.Rig (same store as above, 320,000 texts, 3.0 M samples over 4 days, warm; the machine was busy, so read the ratios, not the digits). First span (one hour, about 26,000 distinct texts):
The second row is the cost of the change: a lookup costs more per text than the join's one pass, so a pattern that matches few texts and a span that carries many gets slower, not faster. That is why the lookup is used for the newest two spans only. Every wider span (the four-hour span and up) keeps the window join: the lookup took 1.3 to 2.7 s there for a pattern every text contains (join 2.4 s) and 6 to 8 s for the widest spans (join 0.7 to 5 s), so using it everywhere would have made finished searches slower. The ruling to test only a span's own digests in every span was therefore applied to the spans where it pays and not past them; the join's repeated whole-table match remains in spans 3 and later, and a search that gets that far still pays it. I did not add carried verdicts: past 1,000 digests the carry stops, and a span carries more than that.
Tests
New class
FindQueriesByTextFirstSpanLiveTests(13 tests). New classFindQueriesByTextLookupSpanLiveTests(2 tests, second round): the statement the tool runs for the newest span examines 7 text rows on a store of 6,000 texts that all name the pattern (6,007 when the tool runs the window join there, so it fails without the change), and a finished all-servers common search over rows in the lookup spans and the later spans lists the rows of the whole-window join in its order. A budget cannot model the lookup without a sleep, because the cost of a whole-table match grows with the store and a test store is small, so the first test counts text rows read, which does not depend on time; the all-servers budget tests above still cover the cut. A store that cannot read a span wider than two hours is modelled without a sleep: the budget's statement hook waits on an advisory lock that a second connection holds, so the source's own timeout cancels the statement, as it does a statement that cannot finish.searched_fromset,completefalse, about 4 hours covered.okand complete. Limit 20 stops with the four newest matches.FindQueriesByTextTests).emptyand complete. These two pass on dev as well and are guards, not fixes.FindQueriesByTextAllServersSpansLiveTests,FindQueriesByTextBudgetKeepsRowsLiveTests,FindQueriesByTextTests) now take the span count and first-span width from the tool. Their newest seeds moved from 1 hour to 30 minutes ago so they stay inside the newest hour.*FindQueriesByText*class: 102 tests, all green on the rig (Darling.Tests, PostgreSQL 18.7), 0 warnings.Gate: not run in the second round (the 55-minute wall clock ran out); the first round's gate lines were
[guard-darling] Total: 4980, Failed: 0,[guard-lite] Total: 1242, Failed: 0,GATE GREEN. CI decides the rest.Considered, not added: widening by measured span time (doubling already bounds the waste to one span), a planner setting for small limits (no store showed a limit-dependent plan).