Skip to content

Name the store's own text-search statements in the PostgreSQL log (#9236) - #9240

Open
erikdarlingdata wants to merge 2 commits into
devfrom
fix/9236-escape-withheld
Open

erikdarlingdata wants to merge 2 commits into
devfrom
fix/9236-escape-withheld

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Fixes #9236

PgLogTextRedactor.RedactStoredStatement withholds any statement that carries ESCAPE '\'. PostgreSQL's default LIKE/ILIKE escape character is already the backslash, so the clause is dropped from the statements that run only on PostgreSQL. Results are the same, and the log can now show the statement. The redactor is unchanged.

Sites changed (PostgreSQL only)

  • DarlingMcpFindQueriesByTextTools.cs: six statements, not five (Query Store, plan cache, all-servers, the two span statements, the matched-digests read).
  • SchemaHistoryReader.cs: the text-search select list and filter, and the name prefix / name contains filters.
  • CollectedDatabases.cs: the database-name search.
  • PgAlertHistoryStore.cs: the dedup-key read.
  • PgSettingScrub.cs: the candidate read (three clauses). Its CandidateSql field is now internal so a test can reach it.
  • Comments that described the clause were updated (CollectedDatabases, PgSettingScrub). One existing test, ServerDatabasesEndpointLiveTests, looked for the old spelling and now looks for the new one.

Sites left alone

  • LongRunningQueryExclusions.BuildSqlPredicates: no dialect parameter. Lite's DuckDB wait-stats read and the PostgreSQL reads both call it, and DuckDB has no default escape character.
  • AttentionWindowSql: the same builders serve Lite (DuckDB) and the Darling PostgreSQL reads, with no dialect parameter.
  • DatabaseExclusionFilter: no dialect parameter. It writes for SQL Server and Lite collectors and for the PostgreSQL target provider.
  • Lite's DuckDbAlertHistoryStore (three statements): DuckDB, never in a PostgreSQL log.
  • The /// comment in DarlingPgSessionStatesReader.cs describes BuildSqlPredicates, which is unchanged.

Giving these builders a dialect flag is a separate change. Until then, a PostgreSQL statement built from them is still withheld in the log.

Test

New PgStatementsReadableInTheLogTests passes the statement the product sends through RedactStoredStatement and asserts text comes back: the find_queries_by_text constants and AllServersJoinSql, CollectedDatabases.NamesSearchLimitedSql and PgSettingScrub.CandidateSql. It has 12 cases.

Revert proof: with DarlingMcpFindQueriesByTextTools.cs and CollectedDatabases.cs put back to dev's text, 9 of 12 cases fail (every one that reads those two files). With the fix, 12 of 12 pass.

No behaviour test reaches the SchemaHistoryReader and PgAlertHistoryStore statements, because both are assembled inside a method that needs a database. Their live classes ran and pass.

Gate and classes

  • pre-push.sh: GATE STOP (the selection is the whole suite). Guard lines: [guard-darling] Total: 4980, Failed: 0 and [guard-lite] Total: 1242, Failed: 0. Build has 0 warnings.
  • Run on a PostgreSQL 18.7 rig, all green: *FindQueriesByText*, QueryTextTrigramIndexTrackerTests, QueryStoreTextSeenLiveTests (117 tests); DarlingMcpSchemaToolsLiveTests, CollectedDatabasesSqlTests, ServerDatabasesEndpointLiveTests, *ExcludedDatabases*, NameOrderWithoutCaseLiveTests, *PgSettingScrub*, AlertResolutionHistoryLiveTests, OpenEngineAlertLiveTests, PgStatementTextScrubLiveTests, PgStatementsReadableInTheLogTests (211 tests).
  • After the last merge of origin/dev (no overlap with these files) the project builds with 0 warnings and the new class passes.

Release note: Store log entries for a timed-out or slow find_queries_by_text call, and a few other store reads, now show the statement instead of "statement withheld".

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review November 21, 2026 07:59
@claude

claude Bot commented Nov 21, 2026

Copy link
Copy Markdown

Review notes, all non-blocking:

  1. Stale doc on the shared pattern builder. PerformanceMonitor.Notifications/AlertContext.cs:789 (BuildDedupKeyLikePattern) still says "pair with ESCAPE '\' on the query". After this PR, the PostgreSQL caller (PgAlertHistoryStore.cs:506) relies on PostgreSQL's default escape. Lite's DuckDB caller (DuckDbAlertHistoryStore.cs:191/261/336) still has to write the clause, because DuckDB has no default escape. One sentence there, or in the #1154 comment at PgAlertHistoryStore.cs:500-505, would stop a later "parity" edit from putting the clause back on the PostgreSQL side.
  2. Revert-proof wording. The body says "9 of 12 cases fail (every one that reads those two files)". Two of the cases read DarlingMcpFindQueriesByTextTools.cs and pass on revert: PlanCacheAllServersProbeSql and PlanCacheProbeSql (:511, :547). Neither contains an ILIKE, so they never carried the clause. The count of 9 is right (8 find_queries cases plus NamesSearchLimitedSql); the parenthetical is not. The PgSettingScrub.CandidateSql case was not part of the revert proof.
  3. AllServersJoinSql(0) / (int.MaxValue) repeat two other cases. They resolve to PlanCacheAllServersLookupSpanSql and PlanCacheAllServersSpanSql, which the theory already lists by name (DarlingMcpFindQueriesByTextTools.cs:396-397). That is harmless; it just means the 12 cases cover 10 distinct statements.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — RedactStoredStatement withholds a statement carrying ESCAPE '\' — Darling/Darling.Tests/PgLogEventsPipelineTests.cs:948 asserts null for exactly ... LIKE 'a\_%' ESCAPE '\' ... (an odd backslash run before a quote).
  • VERIFIED — PostgreSQL's default LIKE/ILIKE escape is the backslash, so results are the same — every changed pattern is a bound parameter built by an escaper that writes \ (SchemaHistoryReader.cs:356,380,385, DarlingMcpFindQueriesByTextTools.cs:589-590, CollectedDatabases.SearchPattern, AlertContextSerializer.BuildDedupKeyLikePattern). The one literal, '%\%%' (PgSettingScrub.cs:238), has a backslash before %, not before a quote. Live tests that would fail if the escape were lost: PercentAndUnderscore_MatchOnlyThemselves, TheRoute_Search_MatchesPercentUnderscoreAndBackslashOnlyAsThemselves, FindSchemaObjects_NamePatterns_AreLiteral_NotLikeWildcards.
  • VERIFIED — The redactor is unchanged — PerformanceMonitor.Collectors/PgLogTextRedactor.cs is not in gh pr diff --name-only.
  • VERIFIED — find_queries_by_text: six statements (Query Store, plan cache, all-servers, the two span statements, matched digests) — six hunks at DarlingMcpFindQueriesByTextTools.cs:172,226,275,308,361,499, inside QueryStoreSql, PlanCacheSql, PlanCacheAllServersSql, PlanCacheAllServersSpanSql, PlanCacheAllServersLookupSpanSql and MatchedDigestsSql. No ILIKE remains with the clause in that file.
  • VERIFIED — SchemaHistoryReader: the text-search select list and filter, and the name prefix / name contains filters — SchemaHistoryReader.cs:361-365, 380, 385, 408-411.
  • VERIFIED — CollectedDatabases database-name search — CollectedDatabases.cs:63.
  • VERIFIED — PgAlertHistoryStore dedup-key read — PgAlertHistoryStore.cs:506.
  • VERIFIED — PgSettingScrub: the candidate read (three clauses), and CandidateSql made internal — PgSettingScrub.cs:212, 238.
  • VERIFIED — The comments were updated, and ServerDatabasesEndpointLiveTests now looks for the new spelling — CollectedDatabases.cs:58, PgSettingScrub.cs:187-188, ServerDatabasesEndpointLiveTests.cs:490,497.
  • VERIFIED — The sites left alone are shared with DuckDB or are DuckDB-only, and a PostgreSQL statement built from them is still withheld — AttentionWindowSql.cs:133,145 (used by DarlingAlertReadAdapter.cs:927-990), LongRunningQueryExclusions.cs:324,331,346 (used by DarlingPgSessionStatesReader), DatabaseExclusionFilter.cs:64,122, and DuckDbAlertHistoryStore.cs:191,261,336. None of these builders takes a dialect parameter.
  • VERIFIED — The new test has 12 cases covering the find_queries constants, AllServersJoinSql, NamesSearchLimitedSql and CandidateSql — PgStatementsReadableInTheLogTests.cs:28-39.
  • VERIFIED — No behaviour test reaches the SchemaHistoryReader and PgAlertHistoryStore statements without a database — both build their SQL inside async methods that take a connection (SchemaHistoryReader.cs:349, PgAlertHistoryStore.cs:495).
  • UNVERIFIED — Revert proof: 9 of 12 fail on dev's text, 12 of 12 pass — needs a test run. The count of 9 matches the tree (8 find_queries cases plus NamesSearchLimitedSql had the clause on dev). The note "every one that reads those two files" is imprecise: the two probe cases read that file and pass on revert (see the PR comment).
  • UNVERIFIED — pre-push.sh guard totals (Darling 4980 / Lite 1242, 0 failed) and a build with 0 warnings — needs a build and a test run.
  • UNVERIFIED — The PostgreSQL 18.7 rig runs of the named live classes (117 + 211 tests) are green — needs the rig.

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Storage/PgAlertHistoryStore.cs:506 — the dedup LIKE now depends on the default escape for the \_/\%/\\ that BuildDedupKeyLikePattern writes. Without an escape, the positive match on an underscored key would miss and the Tier-0 cooldown seed would be lost — covered by HoldEarnedByDeliveryTests.TheTier0Seed_StillSeesAMutedUndeliveredRow_WithItsDedupKey (live; the key orders_db is escaped to orders\_db, so the match fails if the escape is lost)
  2. Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingMcpFindQueriesByTextTools.cs:172 — Query Store text match with escaped %/_ — covered by FindQueriesByTextTests.PercentAndUnderscore_MatchOnlyThemselves
  3. Darling/PerformanceMonitor.Darling.Storage/SchemaHistoryReader.cs:380 — lower(o.object_name) LIKE lower($n) with no clause; lower() keeps the backslash — covered by DarlingMcpSchemaToolsLiveTests.FindSchemaObjects_NamePatterns_AreLiteral_NotLikeWildcards

Traps near this diff

  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding, not a nit" — does not apply. The fix is for PostgreSQL's log redactor, which Lite never feeds. The DuckDB twin keeps ESCAPE '\\' correctly at Lite/Services/DuckDbAlertHistoryStore.cs:191,261,336, because DuckDB has no default escape.
  • "A roster-based census does not see the file it does not roster" — does not apply as a defect. The new theory is a roster (PgStatementsReadableInTheLogTests.cs:28-39) that misses SchemaHistoryReader and PgAlertHistoryStore. The PR body names both omissions and says why, and the shared builders still withheld are listed as left alone on purpose.
  • "Ruling (#5833): tests are load-bearing. … never answer a trap with a test that reads source text or asserts how code is spelled" — does not apply. The new test runs the redactor on the statements and asserts its output, which is behaviour. The edit at ServerDatabasesEndpointLiveTests.cs:497 loosens an existing SQL-spelling pin and adds none.
  • "Every MCP filter is part of the query, not a post-filter on the page" — does not apply. The ILIKE stays ahead of ORDER BY/LIMIT in every changed statement (DarlingMcpFindQueriesByTextTools.cs:172-174, CollectedDatabases.cs:63-65).
  • "A diff that touches every line of a file is a line-ending conversion until proven otherwise" — does not apply. Every hunk is a few lines (gh pr diff 9240: +78/-26).

Divergence (blind read vs PR body)

None of substance. The body's account matches the diff on every site. My blind read found the Darling PostgreSQL statements still built with ESCAPE '\' (AttentionWindowSql, LongRunningQueryExclusions, DatabaseExclusionFilter), and the body discloses those as deliberately left alone. One small mismatch: the revert-proof parenthetical "every one that reads those two files" overstates it, because the two probe cases read that file and never carried the clause. There is also a doc the body does not mention: AlertContext.cs:789 still says to pair the pattern with ESCAPE '\'. Both are in the PR comment as non-blocking notes.

Reviewer checklist

  • Run the PostgreSQL live classes on a rig (FindQueriesByTextTests, ServerDatabasesEndpointLiveTests, DarlingMcpSchemaToolsLiveTests, HoldEarnedByDeliveryTests, PgSettingScrubLiveTests) and confirm that metacharacter searches still match only themselves.
  • On a live store, force a find_queries_by_text timeout and confirm the store log shows the statement text instead of "statement withheld".
  • Open a follow-up issue for a dialect flag on AttentionWindowSql / LongRunningQueryExclusions / DatabaseExclusionFilter, so their PostgreSQL statements stop being withheld.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant