Skip to content

Plan the store table health reads for the rows the store has (#9224) - #9231

Merged
erikdarlingdata merged 6 commits into
devfrom
fix/9224-table-health-catalog-reads
Nov 21, 2026
Merged

erikdarlingdata merged 6 commits into
devfrom
fix/9224-table-health-catalog-reads

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Fixes #9224

Cause

On a store with thousands of chunks the planner estimated millions of rows for the list of store relations, so every join and sort above it was planned for that many rows. The per-table statement hashed the joined rows into 2048 batches and spilled to temp files for a set that really holds a few thousand rows. Two things made the estimate wrong:

  • The member list ran pg_partition_tree in a lateral join over every table root. A set-returning function is estimated at 1000 rows per call, so the list was about 1000 times the number of roots, even though only partitioned roots have a tree.
  • The chunk list came from the timescaledb_information.chunks view, which is estimated at one row.

The per-index statement also spliced in the collector's estimate over every btree index in the database, and that estimate called pg_relation_size on each index a second time, after the read had already sized them.

Fix

  • The member list now has one arm for non-partitioned roots and one arm that walks only the partitioned roots. Same rows.
  • The chunk list comes from pg_inherits (the hypertable root is the parent, the chunk is the child) joined to pg_class, and skips foreign-table chunks, which are the tiered-storage chunks the view skipped. The statement names no column of the chunk catalog. The compression_settings reads (relid, compress_relid) are the catalog columns still named, and both exist on 2.28.1 and on 2.29 and later. TimescaleDB 2.29 rebuilt its chunk catalog (relid is new, table_name and schema_name are gone), so any join on that catalog fails with 42703 on one side of that version. Core inheritance gives the same chunks on every version, with no version check. A chunk dropped by retention has no relation and no inheritance row, as before.
  • The size lists the reads join back by oid are joined as oid = value::oid, so the planner uses the catalog's statistics.
  • The per-index estimate (PgIndexBloatCollector.BuildQueryText) takes a size expression and a filter. The collector's own text is unchanged. The store read keeps the estimate to the indexes it sized, and takes the estimate's size column from its own list. No size function runs a second time.

Measured on a seeded test store

About 4,000 chunks over 40 hypertables (half compressed), about 29,000 catalog relations and 16,000 indexes. Same sizes passed to old and new statements. Timings are noisy because other work ran on the machine, so each pair below ran back to back.

Statement Before After
Per-table read, execution 6.4 s, 53k buffers, 2397 temp blocks 0.7 s, 17k buffers, 355 temp blocks
Per-index read, execution 4.3 s, 306k buffers, 2342 temp blocks, planned 106 billion rows 0.8 to 1.5 s, 350k buffers, 1100 temp blocks, planned 21 thousand rows
Index relation list, planned rows 4.49 million 10.6 thousand (16.3k returned)

Chunk list on a second store of 3,000 chunks over 20 hypertables and 17,000 catalog relations: the relation list returns 13,560 oids both ways, the old and new text compared with EXCEPT in each direction are both empty, and it plans for about 10,000 rows either way. Execution is 78 to 86 ms with the view and 57 ms with pg_inherits.

Rows: the per-table read returns 44 rows and the per-index read 2101 rows. Both sets compared with EXCEPT in each direction are empty. The two relation lists return the same oids. The per-index read's buffer count did not fall: its time was the second set of size calls, and the remaining time is the per-column pg_stats lookups the estimate does by design.

Not changed: the size statements (SELECT s.rel_oid ..., run on up to four connections) already size each relation once. Their time is the file stat per relation. I did not reproduce a 3.6 s WITH targets statement on the test store as a member listing (it ran in about 0.1 s). The per-table read was the slow one that starts that way.

Tests

  • New DarlingStoreTableHealthCatalogReadsLiveTests has two tests. The first seeds 150 tables with indexes, a hypertable with chunks and a partitioned table, and compares the index relation list's planned rows with the rows it returns. On the old member list it fails with "planned 424196 rows, listed 1095". The second adds a temporary table with 600 indexes the read does not list, hands the per-index read its sizes, and counts the rows the estimate produces with EXPLAIN ANALYZE. The estimate must produce no more rows than there are indexes handed in. With the estimate filter reverted it fails with "1333 rows for 689 indexes handed in", and it passes with the filter. The planned row count of the whole statement did not tell the two apart on a small store, so the test counts produced rows instead.
  • The estimate no longer calls pg_relation_size a second time, and no test can see that: nothing differs but the number of file stats.
  • TimescaleDB 2.28.1 was run for real. The runtime carries its libraries, so I installed its extension script on a throwaway cluster, seeded a store with a compressed hypertable, a dropped chunk, a continuous aggregate and a plain table, and ran both Store Health reads. The new statement returns the same rows as dev's on it (29, 10 and 2 chunks, same sizes), with no error. That catalog has table_name and no relid. The version on dev now (2.32.1) runs through the live classes below.
  • Rows before and after are covered by the existing live classes, which run the whole reader on a store with hypertable chunks, partitions and indexes.
  • Classes run: *DarlingStoreTableHealth* (49), *TableHealth* (49), *IndexBloat* (79) and *StoreHealth* (70), all with 0 failed, plus the earlier *PgStatsLookupRows*, *StoreIndexEstimateReason*, *StoreVacuumStalled* and Lite *PgIndexBloat* runs. No test pins SQL text.
  • The Viewer's Store Health tab reads through the same DarlingStoreTableHealthReader, so it gets the same change. Lite's store health is its own DuckDB read and shares no SQL, so it has no change.

Gate

[guard-darling]    Darling.Tests  Total: 4980, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0
[guard-lite]    Lite.Tests  Total: 1242, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0
classes: darling 168, lite 45
GATE STOP: selection is the whole suite (CI's test map answers FULL)

Pushed on green Guard lines and the classes above. CI decides the rest.

Considered, not added

  • Deduplicating the pg_stats lookups for index columns on the same table column (a chunk's two indexes both look up ts). It would roughly halve the remaining estimate time but changes the estimate's shape, and no field number says it matters.

Release note: get_store_table_health and the Viewer's Store Health tab read faster on a store with thousands of chunks.

…alog (#9224)

The chunk catalog's relid column exists only from TimescaleDB 2.29, so the read failed with 42703 on older stores. Core inheritance names the same chunks on every version. The per-index estimate test now counts the rows the estimate produces.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review November 21, 2026 07:06
JOIN pg_catalog.pg_class AS pc ON pc.relnamespace = pn.oid AND pc.relname = ch.chunk_name
/* Chunk membership from core inheritance, not timescaledb_information.chunks: that view joins every chunk to its dimension
slices and to all of pg_class, and its row count is estimated far too low, which let the planner pick the wrong join order
(#9224). No _timescaledb_catalog column is named: TimescaleDB 2.29 rebuilt the chunk catalog, so a column of it exists on

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is wrong as written, and so is the matching PR-body sentence ("The statement names no TimescaleDB catalog column"). Four lines up (L557–558) this same CTE reads _timescaledb_catalog.compression_settings and names cs.relid and cs.compress_relid. TimescaleMemberArms names them again at L576–577. What is true is narrower: no column of the chunk catalog (_timescaledb_catalog.chunk) is named, and that is the catalog 2.29 rebuilt.

This matters because the comment is the record of the version-compatibility guarantee. If a future TimescaleDB reshapes compression_settings the way 2.29 reshaped chunk, the next reader will believe this statement is immune. Suggested wording: "No column of _timescaledb_catalog.chunk is named (2.29 rebuilt that catalog). The compression_settings.relid / compress_relid reads below predate this change and are the remaining catalog dependency." Please make the same correction in the PR body.

@claude

claude Bot commented Nov 21, 2026

Copy link
Copy Markdown

Non-blocking note, a narrow race in the presized per-index read. Before this PR, the est CTE estimated every btree index in the database. Now it estimates only the oids in $1 (EstimateIndexFilterSql, DarlingStoreTableHealthReader.cs:953). Suppose an index is created on a store table after IndexRelationsSql runs but before BuildIndexSql runs. Its row now has no estimate and reads "no figure returned" (NoEstimateRowReason). Before, it got an estimate, with a basis of 0. Either way the size read already counts such an index as empty (the ReadSizesAsync doc says so), so this is consistent with the documented behaviour. I'm noting it only so the change in that reason is a known effect, not a surprise.

claude[bot]
claude Bot previously requested changes Nov 21, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CHANGES REQUESTED

Claims

  • VERIFIED — The member list has one arm for non-partitioned roots and one that walks only partitioned roots; same rows — DarlingStoreTableHealthReader.cs:616-626. The relkind <> 'p' arm keeps kind='table', grouped=false. The 'p' arm keeps 'partitioned', grouped=true, leaves only. This matches the old LATERAL UNION.
  • VERIFIED — Chunk list comes from pg_inherits joined to pg_class, skipping foreign-table chunks — DarlingStoreTableHealthReader.cs:566-568 (inh.inhparent = t.rel_oid, pc.relkind <> 'f'). Continuous-aggregate materializations are still walked because root_rels (L547-551) carries the continuous_aggregates arm.
  • VERIFIED — The size lists are joined as oid = value::oid — DarlingStoreTableHealthReader.cs:709, :995, :996.
  • VERIFIED — PgIndexBloatCollector.BuildQueryText takes a size expression and a filter, and the collector's own text is unchanged — PgIndexBloatCollector.cs:181 passes pg_catalog.pg_relation_size(c.oid) and string.Empty into the same positions (L204, L220). The only other braces in the interpolated body are those two placeholders. const→static readonly has no const-context callers (PgIndexBloatCollector.cs:385, PgStatsLookupRowsLiveTests.cs:160-161, DarlingStoreTableHealthReader.cs:959).
  • VERIFIED — The store read keeps the estimate to the indexes it sized, takes the size column from its own list, and runs no second size function — DarlingStoreTableHealthReader.cs:953-959 (0::bigint plus c.oid::bigint = ANY($1)) and :985 (est_index_bytes from zi.bytes). The caller passes IndexRelationsSql oids as $1 (:1786-1789).
  • VERIFIED — The Viewer's Store Health tab reads through the same reader — PerformanceMonitor.Darling.Viewer/StoreHealthTab.xaml.cs references DarlingStoreTableHealthReader.
  • VERIFIED — Lite's store health shares no SQL — no file under Lite/ references TableHealth or this reader.
  • VERIFIED — No new test pins SQL text — both tests in DarlingStoreTableHealthCatalogReadsLiveTests.cs assert row counts from execution and EXPLAIN plans, not source text.
  • UNVERIFIED — The measurement table (6.4 s→0.7 s, 106 billion→21 thousand planned rows, 4.49M→10.6k) — needs a live seeded store.
  • UNVERIFIED — On a 3,000-chunk store the old and new relation lists are equal under two-way EXCEPT — needs a live store.
  • UNVERIFIED — Test 1 fails on the old member list ("planned 424196 rows, listed 1095") — needs a live run against the reverted SQL.
  • UNVERIFIED — Test 2 fails with the filter reverted ("1333 rows for 689 indexes") — needs a live run. The mechanism is sound: the temp table's indexes are btree and are not excluded by the estimate's nspname filter, while plain_roots drops relpersistence = 't' (:611).
  • UNVERIFIED — The statement returns dev's rows on TimescaleDB 2.28.1 — needs a 2.28.1 cluster.
  • UNVERIFIED — Gate totals and the classes-run counts — needs CI or a local run.
  • UNVERIFIED — The tiered-storage (OSM) chunk is a foreign table that timescaledb_information.chunks skipped — needs the view definition on an OSM-enabled install.
  • REFUTED — "The statement names no TimescaleDB catalog column" (body), echoed in the code comment at DarlingStoreTableHealthReader.cs:561 — the same CTE reads _timescaledb_catalog.compression_settings and names cs.relid and cs.compress_relid (:557-558), as do TimescaleMemberArms at :576-577. Only the chunk catalog is no longer named. The version-compatibility guarantee the comment records is narrower than it says. See the inline comment.

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Storage/DarlingStoreTableHealthReader.cs:567 — chunk membership now comes from pg_inherits. A wrong parent (or a compressed-chunk table counted here as well as in the compressed arm) changes ChunkCount and sizes — covered by DarlingStoreTableHealthLiveTests.HypertableChunks_RollUpToOneRow_WithTheChunkCount (asserts ChunkCount == 3) and ManySmallChunks_AndAFewLargeOnes_ReadTheSameAsTheCollectorAndTheSizeFunctions (compressed chunks, sizes against the size functions)
  2. Darling/PerformanceMonitor.Darling.Storage/DarlingStoreTableHealthReader.cs:954 — the estimate filter. Too wide restores the old cost; too narrow silently drops estimates — covered by DarlingStoreTableHealthCatalogReadsLiveTests.ThePerIndexEstimate_IsKeptToTheIndexesTheReadWasHandedIn (too wide) and DarlingStoreTableHealthLiveTests.AnEstimatedIndexRow_ReadsABasisEqualToItsSize_ForOneIndexAndForARolledUpOne (too narrow: EstimateIndexesUsable 1 and 2)
  3. Darling/PerformanceMonitor.Darling.Storage/DarlingStoreTableHealthReader.cs:985 — est_index_bytes now comes from zi.bytes, not the estimate's own size. A wrong join or the 0::bigint leaking through would zero the basis — covered by DarlingStoreTableHealthLiveTests.AnEstimatedIndexRow_ReadsABasisEqualToItsSize_ForOneIndexAndForARolledUpOne (EstimatedBasisBytes == Bytes, non-null)

Traps near this diff

  • "chunk_compression_stats returns ZERO rows for a hypertable whose compression is off, so a chunk count over it reads 0 (#3610 …). Count chunks from timescaledb_information.chunks." — does not apply — the PR moves off the chunks view, but onto pg_inherits (DarlingStoreTableHealthReader.cs:567), which lists every chunk whether or not it is compressed. It does not use chunk_compression_stats.
  • "timescaledb_information.hypertables never lists a continuous aggregate's materialization … Enumerate continuous_aggregates …" — does not apply — targets still unions continuous_aggregates (:543-545), and chunk_rels walks root_rels (:566), which includes the materialization.
  • "pg_relation_size is the heap alone; TOAST and indexes live elsewhere …" — does not apply — zi.bytes (:985) is pg_relation_size of the index itself (IndexSizesSql, :1516-1518), the same function the estimate used before.
  • "Lite and Darling ship in parity (CONTRIBUTING's two-store rule) …" — does not apply — Lite has no PostgreSQL store-health reader (no TableHealth reference under Lite/). The shared PgIndexBloatCollector.QueryText is byte-identical (PgIndexBloatCollector.cs:181).
  • "Ruling (#5833): tests are load-bearing. … never answer a trap with a test that reads source text or asserts how code is spelled …" — does not apply — the new tests assert executed and planned row counts (DarlingStoreTableHealthCatalogReadsLiveTests.cs:116,174), not SQL text.
  • "InternalsVisibleTo on PerformanceMonitor.Analysis names Lite.Tests only — a helper Darling.Tests must call has to be public (#3688)." — does not apply — the newly internal IndexRelationsSql is in Darling.Storage, whose csproj names Darling.Tests (PerformanceMonitor.Darling.Storage.csproj:45).

Divergence (blind read vs PR body)

  • The body (and the code comment at :561) says the statement names no TimescaleDB catalog column. It still reads _timescaledb_catalog.compression_settings.relid and compress_relid (:557-558, :576-577). The accurate claim is "no chunk-catalog column".
  • The body does not mention one behaviour change from the estimate filter: an index created between the oid list read and the per-index statement used to get an estimate (basis 0) and now reads NoEstimateRowReason. This is minor and consistent with the documented "created after this read counts as empty" (:1539). See the top-level comment.

Reviewer checklist

  • Run DarlingStoreTableHealthCatalogReadsLiveTests against the reverted member list and the reverted filter, to confirm both fail as the body says.
  • On a TimescaleDB store with tiered storage (OSM), confirm the old chunks view did skip the foreign-table chunk, so relkind <> 'f' keeps the rows unchanged.
  • Re-run the store-health reads on 2.28.x and on the current version (2.32.1) and compare rows against dev.

The chunk list comment said no catalog column was named. The compression_settings reads in that statement and in TimescaleMemberArms name relid and compress_relid, so the comment now says those columns are still named and that no column of the chunk catalog is. Comment only.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — The comment now says the compression_settings reads (relid, compress_relid) are the catalog columns still named, here and in TimescaleMemberArms — DarlingStoreTableHealthReader.cs:557 (EXISTS on cs.relid / cs.compress_relid) and :578 (join on cs.relid in TimescaleMemberArms).
  • UNVERIFIED — Those columns exist on TimescaleDB 2.28.1 and 2.29+ — needs a live read against both versions.

Riskiest lines

  1. Darling/PerformanceMonitor.Darling.Storage/DarlingStoreTableHealthReader.cs:561 — the only changed lines are inside a SQL block comment, so there is no behaviour change — UNCOVERED (nothing to cover)
  2. DarlingStoreTableHealthReader.cs:562 — the comment text sits inside a C# verbatim string, so a stray " would break compilation; the new text has none — UNCOVERED (a build would catch it)
  3. DarlingStoreTableHealthReader.cs:564 — the comment closes with */ and no nested /* was added — UNCOVERED (nothing to cover)

Traps near this diff

  • None applied: the range changes comment text only, so I did not check it against individual traps in .github/REVIEW-TRAPS.md.

Divergence (blind read vs PR body)

None. The follow-up range is a comment-only edit, and it matches the code at :557 and :578. I did not re-grade the PR body's claims about earlier commits.

Reviewer checklist

  • Confirm that _timescaledb_catalog.compression_settings(relid, compress_relid) exists on both 2.28.1 and 2.29+, since the comment asserts it.

@github-actions
github-actions Bot dismissed claude[bot]’s stale review November 21, 2026 07:18

Superseded by claude[bot]'s LGTM on 9925e7e: #9231 (review)

@erikdarlingdata
erikdarlingdata merged commit 90021fa into dev Nov 21, 2026
32 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/9224-table-health-catalog-reads branch November 21, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant