feat(cli): add inspect db collation-drift with report output - #6666
divyasharma95 wants to merge 4 commits into
Conversation
7ttp
left a comment
There was a problem hiding this comment.
could you switch the base branch to develop? :D
(that's where features land, main only gets release merges..)
Introduces a structured report model for inspect output (document -> per-format renderers) and ships collation-drift as its first consumer: drift detection for libc and named ICU collations, severity, and the amcheck -> REINDEX CONCURRENTLY -> REFRESH VERSION workflow rendered with the user's real object names. Existing table commands are untouched; an adapter (reportSpecFromTableSpec) makes their later migration byte-identical in text mode.
FORMAT('%I.%I') escapes for identifier position only; embedding the
result in a string literal let a single quote in an index name terminate
the bt_index_check literal, enabling copy-paste SQL injection with the
operator's privileges. Quote the regclass literal properly (quoteLiteral,
'' doubling). REINDEX and ALTER COLLATION use identifier position and
were already safe.
Develop dropped the legacy- prefixes from the inspect engine's files and exports; adapt imports and rename our own engine, exports, and trace name to match (inspect-report.ts, InspectReportSpec, inspectDbCollationDriftCommand, inspect.db.collation-drift).
1ef1d55 to
859e36c
Compare
| JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS k(attnum, ord) | ||
| ON k.attnum <> 0 | ||
| JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum | ||
| JOIN pg_collation c ON c.oid = a.attcollation |
There was a problem hiding this comment.
🟡 Severity: MEDIUM
The index's effective collation is stored in pg_index.indcollation, not necessarily the table column's pg_attribute.attcollation; for example, CREATE INDEX ... (title COLLATE "stale_icu") overrides it. This join omits such indexes, including UNIQUE ones, so stale ordering can survive and admit duplicate keys.
Helpful? Add 👍 / 👎
💡 Fix Suggestion
Suggestion: The fix requires two related changes:
-
In
icu_affected(lines 92–95): Replace the singleunnest(ix.indkey)with a parallel unnest of bothix.indkeyandix.indcollationso each key column carries its index-level collation OID. Then joinpg_collationonk.colloid(the index collation) instead ofa.attcollation(the table column's collation). This ensures an index created withCREATE INDEX ... (title COLLATE "stale_icu")— whereindcollationdiffers fromattcollation— is correctly caught. -
In
libc_affected(around line 61/66): Apply the same unnest change and update the WHERE predicate froma.attcollation = (SELECT oid FROM default_collation)tok.colloid = (SELECT oid FROM default_collation)so indexes whose index-level collation is the database default (even when the column's declared collation is different) are also detected.
Fix for icu_affected (lines 92–95 — same location as the comment):
JOIN LATERAL unnest(ix.indkey, ix.indcollation) WITH ORDINALITY AS k(attnum, colloid, ord)
ON k.attnum <> 0
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum
JOIN pg_collation c ON c.oid = k.colloidFix for libc_affected (lines 61–66 — separate location, apply manually):
JOIN LATERAL unnest(ix.indkey, ix.indcollation) WITH ORDINALITY AS k(attnum, colloid, ord)
ON k.attnum <> 0
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum
CROSS JOIN db_drift d
WHERE am.amname = 'btree'
AND k.colloid = (SELECT oid FROM default_collation)
⚠️ Experimental Feature: This code suggestion is automatically generated. Please review carefully.
| JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS k(attnum, ord) | |
| ON k.attnum <> 0 | |
| JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum | |
| JOIN pg_collation c ON c.oid = a.attcollation | |
| JOIN LATERAL unnest(ix.indkey, ix.indcollation) WITH ORDINALITY AS k(attnum, colloid, ord) | |
| ON k.attnum <> 0 | |
| JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum | |
| JOIN pg_collation c ON c.oid = k.colloid |
Un-export internals (runInspectReport, wrap, non-imported block interfaces) and remove reportSpecFromTableSpec - the Phase 2 adapter has no callers yet and is specified in the design doc; it returns with the migration that uses it.
Done — retargeted to develop, thanks! |
|
|
||
| /** Single-quotes a value as a SQL string literal, escaping embedded quotes. */ | ||
| export function quoteLiteral(value: string): string { | ||
| return `'${value.replaceAll("'", "''")}'`; |
There was a problem hiding this comment.
🟡 Severity: MEDIUM
An attacker-controlled quoted index name can contain backslashes, but this escaping only doubles apostrophes. If standard_conforming_strings is off, a backslash can escape the generated closing quote in the copy-pasteable bt_index_check statement, allowing injected SQL to execute when an operator runs the remediation output.
Helpful? Add 👍 / 👎
💡 Fix Suggestion
Suggestion: Also escape backslashes before escaping single quotes. When standard_conforming_strings is off, PostgreSQL treats \ as an escape character inside single-quoted literals, so a value like foo\' would have the backslash escape the doubled apostrophe and prematurely close the string. Fix: chain a .replaceAll('\\', '\\\\') call first, so every backslash in the input is doubled before any apostrophe escaping happens.
⚠️ Experimental Feature: This code suggestion is automatically generated. Please review carefully.
| return `'${value.replaceAll("'", "''")}'`; | |
| return `'${value.replaceAll("\\\\", "\\\\\\\\").replaceAll("'", "''")}';` |
Summary
Adds a read-only diagnostic that detects collation version drift — the
silent condition where the OS sorting library (glibc or ICU) has been upgraded
since indexes were built, leaving btree indexes on text columns potentially
mis-ordered. Postgres raises no error for this; queries can quietly return
missing rows, sort incorrectly, or let duplicates past a unique constraint.
Alongside the command, this PR introduces a structured report model for
inspect output (
Reportdocument → per-format renderers), withcollation-driftas its first consumer. The 25 existing table commands areuntouched.
Why
Collation drift is a real operational hazard on any fleet that receives OS
upgrades: PG15+ warns about it (
WARNING: database "postgres" has a collation version mismatch) but offers no tooling to answer which indexes are affectedand what do I run, in what order. The failure mode is silent and the
remediation has a strict ordering constraint (rebuild before refreshing
the recorded version) that is easy to get backwards — refreshing first
removes the warning while leaving the corruption.
What it does
default collation (libc, via
pg_database.datcollversion, PG15+) and namedICU collations (
pg_collation.collversion, PG13+),UNION ALLed — acolumn has exactly one collation, so the branches are disjoint.
(wrong order there can admit duplicates), ⚠ otherwise; keys sort first.
real, schema-qualified object names: amcheck (
heapallindexed => true) →REINDEX INDEX CONCURRENTLY→ALTER DATABASE/COLLATION ... REFRESH VERSION, including the ordering warning.table.
--output jsonemits{ rows, report }: raw driver rows keep the existinginspect payload shape; the structured document rides alongside.
The report model
Reportis a document of typed blocks (keyValue,table,callout,sql,steps) with severities, rendered byrenderReportText(table blocksreuse
renderGlamourTable) or emitted as-is in JSON. Design doc:apps/cli/docs/inspect-report-output.md.The engine (
legacy-inspect-report.ts) deliberately mirrors rather thanmodifies
legacy-inspect-query.ts, so this PR cannot affect shippedcommands.
reportSpecFromTableSpecis included so migrating the existingcommands later is a one-line handler change with byte-identical text output
(design doc, Phase 2).
Testing
Full transcripts:
apps/cli/docs/inspect-collation-drift-test-evidence.md.Highlights:
project; JSON mode verified in both states.
returned silently on the fixture (rows are candidates, not confirmed
corruption); REINDEX alone did not clear detection — Postgres only clears
the mismatch on REFRESH, which is exactly why the ordering warning exists;
after REFRESH (
NOTICE: changing version from 73.2 to 153.121) the reportwent green. Postgres's own reindex-time HINT prescribes character-for-
character the statement this command generates.
null-degradation) and statement generators (qualification, quoting, dedup,
heapallindexed). Fullcheck:allgreen.Known limitations
lower(name)) are not detected — the collation livesin the index expression tree, not
pg_attribute. Follow-up would needpg_get_indexdefparsing.Decisions for reviewers
index on
auth.usersis as damaging as one inpublic. Happy to add thestandard filter if preferred.
{ rows, report }— additive; no existing{ rows }consumer breaks when other commands migrate.
(Phase 2/3 in the design doc) — this PR only establishes it.
criticalseverity eventually drive a non-zero exit code (CIgate)? Out of scope here, flagged for discussion.