Skip to content

feat(cli): add inspect db collation-drift with report output - #6666

Open
divyasharma95 wants to merge 4 commits into
supabase:developfrom
divyasharma95:feat/db-collation-check
Open

divyasharma95 wants to merge 4 commits into
supabase:developfrom
divyasharma95:feat/db-collation-check

Conversation

@divyasharma95

Copy link
Copy Markdown

Summary

Adds a read-only diagnostic that detects collation version drift — the
silent condition where the OS sorting library (glibc or ICU) has been upgraded
since indexes were built, leaving btree indexes on text columns potentially
mis-ordered. Postgres raises no error for this; queries can quietly return
missing rows, sort incorrectly, or let duplicates past a unique constraint.

Alongside the command, this PR introduces a structured report model for
inspect output (Report document → per-format renderers), with
collation-drift as its first consumer. The 25 existing table commands are
untouched.

Why

Collation drift is a real operational hazard on any fleet that receives OS
upgrades: PG15+ warns about it (WARNING: database "postgres" has a collation version mismatch) but offers no tooling to answer which indexes are affected
and what do I run, in what order
. The failure mode is silent and the
remediation has a strict ordering constraint (rebuild before refreshing
the recorded version) that is easy to get backwards — refreshing first
removes the warning while leaving the corruption.

What it does

  • Detects both drift sources in a single read-only query: the database
    default collation (libc, via pg_database.datcollversion, PG15+) and named
    ICU collations (pg_collation.collversion, PG13+), UNION ALLed — a
    column has exactly one collation, so the branches are disjoint.
  • Lists affected btree indexes with severity: ✗ for PRIMARY KEY / UNIQUE
    (wrong order there can admit duplicates), ⚠ otherwise; keys sort first.
  • Renders the full remediation workflow in the output, with the user's
    real, schema-qualified object names: amcheck (heapallindexed => true) →
    REINDEX INDEX CONCURRENTLYALTER DATABASE/COLLATION ... REFRESH VERSION, including the ordering warning.
  • Healthy databases get an explicit ✓ line rather than an ambiguous empty
    table.
  • --output json emits { rows, report }: raw driver rows keep the existing
    inspect payload shape; the structured document rides alongside.

The report model

Report is a document of typed blocks (keyValue, table, callout,
sql, steps) with severities, rendered by renderReportText (table blocks
reuse renderGlamourTable) or emitted as-is in JSON. Design doc:
apps/cli/docs/inspect-report-output.md.

The engine (legacy-inspect-report.ts) deliberately mirrors rather than
modifies legacy-inspect-query.ts, so this PR cannot affect shipped
commands. reportSpecFromTableSpec is included so migrating the existing
commands later is a one-line handler change with byte-identical text output
(design doc, Phase 2).

Testing

Full transcripts: apps/cli/docs/inspect-collation-drift-test-evidence.md.
Highlights:

  • Healthy and drift paths verified against local (PG17) and a remote Supabase
    project; JSON mode verified in both states.
  • Full lifecycle executed using only the SQL the command emitted: amcheck
    returned silently on the fixture (rows are candidates, not confirmed
    corruption); REINDEX alone did not clear detection — Postgres only clears
    the mismatch on REFRESH, which is exactly why the ordering warning exists;
    after REFRESH (NOTICE: changing version from 73.2 to 153.121) the report
    went green. Postgres's own reindex-time HINT prescribes character-for-
    character the statement this command generates.
  • Unit tests cover the report builder (severity, block structure, ordering,
    null-degradation) and statement generators (qualification, quoting, dedup,
    heapallindexed). Full check:all green.

Known limitations

  • Expression indexes (lower(name)) are not detected — the collation lives
    in the index expression tree, not pg_attribute. Follow-up would need
    pg_get_indexdef parsing.
  • libc-branch detection requires PG15+; named-ICU requires PG13+.
  • Rows are candidates until amcheck confirms them.

Decisions for reviewers

  1. Internal schemas are included, unlike sibling commands — a mis-ordered
    index on auth.users is as damaging as one in public. Happy to add the
    standard filter if preferred.
  2. JSON envelope { rows, report } — additive; no existing { rows }
    consumer breaks when other commands migrate.
  3. Report model as the forward direction for the other inspect commands
    (Phase 2/3 in the design doc) — this PR only establishes it.
  4. Should critical severity eventually drive a non-zero exit code (CI
    gate)? Out of scope here, flagged for discussion.

@divyasharma95
divyasharma95 requested a review from a team as a code owner September 17, 2026 15:01
Comment thread apps/cli/src/commands/inspect/db/collation-drift/collation-drift.query.ts Outdated

@7ttp 7ttp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you switch the base branch to develop? :D
(that's where features land, main only gets release merges..)

@divyasharma95
divyasharma95 changed the base branch from main to develop September 17, 2026 21:55
Introduces a structured report model for inspect output (document ->
per-format renderers) and ships collation-drift as its first consumer:
drift detection for libc and named ICU collations, severity, and the
amcheck -> REINDEX CONCURRENTLY -> REFRESH VERSION workflow rendered with
the user's real object names. Existing table commands are untouched; an
adapter (reportSpecFromTableSpec) makes their later migration
byte-identical in text mode.
FORMAT('%I.%I') escapes for identifier position only; embedding the
result in a string literal let a single quote in an index name terminate
the bt_index_check literal, enabling copy-paste SQL injection with the
operator's privileges. Quote the regclass literal properly (quoteLiteral,
'' doubling). REINDEX and ALTER COLLATION use identifier position and
were already safe.
Develop dropped the legacy- prefixes from the inspect engine's files and
exports; adapt imports and rename our own engine, exports, and trace name
to match (inspect-report.ts, InspectReportSpec,
inspectDbCollationDriftCommand, inspect.db.collation-drift).
@divyasharma95
divyasharma95 force-pushed the feat/db-collation-check branch from 1ef1d55 to 859e36c Compare September 17, 2026 22:26
Comment on lines +92 to +95
JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS k(attnum, ord)
ON k.attnum <> 0
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum
JOIN pg_collation c ON c.oid = a.attcollation

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Severity: MEDIUM

The index's effective collation is stored in pg_index.indcollation, not necessarily the table column's pg_attribute.attcollation; for example, CREATE INDEX ... (title COLLATE "stale_icu") overrides it. This join omits such indexes, including UNIQUE ones, so stale ordering can survive and admit duplicate keys.
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: The fix requires two related changes:

  1. In icu_affected (lines 92–95): Replace the single unnest(ix.indkey) with a parallel unnest of both ix.indkey and ix.indcollation so each key column carries its index-level collation OID. Then join pg_collation on k.colloid (the index collation) instead of a.attcollation (the table column's collation). This ensures an index created with CREATE INDEX ... (title COLLATE "stale_icu") — where indcollation differs from attcollation — is correctly caught.

  2. In libc_affected (around line 61/66): Apply the same unnest change and update the WHERE predicate from a.attcollation = (SELECT oid FROM default_collation) to k.colloid = (SELECT oid FROM default_collation) so indexes whose index-level collation is the database default (even when the column's declared collation is different) are also detected.

Fix for icu_affected (lines 92–95 — same location as the comment):

  JOIN LATERAL unnest(ix.indkey, ix.indcollation) WITH ORDINALITY AS k(attnum, colloid, ord)
    ON k.attnum <> 0
  JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum
  JOIN pg_collation c ON c.oid = k.colloid

Fix for libc_affected (lines 61–66 — separate location, apply manually):

  JOIN LATERAL unnest(ix.indkey, ix.indcollation) WITH ORDINALITY AS k(attnum, colloid, ord)
    ON k.attnum <> 0
  JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum
  CROSS JOIN db_drift d
  WHERE am.amname = 'btree'
    AND k.colloid = (SELECT oid FROM default_collation)

⚠️ Experimental Feature: This code suggestion is automatically generated. Please review carefully.

Suggested change
JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS k(attnum, ord)
ON k.attnum <> 0
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum
JOIN pg_collation c ON c.oid = a.attcollation
JOIN LATERAL unnest(ix.indkey, ix.indcollation) WITH ORDINALITY AS k(attnum, colloid, ord)
ON k.attnum <> 0
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum
JOIN pg_collation c ON c.oid = k.colloid

Un-export internals (runInspectReport, wrap, non-imported block
interfaces) and remove reportSpecFromTableSpec - the Phase 2 adapter has
no callers yet and is specified in the design doc; it returns with the
migration that uses it.
@divyasharma95

Copy link
Copy Markdown
Author

could you switch the base branch to develop? :D (that's where features land, main only gets release merges..)

Done — retargeted to develop, thanks!


/** Single-quotes a value as a SQL string literal, escaping embedded quotes. */
export function quoteLiteral(value: string): string {
return `'${value.replaceAll("'", "''")}'`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Severity: MEDIUM

An attacker-controlled quoted index name can contain backslashes, but this escaping only doubles apostrophes. If standard_conforming_strings is off, a backslash can escape the generated closing quote in the copy-pasteable bt_index_check statement, allowing injected SQL to execute when an operator runs the remediation output.
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: Also escape backslashes before escaping single quotes. When standard_conforming_strings is off, PostgreSQL treats \ as an escape character inside single-quoted literals, so a value like foo\' would have the backslash escape the doubled apostrophe and prematurely close the string. Fix: chain a .replaceAll('\\', '\\\\') call first, so every backslash in the input is doubled before any apostrophe escaping happens.

⚠️ Experimental Feature: This code suggestion is automatically generated. Please review carefully.

Suggested change
return `'${value.replaceAll("'", "''")}'`;
return `'${value.replaceAll("\\\\", "\\\\\\\\").replaceAll("'", "''")}';`

@divyasharma95
divyasharma95 marked this pull request as draft September 17, 2026 22:51
@divyasharma95 divyasharma95 self-assigned this Sep 17, 2026
@divyasharma95
divyasharma95 marked this pull request as ready for review September 17, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants