perf(plugins): list every schema's tables in one query on Oracle, Dameng, HANA, Trino, CockroachDB and Redshift by datlechin · Pull Request #3708 · TableProApp/TablePro

datlechin · 2026-10-20T16:56:22Z

Summary

Open Quickly, sidebar search, MCP schema search and the backup scope list other schemas' tables through CatalogTableListing. With no all-schema answer from the driver it went straight to one query per schema. It now lists the schemas, drops the system ones, and asks fetchTables(inSchemas:) once. Per schema only when the driver has no set read, or the set read fails for a reason other than a lost connection.
fetchTables(inSchemas:) on Oracle, Dameng, SAP HANA, Trino, CockroachDB and Redshift: the per-schema query widened to IN (...), schema projected on every row. Oracle and Dameng split at 1,000 names (ORA-01795). Redshift still adds external tables per external schema. Trino falls back to per-schema materialized-view reads when the combined one fails.
Naming the schemas keeps Oracle's read off SYS and the other Oracle-maintained schemas, the reason feat(plugins): list every schema's tables in one query on SQL Server and DuckDB #3096 left Oracle on the fallback.
Oracle, PostgreSQL-family and Trino literal and identifier quoting match by code unit. A quote followed by a combining mark (x'́) was left undoubled: PostgreSQL 17 rejects that listing with a syntax error, and in a set read one such schema name would fail every schema.
apple/scripts/probes/build-large-catalog-fixtures.sh builds the Measure the sidebar, Open Quickly and autocomplete on 100 schemas and 50,000 tables #3661 fixtures: PostgreSQL 100 schemas x 500 tables, Oracle the same in a gvenzl/oracle-free container, MySQL one database of 50,000 tables.
New TableProTrinoCoreTests target in TableProCore.
No PluginKit change. Oracle, Dameng, HANA and Trino are registry plugins and need a plugin release to reach users. CockroachDB and Redshift ship in the bundled PostgreSQL plugin.

Server time for the other schemas' tables on the PostgreSQL fixture, PostgreSQL 17.11 local, information_schema query CockroachDB and Redshift use, 3 runs on a loaded machine:

Read	Queries	Time
Per schema	100	2,488 to 3,382 ms
Set read	1	54 to 77 ms

Test plan

CatalogTableListingTests: every schema through the set read (schema list, then one read with the excluded schemas left out); a failed set read falls back per schema and names only the failing schema; a lost connection in the set read fails the listing without per-schema reads.
TableProOracleCoreTests (swift test): 47/47 pass, including the set query, the 1,000-name split, the owner column, the combining-mark escape and SYS qualification of the new query.
TableProTrinoCoreTests (swift test): 3/3 pass.
PostgreSQLDriverTests: Cockroach and Redshift set queries and rows, combining-mark quoting; new builders added to the literal-quoting matrix.
Probe: PostgreSQL fixture built in 234 s; MySQL path built 1,200 tables; Oracle path not run (no Docker here).
Not run locally: the Xcode build, Unit plan, AllPlugins and iOS steps. A disk guard from another project deleted DerivedData's shared module caches every minute while the disk was under 40 GB. Standalone swiftc -typecheck of the PostgreSQL, Oracle, Trino, HANA and Dameng plugin sources and the changed PostgreSQL tests passed. CI covers the rest.

Not in this PR

The UI measurements Measure the sidebar, Open Quickly and autocomplete on 100 schemas and 50,000 tables #3661 asks for (sidebar listing, filter keystroke and expand; Open Quickly open and first result; autocomplete after FROM and schema.; resident memory) and their budgets. They need a quiet machine and the three fixtures.
Snowflake stays per schema: SHOW OBJECTS IN DATABASE caps at 10,000 rows (same reason as feat(plugins): list every schema's tables in one query on SQL Server and DuckDB #3096).
BigQuery (REST list per dataset), SurrealDB (INFO FOR DB per database), R2 SQL (SHOW TABLES per namespace) have no single catalog read. Spanner could take one but needs a live instance for its two dialects.

Docs: TableProApp/docs#67

Refs #3661

…eng, HANA, Trino, CockroachDB and Redshift

# Conflicts: # apple/Packages/TableProOracle/Sources/TableProOracleCore/OracleSchemaQueries.swift

…rmation_schema

J2TeamNNL

Automated review: 7 findings, none a confirmed crash.

J2TeamNNL · 2026-10-21T09:26:13Z

        return merged.sorted { $0.name.localizedStandardCompare($1.name) == .orderedAscending }
    }

+    func fetchTables(inSchemas schemas: [String]) async throws -> [PluginTableInfo]? {


Redshift's set read appends external (Spectrum) tables unsorted, while the single-schema read merges them and re-sorts by name. Consumers relying on name order within a schema see different ordering depending on which path ran.

J2TeamNNL · 2026-10-21T09:26:13Z

+    func fetchTables(inSchemas schemas: [String]) async throws -> [PluginTableInfo]? {
+        guard !schemas.isEmpty else { return [] }
+        let result = try await execute(query: RedshiftTableCatalog.listingQuery(inSchemas: schemas))
+        var tables = result.rows.compactMap { RedshiftTableCatalog.table(fromListingRow: $0.map(\.asText)) }


One svv_external_tables query per external schema, plus an O(n) tables.lazy.filter rescan of the growing list each iteration (quadratic on a 50k-table catalog). Group local names by schema once up front.

J2TeamNNL · 2026-10-21T09:26:13Z

            """
    }

+    static func tables(inSchemas schemas: [String]) -> String {


tables(inSchemas:) builds IN () for an empty list, which does not parse. Only the caller's guard prevents it. Trino and Postgres use a membership(...) helper that emits false for an empty list.

J2TeamNNL · 2026-10-21T09:26:13Z


+    func fetchTables(inSchemas schemas: [String]) async throws -> [PluginTableInfo]? {
+        // DM8 follows Oracle here, so each statement stays under Oracle's 1,000-expression IN limit.
+        let chunkSize = 1_000


Hard-coded chunkSize = 1_000 with each owner bound twice means 2,000 bind parameters per statement, unverified against DM's cap. A failure falls back to per-schema queries, losing the gain on large catalogs. Also duplicates OracleSchemaQueries.maxOwnersPerTableListing.

J2TeamNNL · 2026-10-21T09:26:13Z

+            throw CancellationError()
+        } catch let error as DatabaseError {
+            throw error
+        } catch {


Any non-DatabaseError from the set read (timeout, resource error) falls back to N sequential per-schema reads, multiplying load on an already struggling server on large catalogs.

J2TeamNNL · 2026-10-21T09:26:14Z

    }

+    private func materializedViews(catalog: String, schemas: [String]) async -> Set<TrinoRelationKey> {
+        let sql = TrinoIntrospectionSQL.listMaterializedViews(catalog: catalog, inSchemas: schemas, flavor: flavor)


try? swallows task cancellation, then the per-schema fallback loop runs without Task.checkCancellation(), so work continues after the user cancels.

J2TeamNNL · 2026-10-21T09:26:14Z

+        """
+    }
+
+    public static func table(fromRow row: [String?]) -> PluginTableInfo? {


table(fromRow:) is a near-copy of RedshiftTableCatalog.table(fromListingRow:) and has already drifted (Cockroach uppercases the type before contains("VIEW"), Redshift does not).

J2TeamNNL · 2026-10-21T09:26:14Z

+    }
+
+    func fetchTables(inSchemas schemas: [String]) async throws -> [PluginTableInfo]? {
+        var tables: [PluginTableInfo] = []


Chunks run sequentially and the result is only ordered within each 1,000-owner chunk, not globally by owner/name as the query's ORDER BY implies.

J2TeamNNL · 2026-10-21T09:29:53Z

Review findings from the earlier review, handled in J2TeamNNL@d256453.

Fixed (7):

Redshift set read order: the merged list is now sorted by schema and name, as the single-schema read does.
Redshift per-schema external queries: one svv_external_tables query for all external schemas, and local names are grouped by schema once.
HANA empty list: tables(inSchemas:) emits IN (NULL) for no schemas.
Dameng bind count: 500 owners per statement, so at most 1,000 binds.
Trino cancellation: the materialized view read rethrows cancellation, and the per-schema fallback checks for it.
Cockroach and Redshift row parsing: one implementation. Cockroach calls the Redshift one, which uppercases the type.
Oracle order: the combined result is sorted by owner and name across chunks.

Skipped (1):

CatalogTableListing fallback on a non-DatabaseError: the fallback is the recovery for a set read the engine rejects, and the per-schema loop already fails fast when every schema fails. A timeout cannot be told apart from a rejected set read at that layer.

Not built locally (no Xcode); the edited files parse with swiftc -parse. swiftlint did not run (sourcekit missing).

To take it: git fetch https://github.com/J2TeamNNL/TablePro review-fix/3708 && git cherry-pick d256453916fc8cb728dbea42f20955f62559f40d

datlechin added 2 commits October 20, 2026 23:56

perf(plugins): list every schema's tables in one query on Oracle, Dam…

976fe0f

…eng, HANA, Trino, CockroachDB and Redshift

docs(changelog): note the one-query schema listing

0f0c62b

datlechin mentioned this pull request Oct 20, 2026

docs(open-quickly): list the engines that read other schemas in one query TableProApp/docs#67

Merged

datlechin added 2 commits October 21, 2026 15:21

Merge remote-tracking branch 'origin/main' into perf/3661-large-catalog

7375ad5

# Conflicts: # apple/Packages/TableProOracle/Sources/TableProOracleCore/OracleSchemaQueries.swift

fix(plugins): list a schema set's Presto materialized views from info…

d10f1d0

…rmation_schema

J2TeamNNL reviewed Oct 21, 2026

View reviewed changes

datlechin merged commit 040e52c into main Oct 21, 2026
8 checks passed

datlechin deleted the perf/3661-large-catalog branch October 21, 2026 11:14

Navigation Menu

Repository navigation

perf(plugins): list every schema's tables in one query on Oracle, Dameng, HANA, Trino, CockroachDB and Redshift - #3708

perf(plugins): list every schema's tables in one query on Oracle, Dameng, HANA, Trino, CockroachDB and Redshift#3708
datlechin merged 4 commits into
mainfrom
perf/3661-large-catalog

datlechin commented Oct 20, 2026 •

edited

Loading

Uh oh!

J2TeamNNL left a comment

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

datlechin commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Not in this PR

Uh oh!

J2TeamNNL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

J2TeamNNL commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datlechin commented Oct 20, 2026 •

edited

Loading