feat: add global system tables framework under `sys` database by suxiaogang223 · Pull Request #366 · alibaba/paimon-cpp

suxiaogang223 · 2026-06-13T16:45:47Z

Summary

Add global system table support under the sys database, enabling catalog-level metadata queries.

Closes part of #141.

Changes

Infrastructure

Add GetOptions() to Catalog interface for catalog-level config access
FileSystemCatalog stores and exposes catalog options
New GlobalSystemTableLoader with independent registry for sys tables
Extend SystemTablePath with is_global flag and TryParsePath to detect sys/ paths
FileSystemCatalog handles sys database in ListTables/DatabaseExists/TableExists/LoadTableSchema

Global System Tables

Table	Status	Columns
`sys.catalog_options`	✅ Complete	`key`, `value`
`sys.all_table_options`	✅ Complete	`database_name`, `table_name`, `key`, `value`
`sys.tables`	✅ Complete	`database_name`, `table_name`, `table_type`, `partitioned`, `primary_key`, `record_count`, `file_size_in_bytes`, `file_count`, `last_file_creation_time`
`sys.partitions`	✅ Complete	`database_name`, `table_name`, `partition_name`, `record_count`, `file_size_in_bytes`, `file_count`, `last_update_time`

Integration Tests (4 new tests in SystemTableReadInteTest)

TestReadGlobalCatalogOptions — verifies schema + content
TestReadGlobalAllTableOptions — verifies table options across databases
TestReadGlobalTables — verifies schema, table metadata
TestReadGlobalPartitions — verifies empty result for unpartitioned tables

Files Changed

 include/paimon/catalog/catalog.h                   |   5 +
 src/paimon/CMakeLists.txt                          |   1 +
 src/paimon/core/catalog/catalog.cpp                |   2 +-
 src/paimon/core/catalog/file_system_catalog.cpp    |  39 +-
 src/paimon/core/catalog/file_system_catalog.h      |   6 +-
 .../core/table/system/global_system_tables.cpp     | 370 ++++++++++++++
 .../core/table/system/global_system_tables.h       | 117 +++++
 src/paimon/core/table/system/system_table.cpp      |  28 +-
 src/paimon/core/table/system/system_table.h        |   2 +
 .../core/catalog/file_system_catalog_test.cpp      |  14 +-
 test/inte/read_inte_test.cpp                       | 237 +++++++++
 11 files changed, 810 insertions(+), 10 deletions(-)

Verification

Fedora x86_64, GCC 16:

Build: ✅
paimon-core-test: 36/40 pass (4 failures pre-existing)
paimon-read-inte-test (global tables): 4/4 ✅
paimon-read-inte-test (system tables): same as before

🤖 Generated with Claude Code

- Add GetOptions() to Catalog interface for catalog-level config access - FileSystemCatalog stores and exposes catalog_options - New GlobalSystemTableLoader with independent registry for sys tables - Implement sys.catalog_options, sys.all_table_options, sys.tables - Stub for sys.partitions (manifest aggregation to follow) - Extend SystemTablePath with is_global flag - TryParsePath detects sys/ paths for TableScan/TableRead routing - FileSystemCatalog handles sys database in ListTables, DatabaseExists, TableExists, LoadTableSchema Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ructor

…dent fields

Replace the TODO stubs in TablesSystemTable::BuildRows() with actual manifest entry aggregation. The new AggregateFileStats() helper reads the latest snapshot data files and computes record_count, file_size, file_count, and last_file_creation_time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Replace the stub with actual partition-level file statistics using the AggregateFileStats helper. For each partitioned table, read manifest entries and emit one row per partition with record_count, file_size, file_count, and last_update_time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add 4 tests to SystemTableReadInteTest: - TestReadGlobalCatalogOptions: verifies sys.catalog_options schema and content - TestReadGlobalAllTableOptions: verifies sys.all_table_options with table options - TestReadGlobalTables: verifies sys.tables schema, table_type, partitioned, pk - TestReadGlobalPartitions: verifies sys.partitions returns empty for unpartitioned Tests use a ReadGlobalSystemTable helper that creates the GlobalSystemTableContext with a proper Catalog pointer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

zjw1111

Thanks a lot for building out the global sys system tables — the loader/registry split and reuse of InMemorySystemTable make the design clean and easy to follow.

I left a few inline comments. The most important one is a potential crash on the normal engine read path for sys.tables / sys.partitions / sys.all_table_options (see the comment in system_table.cpp). A couple of smaller notes are below:

global_system_tables.cpp (CatalogOptionsSystemTable::BuildRows, around line 264): this table stores fields via a non-owning std::string_view into context_.catalog_options, while the other tables copy through BinaryString (StringValue). It's safe today because the map is a stable member, but for consistency and to avoid a future dangling-view footgun, would it be possible to use StringValue(...) here too?
file_system_catalog.cpp (line 282): the const_cast<FileSystemCatalog*>(this) looks unnecessary since only const Catalog methods are used later — declaring GlobalSystemTableContext::catalog as const Catalog* would let you drop it. Minor, feel free to skip.

Thanks again!

zjw1111 · 2026-07-09T05:12:58Z

+        // The warehouse is the grandparent of the sys/<table> path
+        context.warehouse = PathUtil::GetParentDirPath(PathUtil::GetParentDirPath(path));
+        context.catalog_options = dynamic_options;
+        // Note: context.catalog is intentionally left as nullptr here.


I think this path can crash for the catalog-dependent tables. The normal engine read path (TableScan::NewScan / TableRead::NewRead) resolves system tables through LoadFromPath, which lands here and never sets context.catalog. Since GlobalSystemTableContext::catalog has no in-class initializer, context.catalog is actually indeterminate here rather than nullptr as the comment says. Later, BuildRows() for sys.tables / sys.partitions / sys.all_table_options calls context_.catalog->ListDatabases(), dereferencing an uninitialized pointer -> UB/segfault. Only sys.catalog_options works on this path.

The integration tests don't catch this because ReadGlobalSystemTable calls GlobalSystemTableLoader::Load directly with a real catalog, bypassing LoadFromPath entirely.

Could you guard this explicitly, e.g. return a Status::NotImplemented(...) for tables that require catalog enumeration when context.catalog == nullptr, and add a test that exercises the path-based (no-catalog) read? That would turn the crash into a controlled error.

zjw1111 · 2026-07-09T05:12:58Z

+/// Context passed to global system table constructors, providing catalog-level
+/// access for enumerating databases, tables, and reading metadata.
+struct GlobalSystemTableContext {
+    Catalog* catalog;  // non-owning pointer


Would it be possible to add a default initializer here, i.e. Catalog* catalog = nullptr;? Right now GlobalSystemTableContext context; in LoadFromPath leaves this member indeterminate rather than null, which makes the null-check suggested in system_table.cpp reliable and matches the intent of the comment there.

zjw1111 · 2026-07-09T05:12:58Z

+    std::string parent_name = PathUtil::GetName(parent);
+
+    // Detect global system table paths: <warehouse>/sys/<table_name>
+    if (parent_name == "sys") {


Minor: this hardcodes the literal "sys", but the repo already exposes Catalog::SYSTEM_DATABASE_NAME (used by FileSystemCatalog::IsSystemDatabase). Could you reuse that constant here to keep the system-database name in one place?

zjw1111

One additional inline note on the sys.tables primary_key column — a schema/data-contract nit. Thanks!

zjw1111 · 2026-07-09T05:27:26Z

+        arrow::field("table_name", arrow::utf8(), /*nullable=*/false),
+        arrow::field("table_type", arrow::utf8(), /*nullable=*/false),
+        arrow::field("partitioned", arrow::boolean(), /*nullable=*/false),
+        arrow::field("primary_key", arrow::utf8(), /*nullable=*/false),


One more small consistency note, thanks for bearing with me.

This field is declared non-nullable, but BuildRows writes NullType() for tables without a primary key:

row.SetField(4, primary_keys_str.empty() ? VariantType(NullType()) : VariantType(StringValue(primary_keys_str)));

So a no-PK table produces a null in a column the schema promises is non-null — a schema/data-contract mismatch. TestReadGlobalTables only covers a PK table, so this path isn't exercised. Would it be possible to either mark this field nullable=true (closer to how Java Paimon expresses it), or emit an empty string for no-PK tables? A no-PK test case would help lock this down too.

lxy-9602 · 2026-07-09T12:48:49Z

-    FileSystemCatalog(const std::shared_ptr<FileSystem>& fs, const std::string& warehouse);
+    FileSystemCatalog(const std::shared_ptr<FileSystem>& fs, const std::string& warehouse,
+                      const std::map<std::string, std::string>& catalog_options = {});



Could you please avoid default parameters in production code?

lxy-9602 · 2026-07-09T12:50:10Z

+    ASSERT_TRUE(std::find(sys_tables.begin(), sys_tables.end(), "all_table_options") !=
+                sys_tables.end());
+    ASSERT_TRUE(std::find(sys_tables.begin(), sys_tables.end(), "tables") != sys_tables.end());
+    ASSERT_TRUE(std::find(sys_tables.begin(), sys_tables.end(), "partitions") != sys_tables.end());


Please rename the test from TestInvalidList to something like TestSystemList.

lxy-9602 · 2026-07-09T12:57:09Z

+    ASSERT_FALSE(sys_tables.empty());
+    // Verify expected global system table names are present
+    ASSERT_TRUE(std::find(sys_tables.begin(), sys_tables.end(), "catalog_options") !=
+                sys_tables.end());


Java gates sys.catalog_options behind catalog-options-table.enabled and keeps it disabled by default, since it may expose catalog-level configuration. Should we mirror that behavior here instead of always listing/loading catalog_options under sys?

lxy-9602 · 2026-07-09T13:04:36Z

+        arrow::field("file_count", arrow::int64(), /*nullable=*/true),
+        arrow::field("last_file_creation_time", arrow::timestamp(arrow::TimeUnit::MILLI),
+                     /*nullable=*/true),
+    });


I noticed that the current C++ sys.tables schema and semantics differ from Java Paimon's sys.tables contract.

In Java, AllTablesTable exposes 14 columns:

database_name, table_name, table_type, partitioned, primary_key,
owner, created_at, created_by, updated_at, updated_by,
record_count, file_size_in_bytes, file_count, last_file_creation_time

In this PR, the C++ implementation exposes only 9 columns:

database_name, table_name, table_type, partitioned, primary_key,
record_count, file_size_in_bytes, file_count, last_file_creation_time

There are also a few semantic/type differences:

primary_key: Java uses a boolean to indicate whether the table has primary keys, while C++ currently returns a string containing the primary key column names, or null when there is no primary key.

table_type: Java reads this from the table option CoreOptions.TYPE with its default value, while C++ derives MANAGED / EXTERNAL from whether data-file.external-paths is set.

last_file_creation_time: Java exposes this as a BIGINT epoch millis value, while C++ exposes it as an Arrow timestamp(ms).

Java includes audit fields (owner, created_at, created_by, updated_at, updated_by), but C++ currently omits them.

Could we align the C++ schema and behavior with Java here? Otherwise clients that rely on the Java sys.tables contract may see different columns, different types, and different values when using the C++ reader.

lxy-9602 · 2026-07-09T13:05:31Z

+        arrow::field("file_count", arrow::int64(), /*nullable=*/true),
+        arrow::field("last_update_time", arrow::timestamp(arrow::TimeUnit::MILLI),
+                     /*nullable=*/true),
+    });


Could we align sys.partitions with Java's AllPartitionsTable contract?

Java exposes:
database_name, table_name, partition_name, record_count, file_size_in_bytes, file_count, last_file_creation_time, done

This PR exposes:
database_name, table_name, partition_name, record_count, file_size_in_bytes, file_count, last_update_time

So C++ is missing done, uses a different time column name/type, and formats partition names differently. Java uses PartitionUtils.buildPartitionName(spec), e.g. dt=2024-07-22/region=cn, while C++ currently returns values like {dt=2024-07-22,region=cn}.

Could we align this table with Java's schema and semantics?

suxiaogang223 mentioned this pull request Jun 13, 2026

fix: isolate external projects from host toolchain library paths #367

Open

suxiaogang223 and others added 6 commits July 2, 2026 16:44

fix: remove unused helper functions in global system tables

40c237f

fix: replace ValueUnsafe() with value() in global system tables

af38088

fix: add default value for catalog_options in FileSystemCatalog const…

633ce7b

…ructor

fix: update TestInvalidList to expect global system table names

6514d09

improve: add table_type detection and clearer TODO for manifest-depen…

92de892

…dent fields

suxiaogang223 force-pushed the codex/global-system-tables branch from e4597fa to 92de892 Compare July 2, 2026 08:45

suxiaogang223 and others added 2 commits July 2, 2026 16:58

suxiaogang223 marked this pull request as ready for review July 2, 2026 12:29

suxiaogang223 force-pushed the codex/global-system-tables branch from 55828f8 to c5d3439 Compare July 2, 2026 13:00

style: apply clang-format fixes from pre-commit

ed5857a

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

zjw1111 reviewed Jul 9, 2026

View reviewed changes

lxy-9602 reviewed Jul 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add global system tables framework under `sys` database#366

feat: add global system tables framework under `sys` database#366
suxiaogang223 wants to merge 10 commits into
alibaba:mainfrom
suxiaogang223:codex/global-system-tables

suxiaogang223 commented Jun 13, 2026 •

edited

Loading

Uh oh!

zjw1111 left a comment

Uh oh!

zjw1111 Jul 9, 2026

Uh oh!

zjw1111 Jul 9, 2026

Uh oh!

zjw1111 Jul 9, 2026

Uh oh!

zjw1111 left a comment

Uh oh!

zjw1111 Jul 9, 2026

Uh oh!

lxy-9602 Jul 9, 2026

Uh oh!

lxy-9602 Jul 9, 2026

Uh oh!

lxy-9602 Jul 9, 2026

Uh oh!

lxy-9602 Jul 9, 2026

Uh oh!

lxy-9602 Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

suxiaogang223 commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Infrastructure

Global System Tables

Integration Tests (4 new tests in SystemTableReadInteTest)

Files Changed

Verification

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

suxiaogang223 commented Jun 13, 2026 •

edited

Loading