Search before asking
Motivation
#192 and #194 added the source-backed primary-key BTree read path. Paimon C++ writers still need the corresponding maintenance path: after compaction changes the active source files of a data level, a missing or stale payload leaves that level uncovered and queries fall back to normal file scans.
Paimon C++ should maintain these payloads during fixed-bucket primary-key writes and compaction, using the existing Java-compatible source metadata, BTree payload format, and index manifests.
Solution
Add the source-backed primary-key BTree maintenance lifecycle for fixed-bucket primary-key tables:
- Validate the Java-equivalent table and index prerequisites.
- Restore committed source-backed payload metadata into bucket writers without mixing Data Evolution payloads.
- Build one payload per indexed field and positive data level from physical source rows, then commit matching index additions and deletions in the same snapshot as the data changes.
- Reconcile missing, stale, duplicate, replaced, removed-definition, and empty-level payloads during compaction.
- Isolate build failures to the affected field and level so reads safely fall back to normal scans and a later maintenance attempt can rebuild the payload.
- Retain live index files during snapshot expiration and orphan cleanup, including tag and branch safety and external-file deletion retries.
Reuse the existing storage formats and internal reader, writer, sort-buffer, path, manifest, and commit abstractions. Do not introduce a new index family or storage protocol.
The implementation is in #245. It keeps maintenance synchronous; Java asynchronous scheduling, manual rebuild actions, realtime writers, and postpone-bucket writers remain outside this scope.
Anything else?
This is a maintenance-path follow-up to the read-path work in #192 and #194. It ports an existing Java capability, so no separate PIP is proposed.
Are you willing to submit a PR?
Search before asking
Motivation
#192 and #194 added the source-backed primary-key BTree read path. Paimon C++ writers still need the corresponding maintenance path: after compaction changes the active source files of a data level, a missing or stale payload leaves that level uncovered and queries fall back to normal file scans.
Paimon C++ should maintain these payloads during fixed-bucket primary-key writes and compaction, using the existing Java-compatible source metadata, BTree payload format, and index manifests.
Solution
Add the source-backed primary-key BTree maintenance lifecycle for fixed-bucket primary-key tables:
Reuse the existing storage formats and internal reader, writer, sort-buffer, path, manifest, and commit abstractions. Do not introduce a new index family or storage protocol.
The implementation is in #245. It keeps maintenance synchronous; Java asynchronous scheduling, manual rebuild actions, realtime writers, and postpone-bucket writers remain outside this scope.
Anything else?
This is a maintenance-path follow-up to the read-path work in #192 and #194. It ports an existing Java capability, so no separate PIP is proposed.
Are you willing to submit a PR?