[Subtask]: Preserve current MatrixOne numeric semantics in Sirius Substrait offload #28968

Parent issue

Parent migration: #28966. Predecessor compatibility evidence: #26159.

Contract and status

Implement MO exact Decimal64/128/256 semantics in embedded Sirius, preserving physical widths, values, precision/scale/nullability, rounding, operation-owned overflow classes and public metadata. Native B #27 advertises complete exact capability mask 16u (total 31u) and statuses 12/13 with ABI-v1 layouts preserved.

Status updated 2026-10-11: C #29757 is merged. D #29862 is implemented as the approved single complete draft PR at c33d09d5fa, normally merged with authoritative main 4022729da2. Development validation passed all 744 unprofiled public executions and all 264 required resource executions; all six allocation profiles had zero warmed growth. Fresh matched Native A/B numeric baselines passed all ten cases, and owning normal/race, all-22 native preparation, combined cuVS/Sirius coexistence and full local SCA passed. Sirius #30 remains the ready native prerequisite. D stays draft until that merge, a verified merged release SDK pin and rebuilt final consumers, uniform complete release campaign/evidence, required CI and reviewer approval. Development and mixed-capture evidence cannot satisfy final acceptance. E/F and Docker remain deferred; this tracker stays open.

• Canonical Q1-Q22 exporter inventory: docs: define exact Sirius numeric contract #29210.
• Exact-revision semantic design approval: docs: correct embedded Sirius numeric semantics #29449, document blob 42a89f09a1d168d02b9583cb3ea7b4de6dbb5634.
• Query-scoped importer hooks: feat: add query-scoped extension import handlers duckdb-substrait#3; unity-build prerequisite fix(tests): qualify importer types in unity builds duckdb-substrait#4, merged revision 95d9ce8d78490db3991ab6145653716aa3ec42c9.
• Sirius fixed-width representation, codecs and checked scalar GPU primitives: feat(numeric): add checked MO exact-decimal primitives sirius#25, merged revision 16135ebfbd1eb79d9a6e41c0858f69bd530f48ae.
• Native A: exact import, typed binding/clone fidelity, scalar GPU execution and conditional masks: feat(numeric): bind MO exact-decimal plans and scalar expressions sirius#26, merged revision c440013c89f1a54a6e5d934aabd0f9d3ef15ed3c.
• Native B merged: feat(numeric): complete embedded MO aggregates and exact capability sirius#27, revision 5ea60cd31955d0dced2adcbcd3df0772207b79ef. Merged root-name follow-up Add set union #28 resolves the original importer crash; consumer fix Add set inner join #29 is merged.
• MO C merged: feat(sirius): lower MO exact decimals for embedded GPU execution #29757 at af5a75c5f9, including the merged kernel/native prerequisites, exact lowering, typed metadata/errors and review corrections.
• Independent public SQL/value/NULL/error/metadata counterexample matrix.
• Actual GPU execution and native-MO equality for all 22 SF1/SF10 queries at streams=2, plus streams 1/4 controls.
• Resource and performance acceptance gates.

The inventory's eight eligible and fourteen declined queries are historical exporter evidence, not eight verified Sirius executions or current all-22 acceptance. Rerun it at each implementation's exact base and inspect the complete reachable bound expression graph, because the first admission failure can hide later unsupported signatures.

Remaining implementation and dependencies

The owner-approved split is in merged #29690.

1. Native B and root-name prerequisites: Sirius Add sort #27/Add set union #28 and importer [#4] Add a new document: CHANGELOG.md #5 are merged.
2. Native consumer prerequisite: Sirius Add set inner join #29 is merged and pinned by C; clean merged release SDK/build/public validation passes as recorded below.
3. MO C feat(sirius): lower MO exact decimals for embedded GPU execution #29757: merged as af5a75c5f9. Capability-scoped exact lowering, Decimal256 publication/reconstruction, public typed errors and query-local schema evidence are delivered, with kernel fix(frontend): publish prepared result metadata after schema retry #29775 and merged native prerequisites inherited. Integration remains opt-in.
4. MO D (next; C is merged): one complete PR must version the real native-MO/embedded-MO runner and complete all-22 SF1/SF10 value/type/NULL/error parity, lifecycle/resource and performance acceptance. Native MO and embedded MO are mandatory routes; equivalent Flight coverage is reported explicitly.

E/F remain under #28966 behind a separate recovery-readiness design and D acceptance.

The dated checkpoints below retain historical validation and review states; the current status and next action are stated above.

Merged native release delivery (2026-10-09)

Sirius #29 merged as 908ffc75a58b3be496b2172795e416326416e7ec; C now pins that merged revision and importer #5 99c7ca3b6f8f3159239e119ed2982d42f98c4690. Clean release SDK regeneration, all 73 fingerprints and actual runtime-package identity checks pass. Default MO, Sirius-only and combined Sirius/cuVS release builds pass. Eight exporter and sixteen MO SDK-verifier tests pass.

At executable C head 59aaa5d234 on MO main 0eb746f37c22f57d2081331b25bf379c80168c81, all 22 native preparations and the complete public numeric fixture pass against the merged release SDK. All 31 terminal events confirm completed GPU tasks, capability 31, exact profile, no fallback, healthy cleanup and zero retained input/result credit. Both default normal/race and Sirius-only owning suites pass for substrait, siriusbridge, cnservice and compile. The independent C99 GPU/credit/cleanup consumer, complete combined bridge suite and same-process cuVS-before/Sirius/cuVS-after pass. Native bridge data/cancellation passes 83 in-process race repetitions.

The latest main merge d7621ea017db4e43dbde9e5a855ba7b2ad920784 adds spill-cache batching and design documentation. Its changed spillio package passes; C source and unspilled fixture inputs are unchanged. Full mandatory pre-push SCA and self-review are required on the submitted C head.

Kernel #29775 now has passing required CI, including pessimistic Compose BVT, overall coverage, unit/SCA and Connector/J pool-reset checks, but remains open. C #29757 stays draft until it inherits the actual merged kernel fix and closes final integrated validation/CI. D implementation begins only after C merges, as one complete bounded public/resource/performance campaign PR. Charged-credit checks and this small fixture do not satisfy D's SF1/SF10, actual-memory/process or performance gates. E/F and both issue closures remain deferred.

Kernel metadata prerequisite and current CI (2026-10-09)

MO kernel #29775 is ready at 50847b7feb9a50545a4cb6a4056f61ad954526e4, based on freshly fetched main c2c1031d0ba4b1034849f6bd3d267252a11eb6c9. It fixes premature result-schema freezing for prepared SELECT when concurrent DDL commits after execute-time binding. An inert phase point independently reproduces the same 20631/HY000 error on clean main b6fa8ccfc7. Headers now use the executed generation before the first positive row, or after successful empty execution; incompatible retries after schema publication remain rejected. Saved results finalize after whole-query success.

The eight public cases pass: both prepared protocols across commit-before-execute and commit-after-binding, empty/nonempty exact BIGINT results, plus empty saved-result completion and exact BIGINT replay. Complete frontend/compiler owning suites pass normally and under race. Focused race validation passes 100 callback repetitions and three full public repetitions in one process. Full mandatory pre-push SCA and lifecycle self-review pass. Exact-head CI now passes, including pessimistic Compose BVT, coverage and CI Required; the kernel PR is still open.

C head 8284b77adad0a3d29881808c0893b51370a53124 has passing shared build, Ubuntu UT, arm64 SCA, UT coverage and Proxy BVT. Its pessimistic Compose BVT fails one statement at pessimistic_transaction/prepare.sql row 10 with the metadata retry error; overall coverage has no valid completed BVT producer and CI Required consequently fails. The earlier PR-size TLS failure passed on targeted retry. C remains draft and must inherit the merged kernel fix and merged #29 before final release/public/owning checks and required CI closure.

Approved D remains one complete PR after merged C: version-2 native-MO/embedded-MO public execution, bounded comparison (32 MiB RAM / 4 GiB scratch), immutable SF1/SF10 manifests, lifecycle/resource evidence and the full 744-execution schedule. Keep every performance gate, including fresh matched 10% numeric baselines and the Flight+MO ratio gate of 1.0 on equivalent available coverage. E/F and issue closure remain deferred.

Native consumer prerequisite and C validation (2026-10-09)

Sirius #29 was validated before merge at clean head 2a39339beb0c2312dc4b71fbdd472e822450637c. It fixes exact NULL predicates over private decimal carriers and condition-free joins left as ANY_JOIN when exact preparation skips DuckDB's ordinary optimizer. NULL predicates use the canonical validity mask; literal TRUE joins use equal constant keys in the existing GPU join, preserving join kind, multiplicity, projection maps and empty-side outer NULLs. FALSE/NULL controls still reject before work starts. No SQL benchmark rewrite, numeric narrowing or fallback is introduced.

Clean frozen-Pixi local validation passes 1,964 assertions / 10 production C ABI cases, 14,879 / 11 exact/ordinary GPU cases, and 852 / 81 binding cases. Tests cover all physical widths, offset slices, empty inputs, NULL literals, materialize/interpreted/JIT evaluation, INNER/LEFT/RIGHT/FULL TRUE joins, duplicate/NULL high-limb payloads and empty sides. Eight SDK exporter tests, all pinned changed-file hooks, an independent C99 GPU/credit/lifecycle consumer and all 73 clean SDK fingerprints pass. Lint, documentation and distribution CI pass; compiler jobs are queued as of this update. Local passes are reported separately from CI.

Joint C development validation passes all 22 native preparations (no reader/GPU work starts during preparation) and the complete public MySQL numeric fixture. The 31 terminal events confirm completed GPU work, capability 31, exact profile, no fallback, healthy cleanup and zero retained input/result credit. The public fixture independently compares native-MO values/metadata/errors, including prepared division, masked errors and empty/all-NULL scalar joins. This uses an explicitly marked development SDK with #29's source, not a merged-pin release claim. Its MO base is 23d8f61252f3017dff218b530817a240206fd762; C now also merges main aee2a0b3764781a8ca908618548dfef75e336263, with exporter owning/vet and full mandatory SCA passing.

Merge order: #29 is merged and pinned in C. Merge kernel #29775, inherit that merged fix in C #29757, complete final integrated validation/CI, then merge C. D implementation begins only after C merges. All full-data/resource/process/performance gates remain open; E/F require separate recovery-readiness approval.

Merged root-name prerequisites and historical evidence

Importer #5 merged as 99c7ca3b6f8f3159239e119ed2982d42f98c4690. Sirius #28 merged as af4dc60152b3e14f263c7fe863b29ac7e154de30 and is C's recorded pin. The original Q1 root-name crash is resolved: canonical decimal carriers are opaque SQL scalars and cannot consume private limb headings; actual name accesses are bounded while ordinary final-STRUCT compatibility is retained.

The final importer fix 6693c3a4d779fb0e15fa07e4ce4166f0572f3952 is tree-identical to its merge. Joint importer/native binding validation passed 1009 assertions / 84 cases, including 157 / 3 canonical importer cases. Clean Sirius head 433cadd43a10442b0825a23234fb6d859207e9d5 is tree-identical to merged #28; its binding 852 / 81, production numeric C ABI 1468 / 8, SDK exporter/independent C consumer/fingerprint and pinned-hook checks passed.

Historical importer distribution CI had seven failures: clean-base STRUCT field-selection failure/crash plus five later failures independently reproduced at clean importer 95d9ce8d78490db3991ab6145653716aa3ec42c9 and bundled DuckDB d8cdaa33fda8df955cc76ef58a280f68f4cd43fa. Those five selected cases reproduced DISTINCT_FROM, TPCH/TPCDS CTE, old literal expected-message mismatch and empty-plan root failures. No assertion was relaxed or skipped, and that distribution run was not green. These scoped historical results do not replace C/D acceptance.

Delivered native evidence

Recorded in merged matrixorigin/sirius#26 and its validation map and raw repetitions:

The Decimal256 conditional child-mask defect was reproduced and fixed while retaining strict carrier-shape assertions and a following exact operation. Tests cover inactive overflow/cast suppression, active error classes and evaluator reuse after failure. The native SDK exports the canonical ExactDecimalLiteral protobuf schema.

At the recorded Native A checkpoint, compiler jobs were queued; lint, documentation build and distribution checks had passed. Local memcheck and kernel timings do not establish full SQL, leak, resource or concurrency acceptance. Complete public numeric statuses and production admission merged in matrixorigin/sirius#27; MO integration subsequently exposed the now-resolved root-name defect.

Native B review evidence

Implementation: matrixorigin/sirius#27, head 827c7100b32c4e856faf58928811f4207653d653, base upstream-dev-merge. The contract, ownership/validation record and retained benchmark runs document supported shapes and remaining acceptance limits.

The matched local SUM64→128 exact/control ratio is 0.7931; evaluator add64 is 0.8165. Every repetition from all three complete runs is retained. The unchanged checked-add64 primitive's pooled historical comparison is +18.15% with varying run medians; fresh matched confirmation remains required in D under the 10% policy. No rollout waiver is claimed.

Full leak checking of the existing nonnumeric native controls reports a 41,386,248-byte default cuDF pinned pool and a generic CUB EmptyKernel diagnostic in the unchanged converter path. Those controls' values/lifecycle assertions pass; this invocation is not clean leak acceptance. D must reconcile process baselines and complete public SQL, lifecycle/resource and performance gates. At the recorded Native B checkpoint, compiler CI was pending; local passes did not imply CI completion.

Acceptance and boundaries

Native MO is the complete Q1-Q22 correctness oracle. Exact support remains embedded-only and MO-reader-only; direct TAE is deferred. Flight performance comparisons are restricted to equivalent supported queries with explicit coverage, rather than extending Flight numerics.

No SQL rewrites, decimal-to-float coercion, unchecked narrowing, fragment execution back in MO, silently relaxed decimal comparisons or CPU/Flight fallback may satisfy acceptance. Agreed floating-point tolerance applies only to approximate columns.

Reuse the approved design's fixed-width coefficient/512-bit scratch contract, MO-bound result types and independent scalar/aggregate/cast error counterexamples. Account numeric buffers and aggregate state through existing admission/spill ownership. More than 10% relevant kernel/query regression blocks rollout unless the policy owner approves a documented correctness/performance tradeoff; correctness is not waived.

The public campaign requires streams=2 plus 1/4 controls, one excluded warmup and five measured suites, with ten Q9 repetitions. Preserve the matched Flight+MO ratio gate of 1.0 on available equivalent coverage, including Q9 where available. Report unavailable comparisons as N/A and never present partial coverage as a full-suite total.

The reader/bridge migration may proceed opt-in while this issue remains open. Close this issue only after complete numeric implementation and public/GPU/resource/performance acceptance; those gates also block default cutover and Flight retirement under #28966.

CI/review correction (2026-10-10)

C #29757 now includes the explicit async histogram-export completion barrier at c4951901727116ed0c86806f772b215a845ca8fb. The previous exact-head Ubuntu CI failure named mometric.TestExporter; the test was observing two async sends before completion. All original batch/sample/counter/gauge/clock assertions remain. Exact normal/race, 100 focused race repetitions, complete metrics owning normal/race and full pre-push SCA pass. Fresh exact-head CI is running.

Kernel #29775's reviewer-found late saved-result failure cleanup is fixed at 6efb5fdfdaad60c2f458963ab143ec5ae5ec812f. The old-head red state regression fails; the corrected error/cancellation controls pass. The public same-connection test persists a batch, injects a producer error after saving, drains it, verifies connection identity, then saves/replays exact BIGINT data. All nine public cases, full frontend/compiler normal/race, three public race repetitions and full SCA pass. The reviewer thread is replied to and resolved, with re-review requested; approval and new CI are still pending. Inherit this prerequisite in C only after actual merge. D implementation remains gated on merged C, and E/F/issue closure stay deferred.

C merged-prerequisite integration (2026-10-10)

Kernel #29775 merged as af3f7232a3; C #29757 now inherits it and the merged MORPC CI-race correction from #29794. C is delivered at b39755ff4e2be20a3c9af464e9a46125ec14d9ea, normally merged with authoritative main 8f48d6fac0.

Final local release/native integration passes default/Sirius/combined builds, all 22 preparations, the public exact-numeric GPU fixture (31 healthy exact-profile/no-fallback events with zero retained credits), nine public prepared/save/replay cases, owning normal/race checks and combined Sirius/cuVS coexistence. Native cancellation passes 68 race repetitions; final planner checks and mandatory SCA pass. The detailed evidence is in C's versioned design record and PR description, with executable revisions stated explicitly.

C is now ready for review. Passing required CI, reviewer approval and the user merge remain before D implementation begins as one complete campaign PR. E/F remain deferred; this tracker stays open.

C deep-review corrections delivered (2026-10-10)

C #29757 is ready for review at a4e34ef05de6a6d02ceb0fa81219cacd452ae6c8, normally merged with authoritative main 62827a1c8b. Fix commit af4f342066 resolves persisted zero-DATE extraction, bound CASE nullability, and an admitted unchecked integer-overflow result. Ordinary embedded integer +/-/* now declines before execution; checked exact-decimal arithmetic and integral aggregates remain eligible.

Both human review findings have actual GPU red/green proofs. Their threads are replied to and resolved, and XuPeng-SH re-review is requested. Final all-22 native preparation, full public GPU normal/race (three repetitions), 34 healthy exact/no-fallback/zero-retained-credit events, owning normal/race, combined release/bridge and full SCA pass. Native pin/ABI and Flight behavior remain unchanged.

Fresh required CI and reviewer approval remain before the user merges C. D follows merged C as one complete campaign PR; E/F remain deferred and this tracker stays open.

D public campaign implementation delivered (2026-10-11)

D #29862 contains the public streaming runner, bounded exact schema/value/duplicate/order comparison, owned service/descendant cleanup, separate allocation profiling, lifecycle/sanitizer/performance evidence and fail-closed acceptance. Early writer/quota failures cancel the unread query before driver close. The shared-CTE compiler fix uses independent MO readers on the same statement snapshot; Sirius #30 preserves retained VARCHAR identity casts and validates destructive bindings. Canonical SQL is unchanged.

All recorded results retain actual source/binary/SDK/capture identities. The development GPU route was slower overall on this hardware; the fresh numeric regression maximum ratio was 1.007914, within the 1.1 policy. Raw sanitizer exit 99 is preserved with exact Native A CUB and no-GPU-work pinned-pool controls and six stable allocation profiles. Full final release acceptance remains incomplete. Merge Sirius #30 next, then advance the verified merged pin and complete the uniform final 744/264 release validation before promoting D. Required CI was inspected once after publication; no CI waiting or automatic merge is requested.