Skip to content

[Feature Request]: Embed Sirius in MatrixOne through CGo and retire Flight offload #28966

Description

@aunjgr

Current scope and status

Embed Sirius and DuckDB in mo-service through C ABI/CGo, using the pinned third_party/sirius submodule and frozen Pixi GPU environment. MO #29547's bounded MO-reader/result bridge and Native A/B are merged. Exact semantic authority remains #29449 document blob 42a89f09a1d168d02b9583cb3ea7b4de6dbb5634; #29690 records the approved delivery sequence.

Status updated 2026-10-09: importer #5 and Sirius #28/#29 are merged. C #29757 pins merged #29 (908ffc75a58b3be496b2172795e416326416e7ec); clean release SDK/build/public and owning validation pass. Kernel #29775 has green required CI and is still open. C remains draft pending the actual kernel merge and final integrated checks/CI. D starts after merged C; SF1/SF10/resource/performance acceptance remains open.

Next: merge kernel #29775, inherit it and finish C #29757's integrated checks/CI, merge C, then implement D as one complete PR. E/F require separate recovery-readiness design; Docker/image work remains separate. Keep #28966 and #28968 open.

Merged native release delivery (2026-10-09)

Sirius #29 merged as 908ffc75a58b3be496b2172795e416326416e7ec; C now pins that merged revision and importer #5 99c7ca3b6f8f3159239e119ed2982d42f98c4690. Clean release SDK regeneration, all 73 fingerprints and actual runtime-package identity checks pass. Default MO, Sirius-only and combined Sirius/cuVS release builds pass. Eight exporter and sixteen MO SDK-verifier tests pass.

At executable C head 59aaa5d234 on MO main 0eb746f37c22f57d2081331b25bf379c80168c81, all 22 native preparations and the complete public numeric fixture pass against the merged release SDK. All 31 terminal events confirm completed GPU tasks, capability 31, exact profile, no fallback, healthy cleanup and zero retained input/result credit. Both default normal/race and Sirius-only owning suites pass for substrait, siriusbridge, cnservice and compile. The independent C99 GPU/credit/cleanup consumer, complete combined bridge suite and same-process cuVS-before/Sirius/cuVS-after pass. Native bridge data/cancellation passes 83 in-process race repetitions.

The latest main merge d7621ea017db4e43dbde9e5a855ba7b2ad920784 adds spill-cache batching and design documentation. Its changed spillio package passes; C source and unspilled fixture inputs are unchanged. Full mandatory pre-push SCA and self-review are required on the submitted C head.

Kernel #29775 now has passing required CI, including pessimistic Compose BVT, overall coverage, unit/SCA and Connector/J pool-reset checks, but remains open. C #29757 stays draft until it inherits the actual merged kernel fix and closes final integrated validation/CI. D implementation begins only after C merges, as one complete bounded public/resource/performance campaign PR. Charged-credit checks and this small fixture do not satisfy D's SF1/SF10, actual-memory/process or performance gates. E/F and both issue closures remain deferred.

Kernel metadata prerequisite and current CI (2026-10-09)

MO kernel #29775 is ready at 50847b7feb9a50545a4cb6a4056f61ad954526e4, based on freshly fetched main c2c1031d0ba4b1034849f6bd3d267252a11eb6c9. It fixes premature result-schema freezing for prepared SELECT when concurrent DDL commits after execute-time binding. An inert phase point independently reproduces the same 20631/HY000 error on clean main b6fa8ccfc7. Headers now use the executed generation before the first positive row, or after successful empty execution; incompatible retries after schema publication remain rejected. Saved results finalize after whole-query success.

The eight public cases pass: both prepared protocols across commit-before-execute and commit-after-binding, empty/nonempty exact BIGINT results, plus empty saved-result completion and exact BIGINT replay. Complete frontend/compiler owning suites pass normally and under race. Focused race validation passes 100 callback repetitions and three full public repetitions in one process. Full mandatory pre-push SCA and lifecycle self-review pass. Exact-head CI now passes, including pessimistic Compose BVT, coverage and CI Required; the kernel PR is still open.

C head 8284b77adad0a3d29881808c0893b51370a53124 has passing shared build, Ubuntu UT, arm64 SCA, UT coverage and Proxy BVT. Its pessimistic Compose BVT fails one statement at pessimistic_transaction/prepare.sql row 10 with the metadata retry error; overall coverage has no valid completed BVT producer and CI Required consequently fails. The earlier PR-size TLS failure passed on targeted retry. C remains draft and must inherit the merged kernel fix and merged #29 before final release/public/owning checks and required CI closure.

Approved D remains one complete PR after merged C: version-2 native-MO/embedded-MO public execution, bounded comparison (32 MiB RAM / 4 GiB scratch), immutable SF1/SF10 manifests, lifecycle/resource evidence and the full 744-execution schedule. Keep every performance gate, including fresh matched 10% numeric baselines and the Flight+MO ratio gate of 1.0 on equivalent available coverage. E/F and issue closure remain deferred.

Native consumer prerequisite and C validation (2026-10-09)

Sirius #29 was validated before merge at clean head 2a39339beb0c2312dc4b71fbdd472e822450637c. It fixes exact NULL predicates over private decimal carriers and condition-free joins left as ANY_JOIN when exact preparation skips DuckDB's ordinary optimizer. NULL predicates use the canonical validity mask; literal TRUE joins use equal constant keys in the existing GPU join, preserving join kind, multiplicity, projection maps and empty-side outer NULLs. FALSE/NULL controls still reject before work starts. No SQL benchmark rewrite, numeric narrowing or fallback is introduced.

Clean frozen-Pixi local validation passes 1,964 assertions / 10 production C ABI cases, 14,879 / 11 exact/ordinary GPU cases, and 852 / 81 binding cases. Tests cover all physical widths, offset slices, empty inputs, NULL literals, materialize/interpreted/JIT evaluation, INNER/LEFT/RIGHT/FULL TRUE joins, duplicate/NULL high-limb payloads and empty sides. Eight SDK exporter tests, all pinned changed-file hooks, an independent C99 GPU/credit/lifecycle consumer and all 73 clean SDK fingerprints pass. Lint, documentation and distribution CI pass; compiler jobs are queued as of this update. Local passes are reported separately from CI.

Joint C development validation passes all 22 native preparations (no reader/GPU work starts during preparation) and the complete public MySQL numeric fixture. The 31 terminal events confirm completed GPU work, capability 31, exact profile, no fallback, healthy cleanup and zero retained input/result credit. The public fixture independently compares native-MO values/metadata/errors, including prepared division, masked errors and empty/all-NULL scalar joins. This uses an explicitly marked development SDK with #29's source, not a merged-pin release claim. Its MO base is 23d8f61252f3017dff218b530817a240206fd762; C now also merges main aee2a0b3764781a8ca908618548dfef75e336263, with exporter owning/vet and full mandatory SCA passing.

Merge order: #29 is merged and pinned in C. Merge kernel #29775, inherit that merged fix in C #29757, complete final integrated validation/CI, then merge C. D implementation begins only after C merges. All full-data/resource/process/performance gates remain open; E/F require separate recovery-readiness approval.

Merged root-name prerequisites and historical evidence

Importer #5 merged as 99c7ca3b6f8f3159239e119ed2982d42f98c4690. Sirius #28 merged as af4dc60152b3e14f263c7fe863b29ac7e154de30 and is C's recorded pin. The original Q1 root-name crash is resolved: canonical decimal carriers are opaque SQL scalars and cannot consume private limb headings; actual name accesses are bounded while ordinary final-STRUCT compatibility is retained.

The final importer fix 6693c3a4d779fb0e15fa07e4ce4166f0572f3952 is tree-identical to its merge. Joint importer/native binding validation passed 1009 assertions / 84 cases, including 157 / 3 canonical importer cases. Clean Sirius head 433cadd43a10442b0825a23234fb6d859207e9d5 is tree-identical to merged #28; its binding 852 / 81, production numeric C ABI 1468 / 8, SDK exporter/independent C consumer/fingerprint and pinned-hook checks passed.

Historical importer distribution CI had seven failures: clean-base STRUCT field-selection failure/crash plus five later failures independently reproduced at clean importer 95d9ce8d78490db3991ab6145653716aa3ec42c9 and bundled DuckDB d8cdaa33fda8df955cc76ef58a280f68f4cd43fa. Those five selected cases reproduced DISTINCT_FROM, TPCH/TPCDS CTE, old literal expected-message mismatch and empty-plan root failures. No assertion was relaxed or skipped, and that distribution run was not green. These scoped historical results do not replace C/D acceptance.

Delivered prerequisites and Native A

The scanner/Sirius TAE fixes already merged remain independently useful; they do not enable embedded direct-TAE input in this milestone. Importer hooks preserve identities and descriptors; Sirius owns numeric arithmetic and execution semantics.

Remaining approved implementation series

The sequence and acceptance contract are recorded in #29690. Each dependent implementation starts from its merged predecessor.

Order Repository Deliverable Required predecessor/status
B Sirius SUM/AVG/MIN/MAX; exact grouping/join/sort keys; spillable full-width states; complete preparation validation; statuses 12/13 and SIRIUS_CAP_MO_EXACT_DECIMAL_V1 (mask value 16u) Merged matrixorigin/sirius#27 at 5ea60cd31955d0dced2adcbcd3df0772207b79ef; root-name follow-up #28 merged; consumer fix #29 merged
C MO Capability-scoped exact lowering, Decimal256 publication/reconstruction, typed public errors and query-local evidence; still opt-in Draft #29757 pins merged #29; native release/public/owning checks pass; pending merged kernel #29775 and integrated CI
D MO Real native-MO/embedded-MO public SQL runner; all-22 SF1/SF10 value/type/NULL/error parity; lifecycle/resource and performance acceptance Merged C
E MO Embedded default and Flight removal with strict configuration/recovery migration Separate recovery-readiness design, D and every acceptance gate
F sidecar Retire superseded MO Flight service/deployment/exclusive tests; preserve independent DuckDB/TAE/HTTP and standalone Sirius tools and historical evidence Separate readiness design, E and a verified available MO release artifact

No CPU/Flight fallback or decimal-to-float coercion can substitute for exact-decimal completion. Do not advertise partial numeric capability. C must use one capability-scoped export profile through validation and serialization; ordinary Flight emission stays unchanged during coexistence.

D requires native MO and embedded MO as the two mandatory routes, with streams=2 and controls at 1/4, one excluded warmup plus five measured suites, and ten Q9 repetitions. Retain the 10% numeric regression policy. The matched Flight+MO ratio gate remains 1.0 on available equivalent coverage, including Q9 where available; unavailable comparisons are N/A. Incomplete Flight coverage is never presented as a full-suite total.

E rejects enabled legacy Flight selections and removed transport settings with actionable migration errors. Disabled Sirius and ordinary CPU behavior remain unchanged. Cutover requires authoritative replay/readiness and zero unresolved old Flight executions; a nil or unready lease manager is not evidence of empty state. Preserve GC protection until old consumers are proven quiescent, and retain the previous release for reconciliation and post-removal rollback.

Native B review evidence

Native B: matrixorigin/sirius#27, head 827c7100b32c4e856faf58928811f4207653d653, targeting upstream-dev-merge. Its contract, ownership/validation record and raw measurements cover full-width aggregate states, exact keys, complete typed preparation and public capability/error exposure.

Local terminal validation passed: CPU arithmetic 52,645 assertions / 8 cases; binding/preparation 831 / 80; native controls 606 / 78; complete affected GPU group 32,349 / 30. Production numeric C ABI passed 1,468 / 8 under memcheck, and numeric/result-codec selection passed 21,294 / 26 under memcheck, both with zero sanitizer errors. Eight SDK exporter tests, the independent C consumer, clean-source SDK fingerprints and pinned changed-file hooks passed. Compiler CI was pending at that recorded Native B checkpoint.

Matched local exact/control ratios are 0.7931 for safe SUM64→128 and 0.8165 for evaluator add64; all three complete runs are retained. Historical unchanged-add64 timing (+18.15% pooled) still needs fresh matched confirmation in D. Existing nonnumeric controls reproduce a generic CUB EmptyKernel diagnostic and a 41,386,248-byte cuDF default pinned-pool report under full leak checking. D must reconcile those process-baseline limits and satisfy every public/resource/lifecycle/performance gate; no rollout waiver or full-suite acceptance is claimed.

Open acceptance gates

Native A's recorded local CPU/GPU, sanitizer, SDK and static checks passed. At that historical checkpoint its compiler jobs were queued; merged status did not imply those jobs passed. Native validation is not the complete public SQL/resource/performance campaign.

Delivery and completion

Use existing directories and dedicated branches in the user's forks. Sirius integration targets upstream-dev-merge; MO/sidecar target main. Importer hooks target its compatible feature/26155-substrait-anti-joins branch pinned by Sirius; do not bundle unrelated importer synchronization.

Pin native consumers only to merged dependency commits. Sirius builds/tests always use frozen Pixi and incremental host builds, not image rebuilds. Intermediate PRs reference this issue and do not close it.

Predecessors #26154, #27586, #26159 and PR #27599 remain historical evidence, not this migration's closure. Preserve reusable tests and explicitly map replacements before deleting transport-only cases.

Close #28968 only after complete numeric/public/resource/performance acceptance. Close #28966 only after E, verified release availability and F are delivered. The opt-in supported-type milestone and Native A are distinct from migration completion.

CI/review correction (2026-10-10)

C #29757 now includes the explicit async histogram-export completion barrier at c4951901727116ed0c86806f772b215a845ca8fb. The previous exact-head Ubuntu CI failure named mometric.TestExporter; the test was observing two async sends before completion. All original batch/sample/counter/gauge/clock assertions remain. Exact normal/race, 100 focused race repetitions, complete metrics owning normal/race and full pre-push SCA pass. Fresh exact-head CI is running.

Kernel #29775's reviewer-found late saved-result failure cleanup is fixed at 6efb5fdfdaad60c2f458963ab143ec5ae5ec812f. The old-head red state regression fails; the corrected error/cancellation controls pass. The public same-connection test persists a batch, injects a producer error after saving, drains it, verifies connection identity, then saves/replays exact BIGINT data. All nine public cases, full frontend/compiler normal/race, three public race repetitions and full SCA pass. The reviewer thread is replied to and resolved, with re-review requested; approval and new CI are still pending. Inherit this prerequisite in C only after actual merge. D implementation remains gated on merged C, and E/F/issue closure stay deferred.

C merged-prerequisite integration (2026-10-10)

Kernel #29775 merged as af3f7232a3; C #29757 now inherits it and the merged MORPC CI-race correction from #29794. C is delivered at b39755ff4e2be20a3c9af464e9a46125ec14d9ea, normally merged with authoritative main 8f48d6fac0.

Final local release/native integration passes default/Sirius/combined builds, all 22 preparations, the public exact-numeric GPU fixture (31 healthy exact-profile/no-fallback events with zero retained credits), nine public prepared/save/replay cases, owning normal/race checks and combined Sirius/cuVS coexistence. Native cancellation passes 68 race repetitions; final planner checks and mandatory SCA pass. The detailed evidence is in C's versioned design record and PR description, with executable revisions stated explicitly.

Only required CI on this delivered C head remains before draft conversion. D implementation follows merged C as one complete campaign PR. E/F remain deferred; this tracker stays open.

Activity

  1. self-assigned this
    on Sep 16, 2026
  2. aunjgr commented on Sep 16, 2026

    @aunjgr
    ContributorAuthor

    Native foundation checkpoint: matrixorigin/sirius#14 (draft), commit a1068810, based on merged #13. Implemented the static C ABI target, coordinator-owned lifecycle, bounded handle/credit accounting, fatal-health ownership, and a generated build-tree SDK validated by an independent CMake C consumer. Validation: 63 assertions/10 native control cases, also under ThreadSanitizer; 313 assertions/21 affected FFI/pipeline/completion/TAE cases; live-GPU C runtime smoke passed. The public capability mask is ENGINE_CONTROL only and query preparation remains UNSUPPORTED. Stages 4-6 (MO input, TAE bindings, incremental results) and the full native C data-path acceptance are not implemented yet; this does not complete the approved four-stage round or permit Flight retirement.

  3. aunjgr commented on Sep 23, 2026

    @aunjgr
    ContributorAuthor

    Decision record for #28966 (owner-approved implementation plan): the GPU toolchain is an optional prerequisite split, implemented in MO #29254 and Sirius #22. Ordinary CPU builds remain independent of CUDA/Pixi. MO GPU builds may keep the existing system CUDA + Conda provider or select an explicit validated manifest; an invalid explicit manifest must fail closed. Sirius exports one locked Pixi CUDA/RAPIDS provider for the combined MO/Sirius binary, and the combined build and package must use that same provider.

    This records the earlier approval of the build-toolchain split and its scope for review. It is not approval of the implementation: CI, code-owner review, and combined GPU evidence remain separate gates. The embedded execution milestone is now MO-reader-only; direct-TAE storage changes were deferred and #29255 was closed.

  4. aunjgr commented on Sep 23, 2026

    @aunjgr
    ContributorAuthor

    Superseding the earlier optional-manifest decision: Pixi is the sole CUDA/RAPIDS build provider for the final embedded Sirius shape. Ordinary CPU-only MatrixOne builds remain Pixi-free. MO GPU builds run under a frozen Pixi environment; combined MO/Sirius builds use the same Sirius mo Pixi prefix for both components. There is no system CUDA/Conda fallback or separate GPU toolchain JSON manifest.

    This changes the implementation contract for MO #29254, Sirius #22, and the draft MO bridge #29203. The combined Sirius/cuVS runtime gate and host-owned NVIDIA driver requirement remain. The revised GPU toolchain design v2 records the owner-directed change; implementation, packaging, and CI still require review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions