Repository navigation

Rebuild Python UDF with typed revisions and Arrow Flight execution - #29152

Open

iamlinjunhong wants to merge 230 commits into

matrixorigin:mainfrom

iamlinjunhong:m-28132

Open

Rebuild Python UDF with typed revisions and Arrow Flight execution#29152
iamlinjunhong wants to merge 230 commits into
matrixorigin:mainfrom
iamlinjunhong:m-28132

Commits on Oct 17, 2026
- refactor(udf): replace legacy Python service with Arrow runtime
  iamlinjunhong
  committed
  b052c43
- fix(udf): tighten runtime naming and lifecycle
  iamlinjunhong
  committed
  58ab8de
- fix(udf): rename Python adapter and fence duplicate calls
  iamlinjunhong
  committed
  e3d1805
- fix(udf): close remaining Python worker contracts
  iamlinjunhong
  committed
  e76c923
- fix(udf): align SQL selection and system account fences
  iamlinjunhong
  committed
  39d9b26
- test(udf): add Python cases to CI BVT
  iamlinjunhong
  committed
  1081c09
- chore(udf): remove stale Python service ignore rules
  iamlinjunhong
  committed
  17127ba
- ci(udf): harden Python BVT build fallback
  iamlinjunhong
  committed
  3db5352
- test(udf): run Python cases in ordinary BVT
  iamlinjunhong
  committed
  5650a69
- test(udf): expand Python boundary BVT coverage
  iamlinjunhong
  committed
  eacdef1
- test(udf): cover null multi-argument calls
  iamlinjunhong
  committed
  645cf78
- fix(udf): close Python execution and type contracts
  iamlinjunhong
  committed
  8731893
- fix(udf): separate active and terminal reservations
  iamlinjunhong
  committed
  0e203be
- test(udf): update decimal normalization baseline
  iamlinjunhong
  committed
  1bc7a02
- fix(udf): harden handler response ownership
  iamlinjunhong
  committed
  7f2d699
- refactor(udf): clarify execution sequence ownership
  iamlinjunhong
  committed
  22fbb07
- fix(udf): encode UUID width in the ABI fingerprint
  iamlinjunhong
  committed
  cd1be2f
- fix(udf): close Python execution channel contracts
  iamlinjunhong
  committed
  d08b98f
- fix(udf): close Python gateway response contracts
  iamlinjunhong
  committed
  9c0957f
- fix(udf): make Python runtime shutdown terminal
  iamlinjunhong
  committed
  43e8b5c
- fix(udf): close Python SQL and runtime contracts
  iamlinjunhong
  committed
  8f59005
- fix(udf): require complete Python type contracts
  iamlinjunhong
  committed
  ae70a44
- fix(udf): enforce one-shot group state transitions
  iamlinjunhong
  committed
  6f4eada
- fix(udf): reject late Flight completion controls
  iamlinjunhong
  committed
  047dc8d
- fix(udf): align Python JSON validation with SQL
  iamlinjunhong
  committed
  327d6f1
- fix(udf): isolate Python handler environment
  iamlinjunhong
  committed
  258a250
- fix(udf): enforce complete Python routine bodies
  iamlinjunhong
  committed
  b29665e
- fix(udf): reject non-finite handler deadlines
  iamlinjunhong
  committed
  ba86a11
- fix(udf): validate Python execution input shapes
  iamlinjunhong
  committed
  2e21ba0
- fix(udf): validate Arrow physical descriptor limits
  iamlinjunhong
  committed
  67c0378
- fix(udf): ignore null decimal payloads
  iamlinjunhong
  committed
  646eea3
- fix(udf): honor SQL integer type equivalence
  iamlinjunhong
  committed
  1c610d6
- fix(udf): reject duplicate invocation opens
  iamlinjunhong
  committed
  4fe5a4d
- fix(udf): reject unknown control envelope fields
  iamlinjunhong
  committed
  66b6ac3
- fix(udf): reject duplicate control JSON fields
  iamlinjunhong
  committed
  9bb3383
- fix(udf): validate opens before ledger admission
  iamlinjunhong
  committed
  bb34714
- fix(udf): enforce Python descriptor shape
  iamlinjunhong
  committed
  3c48292
- fix(udf): reject Arrow bodies on control frames
  iamlinjunhong
  committed
  fe90629
- fix(udf): align Python fencing tuple validation
  iamlinjunhong
  committed
  dd4784d
- fix(udf): reject empty Flight input frames
  iamlinjunhong
  committed
  3fe6303
- fix(udf): encode zero sequence control fields
  iamlinjunhong
  committed
  7d256dc
- fix(udf): enforce control field ownership
  iamlinjunhong
  committed
  c6afa68
- fix(udf): preserve decimal64 signed range
  iamlinjunhong
  committed
  0249316
- fix(udf): reject nonstandard control JSON
  iamlinjunhong
  committed
  566ce4e
- fix(udf): preserve fencing tuple encoding
  iamlinjunhong
  committed
  69bc6e3
- fix(udf): validate IANA timezone identities
  iamlinjunhong
  committed
  37d3c63
- fix(udf): validate input Arrow values
  iamlinjunhong
  committed
  bf2efb0
- fix(udf): reject empty constant inputs
  iamlinjunhong
  committed
  029a301
- fix(udf): classify malformed worker frames
  iamlinjunhong
  committed
  5f89a49
- fix(udf): enforce descriptor value domains
  iamlinjunhong
  committed
  631d08b
- fix(udf): reject invalid control UTF-8
  iamlinjunhong
  committed
  01a06f2
- fix(udf): validate every Python input batch
  iamlinjunhong
  committed
  f1ad756
- feat(udf): implement long-term Python UDF execution
  iamlinjunhong
  committed
  7772e2b
- feat(udf): validate Python definitions before publication
  iamlinjunhong
  committed
  1281c97
- fix(udf): read handler frames across pipe short reads
  iamlinjunhong
  committed
  aff5a90
- fix(udf): keep Arrow encoder cleanup retryable
  iamlinjunhong
  committed
  73f6518
- fix(udf): finalize input before terminal success
  iamlinjunhong
  committed
  d1adacf
- fix(udf): classify invalid handler buffers
  iamlinjunhong
  committed
  f8ca4fd
- fix(udf): reject trailing handler batch bytes
  iamlinjunhong
  committed
  c82efb7
- fix(udf): validate Python handler bindings at create
  iamlinjunhong
  committed
  999c1e4
- fix(udf): validate empty and typed input boundaries
  iamlinjunhong
  committed
  7389c58
- fix(udf): reject unsupported routine types before allocation
  iamlinjunhong
  committed
  03f89c7
- fix(udf): enforce exact external routine selection domain
  iamlinjunhong
  committed
  9ee28ef
- fix(udf): transfer external expression result ownership
  iamlinjunhong
  committed
  abbe2f2
- fix(udf): verify frozen definitions before evaluating arguments
  iamlinjunhong
  committed
  777785a
- fix(udf): skip arguments outside the selected row domain
  iamlinjunhong
  committed
  b6809d6
- fix(udf): preserve canonical JSON tokens across adapters
  iamlinjunhong
  committed
  4c2d56e
- fix(udf): avoid re-expanding shared routine query dependencies
  iamlinjunhong
  committed
  e8bbc2f
- fix(udf): own handler pipes before subsequent allocations
  iamlinjunhong
  committed
  ddad1b1
- docs(udf): describe CI workers without isolation claims
  iamlinjunhong
  committed
  db354e2
- test(udf): cover canonical JSON through real Flight batches
  iamlinjunhong
  committed
  2a5a469
- perf(udf): reduce scalar batch conversion overhead
  iamlinjunhong
  committed
  b5e1c9b
- refactor(udf): share scalar conversion contract
  iamlinjunhong
  committed
  6ff9312
- perf(udf): trim handler process startup
  iamlinjunhong
  committed
  5cb49c6
- perf(udf): trim bounded handler IPC frames
  iamlinjunhong
  committed
  01ac657
- perf(udf): avoid Arrow result stream copies
  iamlinjunhong
  committed
  15c0018
- perf(udf): batch fixed-width input materialization
  iamlinjunhong
  committed
  36a61fc
- perf(udf): reuse variable-width input wrappers
  iamlinjunhong
  committed
  be214d4
- fix(udf): guard empty variable-width inputs
  iamlinjunhong
  committed
  b45b1cc
- perf(udf): reduce handler request copies
  iamlinjunhong
  committed
  b1b9931
- refactor(udf): clarify varlena cache naming
  iamlinjunhong
  committed
  9b909a9
- fix(udf): handle vector inputs in batch sizing
  iamlinjunhong
  committed
  6945d36
- perf(udf): reuse result headers during decode
  iamlinjunhong
  committed
  d375410
- fix(udf): preserve Arrow metadata across input batches
  iamlinjunhong
  committed
  c08368e
- fix(udf): fence supervisor restart during close
  iamlinjunhong
  committed
  0c0f9da
- fix(udf): require explicit Python invocation language
  iamlinjunhong
  committed
  25de061
- fix(udf): require explicit Python definition language
  iamlinjunhong
  committed
  fb7470f
- fix(udf): validate final Python handler binding
  iamlinjunhong
  committed
  e868c85
- fix(udf): fall back when handler group kill is denied
  iamlinjunhong
  committed
  9f9c7a7
- fix(udf): include timezone data in runtime image
  iamlinjunhong
  committed
  fae58c2
- fix(udf): fail closed on broken liveness watches
  iamlinjunhong
  committed
  04a809f
- fix(udf): close gateway admission barrier
  iamlinjunhong
  committed
  651335c
- fix(udf): validate empty selection domains
  iamlinjunhong
  committed
  b23f9ea
- fix(udf): reject noncanonical JSON results
  iamlinjunhong
  committed
  ac4a5c7
- fix(udf): reject opaque text descriptors
  iamlinjunhong
  committed
  d3b3f28
- fix(udf): serialize admission with gateway close
  iamlinjunhong
  committed
  40e4c31
- fix(udf): enforce budget during handler writes
  iamlinjunhong
  committed
  229bfab
- fix(udf): validate Python catalog signature before publish
  iamlinjunhong
  committed
  3506515
- fix(udf): refresh worker lease for readiness
  iamlinjunhong
  committed
  5923ebb
- fix(udf): validate temporal child validity
  iamlinjunhong
  committed
  f8ebb2c
- fix(udf): scope worker group fences by account
  iamlinjunhong
  committed
  021bc81
- fix(udf): scope gateway groups by account
  iamlinjunhong
  committed
  28908ad
- fix(udf): reject noncanonical descriptor fields
  iamlinjunhong
  committed
  634ddd0
- fix(udf): bound worker fencing identities
  iamlinjunhong
  committed
  b46c691
- fix(udf): align Python security IDs with uint32
  iamlinjunhong
  committed
  89581e2
- fix(udf): enforce required worker controls on encode
  iamlinjunhong
  committed
  eb985a5
- fix(udf): bound Python statement timestamps
  iamlinjunhong
  committed
  075b60d
- fix(udf): bound Python descriptor integers
  iamlinjunhong
  committed
  364e2a3
- fix(udf): verify runtime result production
  iamlinjunhong
  committed
  f361a72
- fix(udf): reject boolean contract versions
  iamlinjunhong
  committed
  94956bb
- fix(udf): require canonical timezone kinds
  iamlinjunhong
  committed
  6b517b1
- fix(udf): validate definition schema type
  iamlinjunhong
  committed
  8492fda
- fix(udf): cancel exchanges during worker shutdown
  iamlinjunhong
  committed
  725f4b1
- fix(udf): require canonical routine language
  iamlinjunhong
  committed
  0981883
- fix(udf): reject noncanonical plan languages
  iamlinjunhong
  committed
  78c7aef
- test(udf): inspect scoped gateway group fences
  iamlinjunhong
  committed
  85284df
- fix(udf): align gateway timeout with worker limit
  iamlinjunhong
  committed
  3ba31ba
- fix(udf): preserve large JSON number tokens
  iamlinjunhong
  committed
  0e272d8
- fix(udf): reject noncanonical routine language at bind
  iamlinjunhong
  committed
  46a5f26
- fix(udf): enforce Arrow field nullability
  iamlinjunhong
  committed
  d8a6808
- fix(udf): require object invocation payload
  iamlinjunhong
  committed
  91fb6db
- fix(udf): reject cancelled worker actions
  iamlinjunhong
  committed
  776c1c9
- fix(udf): reject noncanonical catalog languages
  iamlinjunhong
  committed
  09d72e5
- fix(udf): validate module handler bindings
  iamlinjunhong
  committed
  318e97e
- fix(udf): bound resolved artifact sources
  iamlinjunhong
  committed
  cec8cef
- fix(udf): align statement timestamp bounds
  iamlinjunhong
  committed
  3574cc6
- fix(udf): handle bare except in handler validation
  iamlinjunhong
  committed
  b5a31f0
- fix(udf): bound artifact validation resolution
  iamlinjunhong
  committed
  e99dcfe
- fix(udf): inspect mapping pattern handler captures
  iamlinjunhong
  committed
  c61a36f
- fix(udf): stop cleanup reaper on worker shutdown errors
  iamlinjunhong
  committed
  f7ba993
- fix(udf): reject null sql mode context
  iamlinjunhong
  committed
  ce3b268
- fix(udf): validate IANA zones at CN boundary
  iamlinjunhong
  committed
  c97904e
- fix(udf): cap artifact store to wire contract
  iamlinjunhong
  committed
  6f24cdb
- fix(udf): bound control JSON nesting
  iamlinjunhong
  committed
  26a0cb3
- fix(udf): bound validation action JSON
  iamlinjunhong
  committed
  d3e10f7
- fix(udf): reject sub-minute session offsets
  iamlinjunhong
  committed
  40bc0a5
- fix(frontend): stabilize Python replace validation
  iamlinjunhong
  committed
  25307fd
- fix(frontend): reject incomplete Python identities
  iamlinjunhong
  committed
  330c656
- fix(udf): cancel exchange before releasing admission
  iamlinjunhong
  committed
  0349a04
- fix(udf): keep native Flight readers on RPC thread
  iamlinjunhong
  committed
  427a626
- fix(udf): distinguish expected worker shutdown
  iamlinjunhong
  committed
  731977d
- fix(udf): complete handler response frame writes
  iamlinjunhong
  committed
  42c13f2
- fix(udf): validate handler binding before loading
  iamlinjunhong
  committed
  57caa06
- fix(udf): kill handler groups before reaping leaders
  iamlinjunhong
  committed
  ed6221a
- fix(udf): reject stale result acknowledgements
  iamlinjunhong
  committed
  4d1af78
- fix(udf): close Docker contract gaps
  iamlinjunhong
  committed
  6772a1d
- refactor(udf): centralize logical type mapping
  iamlinjunhong
  committed
  6f40965
- fix(udf): preserve legacy python heads on restore
  iamlinjunhong
  committed
  35df98c
- fix(udf): enforce descriptor validity at result boundaries
  iamlinjunhong
  committed
  b5100fb
- fix(udf): cancel gateway operations on close
  iamlinjunhong
  committed
  9f6b05f
- fix(udf): reject incompatible vector descriptor casts
  iamlinjunhong
  committed
  2fb509d
- fix(udf): bound handler response buffering
  iamlinjunhong
  committed
  2c2c483
- udf: add runtime status bridge and finish fencing
  iamlinjunhong
  committed
  e2b34f3
- test: add Arrow compute correctness cases for Python UDF
  iamlinjunhong
  committed
  a71ad83
- test: make Arrow sort null placement explicit
  iamlinjunhong
  committed
  ca8f68a
- test: cover null Arrow take indices
  iamlinjunhong
  committed
  6f1c1fb
- test: make Arrow arithmetic scalar type explicit
  iamlinjunhong
  committed
  e53b15a
- test: make Arrow string options explicit
  iamlinjunhong
  committed
  71b4df8
- test: make Arrow selection safety explicit
  iamlinjunhong
  committed
  c20ac93
- test: cover Arrow vector cardinality rejection
  iamlinjunhong
  committed
  9e7904d
- test: distinguish Arrow string option semantics
  iamlinjunhong
  committed
  d6729a9
- test: bind Arrow scalar arguments to SQL types
  iamlinjunhong
  committed
  c31b4d0
- docs: describe Arrow cardinality coverage
  iamlinjunhong
  committed
  bebd792
- test: clean up BVT service processes
  iamlinjunhong
  committed
  a17fdf5
- test: reap BVT child services
  iamlinjunhong
  committed
  1abe2a7
- test: wait for Python UDF capability handshake
  iamlinjunhong
  committed
  5232162
- test: cover Arrow take bounds errors
  iamlinjunhong
  committed
  4e7ed0f
- test: verify Arrow sort index positions
  iamlinjunhong
  committed
  ea7fc03
- test: cover Arrow float arithmetic
  iamlinjunhong
  committed
  ec2f252
- fix: preserve vector size during Python UDF binding
  iamlinjunhong
  committed
  af7a83a
- fix: keep BVT services alive for test runner
  iamlinjunhong
  committed
  08b30e6
- fix: clean BVT services after test completion
  iamlinjunhong
  committed
  ba898b6
- fix: harden BVT launcher argument handling
  iamlinjunhong
  committed
  dfb5456
- fix: freeze complete Arrow result frames
  iamlinjunhong
  committed
  9cddc05
- fix: verify BVT service liveness during startup
  iamlinjunhong
  committed
  7c214c5
- docs: fix Python UDF BVT generation example
  iamlinjunhong
  committed
  54b9ead
- test: make compose worker readiness verify capabilities
  iamlinjunhong
  committed
  2ec4917
- fix: clean BVT services on startup failure
  iamlinjunhong
  committed
  c00ec47
- ci: preserve BVT logs when setup fails
  iamlinjunhong
  committed
  a2175c0
- fix: avoid waiting on exited BVT processes
  iamlinjunhong
  committed
  1979681
- fix: keep handler quota ownership retryable
  iamlinjunhong
  committed
  8d72e4f
- fix: degrade python runtime initialization failures
  iamlinjunhong
  committed
  ec4fed1
- fix: clean up compose BVT services
  iamlinjunhong
  committed
  251d7b1
- test: execute scalar Python UDF BVT case
  iamlinjunhong
  committed
  24a9f64
- fix(udf): validate overload namespaces and cancel capability waits
  iamlinjunhong
  committed
  54696b5
- test(udf): supply NULL input for scalar control-flow assertion
  iamlinjunhong
  committed
  cb19161
- fix(udf): preserve exact DROP and failed initialization ownership
  iamlinjunhong
  committed
  836191c
- docs(udf): document execution and rollout scope for review
  iamlinjunhong
  committed
  d196887
- fix(ci): add missing license headers for Python UDF files
  iamlinjunhong
  committed
  4e23c79
- fix(udf): satisfy static checks and import ownership
  iamlinjunhong
  committed
  87a5648
- fix(ci): align Python UDF runtime dependencies
  iamlinjunhong
  committed
  c4d18e3
- fix(udf): integrate concurrent CI contract repair
  iamlinjunhong
  committed
  b68fb3f
- fix(udf): remove duplicate frontend import
  iamlinjunhong
  committed
  c510d1d
- fix(udf): preserve Python UDF test dependencies
  iamlinjunhong
  committed
  8dbc08e
- test(optools): isolate Python dependency probe in scheduler harness
  iamlinjunhong
  committed
  46b8872
- test(bootstrap): preserve historical upgrade worker fixture
  iamlinjunhong
  committed
  6abf5f0
- test(udf): cover CI contracts and failure paths
  iamlinjunhong
  committed
  e6f4c58
- fix(optools): retain Python runtime loader path
  iamlinjunhong
  committed
  c4ed457
- docs(udf): clarify test-stage acceptance scope
  iamlinjunhong
  committed
  79765f2
- docs(udf): approve current-stage design contract
  iamlinjunhong
  committed
  31a65af
- docs(udf): link design approval record
  iamlinjunhong
  committed
  720bea0
- docs(udf): record current-stage design approvals
  iamlinjunhong
  committed
  ea5905c
- fix(optools): keep watchdog diagnostics out of UT report
  iamlinjunhong
  committed
  4eeacb6
- fix(udf): preserve recovered catalog and protocol repair
  iamlinjunhong
  committed
  3140cf9
- fix(udf): close catalog and execution protocol admission
  iamlinjunhong
  committed
  2e30885
- fix(udf): close catalog repair admission
  iamlinjunhong
  committed
  166c0eb
- fix(udf): regenerate merged mysql parser
  iamlinjunhong
  committed
  f8f068f
- test(bootstrap): pin historical compensation fixtures
  iamlinjunhong
  committed
  74e15a6
- test(udf): clean rebase upgrade imports
  iamlinjunhong
  committed
  5c24461
- fix(udf): preserve routine catalogs across restore
  iamlinjunhong
  committed
  bdd7486
- test(frontend): mock routine catalog restore schema
  iamlinjunhong
  committed
  516694d
- docs(udf): clarify independent approval status
  iamlinjunhong
  committed
  8437158
- fix(udf): assign fresh protocol after main rebase
  iamlinjunhong
  committed
  c33e229
- fix(udf): preserve official catalog migrations and add v4.0.14 repair
  iamlinjunhong
  committed
  8b57366
- fix(udf): allocate protocol after upstream rebase
  iamlinjunhong
  committed
  ab69051
- optools: target embedded watchdog in stderr regression test
  iamlinjunhong
  committed
  9b784c2
- fix(udf): allocate MORPC 110 after upstream rebase
  iamlinjunhong
  committed
  396d4d9
- test(optools): isolate embedded watchdog fixture by platform
  iamlinjunhong
  committed
  81fe97b
- fix(udf): repair revision restore and preserve published upgrade handlers
  iamlinjunhong
  committed
  d5edfe8
- fix(udf): preserve protocol and integer contracts after rebase
  iamlinjunhong
  committed
  4774020
- docs(udf): record r8 contract after upstream rebase
  iamlinjunhong
  committed
  727ded9
- fix(udf): reconcile Python protocol with index search scans
  iamlinjunhong
  committed
  0611597
- docs(udf): align active protocol references with MORPC 112
  iamlinjunhong
  committed
  c3615ec
- test: isolate embedded admission release channels after main rebase
  iamlinjunhong
  committed
  a0072f0
- fix(udf): advance Python protocol admission after calendar contracts
  iamlinjunhong
  committed
  cfa1172
- fix(udf): allocate MORPC 114 after main rebase
  iamlinjunhong
  committed
  ccd54a4
- fix(udf): allocate MORPC 115 and refresh planner fixtures
  iamlinjunhong
  committed
  f193a6d
- fix(udf): allocate MORPC 116 after upstream rebase
  iamlinjunhong
  committed
  4cf0943
- fix(udf): allocate MORPC 117 after upstream rebase
  iamlinjunhong
  committed
  f831627
- fix(udf): allocate MORPC 118 and isolate Python test helpers
  iamlinjunhong
  committed
  1c87b71

Commits

Commits on Oct 17, 2026