Skip to content

CI: install Python UDF worker dependencies for host coverage tests #29201

Closed
Closed
CI: install Python UDF worker dependencies for host coverage tests#29201
Assignees
iamlinjunhong
Labels
kind/bugSomething isn't workingneeds-triageNeeds evaluation before prioritization. Not yet decided whether to proceed

Description

@iamlinjunhong
iamlinjunhong
opened on Sep 21, 2026

Summary

The host UT and coverage jobs for MatrixOne PR #29152 fail before the Python UDF worker can listen because the hosted test environment does not install the checked-out worker requirements. The stable error is:

ModuleNotFoundError: No module named 'pyarrow'
dial tcp 127.0.0.1:<ephemeral-port>: connect: connection refused

The connection errors are follow-on failures after the worker exits during import.

Source PR and failed checks

The exact run is 35549200935, attempt 1. The complete captured logs show:

  • UT Coverage failed in pkg/udf/python; six real-worker tests and TestRealWorkerStopRecordsCompletion observe the missing pyarrow import and then refused worker connections.
  • Coverage received prerequisites_ready=false and failed with coverage merge blocked: a required producer did not publish a complete coverage profile; no coverage verdict was calculated. No coverage verdict was calculated.
  • CI Required only reports matrixone-ut-coverage: failure.

The runner diagnostics reported 15 GiB memory with substantial free memory and 46 GiB free disk; no OOM or disk-exhaustion signature was present.

Evidence that the failure is in the shared CI setup

The failing job downloaded the shared workflow from the main branch of matrixorigin/CI at commit c76c48c66de9e4684696a02510835a21d4a0aad9. The MatrixOne PR changes the Python UDF worker requirements and real-worker tests, but does not change a GitHub workflow.

The independent upstream fix is already prepared in matrixorigin/CI#457 at head 6f7d4ea2fd90e51e63e21fac9b1afc0c73ed95be. Its exact diff adds the setup action to the existing host UT and coverage jobs and installs pkg/udf/python/worker/requirements.txt in a job-owned Python 3.12 environment, verifies pyarrow.flight, and exports that interpreter for Go subprocesses. The upstream PR is still open and requires review, so the hosted jobs cannot receive this setup yet.

Reproduction

  1. Run the existing host UT or coverage workflow against a checkout containing pkg/udf/python/worker/requirements.txt before the setup action is activated.
  2. Start the real Python worker test path.
  3. The worker imports pyarrow, exits with ModuleNotFoundError, and the Go test reports connection refused for the worker endpoint.
  4. Coverage then receives no complete UT producer profile and reports prerequisites_ready=false.

Resolution

Activate the shared CI setup from CI PR #457, then rerun the failed jobs for MatrixOne run 35549200935 once. This issue is limited to the CI dependency setup; it does not propose weakening or skipping the Python UDF contract tests.

Activity

  1. iamlinjunhong
    added
    kind/bugSomething isn't working
    needs-triageNeeds evaluation before prioritization. Not yet decided whether to proceed
    on Sep 21, 2026

  2. iamlinjunhong commented on Sep 21, 2026

    @iamlinjunhong
    iamlinjunhong
    on Sep 21, 2026
    ContributorAuthor

    Evidence update for the authorized failed-job rerun of MatrixOne run 35549200935.

    Exact head: 6091a55bc442a098a149a17f278f4a2ac08d4823; attempt 2 was accepted and completed with failure. Complete failed log was captured from gh run view 35549200935 --attempt 2 --log-failed (SHA-256 a82e629f28ea42d386a54a29b633112e68071539e30ec6c33017eca76353e86f).

    • UT Coverage job 106272910373, failed step Fail on coverage UT failure: six real-worker tests plus TestRealWorkerStopRecordsCompletion failed. Each worker exited at worker.py:48 with ModuleNotFoundError: No module named 'pyarrow'; the Go tests then reported connection refused and the step exited 1.\n- Coverage job 106279990992, failed step Verify coverage prerequisites: prerequisites_ready: false, followed by coverage merge blocked: a required producer did not publish a complete coverage profile; no coverage verdict was calculated.\n- CI Required job 106280051496, failed step Run actions/github-script@v7: aggregation reported matrixone-ut-coverage: failure and matrixone-coverage-merge: failure.\n\nThe runner diagnostics showed 15 GiB memory, 13 GiB available memory, 46 GiB free disk, and no OOM signal. PR Rebuild Python UDF with typed revisions and Arrow Flight execution #29152 still contains no .github or .ci-workflow change. The shared setup correction remains matrixorigin/CI#457, currently open and awaiting review; it installs the checked-out Python UDF worker requirements in the host jobs. No MatrixOne source change or additional rerun is requested by this update.

  3. iamlinjunhong commented on Sep 22, 2026

    @iamlinjunhong
    iamlinjunhong
    on Sep 22, 2026
    ContributorAuthor

    Evidence update for MatrixOne run 35602821586 at the current PR head.

    The complete failed logs show six real-worker Python UDF tests failing at worker.py:48 with ModuleNotFoundError: No module named 'pyarrow', followed by connection refused; Coverage then reports prerequisites_ready: false and blocks the merge because no complete producer profile was published; CI Required aggregates the producer and merge failures. The MatrixOne PR changes no CI workflow.\n\nThis matches the existing shared-host setup defect tracked here. The independent correction remains matrixorigin/CI#457, which installs the checked-out worker requirements and verifies pyarrow.flight in the existing host UT and coverage jobs. The current run will be rerun once under the authorized CI issue procedure; no test is being skipped and no MatrixOne runtime dependency change is proposed.

  4. iamlinjunhong
    self-assigned this
    on Sep 22, 2026

  5. iamlinjunhong commented on Sep 22, 2026

    @iamlinjunhong
    iamlinjunhong
    on Sep 22, 2026
    ContributorAuthor

    Evidence update for the current exact-head CI run 35689522410 (attempt 1).

    The complete failed logs were read from the three jobs. The failure signatures are:

    • UT Coverage job 106623493880, failed step Fail on coverage UT failure: six real-worker tests in pkg/udf/python failed. Each worker exited at pkg/udf/python/worker/worker.py:48 with ModuleNotFoundError: No module named 'pyarrow'; the Go tests then reported connection refused. There were no build failures.
    • Coverage job 106633088525, failed step Verify coverage prerequisites: prerequisites_ready: false, followed by coverage merge blocked: a required producer did not publish a complete coverage profile; no coverage verdict was calculated.
    • CI Required job 106634500500, failed step Run actions/github-script@v7: aggregation reported matrixone-ut-coverage: failure and matrixone-coverage-merge: failure.

    The UT job fetched matrixorigin/CI shared workflow commit c76c48c66de9e4684696a02510835a21d4a0aad9 from main. MatrixOne PR #29152 adds the worker requirements and real-worker tests, but does not modify .github or .ci-workflow; its optools/run_ut.sh dependency fallback is not used by this shared coverage workflow. The independent setup correction remains matrixorigin/CI#457, which installs the checked-out worker requirements in the host UT/coverage jobs and exposes the prepared interpreter. This is the same open CI setup defect tracked here; no MatrixOne runtime workaround, skipped test, or weakened assertion is proposed.

    The failed jobs for run 35689522410 will be rerun once after this evidence update, as authorized by the intake procedure.

  6. iamlinjunhong
    closed this as completedon Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't workingneeds-triageNeeds evaluation before prioritization. Not yet decided whether to proceed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions