Repository navigation
[CI] TestFixturePIDDuringAcknowledgedTERM times out after 200 TERM acknowledgments #29960
Description
UT Coverage intermittently fails in the shell PID/TERM fixture after completing its 200 acknowledgments. This is a non-PR flaky candidate; the underlying wait/cancellation cause still needs investigation.
Source failure:
- PR: fix async ivfflat include and generated column updates #29698
- Exact head: f0489f5
- Run 38197998268, attempt 1: https://github.com/matrixorigin/matrixone/actions/runs/38197998268
- UT Coverage job 114644976296: https://github.com/matrixorigin/matrixone/actions/runs/38197998268/job/114644976296
- Terminal gate: step 18, Fail on coverage UT failure.
- Linux/x64, Go 1.27.1, CGO_ENABLED=1, matrixone_test tag, coverage set mode, no race detector.
- The complete job log contains zero build failures and only this failing test/package. It excludes compilation, type/static checks and security gates.
Stable signature:
=== RUN TestFixturePIDDuringAcknowledgedTERM
ut_schedule_test.go:1564: PID helper signal execution: context deadline exceeded
context deadline exceeded
MAIN 65911
CHILD 65913 65913
TERMS 200
--- FAIL: TestFixturePIDDuringAcknowledgedTERM (60.07s)
FAIL github.com/matrixorigin/matrixone/optools (152.945s)
The failing producer runs:
go test -mod=readonly -count=1 -json -short -v -tags matrixone_test \
-timeout=20m -p "${COVERAGE_UT_PACKAGE_PARALLELISM}" \
-covermode=set -coverprofile="${coverage_profile}" -coverpkg="${cover_pkgs}" \
${test_scope}Attribution evidence:
- Full job logs show a test-owned one-minute context deadline rather than a compile/static/product error.
- The exact failing test passed three consecutive PR-head runs using matching tags, CGO setting and coverage mode. With GNU Bash 5.2.0 on PATH, this exercises the modern BASHPID branch used by Linux CI: test-body times were 0.04s, 0.02s and 0.11s.
- The complete PR diff changes only pkg/iscp and pkg/sql/plan. This test and its cleanup helper import only the Go standard library and execute a Bash PID/TERM/FIFO protocol; they never call the changed ISCP/planner code. The test, helper and CI harness are unchanged by the PR. Exact-head owning ISCP/planner/parser tests, vet and incremental lint passed before delivery.
- Clean current official up/main 5a5501b passed the same modern-Bash command three consecutive times: 0.05s, 0.02s and 0.03s. Local validation was Go 1.27.1 on Darwin/arm64; Linux procfs emergency cleanup was not exercised. Bash 3.2 fallback controls also passed three PR and three main runs, and are recorded separately.
Reproduction/probe command (run three times at each revision, with GNU Bash 5.2 on PATH):
.agents/skills/mo-dev/scripts/mo-cgo-test \
-count=1 -short -v -timeout=90s -tags matrixone_test \
-covermode=set -coverpkg=./optools \
-run '^TestFixturePIDDuringAcknowledgedTERM$' ./optoolsThe fixture is pure Go plus shell and has no production optools statements to instrument; narrowing coverpkg does not remove a kernel dependency. The 90s outer guard still exceeds its unchanged one-minute internal context.
Independent Linux/x64 occurrence:
- Unrelated PR feat(sirius): add complete public SQL GPU acceptance campaign #29862, head 4230ce3.
- https://github.com/matrixorigin/matrixone/actions/runs/38196458750/job/114640155639
- Complete job log shows the same test, ut_schedule_test.go:1564 context deadline, TERMS 200 and 60.06s elapsed.
The observed chain is Bash TERM/FIFO acknowledgments followed by process join/output completion through runScheduleHarnessCommand. TERMS 200 alone does not prove successful process completion, and no specific root cause is claimed. Search found no CI defect issue for this exact timeout class; the Sirius feature trackers only mention the failed case.
Run 38197998268 is still in progress at intake. No rerun has been requested. Plan: rerun its failed jobs once after the entire workflow reaches a terminal state, then verify that GitHub accepted the next attempt. Preserve all fixture assertions; do not change unrelated kernel code to force this PR green.
Activity
iamlinjunhong commentedon Oct 12, 2026
Additional downstream evidence for PR #29698 at exact head f0489f5, run 38197998268 attempt 1:
prerequisites_ready: false. The first prerequisite gate exits 1 withcoverage merge blocked: a required producer did not publish a complete coverage profile; no coverage verdict was calculated. It never calculates a changed-code coverage verdict.iamlinjunhong commentedon Oct 12, 2026
Terminal-run evidence for PR #29698, exact head f0489f5, run 38197998268 attempt 1:
matrixone-ut-coverage: failure. Both are downstream of the same producer failure.Links:
Rerun accepted: GitHub confirms run 38197998268 attempt 2 at the same exact head. The new UT Coverage job is 114667128477: https://github.com/matrixorigin/matrixone/actions/runs/38197998268/job/114667128477. This is the single failed-jobs rerun for this run; its result is pending.
iamlinjunhong commentedon Oct 12, 2026
New occurrence at PR #29698 exact head cd98fbd: run 38211985663, attempt 1, completed with failure.
The complete failed-job logs were inspected. The sole failing test/package is optools/TestFixturePIDDuringAcknowledgedTERM; build failures are zero. The failure occurs at ut_schedule_test.go:1564 after 60.07s:
The package terminates with failure after 150.275s. No compilation/type/static/security failure or stable kernel product error was found in these logs.
Failed jobs:
Coverage has prerequisites_ready=false and exits with
coverage merge blocked: a required producer did not publish a complete coverage profile; no coverage verdict was calculated. CI Required reportsmatrixone-ut-coverage: failure. Both are downstream of the fixture failure; no changed-code coverage verdict was calculated. Regular Ubuntu UT, shared build, arm64 SCA and both active multi-CN BVT lanes passed at this exact head.Fresh attribution evidence:
All six fresh probes used Go 1.27.1, CGO_ENABLED=1, GNU Bash 5.2.0 on PATH to exercise the modern BASHPID branch, matrixone_test tags and set coverage mode, without race (matching this failed lane):
.agents/skills/mo-dev/scripts/mo-cgo-test \ -count=1 -short -v -timeout=90s -tags matrixone_test \ -covermode=set -coverpkg=./optools \ -run '^TestFixturePIDDuringAcknowledgedTERM$' ./optoolsThis is a pure Go/shell fixture with no production optools statements to instrument, so the focused coverpkg removes no kernel dependency. The outer guard exceeds the unchanged one-minute test deadline.
Comparison qualification: current main includes related fixture hardening from merged PR #29862. It moves acknowledgment outside the TERM handler and disarms the EXIT cleanup trap after joining the sender. Its PR description records the inherited fixture failure reproduced on clean official main and 100 coverage plus 100 race focused passes after correcting synchronization. Therefore the new main passes are explicitly current-main controls using that revised fixture, not claims that an identical unfixed fixture reproduces the CI error. The asserted public fixture contract remains 200 TERM acknowledgments plus main/subshell PID checks. Local probes are Darwin/arm64 and do not exercise Linux procfs emergency cleanup.
Independent Linux/x64 occurrences remain linked in this issue; another open tracker #29985 records the exact same case/deadline-after-TERMS-200 class in run 38184149471, job 114602919136, with a passing same-head optools race control. The maintainer record identifies command completion, pipe ownership and Bash wait/EXIT behavior as the investigation boundary. These independent failures and the isolated dependency closure corroborate the non-PR flaky-candidate attribution; no unproven exact blocking-process cause is asserted.
This is a new run at a new head, with no rerun requested yet. After recording these facts, request one failed-jobs rerun of run 38211985663 and verify its accepted attempt. The earlier run 38197998268 already used its single rerun and passed; it will not be rerun again. No source, test assertion, label, issue lifecycle or review state is changed.
Rerun accepted: GitHub confirms run 38211985663 attempt 2, queued at the same exact head cd98fbd. The new UT Coverage job is 114714302940: https://github.com/matrixorigin/matrixone/actions/runs/38211985663/job/114714302940. This is the single failed-jobs rerun for this run; its execution result is pending.
XuPeng-SH commentedon Oct 12, 2026
mo-bot work ·
mo-bot-local· doneCommit:
1abfd6081b494dc6ab99d53805c1cc209883914dFinal review confirms the existing-fix no-change resolution. The candidate contains PR #29862’s fixture correction, matching validation passes, and no material finding remains.
test: repair CI TERM handshakes, port candidates, and TopN coverage (#…