Skip to content

CI: isolated UT Coverage package exceeds 10-minute deadline during cluster shutdown #29368

Description

@XuPeng-SH

Summary

The pkg/tests/issues/isolated package exhausts its 10-minute go test package deadline in UT Coverage across several unrelated open PRs. The package budget is shared by multiple embedded-cluster tests, so the final running test name is not by itself the cause.

Evidence

Investigation and acceptance

  1. Reproduce or compare against clean main using the same UT Coverage mode and package parallelism; collect per-test timings and the full shutdown goroutine graph.
  2. Determine whether the heartbeat MySQL I/O can outlive CN shutdown and, if so, make the owner cancel/close it so Stopper.Stop terminates. Do not infer that network read is the root cause from one stack alone.
  3. Reduce or isolate the expensive embedded-cluster fixture work so the package finishes with margin under its configured deadline, while preserving each distinct test contract. Avoid simply raising the timeout without resolving the shutdown path and budget.
  4. Verify the affected lifecycle case and the owning package in coverage mode, then a complete UT Coverage producer run.

Activity

  1. XuPeng-SH commented on Sep 24, 2026

    @XuPeng-SH
    ContributorAuthor

    Root cause update: the Coverage workflow gives its whole test phase 3600 seconds, but invokes go test without -timeout, leaving Go's default 600-second deadline on each package. In the failed #29365 report, 11 completed isolated tests consumed 574.35 seconds before TestSpeedupAbortAllTxn began; the package alarm followed 27 seconds later. The final test's shutdown stack is not evidence that its MySQL read caused the 10-minute alarm.

    The CI budget fix is now proposed in matrixorigin/CI#459 (-timeout=20m per package). The separate cancellation ownership concern visible in that stack is recorded as #29371. This issue remains open for validation of the CI change and for reducing the two expensive embedded-cluster fixtures (301.61s and 157.35s in the failed run).

  2. self-assigned this
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions