Skip to content

scripts: the issue_workspace exact-match test fails when a fixture commit's SHA contains 570 (flaky NVX CLI) #1665

Closed

Description

Summary

scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source fails whenever the 12-character SHA prefix of either commit that its fixture creates contains 570. Each time, it fails with AssertionError: True is not false at self.assertFalse(any("570" in name for name in names)), and nothing else in the job fails. In the CI runs created from 2026-11-06 to 2026-11-16, it failed 6 of the 1,554 NVX CLI jobs that completed, 4 on Linux and 2 on Windows. That is about 1 in 260, and 6 of the 21 NVX CLI jobs that failed.

The test checks that issue_workspace.find_issue, when it looks for issue 57, reports none of the fixture's decoys for issue 570. It does so by searching every candidate's name for 570. But the tool names a commit candidate after its SHA prefix, and the fixture's commit SHAs change with the time at which the test runs. A prefix contains 570 with a probability of about 0.24%, so with two commits the test fails about once in 205 runs. Pinning the fixture's commit dates reproduces the failure every time. The tool excludes the decoys correctly, so the defect is in the test.

Incident

  • Workflow: CI run 43902030265, attempt 1, push to feature/nvx-refactor (nvx: separate VM mechanisms from runtime policy #1651) at 263d9ee8dff.
  • Failed job: NVX CLI / Windows, step Validate NVX CLI. It failed in the python -m unittest command of the validate-nvx action's Windows step (action.yml#L25-L34). Of 2,213 tests, 87 were skipped, and this test was the only failure. The run's NVX CLI / Linux job failed for an unrelated reason, which 2b8b51581aa fixed.
  • Runner: GitHub-hosted, image windows-2025-vs2026 20261103.298.1, runner 2.338.2, Python 3.12.10.
  • The test and scripts/issue_workspace.py are byte-identical at all six failing heads and on dev at 28a0c326f5e.

Decisive output:

======================================================================
FAIL: test_find_groups_exact_issue_matches_from_every_source (scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\nvx\nvx\scripts\test_issue_workspace.py", line 283, in test_find_groups_exact_issue_matches_from_every_source
    self.assertFalse(any("570" in name for name in names))
AssertionError: True is not false

----------------------------------------------------------------------
Ran 2213 tests in 679.383s

FAILED (failures=1, skipped=87)

History

I classified the failed NVX CLI jobs of the 1,172 CI runs created from 2026-11-06 to 2026-11-16 18:43 UTC:

  • 783 NVX CLI / Linux jobs completed. 14 failed, 4 of them on this assertion.
  • 771 NVX CLI / Windows jobs completed. 7 failed, 2 of them on this assertion.
  • In each of the 6, this assertion was the job's only failure.
  • The window starts before skills: move deterministic steps into tools, and name skills by task #1337 added the assertion on 2026-11-07, so 6 in 1,554 slightly understates the rate.
Created (UTC) Run Event Head Job Outcome
2026-11-10 01:25 42735627387 push, dev f5ef2dceaee Linux Run failed.
2026-11-14 18:05 43624611183 pull_request, #1591 3e4c9f59ebe Linux Attempt 2 passed; #1591 merged.
2026-11-15 08:26 43694553887 push, dev 45081e3e553 Linux Run failed.
2026-11-16 00:45 43782274623 push, dev 52da532f5e3 Linux Run failed.
2026-11-16 04:23 43803791787 pull_request, #1646 181da97bea4 Windows Attempt 2 passed; #1646 merged.
2026-11-16 15:57 43902030265 push, feature/nvx-refactor (#1651) 263d9ee8dff Windows This incident.

Root cause

Permalinks point to dev at 28a0c326f5e.

  1. The tool names a commit candidate after its SHA. _append_commit_candidates names each commit that mentions the issue f"{commit[:12]} {subject}" (issue_workspace.py#L272-L287). The test's fixture creates two such commits, Fix #57 on dev and Work on issues/57 (test_issue_workspace.py#L236-L239).
  2. The fixture's SHAs change with the clock. _commit (L39-L43) commits with a fixed identity but at the current time, so the SHAs of both commits differ from run to run.
  3. The assertion searches every candidate's name for a bare 570. self.assertFalse(any("570" in name for name in names)) (L283) is meant to catch the decoys: the branch issue-570-unrelated (L241) and pull request Validate the dedicated OpenVMM control-console device #11, Fix #570 (L174-L181). But it also matches a SHA prefix that happens to contain 570. Each of the 10 three-character windows of a 12-character prefix is 570 with a probability of 1 in 4,096, so a prefix contains 570 with a probability of about 0.24%, and either of two prefixes with about 0.49%: about 1 run in 205. The worktree candidate's name, a path under the fixture's random temporary directory (L243), adds about 0.01%.
  4. The tool excludes the decoys. _number_pattern (issue_workspace.py#L227-L228) requires the number to stand alone, so the branch scan (L396-L412) skips issue-570-unrelated. _reference_pattern (L223-L224) does the same for the pull requests' titles and bodies (L326-L338), as the commit searches do for their messages (L432-L438). The test checks pull request Validate the dedicated OpenVMM control-console device #11 separately, with "#11 " (test_issue_workspace.py#L282). The tool keeps a pull request's head ref, here fix-570, only in the candidate's details (issue_workspace.py#L256-L267).

Reproduction

Pinning the fixture's commit dates makes its SHAs deterministic. On Linux (Ubuntu under WSL, Python 3.14.4, Git 2.53.0), with the two files as they are on dev:

GIT_AUTHOR_DATE='1794787201 +0000' GIT_COMMITTER_DATE='1794787201 +0000' \
  python3 -m unittest scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source

This fails 3 times out of 3 with the CI's assertion at line 283, because these dates give the Fix #57 on dev commit the prefix 02e570e6ec14. With 1794787200 +0000, the test passes. Other platforms may produce other SHAs for the same dates.

Impact

Suggested fix

Match the decoy branch's name instead of the bare number:

-        self.assertFalse(any("570" in name for name in names))
+        self.assertFalse(any("issue-570" in name for name in names))
  • issue-570 cannot occur in a hexadecimal SHA prefix, and a temporary directory's random name holds only lowercase letters, digits, and _.
  • With this change, the test passes at both pinned dates above. It still fails at line 283 when _number_pattern loses its (?![0-9]) guard, because the decoy branch then matches issue 57. The "#11 " check on line 282 still covers pull request Validate the dedicated OpenVMM control-console device #11.
  • The change modifies an existing test, so its pull request needs the tests-modified label.

Pinning the commit dates in _commit would also stop the flake, but the assertion would still depend on whichever SHAs those dates produce.

Related work

Metadata

Metadata

Labels

area: ciInitial title-keyword triage suggestion

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions