Skip to content

Fix CI Test Summary silently dropping failed tests on trx filename collision#1946

Merged
badrishc merged 5 commits into
microsoft:mainfrom
hexonal:fix-1944-ci-trx-filename-collision
Jul 23, 2026
Merged

Fix CI Test Summary silently dropping failed tests on trx filename collision#1946
badrishc merged 5 commits into
microsoft:mainfrom
hexonal:fix-1944-ci-trx-filename-collision

Conversation

@hexonal

@hexonal hexonal commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Fixes #1944

Root cause

None of the three dotnet test --logger trx invocations in .github/workflows/ci.yml (standalone/cluster/tsavorite matrix jobs) pass LogFileName=, so VSTest names each .trx file only {hostname}_{date}_{time}_{framework}.trx. GitHub-hosted runners frequently reuse short hostname prefixes, so two concurrent matrix jobs on the same framework can end up producing identically-named trx files.

In the test-summary job, actions/download-artifact@v8 runs with merge-multiple: true, flattening every per-job artifact for a category (standalone/cluster) into one shared directory. When two artifacts contain a same-named file, the concurrent extraction corrupts it (interleaved/truncated XML rather than a clean overwrite). Write-CategorySummary's try { $xml.Load(...) } catch { Write-Warning "Skipping malformed TRX file..." } then silently drops that file — the warning only goes to the raw job log, never into $GITHUB_STEP_SUMMARY — so that job's failed tests vanish from the rendered summary with no indication anything was skipped.

Evidence from the run referenced in #1944 (https://github.com/microsoft/garnet/actions/runs/29697667836?pr=1825)

  • Two jobs failed: Garnet Standalone (ubuntu-latest, net10.0, Release, Garnet.test) and Garnet Cluster (windows-latest, net10.0, Release, Garnet.test.cluster.vectorsets).
  • The failing standalone job's own artifact contains a trx named _runnervm3jd5f_2026-07-19_17_57_27_net10.0.trx.
  • The test-summary job's log contains: WARNING: Skipping malformed TRX file: .../results/standalone/_runnervm3jd5f_2026-07-19_17_57_27_net10.0.trx - Exception calling "Load" ... "The 'TestDefinitions' start tag on line 142 ... does not match the end tag of 'Results'." — an exact filename match to the failing standalone job's own trx, confirming it collided with another job's file during merge-multiple download and was corrupted.
  • The failing cluster job's trx (a different, non-colliding filename) was unaffected, which is exactly why only the cluster failure showed up in the summary and the standalone one silently didn't.

Fix

Give each matrix job's trx file a name derived from its full matrix key (os-framework-configuration-test), matching the existing artifact-naming convention already used a few lines below for actions/upload-artifact. This removes the collision at the source rather than working around it downstream.

Testing

This is a CI workflow change, so it can't be unit-tested locally — the real verification is this PR's own CI run using the new trx naming. I validated the YAML remains syntactically valid (ruby -ryaml) and that --logger "trx;LogFileName=..." is the documented VSTest syntax for overriding the trx output filename. I don't have a way to force a same-hostname collision on demand to reproduce the original symptom directly, so confidence here rests on the log evidence above (the corrupted file's name exactly matching the missing job's own trx) rather than a live before/after repro.

Without an explicit LogFileName, VSTest names trx files only
{hostname}_{timestamp}_{framework}.trx, and GitHub-hosted runners can
reuse short hostname prefixes across concurrent matrix jobs. When the
Test Summary job downloads all per-job artifacts with
merge-multiple: true, two same-named trx files can collide during
extraction and corrupt one of them, which Write-CategorySummary then
silently drops with only a log warning \(not surfaced in the rendered
summary\), causing that job's failed tests to vanish from the report.

Naming each trx file after its full matrix key removes the collision
at the source.
Copilot AI review requested due to automatic review settings July 20, 2026 07:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a CI reporting reliability issue where the test-summary job could silently omit failed tests due to .trx filename collisions across concurrent matrix jobs. It does so by making each dotnet test --logger trx output file name deterministic and unique per matrix key, preventing collisions during actions/download-artifact@v8 with merge-multiple: true.

Changes:

  • Updated all three CI dotnet test invocations (standalone, cluster, Tsavorite) to use --logger "trx;LogFileName=${{ matrix.os }}-${{ matrix.framework }}-${{ matrix.configuration }}-${{ matrix.test }}.trx".
  • Ensured TRX filenames align with the existing artifact naming convention, eliminating cross-job filename collisions at the source.

@hexonal

hexonal commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

CI ran green except for one job: Garnet Standalone (ubuntu-latest, net10.0, Release, Garnet.test), which failed on RespAdminCommandsTests.SeSaveRecoverMultipleKeysTest (a SAVE/RECOVER persistence assertion) — same test, same failure signature I saw flake on an unrelated PR (#1945) a few minutes earlier. This PR only touches trx output filenames in the CI workflow, so this is pre-existing flakiness, not something introduced here.

@hexonal

hexonal commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Update: filed #1950 with a reliable repro (~50% hit rate over 18 local runs) and analysis — this looks like a real, previously-unreported recovery data-loss bug, not simple flakiness.

@badrishc
badrishc merged commit 15d505e into microsoft:main Jul 23, 2026
161 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI bug on Test Summary: Didn't Fully print failed Test

4 participants