Fix CI Test Summary silently dropping failed tests on trx filename collision#1946
Conversation
Without an explicit LogFileName, VSTest names trx files only
{hostname}_{timestamp}_{framework}.trx, and GitHub-hosted runners can
reuse short hostname prefixes across concurrent matrix jobs. When the
Test Summary job downloads all per-job artifacts with
merge-multiple: true, two same-named trx files can collide during
extraction and corrupt one of them, which Write-CategorySummary then
silently drops with only a log warning \(not surfaced in the rendered
summary\), causing that job's failed tests to vanish from the report.
Naming each trx file after its full matrix key removes the collision
at the source.
There was a problem hiding this comment.
Pull request overview
This PR fixes a CI reporting reliability issue where the test-summary job could silently omit failed tests due to .trx filename collisions across concurrent matrix jobs. It does so by making each dotnet test --logger trx output file name deterministic and unique per matrix key, preventing collisions during actions/download-artifact@v8 with merge-multiple: true.
Changes:
- Updated all three CI
dotnet testinvocations (standalone, cluster, Tsavorite) to use--logger "trx;LogFileName=${{ matrix.os }}-${{ matrix.framework }}-${{ matrix.configuration }}-${{ matrix.test }}.trx". - Ensured TRX filenames align with the existing artifact naming convention, eliminating cross-job filename collisions at the source.
|
CI ran green except for one job: |
|
Update: filed #1950 with a reliable repro (~50% hit rate over 18 local runs) and analysis — this looks like a real, previously-unreported recovery data-loss bug, not simple flakiness. |
Fixes #1944
Root cause
None of the three
dotnet test --logger trxinvocations in.github/workflows/ci.yml(standalone/cluster/tsavorite matrix jobs) passLogFileName=, so VSTest names each.trxfile only{hostname}_{date}_{time}_{framework}.trx. GitHub-hosted runners frequently reuse short hostname prefixes, so two concurrent matrix jobs on the same framework can end up producing identically-named trx files.In the
test-summaryjob,actions/download-artifact@v8runs withmerge-multiple: true, flattening every per-job artifact for a category (standalone/cluster) into one shared directory. When two artifacts contain a same-named file, the concurrent extraction corrupts it (interleaved/truncated XML rather than a clean overwrite).Write-CategorySummary'stry { $xml.Load(...) } catch { Write-Warning "Skipping malformed TRX file..." }then silently drops that file — the warning only goes to the raw job log, never into$GITHUB_STEP_SUMMARY— so that job's failed tests vanish from the rendered summary with no indication anything was skipped.Evidence from the run referenced in #1944 (https://github.com/microsoft/garnet/actions/runs/29697667836?pr=1825)
Garnet Standalone (ubuntu-latest, net10.0, Release, Garnet.test)andGarnet Cluster (windows-latest, net10.0, Release, Garnet.test.cluster.vectorsets)._runnervm3jd5f_2026-07-19_17_57_27_net10.0.trx.test-summaryjob's log contains:WARNING: Skipping malformed TRX file: .../results/standalone/_runnervm3jd5f_2026-07-19_17_57_27_net10.0.trx - Exception calling "Load" ... "The 'TestDefinitions' start tag on line 142 ... does not match the end tag of 'Results'."— an exact filename match to the failing standalone job's own trx, confirming it collided with another job's file duringmerge-multipledownload and was corrupted.Fix
Give each matrix job's trx file a name derived from its full matrix key (
os-framework-configuration-test), matching the existing artifact-naming convention already used a few lines below foractions/upload-artifact. This removes the collision at the source rather than working around it downstream.Testing
This is a CI workflow change, so it can't be unit-tested locally — the real verification is this PR's own CI run using the new trx naming. I validated the YAML remains syntactically valid (
ruby -ryaml) and that--logger "trx;LogFileName=..."is the documented VSTest syntax for overriding the trx output filename. I don't have a way to force a same-hostname collision on demand to reproduce the original symptom directly, so confidence here rests on the log evidence above (the corrupted file's name exactly matching the missing job's own trx) rather than a live before/after repro.