Test failure: GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd

[VincentBu]

Failed in: runtime-coreclr outerloop 20260809.3

Failed tests:

coreclr browser wasm Checked no_tiered_compilation @ (Ubuntu.2604.Amd64.Open)AzureLinux.3.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-26.04-helix-webassembly-amd64@sha256:0be8725e08b6d0f58ec0290f3f3b70c449123e78b2b33853e9049df84b912fa8
- GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd

Error message:

Out-of-process wrapper failed with exit code 1.
BEGIN EXECUTION
Running with node
CORE_ROOT: /root/helix/work/correlation
ExePath: /root/helix/work/workitem/e/GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.dll
CLRTestExecutionArguments:
node version: v23.11.1
Timeout in ms: 900000
Running: node --experimental-wasm-exnref --stack-size=8192 /root/helix/work/correlation/corerun.js -c /root/helix/work/correlation -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true /root/helix/work/workitem/e/GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.dll
Using defaults: 1 1200 100000
Using random seed: 246730784
Took 13950 iterations to reach steady state
OOM
3841357900
Test Failed
Kill watchdog with PID 225
Expected: 100
Actual: 1
END EXECUTION - FAILED

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/arcade/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 135
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/arcade/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 117
   at TestLibrary.OutOfProcessTest.ImportPrecomputedResult(String assemblyPath, String testPathPrefix) in /__w/1/s/src/tests/Common/CoreCLRTestLibrary/OutOfProcessTest.cs:line 285
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix) in /__w/1/s/src/tests/Common/CoreCLRTestLibrary/OutOfProcessTest.cs:line 84
   at Program.<<Main>$>g__TestExecutor15|0_16(StreamWriter tempLogSw, StreamWriter statsCsvSw, StreamWriter outOfProcessPlanWriter, <>c__DisplayClass0_0&)

Known Issue Error Message

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Failed test: GC/Features/HeapExpansion/bestfit-finalize",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1545892
Error message validated: [Failed test: GC/Features/HeapExpansion/bestfit-finalize]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/24/2026 11:48:43 AM UTC

Report

Build	Repository	Test	Pull Request
1616793	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134549
1616475	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134527
1616033	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134838
1615904	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134745
1615704	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#132346
1615302	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133265
1615259	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134767
1615080	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134812
1614856	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134284
1614770	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134802
1613957	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133222
1613570	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134729
1613393	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134708
1613197	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#132827
1612573	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1612496	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134108
1612229	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134456
1612090	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134644
1612009	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134527
1611907	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1611772	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134606
1611567	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134387
1611560	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134211
1611422	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134618
1611410	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134560
1611331	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1610287	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134568
1609945	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1609930	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134078
1609811	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134456
1609647	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133265
1609497	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134210
1609481	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134360
1609375	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134312
1609247	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134530
1609039	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134520
1608825	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134502
1608697	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134505
1608554	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1608468	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#131784
1608111	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134471
1608069	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134284
1607863	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134464
1607576	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133986
1606964	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1606549	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133384
1605657	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#132584
1605240	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1604773	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134323
1604307	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1604235	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1604222	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134309
1604207	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133529
1604122	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1604109	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1604099	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1604050	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#132346
1604010	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133656
1603831	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134287
1603793	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134049
1603735	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133188
1603718	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#130079
1603138	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1603094	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134210
1602809	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1602798	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134210
1602582	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#129369
1602553	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1602351	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134121
1602092	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#132260
1601983	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134168
1601800	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134108
1601541	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133662
1601393	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134152
1601196	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134084
1601130	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134010
1600957	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133934
1600360	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134111
1600235	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134113
1599931	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134071
1599469	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#134073
1598267	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1597921	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133977
1597910	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133976
1597894	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133970
1597684	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133827
1597497	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133696
1597202	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1597203	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133603
1596363	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133907
1596317	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1596309	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133773
1595848	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133879
1595718	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133878
1595676	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#132081
1595348	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#129683
1595295	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd
1595144	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133571
1594854	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133833
1594849	dotnet/runtime	GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.cmd	#133834
Displaying 100 of 245 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	40	245

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[JulieLeeMSFT]

@AndyAyersMS, this failure is affecting many pipelines. PTAL.

[tannergooding]

The browser CoreCLR OOM is reproducible and appears to be finalizer starvation, rather than excessive live-request memory or a timeout.

I replayed the normal/non-R2R failure from build 1593908 on #133774 using the original test DLL and browser Wasm Checked runtime from its GC-features Helix payload. The local host was Windows with Node 23.11.1 and tiering disabled, matching the CI Node version and scenario, but not its Linux container.

With unchanged arguments 1 1200 100000 1094640032, the original binary reaches steady state after 15,970 requests, then throws OOM with GC.GetTotalMemory(false) reporting 3,841,467,072 bytes. The CI log reports 3,841,067,724 bytes.

The relevant scheduling behavior is:

• Each Request retains its survivor arrays and a pinned GCHandle; its finalizer releases the handle.
• Go() synchronously constructs requests until five times the steady-state count: 79,850 for this seed.
• Browser FinalizerThread::EnableFinalization defers execution to the host. SystemJS_ScheduleFinalization uses safeSetTimeout(..., 0).
• This synchronous loop never returns control to the JS event loop. Collections continue, but the scheduled finalizers do not run, leaving dead requests' object graphs and pinned handles retained.

An instrumented copy counted completed constructors and finalizers (the latter after pin.Free()), sampling memory once per 1,200 requests:

Run	Constructed requests	Finalizers observed	Managed heap	Result
No scheduling change	59,739	0	3,841,635,812 bytes at OOM	Exit 1
Drain pending finalizers every 1,200 requests	79,850	76,547	376,938,324 bytes at the last observation; maximum sampled value 405,719,424 bytes	Exit 100

The control performed 291 full collections without executing a single Request finalizer. The drain run still used natural collections; no GC.Collect() was added. The sampled maximum is not a continuous peak measurement, and the finalizer counts were observed before post-loop cleanup.

An applicable test workaround is to add this immediately after requests[i] = new Request():

if (OperatingSystem.IsBrowser() && (totalReqs % NumRequests == 0))
{
    // Browser finalizers cannot run while this synchronous loop holds the JS thread.
    // Drain once per live-set-sized batch so dead requests can release their pins
    // and survivor arrays, without forcing collections.
    GC.WaitForPendingFinalizers();
}

On this runtime, WaitForPendingFinalizers drains synchronously outside the collection; it does not require returning to JavaScript. This preserves the random sequence, request count, allocation volume, survivor selection, pinning, and existing success/failure conditions.

The uninstrumented workaround passed the failing seed, seed 0, and a two-outer-iteration run with the original 1200 100000 workload. Original and patched versions also passed on desktop. These were direct test-source compilations against the downloaded CI runtime, not a fresh runtime build or a rerun of the full Helix harness.

This is a workaround for the test, not a general runtime fix: ordinary synchronous user allocation loops can encounter the same lack of finalizer progress. The reproduced failure is on CoreCLR Wasm; these experiments do not establish Mono Wasm behavior or a RyuJIT codegen defect.

Note

This analysis and workaround were drafted by GitHub Copilot from source inspection and local reproduction results.