[Apple][CoreCLR] JIT/Directed/tailcall/mutual_recursion: fatal stack overflow from tail calls across R2R/interpreter transitions

[lewing]

Description

On Apple mobile CoreCLR (reproduced on maccatalyst-arm64, Release, ReadyToRun), JIT/Directed/tailcall/mutual_recursion crashes the Directed_do merged runner with a fatal stack overflow.

The test is compiled with F# tail calls (--tailcalls+) and depends on them. Its mutually recursive calls alternate between ReadyToRun code and interpreted code, and the tail calls are not honored across those transitions, so the native stack grows until the process aborts:

EEPolicy::HandleFatalStackOverflow
HandleHardwareException
...
InterpExecMethod
ExecuteInterpretedMethod
InterpreterStubRetI8
CallJittedMethodRetI8
InterpExecMethod
ExecuteInterpretedMethod
InterpreterStubRetI8
CallJittedMethodRetI8
... (repeats)

The test previously failed earlier on Apple, with FileNotFoundException for FSharp.Core, because the app bundle didn't include FSharp.Core. It now reaches this crash once FSharp.Core is bundled (#134767).

Configuration

• Fails: maccatalyst-arm64 CoreCLR, Release, ReadyToRun (no JIT; code that isn't precompiled runs in the interpreter).
• Passes: android-arm64 CoreCLR on the emulator, which has a JIT (Directed_do: 41 passed, 0 failed).
• Related but closed: [wasm][R2R] mutual_recursion crashes with SIGSEGV #133464 (browser-wasm R2R SIGSEGV in the same test).

Expected behavior

Tail calls between R2R and interpreted code don't grow the stack, and the test returns 100.

Notes

xharness reported SUCCESS for this crashing run: testEndSignalDetected was false and no results file was written. That will be tracked separately.

Note

This issue was drafted with GitHub Copilot.

[dotnet-policy-service]

Tagging subscribers to 'os-maccatalyst': @vitek-karas, @kotlarmilos, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @BrzVlad, @janvorli
See info in area-owners.md if you want to be subscribed.