Skip to content

Guard the stack when checking long sequence expressions - #20480

Open
xperiandri wants to merge 3 commits into
dotnet:mainfrom
xperiandri:fix/seq-expr-stack-guard
Open

Guard the stack when checking long sequence expressions#20480
xperiandri wants to merge 3 commits into
dotnet:mainfrom
xperiandri:fix/seq-expr-stack-guard

Conversation

@xperiandri

Copy link
Copy Markdown
Contributor

Description

Checking a long seq { … } body overflows the stack. TcSequenceExpression walks the expr; rest spine with tcSequenceExprBody → tcSequenceExprBodyAsSequenceOrStatement → tryTcSequenceExprBody, one recursion per element, and none of those three functions is stack-guarded — only the leaves are, via TcExpr. Once the spine has consumed a thread down to the StackGuard margin every leaf hops to a fresh thread and returns, while the spine keeps growing on the original thread until it hits the guard page. The crash therefore lands a few frames past a leaf, inside the constraint solver:

   at FSharp.Compiler.ConstraintSolver.PostponeOnFailedMemberConstraintResolution(...)
   at FSharp.Compiler.ConstraintSolver.AddCxTypeEqualsType(...)
   at FSharp.Compiler.CheckSequenceExpressions+tcSequenceExprBodyAsSequenceOrStatement@415.Invoke(...)
Repeated N times:
   at FSharp.Compiler.CheckSequenceExpressions+tryTcSequenceExprBody@44.Invoke(...)
   at FSharp.Compiler.CheckSequenceExpressions+tcSequenceExprBodyAsSequenceOrStatement@415.Invoke(...)
   at FSharp.Compiler.CheckSequenceExpressions+tcSequenceExprBody@405.Invoke(...)

Found with a 183-element data file (ISO language codes as record literals in a seq) that crashed a Debug VSIX in Visual Studio. Measured on a thread-pool thread with FSharpChecker.ParseAndCheckFileInProject over seq { { Name = "n1"; Id = 1 }; … }:

Compiler Elements that overflow
This tree, Debug 183
This tree, Release between 400 and 600
SDK 11.0.100-preview.6 3000 (1000 passes)

The fix runs tcSequenceExprBodyAsSequenceOrStatement — the funnel every spine step goes through — under cenv.stackGuard.Guard, the same shape as TranslateComputationExpression in CheckComputationExpressions.fs.

With the spine guarded the next wall is post-inference checking: the checked seq is a right-nested Seq.append (Seq.singleton e1) (Seq.delay (fun () -> …)) tree, and CheckNoReraise walked it with freeInExpr CollectLocals, whose stackGuard is NoneaccFreeInExprNonLinearImpl then recursed once per element (676 frames in Debug):

Repeated 676 times:
   at FSharp.Compiler.TypedTreeOps.ExprFreeVars.accFreeInExprNonLinearImpl(FreeVarOptions, Expr, FreeVars)
   at FSharp.Compiler.TypedTreeOps.ExprFreeVars.accFreeInExprs(...)
   at FSharp.Compiler.TypedTreeOps.ExprFreeVars.accFreeInExprNonLinearImpl(...)
   at FSharp.Compiler.PostTypeCheckSemanticChecks.CheckNoReraise(cenv, FSharpOption`1<FreeVars>, Expr)

CheckNoReraise now uses CollectLocalsWithStackGuard(), as CheckEscapes in the same file already does. The other freeInExpr CollectLocals call sites are in the optimizer and IlxGen, outside the IDE checking path, and are left alone here.

Checklist

  • Test cases added — Long sequence of implicit yields does not overflow the stack in Language/SequenceExpressions/SequenceExpressionTests.fs (5000 elements; crashes the test host without the fix in both Debug and Release)
  • Performance benchmarks added in case of performance changes — n/a
  • Release notes entry updated — docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

`TcSequenceExpression` recursed once per `expr; rest` node of a `seq { }`
body through `tcSequenceExprBody`, `tcSequenceExprBodyAsSequenceOrStatement`
and `tryTcSequenceExprBody` with no stack guard on that spine; only the
leaves were guarded, via `TcExpr`. On a 1 MB thread-pool thread a few
hundred (Debug) to a few thousand (Release) implicit-yield elements ran
the thread out of stack a few frames past a leaf.

With the spine guarded, `CheckNoReraise` then walked the right-nested
`Seq.append`/`Seq.delay` tree with `freeInExpr CollectLocals`, whose
`stackGuard` is `None`, and overflowed in `accFreeInExprNonLinearImpl`.

Run `tcSequenceExprBodyAsSequenceOrStatement` under `cenv.stackGuard` and
give `CheckNoReraise` the guarded free-variable options `CheckEscapes`
already uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

❗ Release notes required

You can open this PR in browser to add release notes: open in github.dev


✅ Found changes and release notes in following paths:

Change path Release notes path Description
`src/Compiler` docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@xperiandri

Copy link
Copy Markdown
Contributor Author

Two related things found while verifying this, both pre-existing and deliberately not addressed here:

1. The same input still overflows in codegen (fsc/fsi, not the IDE check). With both guards in place, a 1000-element seq compiled by the Debug fsi dies in LowerSequenceExpressions, which walks the right-nested Seq.delay tree with isVarFreeInExprfreeInExpr CollectTyparsAndLocals (TypedTreeOps.Transforms.fs), again with stackGuard = None:

Repeated 658 times:
   at FSharp.Compiler.TypedTreeOps.ExprFreeVars.accFreeInExprNonLinearImpl(FreeVarOptions, Expr, FreeVars)
--------------------------------
   at FSharp.Compiler.TypedTreeOps.SeqExprPatterns.isVarFreeInExpr(Val, Expr)
   at FSharp.Compiler.TypedTreeOps.SeqExprPatterns.|SeqDelay|_|(TcGlobals, Expr)
   at FSharp.Compiler.LowerSequenceExpressions+ConvertSeqExprCode@158.Invoke(...)
   at FSharp.Compiler.LowerSequenceExpressions.ConvertSequenceExprToObject(TcGlobals, ImportMap, Expr)
   at FSharp.Compiler.IlxGen.GenExprPreSteps(cenv, CodeGenBuffer, IlxGenEnv, Expr, sequel)

There are ~25 more freeInExpr CollectLocals / CollectTyparsAndLocals call sites in the optimizer and IlxGen with the same exposure. The systemic fix would be a shared StackGuard inside the default FreeVarOptions presets (safe now that #18971 made the depth counter ThreadLocal), but that puts a guard on the hottest traversal in the compiler and wants --times numbers before/after, so it belongs in its own PR.

2. Type checking a long seq is superlinear. Measured with FSharpChecker.ParseAndCheckFileInProject on a Debug build: 1000 elements → 16 s, 2000 → 258 s. It was the same before this change (the 2000-element run took ~5 min before overflowing), so it is not caused by the guard; stripTyparEqnsAux chains show up at the top of several traces, which points at the per-element UnifyTypes genOuterTy (mkSeqTy freshTypar) in the implicit-yield branch. Also a separate issue.

Happy to open issues for both if maintainers prefer tracking them that way.

@github-actions github-actions Bot added the AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files label Sep 7, 2026
@github-project-automation github-project-automation Bot moved this from New to In Progress in F# Compiler and Tooling Sep 8, 2026
Comment thread docs/release-notes/.FSharp.Compiler.Service/11.0.100.md Outdated
@T-Gro
T-Gro enabled auto-merge (squash) September 8, 2026 07:40
Co-authored-by: Tomas Grosup <tomasgrosup@microsoft.com>
auto-merge was automatically disabled September 8, 2026 11:53

Head branch was pushed to by a user without write access

@T-Gro T-Gro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖🕵️ LGTM ✅

Dimensions covered (expand)
  • Checking, stack guards and cancellation context
  • Test and diagnostic fidelity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants