Skip to content

Run feature factories outside the OperationFeatureCollection write lock - #10168

Open
apereiratl wants to merge 2 commits into
ChilliCream:mainfrom
apereiratl:fix/operation-feature-collection-deadlock
Open

Run feature factories outside the OperationFeatureCollection write lock#10168
apereiratl wants to merge 2 commits into
ChilliCream:mainfrom
apereiratl:fix/operation-feature-collection-deadlock

Conversation

@apereiratl

Copy link
Copy Markdown

GetOrSetSafe invoked the caller's factory while holding _writeLock, which lets two execution locks be taken in opposite orders:

Operation.GetSelectionSet takes the operation lock, then CompileSelectionSet writes selection features (OperationCompiler sets the optimizer feature), so it holds the operation lock and wants the feature lock.

GetOrCreateSelectorExpression takes the feature lock, then the factory builds the selector expression, which calls Operation.GetSelectionSet, so it holds the feature lock and wants the operation lock.

Two threads hitting both paths on one operation deadlock permanently. Every later request completing a composite value then blocks on the same operation lock, the pool injects threads to replace them, and the server stops answering. Only the first execution of an operation compiles selection sets and selector expressions, so it takes a schema with many distinct operations to hit reliably, but it needs no unusual input and does not recover.

The factory now runs before the lock is taken, which is entered only to store the result. Racing callers can each build a value, the first stored wins, and every caller receives that instance, matching ConcurrentDictionary.GetOrAdd.

GetOrSetSafe invoked the caller's factory while holding _writeLock, which lets two
execution locks be taken in opposite orders:

  Operation.GetSelectionSet takes the operation lock, then CompileSelectionSet writes
  selection features (OperationCompiler sets the optimizer feature), so it holds the
  operation lock and wants the feature lock.

  GetOrCreateSelectorExpression takes the feature lock, then the factory builds the
  selector expression, which calls Operation.GetSelectionSet, so it holds the feature
  lock and wants the operation lock.

Two threads hitting both paths on one operation deadlock permanently. Every later
request completing a composite value then blocks on the same operation lock, the pool
injects threads to replace them, and the server stops answering. Only the first
execution of an operation compiles selection sets and selector expressions, so it
takes a schema with many distinct operations to hit reliably, but it needs no unusual
input and does not recover.

The factory now runs before the lock is taken, which is entered only to store the
result. Racing callers can each build a value, the first stored wins, and every caller
receives that instance, matching ConcurrentDictionary.GetOrAdd.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request fixes a deadlock risk in OperationFeatureCollection by ensuring GetOrSetSafe executes feature factories outside the collection’s _writeLock, only taking the lock to store the computed value (double-checking for an existing value first). This aligns the behavior with ConcurrentDictionary.GetOrAdd semantics under contention.

Changes:

  • Run GetOrSetSafe factories outside _writeLock for both operation-level and selection-level feature storage, then lock only to publish the winning instance.
  • Add concurrency-focused tests to ensure factories aren’t invoked while holding the write lock and that racing callers observe the same stored instance.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/HotChocolate/Core/src/Types/Execution/Processing/OperationFeatureCollection.cs Updates operation-level GetOrSetSafe to invoke factories outside _writeLock and publish the first-created value.
src/HotChocolate/Core/src/Types/Execution/Processing/OperationFeatureCollection.Selections.cs Applies the same “factory outside lock” pattern for selection-scoped features.
src/HotChocolate/Core/test/Execution.Tests/Processing/OperationFeatureCollectionTests.cs Adds tests intended to validate no lock is held during factory execution and that racing callers return the same stored instance.
Suppressed comments (1)

src/HotChocolate/Core/test/Execution.Tests/Processing/OperationFeatureCollectionTests.cs:65

  • Same issue as the selection-factory test: the waits inside the factory don’t assert success. If the writer only completes after the factory returns (because the write lock was held), the event may still be set before the outer assertions, letting the test pass even though the lock was held during factory execution.
            concurrentWriteFinished.Wait(s_timeout);
            writer.Wait(s_timeout);
            return new Feature();

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Review feedback: building a feature exactly once is deliberate, because creation is
costly where this is used, so the previous commit's GetOrAdd semantics were the wrong
trade. Creation is now gated per feature by a Lazy, which keeps it to once while the
build itself happens with no lock held.

A single write lock guards every feature of the operation, which is the only reason
the two deadlocking paths are in each other's way: they touch different features.
Building a projection selector calls Operation.GetSelectionSet and so wants the
operation lock, while compiling a selection set holds the operation lock and writes an
optimizer feature here. Gating per feature lets both proceed.

Note that purity of the built value does not avoid this. CreateSelectorExpression is
pure, and still deadlocks, because it acquires the operation lock while this lock is
held; the cycle is in the lock order, not in the value.

Racing callers may each allocate a Lazy, but only the one that wins the slot is
evaluated, so the factory runs once per feature. Lazy<object> rather than
Lazy<TFeature> because the trimmer requires a parameterless constructor on the type
argument, which a feature need not have, and the value is stored as object regardless.

The two deadlock tests asserted after the factory returned, by which point the lock is
released and the concurrent write completes on its own, so they passed even when the
lock was held. They now assert on the wait inside the factory. A third test covers
single creation under a race, which fails against the previous commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants