Run feature factories outside the OperationFeatureCollection write lock - #10168
Open
apereiratl wants to merge 2 commits into
Open
Run feature factories outside the OperationFeatureCollection write lock#10168apereiratl wants to merge 2 commits into
apereiratl wants to merge 2 commits into
Conversation
GetOrSetSafe invoked the caller's factory while holding _writeLock, which lets two execution locks be taken in opposite orders: Operation.GetSelectionSet takes the operation lock, then CompileSelectionSet writes selection features (OperationCompiler sets the optimizer feature), so it holds the operation lock and wants the feature lock. GetOrCreateSelectorExpression takes the feature lock, then the factory builds the selector expression, which calls Operation.GetSelectionSet, so it holds the feature lock and wants the operation lock. Two threads hitting both paths on one operation deadlock permanently. Every later request completing a composite value then blocks on the same operation lock, the pool injects threads to replace them, and the server stops answering. Only the first execution of an operation compiles selection sets and selector expressions, so it takes a schema with many distinct operations to hit reliably, but it needs no unusual input and does not recover. The factory now runs before the lock is taken, which is entered only to store the result. Racing callers can each build a value, the first stored wins, and every caller receives that instance, matching ConcurrentDictionary.GetOrAdd.
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request fixes a deadlock risk in OperationFeatureCollection by ensuring GetOrSetSafe executes feature factories outside the collection’s _writeLock, only taking the lock to store the computed value (double-checking for an existing value first). This aligns the behavior with ConcurrentDictionary.GetOrAdd semantics under contention.
Changes:
- Run
GetOrSetSafefactories outside_writeLockfor both operation-level and selection-level feature storage, then lock only to publish the winning instance. - Add concurrency-focused tests to ensure factories aren’t invoked while holding the write lock and that racing callers observe the same stored instance.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/HotChocolate/Core/src/Types/Execution/Processing/OperationFeatureCollection.cs | Updates operation-level GetOrSetSafe to invoke factories outside _writeLock and publish the first-created value. |
| src/HotChocolate/Core/src/Types/Execution/Processing/OperationFeatureCollection.Selections.cs | Applies the same “factory outside lock” pattern for selection-scoped features. |
| src/HotChocolate/Core/test/Execution.Tests/Processing/OperationFeatureCollectionTests.cs | Adds tests intended to validate no lock is held during factory execution and that racing callers return the same stored instance. |
Suppressed comments (1)
src/HotChocolate/Core/test/Execution.Tests/Processing/OperationFeatureCollectionTests.cs:65
- Same issue as the selection-factory test: the waits inside the factory don’t assert success. If the writer only completes after the factory returns (because the write lock was held), the event may still be set before the outer assertions, letting the test pass even though the lock was held during factory execution.
concurrentWriteFinished.Wait(s_timeout);
writer.Wait(s_timeout);
return new Feature();
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Review feedback: building a feature exactly once is deliberate, because creation is costly where this is used, so the previous commit's GetOrAdd semantics were the wrong trade. Creation is now gated per feature by a Lazy, which keeps it to once while the build itself happens with no lock held. A single write lock guards every feature of the operation, which is the only reason the two deadlocking paths are in each other's way: they touch different features. Building a projection selector calls Operation.GetSelectionSet and so wants the operation lock, while compiling a selection set holds the operation lock and writes an optimizer feature here. Gating per feature lets both proceed. Note that purity of the built value does not avoid this. CreateSelectorExpression is pure, and still deadlocks, because it acquires the operation lock while this lock is held; the cycle is in the lock order, not in the value. Racing callers may each allocate a Lazy, but only the one that wins the slot is evaluated, so the factory runs once per feature. Lazy<object> rather than Lazy<TFeature> because the trimmer requires a parameterless constructor on the type argument, which a feature need not have, and the value is stored as object regardless. The two deadlock tests asserted after the factory returned, by which point the lock is released and the concurrent write completes on its own, so they passed even when the lock was held. They now assert on the wait inside the factory. A third test covers single creation under a race, which fails against the previous commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GetOrSetSafe invoked the caller's factory while holding _writeLock, which lets two execution locks be taken in opposite orders:
Operation.GetSelectionSet takes the operation lock, then CompileSelectionSet writes selection features (OperationCompiler sets the optimizer feature), so it holds the operation lock and wants the feature lock.
GetOrCreateSelectorExpression takes the feature lock, then the factory builds the selector expression, which calls Operation.GetSelectionSet, so it holds the feature lock and wants the operation lock.
Two threads hitting both paths on one operation deadlock permanently. Every later request completing a composite value then blocks on the same operation lock, the pool injects threads to replace them, and the server stops answering. Only the first execution of an operation compiles selection sets and selector expressions, so it takes a schema with many distinct operations to hit reliably, but it needs no unusual input and does not recover.
The factory now runs before the lock is taken, which is entered only to store the result. Racing callers can each build a value, the first stored wins, and every caller receives that instance, matching ConcurrentDictionary.GetOrAdd.