Skip to content

keep every pending registration per task instead of replacing it - #137

Merged
JoshVanL merged 3 commits into
dapr:mainfrom
cicoyle:fix-data-races
Oct 9, 2026
Merged

JoshVanL merged 3 commits into
dapr:mainfrom
cicoyle:fix-data-races

Conversation

@cicoyle

@cicoyle cicoyle commented Oct 7, 2026

Copy link
Copy Markdown

The local tasks backend kept one completion registration per task (activity: instance ID + task ID, workflow: instance ID) and stored a new one with a plain sync.Map.Store. A second registration for the same task replaced the first. Complete and cancel only reached the newer one, and the older one's deregister was a no-op, so the older execution stayed pending until the worker context ended.

In Dapr this happens when a workflow instance is recreated under the same ID while an activity of the previous run is still executing on the same sidecar.

Stress tests showed sidecars holding about 90 such executions each (about 250 goroutines and 30 MB), released only at shutdown with failed before receiving activity result. With WithMaxParallelism, each stranded execution also held a worker slot forever.

Signed-off-by: Cassandra Coyle <cassie@diagrid.io>
@cicoyle
cicoyle requested a review from a team as a code owner October 7, 2026 20:10
Copilot AI balanced review requested due to automatic review settings October 7, 2026 20:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Tokenless completions can settle overlapping executions with another run’s result.

1 open finding
What changed in this PR

Keeps all pending registrations for each local task so overlapping executions are not stranded by newer registrations.

Changes:

  • Replaces single-registration storage with a synchronized callback registry.
  • Adds delivery, deregistration, concurrency, and executor regression tests.
  • Updates the callback contract and enables backend race tests in CI.
File Description
backend/​local/​task.go Retains and delivers to every same-key registration.
backend/​local/​task_test.go Tests multiple registrations and concurrent use.
backend/​local/​executor_test.go Tests overlapping executions with distinct completion tokens.
backend/​backend.go Documents delivery to all pending callbacks.
.github/​workflows/​pr-validation.yml Runs backend tests with race detection.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread backend/local/task.go Outdated
Signed-off-by: Cassandra Coyle <cassie@diagrid.io>
Comment thread backend/backend.go
Comment thread backend/local/task.go
Comment thread backend/local/task.go Outdated
Comment thread .github/workflows/pr-validation.yml Outdated
…tokenless completions, run all tests with -race

Signed-off-by: Cassandra Coyle <cassie@diagrid.io>
@JoshVanL
JoshVanL merged commit 9268ad2 into dapr:main Oct 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants