Skip to content

fix: preserve early workflow events in streams transport - #40981

Open
Zhengcy05 wants to merge 1 commit into
langgenius:mainfrom
Zhengcy05:fix/streams-starting-cursor
Open

fix: preserve early workflow events in streams transport#40981
Zhengcy05 wants to merge 1 commit into
langgenius:mainfrom
Zhengcy05:fix/streams-starting-cursor

Conversation

@Zhengcy05

Copy link
Copy Markdown

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Fixes #40948.

Redis Streams subscriptions currently start from $, while streaming workflow tasks are enqueued before the listener performs its first XREAD. A fast worker can therefore publish workflow_started and early node events before $ is resolved, causing those events to be skipped even though they remain stored in Redis.

This PR captures the stream tip before dispatching a fresh workflow task and uses it as that subscription's starting cursor. Subscriptions without an explicit cursor continue to default to $, preserving the live-only behavior introduced to avoid replaying stale workflow_paused events in #34040.

Changes include:

  • Add a cursor-capable topic protocol and per-subscription starting cursors for Redis Streams.
  • Treat a missing or empty stream as cursor 0-0, since workflow streams are created lazily by the first XADD.
  • Capture the stream tip before Celery task dispatch for fresh Workflow and Advanced Chat runs.
  • Pass the captured topic and cursor through the event generator to the final Streams subscription.
  • Correct the Streams test double so $ only observes entries appended after the initial read.
  • Add regression coverage for explicit cursors while retaining coverage for the default $ behavior.

Validation:

  • uv run --project api pytest -q api/tests/unit_tests/libs/broadcast_channel/redis/test_streams_channel_unit_tests.py api/tests/unit_tests/services/test_app_generate_service_streaming_integration.py api/tests/unit_tests/services/test_app_generate_service.py — 71 passed.
  • Focused Ruff checks passed for all changed files.
  • Ruff format checks passed for all changed files.
  • Focused Pyrefly checks passed for all changed production files.
  • Docker-backed integration tests were not run locally because they are CI-only.

Screenshots

N/A — backend event-delivery fix.

Checklist

  • This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs)
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] workflow_started is dropped for concurrent streaming runs with the streams event transport (regression of #32518 via #34030)

1 participant