Skip to content

fix(infer): share a background event loop across synchronous requests - #10080

Open
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:fix/sync-infer-event-loop
Open

fix(infer): share a background event loop across synchronous requests#10080
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:fix/sync-infer-event-loop

Conversation

@taking-lying-flat

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Interleaving two streams returned by synchronous infer(..., stream=True) can hang the second consumer: each iterator starts a thread that calls run_until_complete() on the same cached event loop. Once the first stream is running, the second thread raises RuntimeError: This event loop is already running outside the coroutine's error handler, leaving its result queue empty.

Run the engine's shared loop in one daemon thread using the existing utility, serialize lazy initialization, and submit both streaming and non-streaming work with asyncio.run_coroutine_threadsafe(). Reusing the loop preserves background tasks bound to the engine's first loop.

This change only modifies the synchronous bridge in swift/infer_engine/infer_engine.py.

Experiment results

  • Four local, model-free regression checks passed: interleaved streams with non-streaming inference, concurrent first use, a persistent background handler across batches, and direct async inference.
  • All five existing tests in tests.infer.test_infer_engine and tests.infer.test_infer_engine_limits passed.
  • All applicable pre-commit checks passed.

Regression scripts were kept outside the repository; the PR contains production code only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant