perf(plugins): run scheduled updates off the render thread - #407
Merged
Conversation
plugin update() executed inline in the render loop — execute_update's internal thread.join(timeout=30) blocked it, so one slow plugin HTTP fetch froze scrolling for the whole fetch (up to 30s; DNS-retry storms made this a regular occurrence on flaky networks). Scheduling stays on the render thread and keeps every existing gate (enabled, circuit breaker, can_execute, interval); due updates are now enqueued to a single background worker (serialized — same one-at-a-time execution as before, no thundering herd). RUNNING is set at enqueue so can_execute blocks re-entry alongside the pending-set dedup. Per-plugin locks make the old implicit update/display no-overlap guarantee explicit: the worker holds the plugin's lock through its update; the display side try-locks and, when the plugin is mid-update, holds the last frame for that iteration — reported as success so a mid-update skip never advances the rotation. Unlike before, the guarantee now also holds across the post-timeout window (previously the lingering update thread overlapped display()). Deadlock-free by construction: the worker takes one lock; display never blocks. Timeout semantics unchanged (lingering daemon thread documented). Kill switch: plugin_system.synchronous_updates: true restores the inline path. 8 new concurrency tests (non-blocking scheduler, overlap assertion under a hammering display loop, lock release on failure/timeout paths, dedup, kill switch); 4-min devpi soak clean (updates completing, rotation advancing, no stuck RUNNING states). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqzC1nzTWL4kaqgMaQZFam
Contributor
|
Warning Review limit reached
Next review available in: 37 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthrough
ChangesAsynchronous plugin update flow
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant PluginManager
participant UpdateWorker
participant DisplayController
participant PluginExecutor
PluginManager->>UpdateWorker: enqueue scheduled update
UpdateWorker->>PluginExecutor: execute plugin update
DisplayController->>PluginManager: try acquire plugin lock
PluginManager-->>DisplayController: skip display while update runs
UpdateWorker-->>PluginManager: record completion and release lock
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 32 |
| Duplication | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Owner
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Owner
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/display_controller.py`:
- Around line 1864-1868: Update the display flow around _display_lock_or_skip
and its result handling to track whether display() actually executed, rather
than treating a skipped frame as success. Only record health success and clear
force_change when the display call ran; preserve the existing last-frame
behavior when the lock is busy so a subsequent mode-switch render retains its
required clear.
In `@src/plugin_system/plugin_manager.py`:
- Around line 831-835: Update DisplayController.cleanup() to call
PluginManager.stop_update_worker() before tearing down display or cache-backed
resources, ensuring in-flight updates stop first. Enhance stop_update_worker()
to detect when the worker remains alive after join(timeout) and log that the
shutdown timeout expired.
- Around line 114-120: Update the configuration handling around
_synchronous_updates to validate that plugin_system is a mapping and
synchronous_updates is a boolean, rather than coercing arbitrary values with
bool(). Catch only the expected configuration exceptions, keep the safe
synchronous-mode behavior when validation or loading fails, and log a clear
message containing the failure context.
- Around line 816-823: Coordinate queued updates with unloading in
_execute_update_now’s worker path: acquire the plugin lock before looking up the
instance, re-fetch it while holding the lock, and skip absent plugins without
restoring their state to ENABLED. In test/test_async_plugin_updates.py lines
177-182, invoke unload_plugin() deterministically and assert the update is
skipped, pending tracking is cleared, and the plugin remains unloaded.
- Around line 820-823: Move plugin update lock ownership from the
timeout-wrapped caller into the actual executor target around
_execute_update_now, so the lock remains held until the daemon operation
finishes; in src/plugin_system/plugin_manager.py lines 820-823, update the
executor target accordingly, and in lines 841-846 retain RUNNING/pending
lifecycle state until the underlying update exits. Apply the same target-owned
locking to the display operation in src/display_controller.py lines 1864-1887,
and add a timeout regression in test/test_async_plugin_updates.py lines 3-10
proving exclusion remains active after PluginExecutor returns.
In `@test/test_async_plugin_updates.py`:
- Around line 177-182: The test_unloaded_while_queued_is_harmless test must
exercise the public unload_plugin lifecycle rather than deleting pm.plugins
directly. Queue the plugin update behind a deterministic blocker, call
unload_plugin(), then release the blocker and assert the plugin’s update was not
called, its update is no longer pending, and its lifecycle state is unloaded;
avoid timing-based sleeps and keep the test independent and Raspberry Pi
compatible.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 33506f0d-6cc3-4399-a2e8-2539849cb863
📒 Files selected for processing (4)
src/display_controller.pysrc/plugin_system/plugin_manager.pytest/test_async_plugin_updates.pytest/test_plugin_system.py
…validation, and lock-lifetime gaps in async updates Addresses PR #407 review findings: - display_controller: only clear force_change / record health success when display() actually ran this frame, not when the frame was skipped because the plugin's lock was busy (a skip must preserve a pending mode-switch force_clear). - plugin_manager: replace bool() coercion of synchronous_updates with explicit isinstance validation of plugin_system/synchronous_updates, failing safe to synchronous mode (with a logged reason) on malformed config instead of silently defaulting to async. - plugin_manager: _update_worker_loop now acquires the plugin lock before looking up its instance and re-checks under the lock, so an unloaded plugin's lifecycle state is never resurrected to ENABLED. - plugin_manager + display_controller: move lock ownership (and, for updates, RUNNING/pending lifecycle bookkeeping) into the actual update()/ display() call itself rather than the timeout-wrapped caller, so the lock stays held for the real operation's duration even after PluginExecutor's own join(timeout) elapses and a lingering daemon thread keeps running in the background. - DisplayController.cleanup() now stops the update worker before tearing down display/cache resources; stop_update_worker() logs when the join times out instead of failing silently. - test_async_plugin_updates: rewrite test_unloaded_while_queued_is_harmless to exercise the public unload_plugin() lifecycle (via a deterministic blocker) instead of deleting pm.plugins directly, and add a regression test proving the plugin lock stays held through PluginExecutor's own timeout while the real update() call is still running. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KEZK1P1Q1fu5pcuVrkrCFZ
# Conflicts: # src/plugin_system/plugin_manager.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 5 of the performance series — the biggest UX defect found in the deep-dive: plugin
update()ran inline in the render loop (execute_update'sthread.join(timeout=30)blocked it), so one slow HTTP fetch froze scrolling for the whole fetch. On a flaky network this is constant: we watched soccer's ESPN DNS-retry storms stall the devpi repeatedly.Design (safety-first):
can_execute, interval). Only execution moves, to a single serialized worker (same one-at-a-time behavior as before; no concurrent fetch herd).RUNNINGset at enqueue + a pending-set gives double re-entry protection.display(); now the lock holds through that window too. The display side try-locks; if the plugin is mid-update it holds the last frame for that iteration and reports success, so a skip never falsely advances rotation. Deadlock-free by construction (worker holds one lock; display never blocks).plugin_system.synchronous_updates: truerestores the inline path with no deploy.state_manageris RLock-guarded;health_trackeruses whole-value per-key writes (worst case one-cycle staleness, self-correcting — no worse than today's post-timeout overlap).Verification
test_circuit_breaker, is the mock drift fixed in fix(core): harden text-measurement caches; surface snapshot failures #400).🤖 Generated with Claude Code
https://claude.ai/code/session_01FqzC1nzTWL4kaqgMaQZFam
Summary by CodeRabbit
New Features
Bug Fixes