Skip to content

Recover sync after watcher overflow - #29

Open
raphaeldelio wants to merge 8 commits into
redis:mainfrom
raphaeldelio:fix/sync-watcher-overflow
Open

Recover sync after watcher overflow#29
raphaeldelio wants to merge 8 commits into
redis:mainfrom
raphaeldelio:fix/sync-watcher-overflow

Conversation

@raphaeldelio

@raphaeldelio raphaeldelio commented Sep 10, 2026

Copy link
Copy Markdown

Problem

AFS sync watches local file changes and queues notifications for the sync daemon. The queue holds 1,024 events by default. A burst of changes, such as installing Python packages, can fill it and cause notifications to be dropped. The operating system watcher can also lose events.

AFS did not recover those missed changes automatically. Files could remain missing or outdated in Redis even though they existed locally, leaving an incomplete environment when restored in a new session.

Solution

When events are lost, request a scan of the current local and remote trees to synchronize the missed changes. The recovery signal has its own single slot, so it can still be recorded when the event queue is full. Repeated requests are combined, new losses during a scan trigger another pass, and failed scans are retried. Recovery also restores directory watches.

The scan coordinates with uploads and renames already in progress. The added checks address races that created unnecessary conflict copies or deleted local files after a failed remote rename. Actual file conflicts still preserve the local version.

Add sync.watcherQueueCapacity so users can tune the event queue without changing code. The default remains 1,024.

Validation

The full CLI race suite passed three times; builds and static checks passed. Linux ARM64 regressions passed 51 tests twice.

An AgentCore build combining this PR with #28 restored all 1,750 stress files and the complete 7,081 entry pandas and NumPy environment. All 4,142 package hashes passed, with no manual reconciliation or unwanted conflict copies.

The PRs can merge in either order. This change does not add a save completion API or protect local writes that have not reached Redis before a runtime stops.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant