Fix HTTP 421 partition sharding and add trust-token auth mode#6
Open
mslaughter21228 wants to merge 13 commits into
Open
Fix HTTP 421 partition sharding and add trust-token auth mode#6mslaughter21228 wants to merge 13 commits into
mslaughter21228 wants to merge 13 commits into
Conversation
driver.py: detect Apple account shard via the setup/ws/1/validate
endpoint before authenticating. Patches api._setup_endpoint to
p{N}-setup.icloud.com to prevent 421 errors for users whose Apple
ID lives on a non-default partition.
auth.py: rewrite to include the same partition detection, add a
--trust-token flag for injecting a browser-extracted HSA2 trust
token directly into the pyicloud session. This allows authentication
to succeed when 2FA push notifications are not delivered to Apple
devices (reproduced on iOS 26 beta).
finish-setup.sh: interactive helper script for first-time credential
configuration and systemd user service registration.
- Add sync_paths config option (whitelist of iCloud paths to hydrate) - Add _path_allowed() method to ICloudSyncEngine gating ensure_local_file and _schedule_download_with_delay — paths outside sync_paths get stubs only - Switch warmup_mode default to lazy to prevent background hydration competing with FUSE open() calls on large directories - Add hydrate_dir.py: standalone script to pre-hydrate a directory via drive.get_file(docwsid) bypassing FUSE, safe to run while service is stopped - Bump warmup_workers to 4, remote_refresh_interval to 3600s
Fixes backslash octal escapes (\374 -> ü), escaped chars (\_ -> _), bare backslashes, CIFS-illegal chars (< > : " | ? *), trailing spaces and dots on directory names. Updates SQLite state DB to match. Safe to run before copying from mirror to NAS mount.
…afety - driver.py: init_icloud() accepts require_session=False so auth failure parks the service in unauthenticated mode instead of crashing and triggering Apple lockout via Restart=on-failure. All write ops return EACCES when unauthenticated; read ops serve local cache where possible. sync_engine guarded against None in open/read paths. - auth.py: add --force-sms flag that calls _request_sms_2fa_code() directly, bypassing the push_mode=='sms' guard that iOS 26 beta breaks (Apple returns push_mode='push' on the trusted phone number). Add --debug flag to print Apple's reported auth mode and phone metadata. Improved user messaging throughout. - icloudctl: pass through all auth flags ($@) so --force-sms, --debug, and --trust-token reach auth.py. Updated usage text. - fix_filenames.py: replace substring replace() in directory DB rename with safe SUBSTR-based prefix match, preventing silent corruption of sibling paths sharing a common prefix. - .gitignore: add *.bak
On every service start, _reconcile_persistent_cache() was calling file_sha256() on every hydrated file in the mirror — reading the full contents of every downloaded file just to recompute a checksum that was already stored in the DB. With 18k+ entries this caused a 4.5-minute startup delay and 11.5 GB memory usage before the service was OOM-killed. Fix: only recompute the SHA256 when size or mtime has changed from the stored value, or when no checksum is recorded. Unchanged files reuse the existing stored checksum. This reduces cold-start reconciliation from minutes to seconds for large caches.
Allows carving large subdirectories out of an allowed sync_path without having to restructure the whole allow-list. exclude_paths is evaluated before sync_paths so it always wins. Wired through ICloudSyncEngine constructor, ICloudFS.init_local_cache(), main() config parsing, and documented in config.example.yaml. Live config updated to exclude /Downloads/Michael Priority until the rest of /Downloads is stable.
Blocking hydration command that ensures all eligible files are fully local before copy tasks run. Reads sync_paths and exclude_paths from config.yaml — same rules as the FUSE driver, single source of truth. - Queries state DB for unhydrated files (fast, no filesystem walk) - Filters using path_allowed() mirroring ICloudSyncEngine._path_allowed() - Opens each file through the FUSE mount to trigger lazy hydration - Reports progress every 25 files or 30 seconds with ETA - --dry-run lists what would be hydrated without downloading - --verbose prints each file as it downloads - Idempotent: safe to interrupt and re-run - Exit 0 = all done, exit 1 = partial failures, exit 2 = config error Mount dir resolved from config.yaml > icloud.env > ~/iCloud default.
- _crawl_remote_snapshot: skip recursing into folders that cannot lead to a sync_path. With sync_paths: [/Downloads] this avoids crawling the entire 19k-entry iCloud library — only /Downloads and its children are enumerated, reducing sync time dramatically. - SIGUSR1 handler was already in driver.py but the running process predated the edit; restart picks it up.
SIGUSR1 arriving during the ~75s reconcile pass was killing the process because Python's default SIGUSR1 handler (terminate) was still active. Fix: move all signal.signal() calls to before init_icloud/init_local_cache so handlers are live from process start. The SIGUSR1 handler now also waits up to 120s for sync_engine to become available if it fires during the reconcile window.
get_children(force=True) is a blocking HTTP call with no timeout. If Apple API stalls on any folder the crawl hangs indefinitely and icloudctl sync times out with no log output. Wrap each get_children call in a ThreadPoolExecutor future with a 60s per-folder deadline. Timed-out folders are logged as warnings and skipped so the crawl continues rather than freezing forever.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
driver.py: detect Apple account shard via the setup/ws/1/validate endpoint before authenticating. Patches api._setup_endpoint to p{N}-setup.icloud.com to prevent 421 errors for users whose Apple ID lives on a non-default partition.
auth.py: rewrite to include the same partition detection, add a --trust-token flag for injecting a browser-extracted HSA2 trust token directly into the pyicloud session. This allows authentication to succeed when 2FA push notifications are not delivered to Apple devices (reproduced on iOS 26 beta).
finish-setup.sh: interactive helper script for first-time credential configuration and systemd user service registration.