refactor(concurrent): introduce authoritative VerifiedProgress and harden completion lifecycle by superGekFordJ · Pull Request #633 · SurgeDM/Surge

superGekFordJ · 2026-08-26T08:02:46Z

Following up on discussions in #632, this PR introduces VerifiedProgress (VP) as the single authoritative Source of Truth (SoT) for download completion, pause/resume accounting, and worker lifecycles.

When previously refactoring our engine to adopt VerifiedProgress, I encountered and debugged these exact edge-case pitfalls behind #579 / #624. They ultimately broke down into 5 interconnected root causes:

HedgeWork Lacking Progress Guards: Idle workers popping hedged tasks lacked a physical progress check (VP >= totalSize), firing redundant HTTP requests that triggered 404/429 storms (feat: make concurrent downloads throttle-aware #632 removed hedging to mitigate this, though with VP as SoT it can be cleanly revisited later).
Permanent HTTP (404) Burning Retries: 404 status codes wrapped in ErrPermanentHTTP still went through generic sleep-retries 3 times, worsening server rate limits.
Completion Monitor Missing Active Cancel: runCompletionMonitor only called queue.Close(), failing to cancel in-flight workers stuck in resp.Body.Read() or retry sleeps.
Spurious Error Propagation & State Inconsistency: Trailing hedged/retrying workers eventually failing 404 bubbled up errors to Download(), causing UI failure reports even after main workers had already written all data.
Early-EOF Silent Data Loss & Fragile Counters: Server-side premature io.EOF before StopAt silently returned nil (dropping bytes), while completion relied on a volatile Downloaded counter rather than on-disk physical verification.

Most of these issues ultimately stemmed from the lack of a single authoritative physical pointer, which led different subsystems (Queue, Worker, Monitor, Resume) into desynchronized state guesswork.

By establishing VerifiedProgress (grounded in physical WriteAt and bitmap deduplication) as the strong SoT, this PR cleans up these module boundaries:

Completion & Workers are decoupled: A download strictly completes when $VP \ge FileSize$. In-flight workers are actively canceled on completion, and workers check VP on pop to exit early and cleanly release concurrency gates.
Errors fail fast or retry safely: 404s break out immediately, Early-EOFs are caught and requeued, and spurious trailing errors can no longer fail a finished download.
Resume is decoupled: Restored progress is recomputed strictly from physical chunk bitmaps, ensuring on-disk completed chunks are never accidentally erased or overwritten by remaining task overlaps.

Side note on future ideas

While intentionally excluded here to keep this PR focused and clean, this solid VP foundation opens the door for future improvements—such as safely re-introducing end-game hedging without race conditions, or evolving the concurrency gate from goroutine blocking to lightweight connection draining (Drain) for better TCP/TLS socket reuse.

Summary by CodeRabbit

Bug Fixes
- Improved download completion detection using verified progress.
- Preserved completed chunks and accurate progress when resuming downloads.
- Improved handling of early connection termination, HTTP errors, retries, and partial responses.
- Prevented tasks from being incorrectly requeued or workers from continuing after completion.
- Improved pause, resume, and state snapshot behavior at completion boundaries.
Performance
- Reduced contention when tracking download progress.
Tests
- Added extensive coverage for resume, retries, completion, cancellation, HTTP errors, and early-termination scenarios.

github-actions · 2026-08-26T08:03:27Z

Binary Size Analysis

⚠️ Size Increased

Version	Human Readable	Raw Bytes
Main	18.09 MB	`18972964`
PR	18.10 MB	`18981156`
Difference	8.00 KB	`8192`

coderabbitai · 2026-08-26T08:03:35Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8ae57015-5f24-475e-89fa-571bda48fd30

📥 Commits

Reviewing files that changed from the base of the PR and between 6f7951b and 5a32da1.

📒 Files selected for processing (17)

internal/progress/accuracy_test.go
internal/progress/bitmap.go
internal/progress/bitmap_test.go
internal/progress/bytes.go
internal/scheduler/main_test.go
internal/strategy/concurrent/completion_monitor_cancel_test.go
internal/strategy/concurrent/downloader.go
internal/strategy/concurrent/downloader_helpers_test.go
internal/strategy/concurrent/early_eof_requeue_test.go
internal/strategy/concurrent/handlepause_vp_guard_test.go
internal/strategy/concurrent/main_test.go
internal/strategy/concurrent/permanent_http_test.go
internal/strategy/concurrent/prewarm_test.go
internal/strategy/concurrent/resume_antiregression_vp_guard_test.go
internal/strategy/concurrent/resume_stopat_clamp_test.go
internal/strategy/concurrent/worker.go
internal/strategy/concurrent/worker_vp_early_exit_test.go

💤 Files with no reviewable changes (1)

internal/strategy/concurrent/downloader_helpers_test.go

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Walkthrough

Walkthrough

Changes

Download correctness

Layer / File(s)	Summary
Progress restoration and counter layout `internal/progress/bitmap.go`, `internal/progress/*_test.go`, `internal/progress/bytes.go`	Bitmap recalculation preserves restored completed chunks as fully verified. `ByteTracker` counters use cache-line padding.
Completion monitoring and resume state `internal/strategy/concurrent/downloader.go`, `internal/strategy/concurrent/completion_monitor_cancel_test.go`, `internal/strategy/concurrent/handlepause_vp_guard_test.go`, `internal/strategy/concurrent/resume_antiregression_vp_guard_test.go`	Completion now depends on verified progress. Resume and pause snapshots preserve bitmap-derived progress and handle nil state safely.
Worker retry and boundary handling `internal/strategy/concurrent/worker.go`, `internal/strategy/concurrent/early_eof_requeue_test.go`, `internal/strategy/concurrent/permanent_http_test.go`, `internal/strategy/concurrent/prewarm_test.go`, `internal/strategy/concurrent/resume_stopat_clamp_test.go`, `internal/strategy/concurrent/worker_vp_early_exit_test.go`	Workers handle completion cancellation, permanent HTTP errors, early EOF, and `StopAt` clamping without incorrect retries, requeueing, or progress accounting.

Test execution cleanup

Layer / File(s)	Summary
Explicit test and leak handling `internal/scheduler/main_test.go`, `internal/strategy/concurrent/main_test.go`	Test mains now clean network resources, run goleak verification explicitly, report leaks, and preserve test exit codes.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CompletionMonitor
  participant Downloader
  participant Worker
  participant TaskQueue
  CompletionMonitor->>Downloader: observe VerifiedProgress
  CompletionMonitor->>Worker: cancel active tasks
  CompletionMonitor->>TaskQueue: close and drain queued tasks
  Worker->>Downloader: exit without requeue during completion

Suggested reviewers: supercoolpencil

Merge Risk: ⚪ Minimal · up to 5a32d

The completion lifecycle changes introduce no actionable merge-blocking risk based on the current evidence; the PR is merge-ready after normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 2.70% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 37 functions across 16 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the primary changes: introducing authoritative VerifiedProgress and strengthening concurrent completion handling.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

superGekFordJ · 2026-09-09T14:25:31Z

Hey @SuperCoolPencil,

A quick update on my recent testing:

Full Sanitized Log (~500MB download under 403 rate-limits)

[2026-09-09 12:06:45] Queued enqueue command (via Orchestrator): https://cdn.example.org/download/sample_video.mp4 -> 
[2026-09-09 12:06:45] Lifecycle: Enqueue https://cdn.example.org/download/sample_video.mp4 (Filename: )
[2026-09-09 12:06:45] Probing server: https://cdn.example.org/download/sample_video.mp4
[2026-09-09 12:06:45] NetworkPool: creating new shared transport (proxy=, limit=0)
[2026-09-09 12:06:45] NetworkPool: AcquireTransport (key={proxyURL: customDNS: maxConns:0}, refs=1)
[2026-09-09 12:06:46] Probe response status: 206
[2026-09-09 12:06:46] Content-Range header: bytes 0-0/538465527
[2026-09-09 12:06:46] Range supported, file size: 538465527
[2026-09-09 12:06:46] Filename from URL path: sample_video.mp4
[2026-09-09 12:06:46] Detected MIME: application/octet-stream
[2026-09-09 12:06:46] Final resolved filename: sample_video.mp4
[2026-09-09 12:06:46] Probe complete - filename: sample_video.mp4, size: 538465527, range: true
[2026-09-09 12:06:46] NetworkPool: ReleaseTransport (key={proxyURL: customDNS: maxConns:0}, refs=0)
[2026-09-09 12:06:46] Destination path: D:\downloads\sample_video.mp4
[2026-09-09 12:06:46] Using concurrent downloader
[2026-09-09 12:06:46] Calling Download with mirrors: []
[2026-09-09 12:06:46] ConcurrentDownloader.Download: https://cdn.example.org/download/sample_video.mp4 -> D:\downloads\sample_video.mp4 (size: 538465527, mirrors: 0)
[2026-09-09 12:06:46] NetworkPool: AcquireTransport (key={proxyURL: customDNS: maxConns:0}, refs=1)
[2026-09-09 12:06:46] Pre-warming complete: 16 connections hot
[2026-09-09 12:06:46] InitBitmap: Total=538465527, ChunkSize=33652736
[2026-09-09 12:06:46] Worker 8 started
[2026-09-09 12:06:46] Worker 8: Setting range 0-33652736 to Downloading
[2026-09-09 12:06:46] Worker 0 started
[2026-09-09 12:06:46] Worker 14 started
[2026-09-09 12:06:46] Worker 14: Setting range 33652736-67305472 to Downloading
[2026-09-09 12:06:46] Worker 12 started
[2026-09-09 12:06:46] Worker 12: Setting range 67305472-100958208 to Downloading
[2026-09-09 12:06:46] Worker 0: Setting range 100958208-134610944 to Downloading
[2026-09-09 12:06:46] Worker 15 started
[2026-09-09 12:06:46] Worker 15: Setting range 134610944-168263680 to Downloading
[2026-09-09 12:06:46] Worker 4 started
[2026-09-09 12:06:46] Worker 4: Setting range 168263680-201916416 to Downloading
[2026-09-09 12:06:46] Worker 13 started
[2026-09-09 12:06:46] Worker 3 started
[2026-09-09 12:06:46] Worker 6 started
[2026-09-09 12:06:46] Worker 5 started
[2026-09-09 12:06:46] Worker 5: Setting range 302874624-336527360 to Downloading
[2026-09-09 12:06:46] Worker 10 started
[2026-09-09 12:06:46] Worker 10: Setting range 336527360-370180096 to Downloading
[2026-09-09 12:06:46] Worker 7 started
[2026-09-09 12:06:46] Worker 7: Setting range 370180096-403832832 to Downloading
[2026-09-09 12:06:46] Worker 2 started
[2026-09-09 12:06:46] Worker 13: Setting range 201916416-235569152 to Downloading
[2026-09-09 12:06:46] Worker 9 started
[2026-09-09 12:06:46] Worker 9: Setting range 437485568-471138304 to Downloading
[2026-09-09 12:06:46] Worker 1 started
[2026-09-09 12:06:46] Worker 1: Setting range 471138304-504791040 to Downloading
[2026-09-09 12:06:46] Worker 3: Setting range 235569152-269221888 to Downloading
[2026-09-09 12:06:46] Worker 6: Setting range 269221888-302874624 to Downloading
[2026-09-09 12:06:46] Worker 11 started
[2026-09-09 12:06:46] Worker 11: Setting range 504791040-538465527 to Downloading
[2026-09-09 12:06:46] Worker 2: Setting range 403832832-437485568 to Downloading
[2026-09-09 12:06:47] Worker 7: Task offset=370180096 length=33652736 took 426.7515ms
[2026-09-09 12:06:47] Worker 12: Task offset=67305472 length=33652736 took 429.3851ms
[2026-09-09 12:06:47] Worker 10: Task offset=336527360 length=33652736 took 428.8531ms
[2026-09-09 12:06:47] Worker 0: Task offset=100958208 length=33652736 took 431.9425ms
[2026-09-09 12:06:47] Worker 13: Task offset=201916416 length=33652736 took 430.8875ms
[2026-09-09 12:06:47] Worker 15: Task offset=134610944 length=33652736 took 498.6182ms
[2026-09-09 12:06:47] Worker 3: Task offset=235569152 length=33652736 took 504.3885ms
[2026-09-09 12:06:47] Worker 6: Task offset=269221888 length=33652736 took 505.4434ms
[2026-09-09 12:06:47] Worker 2: Task offset=403832832 length=33652736 took 506.2708ms
[2026-09-09 12:06:47] Worker 9: Task offset=437485568 length=33652736 took 511.5133ms
[2026-09-09 12:06:47] Worker 5: Task offset=302874624 length=33652736 took 515.2725ms
[2026-09-09 12:06:47] Worker 14: Task offset=33652736 length=33652736 took 516.8207ms
[2026-09-09 12:06:47] Worker 11: Task offset=504791040 length=33674487 took 517.3568ms
[2026-09-09 12:06:47] Worker 8: Task offset=0 length=33652736 took 520.5151ms
[2026-09-09 12:06:47] Worker 7: Setting range 370180096-403832832 to Downloading
[2026-09-09 12:06:47] Worker 12: Setting range 67305472-100958208 to Downloading
[2026-09-09 12:06:47] Worker 10: Setting range 336527360-370180096 to Downloading
[2026-09-09 12:06:47] Worker 13: Setting range 201916416-235569152 to Downloading
[2026-09-09 12:06:47] Worker 0: Setting range 100958208-134610944 to Downloading
[2026-09-09 12:06:47] Worker 15: Setting range 134610944-168263680 to Downloading
[2026-09-09 12:06:47] Worker 3: Setting range 235569152-269221888 to Downloading
[2026-09-09 12:06:47] Worker 6: Setting range 269221888-302874624 to Downloading
[2026-09-09 12:06:47] Worker 2: Setting range 403832832-437485568 to Downloading
[2026-09-09 12:06:47] Worker 9: Setting range 437485568-471138304 to Downloading
[2026-09-09 12:06:47] Worker 5: Setting range 302874624-336527360 to Downloading
[2026-09-09 12:06:47] Worker 14: Setting range 33652736-67305472 to Downloading
[2026-09-09 12:06:47] Worker 11: Setting range 504791040-538465527 to Downloading
[2026-09-09 12:06:47] Worker 8: Setting range 0-33652736 to Downloading
[2026-09-09 12:06:47] Worker 15: Task offset=134610944 length=33652736 took 141.76ms
[2026-09-09 12:06:47] Worker 11: Task offset=504791040 length=33674487 took 128.872ms
[2026-09-09 12:06:47] Worker 14: Task offset=33652736 length=33652736 took 131.4743ms
[2026-09-09 12:06:47] Worker 8: Task offset=0 length=33652736 took 128.3644ms
[2026-09-09 12:06:47] Worker 6: Task offset=269221888 length=33652736 took 142.4565ms
[2026-09-09 12:06:47] Worker 2: Task offset=403832832 length=33652736 took 143.4896ms
[2026-09-09 12:06:47] Worker 5: Task offset=302874624 length=33652736 took 198.9904ms
[2026-09-09 12:06:47] Worker 12: Task offset=67305472 length=33652736 took 292.8789ms
[2026-09-09 12:06:47] Worker 3: Task offset=235569152 length=33652736 took 218.8502ms
[2026-09-09 12:06:47] Worker 9: Task offset=437485568 length=33652736 took 212.0906ms
[2026-09-09 12:06:47] Worker 7: Task offset=370180096 length=33652736 took 300.6837ms
[2026-09-09 12:06:47] Worker 0: Task offset=100958208 length=33652736 took 296.5392ms
[2026-09-09 12:06:47] Worker 10: Task offset=336527360 length=33652736 took 299.6945ms
[2026-09-09 12:06:47] Worker 13: Task offset=201916416 length=33652736 took 298.2143ms
[2026-09-09 12:06:47] Worker 4: Task offset=168263680 length=33652736 took 1.1937078s
[2026-09-09 12:06:47] Balancer: stole 16 MiB from worker 11 (new range: 521625600-538465527)
[2026-09-09 12:06:47] Worker 4: Setting range 521625600-538465527 to Downloading
[2026-09-09 12:06:47] Worker 1: Task offset=471138304 length=33652736 took 1.2576371s
[2026-09-09 12:06:47] Balancer: stole 16 MiB from worker 3 (new range: 252395520-269221888)
[2026-09-09 12:06:47] Worker 1: Setting range 252395520-269221888 to Downloading
[2026-09-09 12:06:48] Worker 15: Setting range 134610944-168263680 to Downloading
[2026-09-09 12:06:48] Worker 11: Setting range 504791040-521625600 to Downloading
[2026-09-09 12:06:48] Worker 14: Setting range 33652736-67305472 to Downloading
[2026-09-09 12:06:48] Worker 8: Setting range 0-33652736 to Downloading
[2026-09-09 12:06:48] Worker 6: Setting range 269221888-302874624 to Downloading
[2026-09-09 12:06:48] Worker 2: Setting range 403832832-437485568 to Downloading
[2026-09-09 12:06:48] Worker 5: Setting range 302874624-336527360 to Downloading
[2026-09-09 12:06:48] Worker 12: Setting range 67305472-100958208 to Downloading
[2026-09-09 12:06:48] Worker 3: Setting range 235569152-252395520 to Downloading
[2026-09-09 12:06:48] Worker 9: Setting range 437485568-471138304 to Downloading
[2026-09-09 12:06:48] Worker 0: Setting range 100958208-134610944 to Downloading
[2026-09-09 12:06:48] Worker 7: Setting range 370180096-403832832 to Downloading
[2026-09-09 12:06:48] Worker 10: Setting range 336527360-370180096 to Downloading
[2026-09-09 12:06:48] Worker 13: Setting range 201916416-235569152 to Downloading
[2026-09-09 12:06:48] Worker 1: Task offset=252395520 length=16826368 took 777.4959ms
[2026-09-09 12:06:48] Balancer: stole 16 MiB from worker 13 (new range: 218742784-235569152)
[2026-09-09 12:06:48] Worker 1: Setting range 218742784-235569152 to Downloading
[2026-09-09 12:06:48] Worker 15: Task offset=134610944 length=33652736 took 246.8044ms
[2026-09-09 12:06:48] Worker 15: task at offset 134610944 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 12: Task offset=67305472 length=33652736 took 165.9581ms
[2026-09-09 12:06:48] Worker 12: task at offset 67305472 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 2: Task offset=403832832 length=33652736 took 237.884ms
[2026-09-09 12:06:48] Worker 2: task at offset 403832832 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 5: Task offset=302874624 length=33652736 took 173.9506ms
[2026-09-09 12:06:48] Worker 5: task at offset 302874624 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 8: Task offset=0 length=33652736 took 241.063ms
[2026-09-09 12:06:48] Worker 8: task at offset 0 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 10: Task offset=336527360 length=33652736 took 160.7902ms
[2026-09-09 12:06:48] Worker 10: task at offset 336527360 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 6: Task offset=269221888 length=33652736 took 243.3913ms
[2026-09-09 12:06:48] Worker 6: task at offset 269221888 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 14: Task offset=33652736 length=33652736 took 244.4135ms
[2026-09-09 12:06:48] Worker 14: task at offset 33652736 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 4: Task offset=521625600 length=16839927 took 931.5025ms
[2026-09-09 12:06:48] Balancer: stole 16 MiB from worker 9 (new range: 454311936-471138304)
[2026-09-09 12:06:48] Worker 4: Setting range 454311936-471138304 to Downloading
[2026-09-09 12:06:48] Worker 0: Task offset=100958208 length=33652736 took 239.7192ms
[2026-09-09 12:06:48] Worker 0: task at offset 100958208 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 9: Task offset=437485568 length=33652736 took 243.9047ms
[2026-09-09 12:06:48] Worker 9: task at offset 437485568 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 1: Task offset=218742784 length=16826368 took 133.1342ms
[2026-09-09 12:06:48] Worker 7: Task offset=370180096 length=33652736 took 243.4634ms
[2026-09-09 12:06:48] Worker 7: task at offset 370180096 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 3: Task offset=235569152 length=16826368 took 250.3866ms
[2026-09-09 12:06:48] Worker 3: task at offset 235569152 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:48] Worker 11: Task offset=504791040 length=16834560 took 339.3518ms
[2026-09-09 12:06:48] Worker 11: task at offset 504791040 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:49] Worker 1: Setting range 218742784-235569152 to Downloading
[2026-09-09 12:06:49] Worker 15: Setting range 134610944-168263680 to Downloading
[2026-09-09 12:06:49] Worker 2: Setting range 403832832-437485568 to Downloading
[2026-09-09 12:06:49] Worker 12: Setting range 67305472-100958208 to Downloading
[2026-09-09 12:06:49] Worker 5: Setting range 302874624-336527360 to Downloading
[2026-09-09 12:06:49] Worker 10: Setting range 336527360-370180096 to Downloading
[2026-09-09 12:06:49] Worker 8: Setting range 0-33652736 to Downloading
[2026-09-09 12:06:49] Worker 14: Setting range 33652736-67305472 to Downloading
[2026-09-09 12:06:49] Worker 6: Setting range 269221888-302874624 to Downloading
[2026-09-09 12:06:49] Worker 0: Setting range 100958208-134610944 to Downloading
[2026-09-09 12:06:49] Worker 9: Setting range 437485568-454311936 to Downloading
[2026-09-09 12:06:49] Worker 13: Task offset=201916416 length=33652736 took 740.9359ms
[2026-09-09 12:06:49] Worker stopped early due to stealing
[2026-09-09 12:06:49] Balancer: stole 16 MiB from worker 5 (new range: 319700992-336527360)
[2026-09-09 12:06:49] Worker 13: Setting range 319700992-336527360 to Downloading
[2026-09-09 12:06:49] Worker 7: Setting range 370180096-403832832 to Downloading
[2026-09-09 12:06:49] Worker 3: Setting range 235569152-252395520 to Downloading
[2026-09-09 12:06:49] Worker 11: Setting range 504791040-521625600 to Downloading
[2026-09-09 12:06:49] Worker 13: Task offset=319700992 length=16826368 took 395.6602ms
[2026-09-09 12:06:49] Worker 15: Task offset=134610944 length=33652736 took 479.5273ms
[2026-09-09 12:06:49] Worker 2: Task offset=403832832 length=33652736 took 480.5821ms
[2026-09-09 12:06:49] Worker 12: Task offset=67305472 length=33652736 took 481.0977ms
[2026-09-09 12:06:49] Worker 8: Task offset=0 length=33652736 took 480.5925ms
[2026-09-09 12:06:49] Worker 3: Task offset=235569152 length=16826368 took 395.7276ms
[2026-09-09 12:06:49] Worker 10: Task offset=336527360 length=33652736 took 481.1053ms
[2026-09-09 12:06:49] Worker 0: Task offset=100958208 length=33652736 took 457.0419ms
[2026-09-09 12:06:49] Worker 14: Task offset=33652736 length=33652736 took 548.9661ms
[2026-09-09 12:06:49] Worker 11: Task offset=504791040 length=16834560 took 459.2039ms
[2026-09-09 12:06:49] Worker 6: Task offset=269221888 length=33652736 took 557.2902ms
[2026-09-09 12:06:49] Worker 9: Task offset=437485568 length=16826368 took 483.96ms
[2026-09-09 12:06:49] Worker 7: Task offset=370180096 length=33652736 took 479.8526ms
[2026-09-09 12:06:49] Worker 1: Task offset=218742784 length=16826368 took 590.0966ms
[2026-09-09 12:06:50] Worker 5: Task offset=302874624 length=33652736 took 849.3845ms
[2026-09-09 12:06:50] Worker stopped early due to stealing
[2026-09-09 12:06:50] Balancer: stole 16 MiB from worker 14 (new range: 50479104-67305472)
[2026-09-09 12:06:50] Worker 5: Setting range 50479104-67305472 to Downloading
[2026-09-09 12:06:50] Worker 13: Setting range 319700992-336527360 to Downloading
[2026-09-09 12:06:50] Worker 15: Setting range 134610944-168263680 to Downloading
[2026-09-09 12:06:50] Worker 2: Setting range 403832832-437485568 to Downloading
[2026-09-09 12:06:50] Worker 12: Setting range 67305472-100958208 to Downloading
[2026-09-09 12:06:50] Worker 3: Setting range 235569152-252395520 to Downloading
[2026-09-09 12:06:50] Worker 8: Setting range 0-33652736 to Downloading
[2026-09-09 12:06:50] Worker 10: Setting range 336527360-370180096 to Downloading
[2026-09-09 12:06:50] Worker 0: Setting range 100958208-134610944 to Downloading
[2026-09-09 12:06:50] Worker 14: Setting range 33652736-50479104 to Downloading
[2026-09-09 12:06:50] Worker 11: Setting range 504791040-521625600 to Downloading
[2026-09-09 12:06:50] Worker 6: Setting range 269221888-302874624 to Downloading
[2026-09-09 12:06:50] Worker 9: Setting range 437485568-454311936 to Downloading
[2026-09-09 12:06:50] Worker 7: Setting range 370180096-403832832 to Downloading
[2026-09-09 12:06:50] Worker 5: Task offset=50479104 length=16826368 took 205.7771ms
[2026-09-09 12:06:50] Worker 13: Task offset=319700992 length=16826368 took 180.4743ms
[2026-09-09 12:06:50] Worker 14: Task offset=33652736 length=16826368 took 107.5725ms
[2026-09-09 12:06:50] Worker 6: Task offset=269221888 length=33652736 took 99.4875ms
[2026-09-09 12:06:50] Worker 10: Task offset=336527360 length=33652736 took 179.9803ms
[2026-09-09 12:06:50] Worker 7: Task offset=370180096 length=33652736 took 99.4337ms
[2026-09-09 12:06:50] Worker 9: Task offset=437485568 length=16826368 took 100.4824ms
[2026-09-09 12:06:50] Worker 3: Task offset=235569152 length=16826368 took 245.6588ms
[2026-09-09 12:06:50] Worker 15: Task offset=134610944 length=33652736 took 259.1829ms
[2026-09-09 12:06:50] Worker 0: Task offset=100958208 length=33652736 took 205.3847ms
[2026-09-09 12:06:50] Worker 12: Task offset=67305472 length=33652736 took 262.3047ms
[2026-09-09 12:06:50] Worker 8: Task offset=0 length=33652736 took 264.3743ms
[2026-09-09 12:06:50] Worker 11: Task offset=504791040 length=16834560 took 193.0107ms
[2026-09-09 12:06:50] Worker 1: Setting range 218742784-235569152 to Downloading
[2026-09-09 12:06:50] Worker 4: Task offset=454311936 length=16826368 took 1.8923193s
[2026-09-09 12:06:50] Balancer: stole 16 MiB from worker 12 (new range: 84131840-100958208)
[2026-09-09 12:06:50] Worker 4: Setting range 84131840-100958208 to Downloading
[2026-09-09 12:06:50] Worker 5: Setting range 50479104-67305472 to Downloading
[2026-09-09 12:06:50] Worker 5: Task offset=50479104 length=16826368 took 136.0197ms
[2026-09-09 12:06:50] Worker 1: Task offset=218742784 length=16826368 took 287.016ms
[2026-09-09 12:06:50] Worker 1: task at offset 218742784 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:50] Worker 2: Task offset=403832832 length=33652736 took 788.8708ms
[2026-09-09 12:06:50] Balancer: stole 16 MiB from worker 7 (new range: 387006464-403832832)
[2026-09-09 12:06:50] Worker 2: Setting range 387006464-403832832 to Downloading
[2026-09-09 12:06:51] Worker 13: Setting range 319700992-336527360 to Downloading
[2026-09-09 12:06:51] Worker 14: Setting range 33652736-50479104 to Downloading
[2026-09-09 12:06:51] Worker 6: Setting range 269221888-302874624 to Downloading
[2026-09-09 12:06:51] Worker 10: Setting range 336527360-370180096 to Downloading
[2026-09-09 12:06:51] Worker 7: Setting range 370180096-387006464 to Downloading
[2026-09-09 12:06:51] Worker 9: Setting range 437485568-454311936 to Downloading
[2026-09-09 12:06:51] Worker 3: Setting range 235569152-252395520 to Downloading
[2026-09-09 12:06:51] Worker 15: Setting range 134610944-168263680 to Downloading
[2026-09-09 12:06:51] Worker 0: Setting range 100958208-134610944 to Downloading
[2026-09-09 12:06:51] Worker 12: Setting range 67305472-84131840 to Downloading
[2026-09-09 12:06:51] Worker 8: Setting range 0-33652736 to Downloading
[2026-09-09 12:06:51] Worker 11: Setting range 504791040-521625600 to Downloading
[2026-09-09 12:06:51] Worker 4: Task offset=84131840 length=16826368 took 555.8849ms
[2026-09-09 12:06:51] Balancer: stole 16 MiB from worker 15 (new range: 151437312-168263680)
[2026-09-09 12:06:51] Worker 4: Setting range 151437312-168263680 to Downloading
[2026-09-09 12:06:51] Worker 15: Task offset=134610944 length=33652736 took 139.5447ms
[2026-09-09 12:06:51] Worker 15: task at offset 134610944 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 13: Task offset=319700992 length=16826368 took 218.8369ms
[2026-09-09 12:06:51] Worker 13: task at offset 319700992 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 4: Task offset=151437312 length=16826368 took 93.9398ms
[2026-09-09 12:06:51] Worker 11: Task offset=504791040 length=16834560 took 128.9653ms
[2026-09-09 12:06:51] Worker 11: task at offset 504791040 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 9: Task offset=437485568 length=16826368 took 216.279ms
[2026-09-09 12:06:51] Worker 9: task at offset 437485568 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 7: Task offset=370180096 length=16826368 took 217.2942ms
[2026-09-09 12:06:51] Worker 7: task at offset 370180096 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 10: Task offset=336527360 length=33652736 took 220.5902ms
[2026-09-09 12:06:51] Worker 10: task at offset 336527360 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 2: Task offset=387006464 length=16826368 took 483.0396ms
[2026-09-09 12:06:51] Worker 3: Task offset=235569152 length=16826368 took 226.5311ms
[2026-09-09 12:06:51] Worker 3: task at offset 235569152 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 1: Setting range 218742784-235569152 to Downloading
[2026-09-09 12:06:51] Worker 0: Task offset=100958208 length=33652736 took 224.8263ms
[2026-09-09 12:06:51] Worker 0: task at offset 100958208 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 8: Task offset=0 length=33652736 took 222.3466ms
[2026-09-09 12:06:51] Worker 8: task at offset 0 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 14: Task offset=33652736 length=16826368 took 318.5357ms
[2026-09-09 12:06:51] Worker 14: task at offset 33652736 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:51] Worker 5: Setting range 50479104-67305472 to Downloading
[2026-09-09 12:06:51] Worker 4: Setting range 151437312-168263680 to Downloading
[2026-09-09 12:06:51] Worker 2: Setting range 387006464-403832832 to Downloading
[2026-09-09 12:06:51] Worker 12: Task offset=67305472 length=16826368 took 616.5921ms
[2026-09-09 12:06:51] Balancer: stole 8.0 MiB from worker 5 (new range: 58892288-67305472)
[2026-09-09 12:06:51] Worker 12: Setting range 58892288-67305472 to Downloading
[2026-09-09 12:06:51] Worker 15: Setting range 134610944-151437312 to Downloading
[2026-09-09 12:06:51] Worker 13: Setting range 319700992-336527360 to Downloading
[2026-09-09 12:06:51] Worker 11: Setting range 504791040-521625600 to Downloading
[2026-09-09 12:06:51] Worker 7: Setting range 370180096-387006464 to Downloading
[2026-09-09 12:06:51] Worker 9: Setting range 437485568-454311936 to Downloading
[2026-09-09 12:06:51] Worker 10: Setting range 336527360-370180096 to Downloading
[2026-09-09 12:06:51] Worker 2: Task offset=387006464 length=16826368 took 60.2598ms
[2026-09-09 12:06:51] Worker 3: Setting range 235569152-252395520 to Downloading
[2026-09-09 12:06:51] Worker 0: Setting range 100958208-134610944 to Downloading
[2026-09-09 12:06:51] Worker 8: Setting range 0-33652736 to Downloading
[2026-09-09 12:06:51] Worker 14: Setting range 33652736-50479104 to Downloading
[2026-09-09 12:06:51] Worker 1: Task offset=218742784 length=16826368 took 533.4765ms
[2026-09-09 12:06:51] Worker 12: Task offset=58892288 length=8413184 took 135.4293ms
[2026-09-09 12:06:51] Worker 5: Task offset=50479104 length=16826368 took 311.0566ms
[2026-09-09 12:06:51] Worker 5: task at offset 50479104 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:52] Worker 6: Task offset=269221888 length=33652736 took 903.4663ms
[2026-09-09 12:06:52] Balancer: stole 16 MiB from worker 10 (new range: 353353728-370180096)
[2026-09-09 12:06:52] Worker 6: Setting range 353353728-370180096 to Downloading
[2026-09-09 12:06:52] Worker 0: Task offset=100958208 length=33652736 took 392.8033ms
[2026-09-09 12:06:52] Worker 7: Task offset=370180096 length=16826368 took 480.0604ms
[2026-09-09 12:06:52] Worker 6: Task offset=353353728 length=16826368 took 299.4454ms
[2026-09-09 12:06:52] Worker 15: Task offset=134610944 length=16826368 took 486.3076ms
[2026-09-09 12:06:52] Worker 10: Task offset=336527360 length=33652736 took 482.0948ms
[2026-09-09 12:06:52] Worker 13: Task offset=319700992 length=16826368 took 489.0487ms
[2026-09-09 12:06:52] Worker 9: Task offset=437485568 length=16826368 took 486.9414ms
[2026-09-09 12:06:52] Worker 14: Task offset=33652736 length=16826368 took 389.3871ms
[2026-09-09 12:06:52] Worker 1: Setting range 218742784-235569152 to Downloading
[2026-09-09 12:06:52] Worker 12: Setting range 58892288-67305472 to Downloading
[2026-09-09 12:06:52] Worker 4: Task offset=151437312 length=16826368 took 629.5996ms
[2026-09-09 12:06:52] Balancer: stole 16 MiB from worker 0 (new range: 117784576-134610944)
[2026-09-09 12:06:52] Worker 4: Setting range 117784576-134610944 to Downloading
[2026-09-09 12:06:52] Worker 3: Task offset=235569152 length=16826368 took 504.191ms
[2026-09-09 12:06:52] Worker 5: Setting range 50479104-58892288 to Downloading
[2026-09-09 12:06:52] Worker 2: Setting range 387006464-403832832 to Downloading
[2026-09-09 12:06:52] Worker 0: Setting range 100958208-117784576 to Downloading
[2026-09-09 12:06:52] Worker 7: Setting range 370180096-387006464 to Downloading
[2026-09-09 12:06:52] Worker 6: Setting range 353353728-370180096 to Downloading
[2026-09-09 12:06:52] Worker 15: Setting range 134610944-151437312 to Downloading
[2026-09-09 12:06:52] Worker 10: Setting range 336527360-353353728 to Downloading
[2026-09-09 12:06:52] Worker 13: Setting range 319700992-336527360 to Downloading
[2026-09-09 12:06:52] Worker 9: Setting range 437485568-454311936 to Downloading
[2026-09-09 12:06:52] Worker 14: Setting range 33652736-50479104 to Downloading
[2026-09-09 12:06:52] Worker 3: Setting range 235569152-252395520 to Downloading
[2026-09-09 12:06:52] Worker 10: Task offset=336527360 length=16826368 took 110.9618ms
[2026-09-09 12:06:52] Worker 6: Task offset=353353728 length=16826368 took 112.509ms
[2026-09-09 12:06:52] Worker 11: Task offset=504791040 length=16834560 took 998.9695ms
[2026-09-09 12:06:52] Balancer: stole 10 MiB from worker 8 (new range: 22593536-33652736)
[2026-09-09 12:06:52] Worker 14: Task offset=33652736 length=16826368 took 110.3297ms
[2026-09-09 12:06:52] Worker 11: Setting range 22593536-33652736 to Downloading
[2026-09-09 12:06:52] Worker 1: Task offset=218742784 length=16826368 took 488.5735ms
[2026-09-09 12:06:52] Worker 2: Task offset=387006464 length=16826368 took 167.7467ms
[2026-09-09 12:06:52] Worker 2: task at offset 387006464 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:52] Worker 7: Task offset=370180096 length=16826368 took 119.8333ms
[2026-09-09 12:06:52] Worker 9: Task offset=437485568 length=16826368 took 113.0492ms
[2026-09-09 12:06:52] Worker 5: Task offset=50479104 length=8413184 took 376.0055ms
[2026-09-09 12:06:52] Worker 0: Task offset=100958208 length=16826368 took 121.9287ms
[2026-09-09 12:06:52] Worker 13: Task offset=319700992 length=16826368 took 115.7519ms
[2026-09-09 12:06:52] Worker 12: Task offset=58892288 length=8413184 took 487.1302ms
[2026-09-09 12:06:52] Worker 4: Task offset=117784576 length=16826368 took 473.7305ms
[2026-09-09 12:06:53] Worker 8: Task offset=0 length=33652736 took 1.121678s
[2026-09-09 12:06:53] Worker stopped early due to stealing
[2026-09-09 12:06:53] Balancer: stole 8.0 MiB from worker 4 (new range: 126197760-134610944)
[2026-09-09 12:06:53] Worker 8: Setting range 126197760-134610944 to Downloading
[2026-09-09 12:06:53] Worker 5: Setting range 50479104-58892288 to Downloading
[2026-09-09 12:06:53] Worker 4: Setting range 117784576-126197760 to Downloading
[2026-09-09 12:06:53] Worker 2: Setting range 387006464-403832832 to Downloading
[2026-09-09 12:06:53] Worker 3: Task offset=235569152 length=16826368 took 558.9726ms
[2026-09-09 12:06:53] Worker 4: Task offset=117784576 length=8413184 took 135.8125ms
[2026-09-09 12:06:53] Worker 5: Task offset=50479104 length=8413184 took 139.0198ms
[2026-09-09 12:06:53] Worker 11: Task offset=22593536 length=11059200 took 542.1329ms
[2026-09-09 12:06:53] Worker 15: Task offset=134610944 length=16826368 took 691.5355ms
[2026-09-09 12:06:53] Balancer: stole 8.0 MiB from worker 10 (new range: 344940544-353353728)
[2026-09-09 12:06:53] Worker 15: Setting range 344940544-353353728 to Downloading
[2026-09-09 12:06:53] Worker 6: Setting range 353353728-370180096 to Downloading
[2026-09-09 12:06:53] Worker 10: Setting range 336527360-344940544 to Downloading
[2026-09-09 12:06:53] Worker 14: Setting range 33652736-50479104 to Downloading
[2026-09-09 12:06:53] Worker 1: Setting range 218742784-235569152 to Downloading
[2026-09-09 12:06:53] Worker 7: Setting range 370180096-387006464 to Downloading
[2026-09-09 12:06:53] Worker 9: Setting range 437485568-454311936 to Downloading
[2026-09-09 12:06:53] Worker 0: Setting range 100958208-117784576 to Downloading
[2026-09-09 12:06:53] Worker 13: Setting range 319700992-336527360 to Downloading
[2026-09-09 12:06:53] Worker 12: Setting range 58892288-67305472 to Downloading
[2026-09-09 12:06:53] Worker 8: Task offset=126197760 length=8413184 took 628.3753ms
[2026-09-09 12:06:53] Balancer: stole 8.0 MiB from worker 13 (new range: 328114176-336527360)
[2026-09-09 12:06:53] Worker 8: Setting range 328114176-336527360 to Downloading
[2026-09-09 12:06:53] Worker 11: Setting range 22593536-33652736 to Downloading
[2026-09-09 12:06:53] Worker 14: Task offset=33652736 length=16826368 took 250.4718ms
[2026-09-09 12:06:53] Worker 14: task at offset 33652736 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:53] Worker 6: Task offset=353353728 length=16826368 took 253.5676ms
[2026-09-09 12:06:53] Worker 6: task at offset 353353728 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:53] Worker 15: Task offset=344940544 length=8413184 took 475.6817ms
[2026-09-09 12:06:53] Worker 7: Task offset=370180096 length=16826368 took 250.6741ms
[2026-09-09 12:06:53] Worker 7: task at offset 370180096 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:53] Worker 9: Task offset=437485568 length=16826368 took 250.6606ms
[2026-09-09 12:06:53] Worker 9: task at offset 437485568 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:53] Worker 8: Task offset=328114176 length=8413184 took 213.7306ms
[2026-09-09 12:06:53] Worker 11: Task offset=22593536 length=11059200 took 112.624ms
[2026-09-09 12:06:53] Worker 0: Task offset=100958208 length=16826368 took 252.3034ms
[2026-09-09 12:06:53] Worker 0: task at offset 100958208 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:53] Worker 13: Task offset=319700992 length=16826368 took 252.5287ms
[2026-09-09 12:06:53] Worker 13: task at offset 319700992 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:53] Worker 10: Task offset=336527360 length=8413184 took 259.2706ms
[2026-09-09 12:06:53] Worker 10: task at offset 336527360 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:53] Worker 12: Task offset=58892288 length=8413184 took 253.0333ms
[2026-09-09 12:06:53] Worker 12: task at offset 58892288 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:54] Worker 3: Setting range 235569152-252395520 to Downloading
[2026-09-09 12:06:54] Worker 4: Setting range 117784576-126197760 to Downloading
[2026-09-09 12:06:54] Worker 5: Setting range 50479104-58892288 to Downloading
[2026-09-09 12:06:54] Worker 1: Task offset=218742784 length=16826368 took 608.0103ms
[2026-09-09 12:06:54] Balancer: stole 8.0 MiB from worker 3 (new range: 243982336-252395520)
[2026-09-09 12:06:54] Worker 1: Setting range 243982336-252395520 to Downloading
[2026-09-09 12:06:54] Worker 15: Setting range 344940544-353353728 to Downloading
[2026-09-09 12:06:54] Worker 8: Setting range 328114176-336527360 to Downloading
[2026-09-09 12:06:54] Worker 14: Setting range 33652736-50479104 to Downloading
[2026-09-09 12:06:54] Worker 6: Setting range 353353728-370180096 to Downloading
[2026-09-09 12:06:54] Worker 7: Setting range 370180096-387006464 to Downloading
[2026-09-09 12:06:54] Worker 9: Setting range 437485568-454311936 to Downloading
[2026-09-09 12:06:54] Worker 0: Setting range 100958208-117784576 to Downloading
[2026-09-09 12:06:54] Worker 13: Setting range 319700992-328114176 to Downloading
[2026-09-09 12:06:54] Worker 10: Setting range 336527360-344940544 to Downloading
[2026-09-09 12:06:54] Worker 12: Setting range 58892288-67305472 to Downloading
[2026-09-09 12:06:54] Worker 2: Task offset=387006464 length=16826368 took 1.0750872s
[2026-09-09 12:06:54] Balancer: stole 8.0 MiB from worker 0 (new range: 109371392-117784576)
[2026-09-09 12:06:54] Worker 2: Setting range 109371392-117784576 to Downloading
[2026-09-09 12:06:54] Worker 5: Task offset=50479104 length=8413184 took 235.0811ms
[2026-09-09 12:06:54] Worker 5: task at offset 50479104 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:54] Worker 3: Task offset=235569152 length=16826368 took 239.2363ms
[2026-09-09 12:06:54] Worker 3: task at offset 235569152 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:54] Worker 15: Task offset=344940544 length=8413184 took 126.1095ms
[2026-09-09 12:06:54] Worker 4: Task offset=117784576 length=8413184 took 238.1995ms
[2026-09-09 12:06:54] Worker 4: task at offset 117784576 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:54] Worker 1: Task offset=243982336 length=8413184 took 170.2552ms
[2026-09-09 12:06:54] Worker 8: Task offset=328114176 length=8413184 took 314.0216ms
[2026-09-09 12:06:54] Balancer: stole 8.0 MiB from worker 6 (new range: 361766912-370180096)
[2026-09-09 12:06:54] Worker 8: Setting range 361766912-370180096 to Downloading
[2026-09-09 12:06:54] Worker 11: Setting range 22593536-33652736 to Downloading
[2026-09-09 12:06:54] Worker 1: Setting range 243982336-252395520 to Downloading
[2026-09-09 12:06:54] Worker 1: Task offset=243982336 length=8413184 took 69.9189ms
[2026-09-09 12:06:54] Worker 6: Task offset=353353728 length=16826368 took 500.3713ms
[2026-09-09 12:06:54] Worker 7: Task offset=370180096 length=16826368 took 499.8491ms
[2026-09-09 12:06:54] Worker 11: Task offset=22593536 length=11059200 took 197.2243ms
[2026-09-09 12:06:54] Worker 11: task at offset 22593536 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:54] Worker 2: Task offset=109371392 length=8413184 took 476.0098ms
[2026-09-09 12:06:54] Worker 0: Task offset=100958208 length=16826368 took 498.7898ms
[2026-09-09 12:06:54] Worker 9: Task offset=437485568 length=16826368 took 501.9375ms
[2026-09-09 12:06:54] Worker 10: Task offset=336527360 length=8413184 took 497.9734ms
[2026-09-09 12:06:54] Worker 12: Task offset=58892288 length=8413184 took 497.9631ms
[2026-09-09 12:06:54] Worker 14: Task offset=33652736 length=16826368 took 511.428ms
[2026-09-09 12:06:54] Worker 5: Setting range 50479104-58892288 to Downloading
[2026-09-09 12:06:54] Worker 3: Setting range 235569152-243982336 to Downloading
[2026-09-09 12:06:54] Worker 4: Setting range 117784576-126197760 to Downloading
[2026-09-09 12:06:55] Worker 13: Task offset=319700992 length=8413184 took 701.1469ms
[2026-09-09 12:06:55] Balancer: stole 8.0 MiB from worker 14 (new range: 42065920-50479104)
[2026-09-09 12:06:55] Worker 13: Setting range 42065920-50479104 to Downloading
[2026-09-09 12:06:55] Worker 8: Task offset=361766912 length=8413184 took 589.5313ms
[2026-09-09 12:06:55] Balancer: stole 8.0 MiB from worker 7 (new range: 378593280-387006464)
[2026-09-09 12:06:55] Worker 8: Setting range 378593280-387006464 to Downloading
[2026-09-09 12:06:55] Worker 15: Setting range 344940544-353353728 to Downloading
[2026-09-09 12:06:55] Worker 6: Setting range 353353728-361766912 to Downloading
[2026-09-09 12:06:55] Worker 7: Setting range 370180096-378593280 to Downloading
[2026-09-09 12:06:55] Worker 2: Setting range 109371392-117784576 to Downloading
[2026-09-09 12:06:55] Worker 0: Setting range 100958208-109371392 to Downloading
[2026-09-09 12:06:55] Worker 10: Setting range 336527360-344940544 to Downloading
[2026-09-09 12:06:55] Worker 9: Setting range 437485568-454311936 to Downloading
[2026-09-09 12:06:55] Worker 12: Setting range 58892288-67305472 to Downloading
[2026-09-09 12:06:55] Worker 14: Setting range 33652736-42065920 to Downloading
[2026-09-09 12:06:55] Worker 11: Setting range 22593536-33652736 to Downloading
[2026-09-09 12:06:55] Worker 13: Task offset=42065920 length=8413184 took 348.5146ms
[2026-09-09 12:06:55] Worker 8: Task offset=378593280 length=8413184 took 248.684ms
[2026-09-09 12:06:55] Worker 0: Task offset=100958208 length=8413184 took 152.5388ms
[2026-09-09 12:06:55] Worker 4: Task offset=117784576 length=8413184 took 527.4716ms
[2026-09-09 12:06:55] Worker 10: Task offset=336527360 length=8413184 took 151.4852ms
[2026-09-09 12:06:55] Worker 5: Task offset=50479104 length=8413184 took 529.002ms
[2026-09-09 12:06:55] Worker 9: Task offset=437485568 length=16826368 took 151.4852ms
[2026-09-09 12:06:55] Worker 15: Task offset=344940544 length=8413184 took 227.3355ms
[2026-09-09 12:06:55] Worker 15: task at offset 344940544 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:55] Worker 7: Task offset=370180096 length=8413184 took 154.6088ms
[2026-09-09 12:06:55] Worker 12: Task offset=58892288 length=8413184 took 151.9961ms
[2026-09-09 12:06:55] Worker 11: Task offset=22593536 length=11059200 took 55.329ms
[2026-09-09 12:06:55] Worker 2: Task offset=109371392 length=8413184 took 156.2111ms
[2026-09-09 12:06:55] Worker 14: Task offset=33652736 length=8413184 took 175.0486ms
[2026-09-09 12:06:55] Worker 3: Task offset=235569152 length=8413184 took 744.7076ms
[2026-09-09 12:06:55] Balancer: stole 8.0 MiB from worker 9 (new range: 445898752-454311936)
[2026-09-09 12:06:55] Worker 3: Setting range 445898752-454311936 to Downloading
[2026-09-09 12:06:55] Worker 1: Setting range 243982336-252395520 to Downloading
[2026-09-09 12:06:55] Worker 6: Task offset=353353728 length=8413184 took 481.4582ms
[2026-09-09 12:06:55] Balancer: stole 5.3 MiB from worker 11 (new range: 28123136-33652736)
[2026-09-09 12:06:55] Worker 6: Setting range 28123136-33652736 to Downloading
[2026-09-09 12:06:55] Worker 5: Setting range 50479104-58892288 to Downloading
[2026-09-09 12:06:55] Worker 4: Setting range 117784576-126197760 to Downloading
[2026-09-09 12:06:55] Worker 13: Setting range 42065920-50479104 to Downloading
[2026-09-09 12:06:55] Worker 8: Setting range 378593280-387006464 to Downloading
[2026-09-09 12:06:55] Worker 11: Setting range 22593536-28123136 to Downloading
[2026-09-09 12:06:55] Worker 3: Task offset=445898752 length=8413184 took 224.4207ms
[2026-09-09 12:06:55] Worker 15: Setting range 344940544-353353728 to Downloading
[2026-09-09 12:06:55] Worker 8: Task offset=378593280 length=8413184 took 112.2372ms
[2026-09-09 12:06:56] Worker 6: Task offset=28123136 length=5529600 took 279.3225ms
[2026-09-09 12:06:56] Balancer: stole 4.0 MiB from worker 10 (new range: 340733952-344940544)
[2026-09-09 12:06:56] Worker 6: Setting range 340733952-344940544 to Downloading
[2026-09-09 12:06:56] Worker 1: Task offset=243982336 length=8413184 took 437.0695ms
[2026-09-09 12:06:56] Balancer: stole 4.0 MiB from worker 5 (new range: 54685696-58892288)
[2026-09-09 12:06:56] Worker 1: Setting range 54685696-58892288 to Downloading
[2026-09-09 12:06:56] Worker 10: Setting range 336527360-340733952 to Downloading
[2026-09-09 12:06:56] Worker 9: Setting range 437485568-445898752 to Downloading
[2026-09-09 12:06:56] Worker 0: Setting range 100958208-109371392 to Downloading
[2026-09-09 12:06:56] Worker 12: Setting range 58892288-67305472 to Downloading
[2026-09-09 12:06:56] Worker 7: Setting range 370180096-378593280 to Downloading
[2026-09-09 12:06:56] Worker 2: Setting range 109371392-117784576 to Downloading
[2026-09-09 12:06:56] Worker 14: Setting range 33652736-42065920 to Downloading
[2026-09-09 12:06:56] Worker 3: Setting range 445898752-454311936 to Downloading
[2026-09-09 12:06:56] Worker 3: Task offset=445898752 length=8413184 took 159.691ms
[2026-09-09 12:06:56] Worker 13: Task offset=42065920 length=8413184 took 601.024ms
[2026-09-09 12:06:56] Worker 11: Task offset=22593536 length=5529600 took 599.3052ms
[2026-09-09 12:06:56] Worker 14: Task offset=33652736 length=8413184 took 171.3042ms
[2026-09-09 12:06:56] Worker 14: task at offset 33652736 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:56] Worker 12: Task offset=58892288 length=8413184 took 201.6923ms
[2026-09-09 12:06:56] Worker 12: task at offset 58892288 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:56] Worker 9: Task offset=437485568 length=8413184 took 202.8105ms
[2026-09-09 12:06:56] Worker 9: task at offset 437485568 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:56] Worker 7: Task offset=370180096 length=8413184 took 202.744ms
[2026-09-09 12:06:56] Worker 7: task at offset 370180096 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:56] Worker 4: Task offset=117784576 length=8413184 took 603.558ms
[2026-09-09 12:06:56] Worker 1: Task offset=54685696 length=4206592 took 323.1171ms
[2026-09-09 12:06:56] Worker 10: Task offset=336527360 length=4206592 took 203.8915ms
[2026-09-09 12:06:56] Worker 10: task at offset 336527360 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:56] Worker 2: Task offset=109371392 length=8413184 took 201.3465ms
[2026-09-09 12:06:56] Worker 2: task at offset 109371392 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:56] Worker 0: Task offset=100958208 length=8413184 took 215.7157ms
[2026-09-09 12:06:56] Worker 0: task at offset 100958208 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:56] Worker 5: Task offset=50479104 length=8413184 took 684.6249ms
[2026-09-09 12:06:56] Worker 6: Task offset=340733952 length=4206592 took 514.1511ms
[2026-09-09 12:06:56] Balancer: stole 4.0 MiB from worker 3 (new range: 450105344-454311936)
[2026-09-09 12:06:56] Worker 6: Setting range 450105344-454311936 to Downloading
[2026-09-09 12:06:56] Worker 8: Setting range 378593280-387006464 to Downloading
[2026-09-09 12:06:56] Worker 15: Task offset=344940544 length=8413184 took 838.4548ms
[2026-09-09 12:06:56] Balancer: stole 4.0 MiB from worker 8 (new range: 382799872-387006464)
[2026-09-09 12:06:56] Worker 15: Setting range 382799872-387006464 to Downloading
[2026-09-09 12:06:56] Worker 1: Setting range 54685696-58892288 to Downloading
[2026-09-09 12:06:56] Worker 15: Task offset=382799872 length=4206592 took 158.225ms
[2026-09-09 12:06:56] Worker 14: Setting range 33652736-42065920 to Downloading
[2026-09-09 12:06:56] Worker 12: Setting range 58892288-67305472 to Downloading
[2026-09-09 12:06:56] Worker 9: Setting range 437485568-445898752 to Downloading
[2026-09-09 12:06:56] Worker 7: Setting range 370180096-378593280 to Downloading
[2026-09-09 12:06:56] Worker 10: Setting range 336527360-340733952 to Downloading
[2026-09-09 12:06:56] Worker 2: Setting range 109371392-117784576 to Downloading
[2026-09-09 12:06:56] Worker 0: Setting range 100958208-109371392 to Downloading
[2026-09-09 12:06:56] Worker 8: Task offset=378593280 length=8413184 took 219.3977ms
[2026-09-09 12:06:56] Worker stopped early due to stealing
[2026-09-09 12:06:56] Balancer: stole 4.0 MiB from worker 12 (new range: 63098880-67305472)
[2026-09-09 12:06:56] Worker 8: Setting range 63098880-67305472 to Downloading
[2026-09-09 12:06:57] Worker 6: Task offset=450105344 length=4206592 took 548.6361ms
[2026-09-09 12:06:57] Balancer: stole 4.0 MiB from worker 0 (new range: 105164800-109371392)
[2026-09-09 12:06:57] Worker 6: Setting range 105164800-109371392 to Downloading
[2026-09-09 12:06:57] Worker 11: Setting range 22593536-28123136 to Downloading
[2026-09-09 12:06:57] Worker 13: Setting range 42065920-50479104 to Downloading
[2026-09-09 12:06:57] Worker 3: Setting range 445898752-450105344 to Downloading
[2026-09-09 12:06:57] Worker 4: Setting range 117784576-126197760 to Downloading
[2026-09-09 12:06:57] Worker 5: Setting range 50479104-54685696 to Downloading
[2026-09-09 12:06:57] Worker 15: Setting range 382799872-387006464 to Downloading
[2026-09-09 12:06:57] Worker 13: Task offset=42065920 length=8413184 took 199.7918ms
[2026-09-09 12:06:57] Worker 13: task at offset 42065920 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:57] Worker 1: Task offset=54685696 length=4206592 took 598.9727ms
[2026-09-09 12:06:57] Worker 4: Task offset=117784576 length=8413184 took 201.1935ms
[2026-09-09 12:06:57] Worker 4: task at offset 117784576 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:57] Worker 14: Task offset=33652736 length=8413184 took 502.0346ms
[2026-09-09 12:06:57] Worker 7: Task offset=370180096 length=8413184 took 501.54ms
[2026-09-09 12:06:57] Worker 15: Task offset=382799872 length=4206592 took 107.9537ms
[2026-09-09 12:06:57] Worker 5: Task offset=50479104 length=4206592 took 121.034ms
[2026-09-09 12:06:57] Worker 5: task at offset 50479104 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:57] Worker 3: Task offset=445898752 length=4206592 took 205.5441ms
[2026-09-09 12:06:57] Worker 3: task at offset 445898752 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:57] Worker 10: Task offset=336527360 length=4206592 took 502.5717ms
[2026-09-09 12:06:57] Worker 12: Task offset=58892288 length=8413184 took 506.1747ms
[2026-09-09 12:06:57] Worker 11: Task offset=22593536 length=5529600 took 208.0999ms
[2026-09-09 12:06:57] Worker 11: task at offset 22593536 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:57] Worker 8: Task offset=63098880 length=4206592 took 547.7481ms
[2026-09-09 12:06:57] Worker 2: Task offset=109371392 length=8413184 took 604.7457ms
[2026-09-09 12:06:57] Worker 0: Task offset=100958208 length=8413184 took 608.3017ms
[2026-09-09 12:06:57] Worker 6: Task offset=105164800 length=4206592 took 570.3047ms
[2026-09-09 12:06:57] Balancer: stole 4.0 MiB from worker 7 (new range: 374386688-378593280)
[2026-09-09 12:06:57] Worker 6: Setting range 374386688-378593280 to Downloading
[2026-09-09 12:06:57] Worker 9: Task offset=437485568 length=8413184 took 776.785ms
[2026-09-09 12:06:57] Balancer: stole 4.0 MiB from worker 14 (new range: 37859328-42065920)
[2026-09-09 12:06:57] Worker 9: Setting range 37859328-42065920 to Downloading
[2026-09-09 12:06:57] Worker 14: Setting range 33652736-37859328 to Downloading
[2026-09-09 12:06:57] Worker 7: Setting range 370180096-374386688 to Downloading
[2026-09-09 12:06:57] Worker 10: Setting range 336527360-340733952 to Downloading
[2026-09-09 12:06:57] Worker 12: Setting range 58892288-63098880 to Downloading
[2026-09-09 12:06:57] Worker 8: Setting range 63098880-67305472 to Downloading
[2026-09-09 12:06:57] Worker 13: Setting range 42065920-50479104 to Downloading
[2026-09-09 12:06:57] Worker 4: Setting range 117784576-126197760 to Downloading
[2026-09-09 12:06:57] Worker 5: Setting range 50479104-54685696 to Downloading
[2026-09-09 12:06:57] Worker 3: Setting range 445898752-450105344 to Downloading
[2026-09-09 12:06:57] Worker 9: Task offset=37859328 length=4206592 took 228.073ms
[2026-09-09 12:06:57] Worker 2: Setting range 109371392-117784576 to Downloading
[2026-09-09 12:06:57] Worker 11: Setting range 22593536-28123136 to Downloading
[2026-09-09 12:06:57] Worker 0: Setting range 100958208-105164800 to Downloading
[2026-09-09 12:06:58] Worker 14: Task offset=33652736 length=4206592 took 159.2079ms
[2026-09-09 12:06:58] Worker 7: Task offset=370180096 length=4206592 took 165.9341ms
[2026-09-09 12:06:58] Worker 12: Task offset=58892288 length=4206592 took 174.6494ms
[2026-09-09 12:06:58] Worker 6: Task offset=374386688 length=4206592 took 390.7597ms
[2026-09-09 12:06:58] Balancer: stole 4.0 MiB from worker 4 (new range: 121991168-126197760)
[2026-09-09 12:06:58] Worker 6: Setting range 121991168-126197760 to Downloading
[2026-09-09 12:06:58] Worker 10: Task offset=336527360 length=4206592 took 350.7026ms
[2026-09-09 12:06:58] Balancer: stole 4.0 MiB from worker 13 (new range: 46272512-50479104)
[2026-09-09 12:06:58] Worker 10: Setting range 46272512-50479104 to Downloading
[2026-09-09 12:06:58] Worker 1: Setting range 54685696-58892288 to Downloading
[2026-09-09 12:06:58] Worker 15: Setting range 382799872-387006464 to Downloading
[2026-09-09 12:06:58] Worker 9: Setting range 37859328-42065920 to Downloading
[2026-09-09 12:06:58] Worker 8: Task offset=63098880 length=4206592 took 523.8697ms
[2026-09-09 12:06:58] Worker 5: Task offset=50479104 length=4206592 took 498.5882ms
[2026-09-09 12:06:58] Worker 4: Task offset=117784576 length=8413184 took 499.6808ms
[2026-09-09 12:06:58] Worker 1: Task offset=54685696 length=4206592 took 201.4504ms
[2026-09-09 12:06:58] Worker 1: task at offset 54685696 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:58] Worker 15: Task offset=382799872 length=4206592 took 198.3566ms
[2026-09-09 12:06:58] Worker 15: task at offset 382799872 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:58] Worker 9: Task offset=37859328 length=4206592 took 96.0975ms
[2026-09-09 12:06:58] Worker 0: Task offset=100958208 length=4206592 took 484.7763ms
[2026-09-09 12:06:58] Worker 3: Task offset=445898752 length=4206592 took 573.4317ms
[2026-09-09 12:06:58] Worker 2: Task offset=109371392 length=8413184 took 574.0041ms
[2026-09-09 12:06:58] Worker 10: Task offset=46272512 length=4206592 took 328.6622ms
[2026-09-09 12:06:58] Worker 13: Task offset=42065920 length=8413184 took 589.4369ms
[2026-09-09 12:06:58] Worker 6: Task offset=121991168 length=4206592 took 481.4874ms
[2026-09-09 12:06:58] Balancer: stole 4.0 MiB from worker 2 (new range: 113577984-117784576)
[2026-09-09 12:06:58] Worker 6: Setting range 113577984-117784576 to Downloading
[2026-09-09 12:06:58] Worker 11: Task offset=22593536 length=5529600 took 731.6231ms
[2026-09-09 12:06:58] Balancer: stole 2.0 MiB from worker 0 (new range: 103059456-105164800)
[2026-09-09 12:06:58] Worker 11: Setting range 103059456-105164800 to Downloading
[2026-09-09 12:06:58] Worker 14: Setting range 33652736-37859328 to Downloading
[2026-09-09 12:06:58] Worker 7: Setting range 370180096-374386688 to Downloading
[2026-09-09 12:06:58] Worker 12: Setting range 58892288-63098880 to Downloading
[2026-09-09 12:06:58] Worker 4: Setting range 117784576-121991168 to Downloading
[2026-09-09 12:06:58] Worker 5: Setting range 50479104-54685696 to Downloading
[2026-09-09 12:06:58] Worker 4: Task offset=117784576 length=4206592 took 69.3521ms
[2026-09-09 12:06:58] Worker 12: Task offset=58892288 length=4206592 took 92.0841ms
[2026-09-09 12:06:58] Worker 12: task at offset 58892288 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:58] Worker 3: Setting range 445898752-450105344 to Downloading
[2026-09-09 12:06:58] Worker 10: Setting range 46272512-50479104 to Downloading
[2026-09-09 12:06:58] Worker 13: Setting range 42065920-46272512 to Downloading
[2026-09-09 12:06:58] Worker 1: Setting range 54685696-58892288 to Downloading
[2026-09-09 12:06:58] Worker 15: Setting range 382799872-387006464 to Downloading
[2026-09-09 12:06:58] Worker 7: Task offset=370180096 length=4206592 took 152.4257ms
[2026-09-09 12:06:58] Worker 7: task at offset 370180096 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:59] Worker 14: Task offset=33652736 length=4206592 took 187.5307ms
[2026-09-09 12:06:59] Worker 14: task at offset 33652736 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:59] Worker 5: Task offset=50479104 length=4206592 took 155.6822ms
[2026-09-09 12:06:59] Worker 11: Task offset=103059456 length=2105344 took 357.345ms
[2026-09-09 12:06:59] Balancer: stole 2.0 MiB from worker 6 (new range: 115679232-117784576)
[2026-09-09 12:06:59] Worker 11: Setting range 115679232-117784576 to Downloading
[2026-09-09 12:06:59] Worker 6: Task offset=113577984 length=4206592 took 582.8642ms
[2026-09-09 12:06:59] Worker stopped early due to stealing
[2026-09-09 12:06:59] Balancer: stole 2.0 MiB from worker 4 (new range: 119885824-121991168)
[2026-09-09 12:06:59] Worker 6: Setting range 119885824-121991168 to Downloading
[2026-09-09 12:06:59] Worker 8: Setting range 63098880-67305472 to Downloading
[2026-09-09 12:06:59] Worker 9: Setting range 37859328-42065920 to Downloading
[2026-09-09 12:06:59] Worker 0: Setting range 100958208-103059456 to Downloading
[2026-09-09 12:06:59] Worker 2: Setting range 109371392-113577984 to Downloading
[2026-09-09 12:06:59] Worker 3: Task offset=445898752 length=4206592 took 474.486ms
[2026-09-09 12:06:59] Worker 0: Task offset=100958208 length=2101248 took 145.6367ms
[2026-09-09 12:06:59] Worker 0: task at offset 100958208 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:59] Worker 9: Task offset=37859328 length=4206592 took 150.7391ms
[2026-09-09 12:06:59] Worker 9: task at offset 37859328 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:59] Worker 11: Task offset=115679232 length=2105344 took 356.97ms
[2026-09-09 12:06:59] Worker 8: Task offset=63098880 length=4206592 took 152.9454ms
[2026-09-09 12:06:59] Worker 8: task at offset 63098880 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:59] Worker 6: Task offset=119885824 length=2105344 took 263.3277ms
[2026-09-09 12:06:59] Worker 2: Task offset=109371392 length=4206592 took 73.1374ms
[2026-09-09 12:06:59] Worker 2: task at offset 109371392 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:59] Worker 12: Setting range 58892288-63098880 to Downloading
[2026-09-09 12:06:59] Worker 7: Setting range 370180096-374386688 to Downloading
[2026-09-09 12:06:59] Worker 15: Task offset=382799872 length=4206592 took 525.1194ms
[2026-09-09 12:06:59] Worker 1: Task offset=54685696 length=4206592 took 527.5981ms
[2026-09-09 12:06:59] Worker 14: Setting range 33652736-37859328 to Downloading
[2026-09-09 12:06:59] Worker 10: Task offset=46272512 length=4206592 took 583.342ms
[2026-09-09 12:06:59] Balancer: stole 2.0 MiB from worker 1 (new range: 56786944-58892288)
[2026-09-09 12:06:59] Worker 10: Setting range 56786944-58892288 to Downloading
[2026-09-09 12:06:59] Worker 13: Task offset=42065920 length=4206592 took 591.0436ms
[2026-09-09 12:06:59] Balancer: stole 2.0 MiB from worker 7 (new range: 372281344-374386688)
[2026-09-09 12:06:59] Worker 13: Setting range 372281344-374386688 to Downloading
[2026-09-09 12:06:59] Worker 4: Setting range 117784576-119885824 to Downloading
[2026-09-09 12:06:59] Worker 11: Setting range 115679232-117784576 to Downloading
[2026-09-09 12:06:59] Worker 6: Setting range 119885824-121991168 to Downloading
[2026-09-09 12:06:59] Worker 5: Setting range 50479104-54685696 to Downloading
[2026-09-09 12:06:59] Worker 15: Setting range 382799872-387006464 to Downloading
[2026-09-09 12:06:59] Worker 1: Setting range 54685696-56786944 to Downloading
[2026-09-09 12:06:59] Worker 0: Setting range 100958208-103059456 to Downloading
[2026-09-09 12:06:59] Worker 9: Setting range 37859328-42065920 to Downloading
[2026-09-09 12:06:59] Worker 8: Setting range 63098880-67305472 to Downloading
[2026-09-09 12:06:59] Worker 2: Setting range 109371392-113577984 to Downloading
[2026-09-09 12:06:59] Worker 4: Task offset=117784576 length=2101248 took 203.9064ms
[2026-09-09 12:06:59] Worker 4: task at offset 117784576 failed after 3 retries: unexpected status: 403
[2026-09-09 12:06:59] Worker 12: Task offset=58892288 length=4206592 took 502.5873ms
[2026-09-09 12:06:59] Worker 1: Task offset=54685696 length=2101248 took 46.118ms
[2026-09-09 12:06:59] Worker 15: Task offset=382799872 length=4206592 took 49.2638ms
[2026-09-09 12:06:59] Worker 10: Task offset=56786944 length=2105344 took 410.0628ms
[2026-09-09 12:06:59] Worker 6: Task offset=119885824 length=2105344 took 124.4498ms
[2026-09-09 12:06:59] Worker 11: Task offset=115679232 length=2105344 took 125.9811ms
[2026-09-09 12:07:00] Worker 7: Task offset=370180096 length=4206592 took 528.8076ms
[2026-09-09 12:07:00] Worker stopped early due to stealing
[2026-09-09 12:07:00] Balancer: stole 2.0 MiB from worker 12 (new range: 60993536-63098880)
[2026-09-09 12:07:00] Worker 7: Setting range 60993536-63098880 to Downloading
[2026-09-09 12:07:00] Worker 14: Task offset=33652736 length=4206592 took 501.1695ms
[2026-09-09 12:07:00] Worker 13: Task offset=372281344 length=2105344 took 480.5962ms
[2026-09-09 12:07:00] Worker 5: Task offset=50479104 length=4206592 took 242.5719ms
[2026-09-09 12:07:00] Balancer: stole 2.0 MiB from worker 8 (new range: 65200128-67305472)
[2026-09-09 12:07:00] Worker 5: Setting range 65200128-67305472 to Downloading
[2026-09-09 12:07:00] Worker 3: Setting range 445898752-450105344 to Downloading
[2026-09-09 12:07:00] Worker 12: Setting range 58892288-60993536 to Downloading
[2026-09-09 12:07:00] Worker 10: Setting range 56786944-58892288 to Downloading
[2026-09-09 12:07:00] Worker 14: Setting range 33652736-37859328 to Downloading
[2026-09-09 12:07:00] Worker 13: Setting range 372281344-374386688 to Downloading
[2026-09-09 12:07:00] Worker 3: Task offset=445898752 length=4206592 took 222.4188ms
[2026-09-09 12:07:00] Worker 3: task at offset 445898752 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:00] Worker 4: Setting range 117784576-119885824 to Downloading
[2026-09-09 12:07:00] Worker 5: Task offset=65200128 length=2105344 took 375.5788ms
[2026-09-09 12:07:00] Worker 7: Task offset=60993536 length=2105344 took 426.387ms
[2026-09-09 12:07:00] Worker 0: Task offset=100958208 length=2101248 took 524.5901ms
[2026-09-09 12:07:00] Worker 2: Task offset=109371392 length=4206592 took 522.488ms
[2026-09-09 12:07:00] Worker 10: Task offset=56786944 length=2105344 took 100.7955ms
[2026-09-09 12:07:00] Worker 12: Task offset=58892288 length=2101248 took 172.1934ms
[2026-09-09 12:07:00] Balancer: stole 2.0 MiB from worker 14 (new range: 35753984-37859328)
[2026-09-09 12:07:00] Worker 12: Setting range 35753984-37859328 to Downloading
[2026-09-09 12:07:00] Worker 8: Task offset=63098880 length=4206592 took 601.9152ms
[2026-09-09 12:07:00] Worker 9: Task offset=37859328 length=4206592 took 664.3582ms
[2026-09-09 12:07:00] Balancer: stole 2.0 MiB from worker 15 (new range: 384901120-387006464)
[2026-09-09 12:07:00] Worker 9: Setting range 384901120-387006464 to Downloading
[2026-09-09 12:07:00] Worker 1: Setting range 54685696-56786944 to Downloading
[2026-09-09 12:07:00] Worker 15: Setting range 382799872-384901120 to Downloading
[2026-09-09 12:07:00] Worker 6: Setting range 119885824-121991168 to Downloading
[2026-09-09 12:07:00] Worker 11: Setting range 115679232-117784576 to Downloading
[2026-09-09 12:07:00] Worker 5: Setting range 65200128-67305472 to Downloading
[2026-09-09 12:07:00] Worker 7: Setting range 60993536-63098880 to Downloading
[2026-09-09 12:07:00] Worker 2: Setting range 109371392-113577984 to Downloading
[2026-09-09 12:07:00] Worker 0: Setting range 100958208-103059456 to Downloading
[2026-09-09 12:07:00] Worker 7: Task offset=60993536 length=2105344 took 74.2395ms
[2026-09-09 12:07:00] Worker 1: Task offset=54685696 length=2101248 took 176.4573ms
[2026-09-09 12:07:00] Worker 1: task at offset 54685696 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:00] Worker 12: Task offset=35753984 length=2105344 took 405.534ms
[2026-09-09 12:07:00] Worker 6: Task offset=119885824 length=2105344 took 174.3256ms
[2026-09-09 12:07:00] Worker 6: task at offset 119885824 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:00] Worker 11: Task offset=115679232 length=2105344 took 174.8174ms
[2026-09-09 12:07:00] Worker 11: task at offset 115679232 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:00] Worker 8: Setting range 63098880-65200128 to Downloading
[2026-09-09 12:07:00] Worker 5: Task offset=65200128 length=2105344 took 80.4864ms
[2026-09-09 12:07:00] Worker 2: Task offset=109371392 length=4206592 took 78.8565ms
[2026-09-09 12:07:00] Worker 0: Task offset=100958208 length=2101248 took 79.7737ms
[2026-09-09 12:07:00] Worker 4: Task offset=117784576 length=2101248 took 482.019ms
[2026-09-09 12:07:00] Worker 15: Task offset=382799872 length=2101248 took 180.976ms
[2026-09-09 12:07:00] Worker 15: task at offset 382799872 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:00] Worker 9: Task offset=384901120 length=2105344 took 340.1056ms
[2026-09-09 12:07:00] Worker 3: Setting range 445898752-450105344 to Downloading
[2026-09-09 12:07:00] Worker 14: Task offset=33652736 length=4206592 took 580.5574ms
[2026-09-09 12:07:00] Worker stopped early due to stealing
[2026-09-09 12:07:00] Balancer: stole 2.0 MiB from worker 3 (new range: 448000000-450105344)
[2026-09-09 12:07:00] Worker 14: Setting range 448000000-450105344 to Downloading
[2026-09-09 12:07:01] Worker 13: Task offset=372281344 length=2105344 took 682.9545ms
[2026-09-09 12:07:01] Balancer: stole 2.0 MiB from worker 2 (new range: 111472640-113577984)
[2026-09-09 12:07:01] Worker 13: Setting range 111472640-113577984 to Downloading
[2026-09-09 12:07:01] Worker 10: Setting range 56786944-58892288 to Downloading
[2026-09-09 12:07:01] Worker 12: Setting range 35753984-37859328 to Downloading
[2026-09-09 12:07:01] Worker 4: Setting range 117784576-119885824 to Downloading
[2026-09-09 12:07:01] Worker 9: Setting range 384901120-387006464 to Downloading
[2026-09-09 12:07:01] Worker 1: Setting range 54685696-56786944 to Downloading
[2026-09-09 12:07:01] Worker 4: Task offset=117784576 length=2101248 took 95.7422ms
[2026-09-09 12:07:01] Worker 3: Task offset=445898752 length=4206592 took 478.0856ms
[2026-09-09 12:07:01] Worker 6: Setting range 119885824-121991168 to Downloading
[2026-09-09 12:07:01] Worker 13: Task offset=111472640 length=2105344 took 310.958ms
[2026-09-09 12:07:01] Worker 10: Task offset=56786944 length=2105344 took 175.2925ms
[2026-09-09 12:07:01] Worker 10: task at offset 56786944 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:01] Worker 14: Task offset=448000000 length=2105344 took 422.3619ms
[2026-09-09 12:07:01] Worker 11: Setting range 115679232-117784576 to Downloading
[2026-09-09 12:07:01] Worker 12: Task offset=35753984 length=2105344 took 102.1986ms
[2026-09-09 12:07:01] Worker 15: Setting range 382799872-384901120 to Downloading
[2026-09-09 12:07:01] Worker 9: Task offset=384901120 length=2105344 took 147.4991ms
[2026-09-09 12:07:01] Worker 8: Task offset=63098880 length=2101248 took 698.4886ms
[2026-09-09 12:07:01] Worker 7: Setting range 60993536-63098880 to Downloading
[2026-09-09 12:07:01] Worker 5: Setting range 65200128-67305472 to Downloading
[2026-09-09 12:07:01] Worker 2: Setting range 109371392-111472640 to Downloading
[2026-09-09 12:07:01] Worker 0: Setting range 100958208-103059456 to Downloading
[2026-09-09 12:07:01] Worker 3: Setting range 445898752-448000000 to Downloading
[2026-09-09 12:07:01] Worker 14: Setting range 448000000-450105344 to Downloading
[2026-09-09 12:07:01] Worker 13: Setting range 111472640-113577984 to Downloading
[2026-09-09 12:07:01] Worker 15: Task offset=382799872 length=2101248 took 472.3042ms
[2026-09-09 12:07:01] Worker 0: Task offset=100958208 length=2101248 took 172.8439ms
[2026-09-09 12:07:01] Worker 0: task at offset 100958208 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:01] Worker 13: Task offset=111472640 length=2105344 took 76.2452ms
[2026-09-09 12:07:01] Worker 6: Task offset=119885824 length=2105344 took 476.4352ms
[2026-09-09 12:07:01] Worker 1: Task offset=54685696 length=2101248 took 489.4846ms
[2026-09-09 12:07:01] Worker 3: Task offset=445898752 length=2101248 took 89.3552ms
[2026-09-09 12:07:01] Worker 14: Task offset=448000000 length=2105344 took 88.7719ms
[2026-09-09 12:07:01] Worker 2: Task offset=109371392 length=2101248 took 187.063ms
[2026-09-09 12:07:01] Worker 2: task at offset 109371392 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:01] Worker 7: Task offset=60993536 length=2105344 took 195.341ms
[2026-09-09 12:07:01] Worker 7: task at offset 60993536 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:01] Worker 10: Setting range 56786944-58892288 to Downloading
[2026-09-09 12:07:01] Worker 11: Task offset=115679232 length=2105344 took 531.0615ms
[2026-09-09 12:07:02] Worker 5: Task offset=65200128 length=2105344 took 351.6324ms
[2026-09-09 12:07:02] Worker 4: Setting range 117784576-119885824 to Downloading
[2026-09-09 12:07:02] Worker 12: Setting range 35753984-37859328 to Downloading
[2026-09-09 12:07:02] Worker 15: Setting range 382799872-384901120 to Downloading
[2026-09-09 12:07:02] Worker 6: Setting range 119885824-121991168 to Downloading
[2026-09-09 12:07:02] Worker 1: Setting range 54685696-56786944 to Downloading
[2026-09-09 12:07:02] Worker 0: Setting range 100958208-103059456 to Downloading
[2026-09-09 12:07:02] Worker 2: Setting range 109371392-111472640 to Downloading
[2026-09-09 12:07:02] Worker 7: Setting range 60993536-63098880 to Downloading
[2026-09-09 12:07:02] Worker 10: Task offset=56786944 length=2105344 took 510.5996ms
[2026-09-09 12:07:02] Worker 15: Task offset=382799872 length=2101248 took 134.8655ms
[2026-09-09 12:07:02] Worker 1: Task offset=54685696 length=2101248 took 122.5165ms
[2026-09-09 12:07:02] Worker 6: Task offset=119885824 length=2105344 took 135.4162ms
[2026-09-09 12:07:02] Worker 12: Task offset=35753984 length=2105344 took 251.2981ms
[2026-09-09 12:07:02] Worker 4: Task offset=117784576 length=2101248 took 421.6901ms
[2026-09-09 12:07:02] Worker 13: Setting range 111472640-113577984 to Downloading
[2026-09-09 12:07:02] Worker 3: Setting range 445898752-448000000 to Downloading
[2026-09-09 12:07:02] Worker 14: Setting range 448000000-450105344 to Downloading
[2026-09-09 12:07:02] Worker 10: Setting range 56786944-58892288 to Downloading
[2026-09-09 12:07:02] Worker 7: Task offset=60993536 length=2105344 took 522.0681ms
[2026-09-09 12:07:02] Worker 0: Task offset=100958208 length=2101248 took 541.337ms
[2026-09-09 12:07:02] Worker 2: Task offset=109371392 length=2101248 took 528.7172ms
[2026-09-09 12:07:02] Worker 10: Task offset=56786944 length=2105344 took 107.4164ms
[2026-09-09 12:07:02] Worker 3: Task offset=445898752 length=2101248 took 231.8399ms
[2026-09-09 12:07:02] Worker 3: task at offset 445898752 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:03] Worker 14: Task offset=448000000 length=2105344 took 309.6491ms
[2026-09-09 12:07:03] Worker 13: Task offset=111472640 length=2105344 took 455.4769ms
[2026-09-09 12:07:03] Worker 15: Setting range 382799872-384901120 to Downloading
[2026-09-09 12:07:03] Worker 6: Setting range 119885824-121991168 to Downloading
[2026-09-09 12:07:03] Worker 1: Setting range 54685696-56786944 to Downloading
[2026-09-09 12:07:03] Worker 7: Setting range 60993536-63098880 to Downloading
[2026-09-09 12:07:03] Worker 0: Setting range 100958208-103059456 to Downloading
[2026-09-09 12:07:03] Worker 2: Setting range 109371392-111472640 to Downloading
[2026-09-09 12:07:03] Worker 1: Task offset=54685696 length=2101248 took 206.0395ms
[2026-09-09 12:07:03] Worker 1: task at offset 54685696 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:03] Worker 2: Task offset=109371392 length=2101248 took 99.7596ms
[2026-09-09 12:07:03] Worker 0: Task offset=100958208 length=2101248 took 100.8003ms
[2026-09-09 12:07:03] Worker 15: Task offset=382799872 length=2101248 took 208.181ms
[2026-09-09 12:07:03] Worker 15: task at offset 382799872 failed after 3 retries: unexpected status: 403
[2026-09-09 12:07:03] Worker 3: Setting range 445898752-448000000 to Downloading
[2026-09-09 12:07:03] Worker 7: Task offset=60993536 length=2105344 took 159.7052ms
[2026-09-09 12:07:03] Worker 6: Task offset=119885824 length=2105344 took 323.3059ms
[2026-09-09 12:07:03] Worker 10: Setting range 56786944-58892288 to Downloading
[2026-09-09 12:07:03] Worker 1: Setting range 54685696-56786944 to Downloading
[2026-09-09 12:07:03] Worker 15: Setting range 382799872-384901120 to Downloading
[2026-09-09 12:07:04] Worker 10: Task offset=56786944 length=2105344 took 276.425ms
[2026-09-09 12:07:04] Worker 3: Task offset=445898752 length=2101248 took 581.527ms
[2026-09-09 12:07:04] Worker 0: Setting range 100958208-103059456 to Downloading
[2026-09-09 12:07:04] Worker 2: Setting range 109371392-111472640 to Downloading
[2026-09-09 12:07:04] Worker 1: Task offset=54685696 length=2101248 took 517.7547ms
[2026-09-09 12:07:04] Worker 15: Task offset=382799872 length=2101248 took 518.296ms
[2026-09-09 12:07:04] Worker 0: Task offset=100958208 length=2101248 took 283.952ms
[2026-09-09 12:07:04] Worker 2: Task offset=109371392 length=2101248 took 287.622ms
[2026-09-09 12:07:04] Worker 1: Setting range 54685696-56786944 to Downloading
[2026-09-09 12:07:04] Worker 15: Setting range 382799872-384901120 to Downloading
[2026-09-09 12:07:05] Worker 1: Task offset=54685696 length=2101248 took 335.4173ms
[2026-09-09 12:07:05] Worker 15: Task offset=382799872 length=2101248 took 336.0272ms
[2026-09-09 12:07:05] Worker 15 finished
[2026-09-09 12:07:05] Worker 6 finished
[2026-09-09 12:07:05] Worker 8 finished
[2026-09-09 12:07:05] Worker 9 finished
[2026-09-09 12:07:05] Worker 10 finished
[2026-09-09 12:07:05] Worker 3 finished
[2026-09-09 12:07:05] Worker 0 finished
[2026-09-09 12:07:05] Worker 2 finished
[2026-09-09 12:07:05] Worker 1 finished
[2026-09-09 12:07:05] Worker 14 finished
[2026-09-09 12:07:05] Worker 11 finished
[2026-09-09 12:07:05] Worker 12 finished
[2026-09-09 12:07:05] Worker 4 finished
[2026-09-09 12:07:05] Worker 5 finished
[2026-09-09 12:07:05] Worker 13 finished
[2026-09-09 12:07:05] Worker 7 finished
[2026-09-09 12:07:05] NetworkPool: ReleaseTransport (key={proxyURL: customDNS: maxConns:0}, refs=0)
[2026-09-09 12:07:15] NetworkPool: idle timeout reached, evicting transport

As shown in the log, the download went through over 40 task-stealing adjustments across workers as well as repeated 403 backoffs, yet the file remained completely intact with a matching hash. Earlier in downstream testing when I was working on errored task resume, an imperfect interaction with the soft-403 limits caused errors to bubble up directly. A 5GB file ended up going through around 8 resume cycles, and the final hash still came out 100% clean. Between this trace and that experience, the VP mechanism has proven to be very solid and trustworthy, at least on Windows.

One unexpected thing I noticed from the trace is that the AIMD/adaptive stealing seems to kick in right at the start of the download, which is a bit earlier than expected. To address this, maybe a simple tail guard like activeWorkers > 0 && totalRemaining < int64(activeWorkers)*defaultMinChunk could help.

It’s reassuring to see the underlying state machine become this solid, which significantly reduces the need for defensive handling on the UI layer. Looking forward to seeing this PR merged so we can build further improvements on top of a reliable physical ground truth.

SuperCoolPencil · 2026-09-09T14:31:12Z

Hi @superGekFordJ

As always thanks so much for the PR.

I have exams next week and just a lot going on right now in general so getting to this may take a bit of time. Just wanted to let you know :)

superGekFordJ · 2026-09-09T15:00:09Z

Don't worry at all, no rush whatsoever! I just happened to run into these findings while testing and wanted to write them down here to keep track. Best of luck with your exams! :)

A vanished URL (404/401/410/416) still wrapped ErrPermanentHTTP but then burned GetMaxTaskRetries (default 3) plus the single-mirror RetryBaseDelay backoff. Break the inner retry loop before genericAttempt++ so the existing lastErr residual Push still runs and the concurrency gate is released. 403 stays errSoftForbidden so confirmation-then-escalate still runs. 429/503 with Retry-After and ENOSPC paths are unchanged.

A clean io.EOF from a truncated body (chunked, no Content-Length) used to return success even when the HTTP offset was still short of StopAt, leaving a hole. Surface that as a retryable early-EOF error so the existing retry and residual requeue path can finish the range. UnexpectedEOF and other read errors stay on the read-error path. Exact-range completion without a connection FIN is unchanged.

queue.Close only wakes Pop waiters; workers blocked in resp.Body.Read would keep executeWorkers waiting on those readers. When the completion condition fires, cancel every registered ActiveTask then Close. Completion still keys off Downloaded. Cancelling in-flight writers when that counter is inflated can freeze holes those writers would have filled, and Push after Close can re-stick a tarpit worker because this change does not drain the queue.

Completion cancel reused the health-stall path, which Push'es a remainder onto a queue that ignores Close. The monitor has already returned, so a tarpit GET can hang executeWorkers again. Mark completing before cancel, skip requeue, and return so health stalls still rotate. Also snapshot StopAt once for the early-EOF error string, relax the permanent-HTTP timing bound below the first single-mirror backoff, and assert the early-EOF primary was contacted.

…ite clamp Downloaded-full is no longer a cancel trigger; the popped VP-full task must stay consumed.

…lse sharing

SuperCoolPencil force-pushed the feat/verified-progress branch from 5a32da1 to 17fce5c Compare August 26, 2026 21:30

superGekFordJ changed the title ~~feat(concurrent): introduce authoritative VerifiedProgress and harden completion lifecycle~~ refactor(concurrent): introduce authoritative VerifiedProgress and harden completion lifecycle Aug 27, 2026

SuperCoolPencil force-pushed the feat/verified-progress branch from 17fce5c to 711530c Compare September 7, 2026 05:48

superGekFordJ mentioned this pull request Sep 8, 2026

fix(scheduler): prioritize physical success over late pause and guard completion boundary #648

Merged

superGekFordJ added 12 commits September 19, 2026 05:28

fix(progress): trust restored bitmap when resuming verified progress

4c0e3c4

fix(concurrent): complete, pause, and exit workers on verified progress

998c92a

test(concurrent): align monitor names with VP completion and cover wr…

cdc04df

…ite clamp Downloaded-full is no longer a cancel trigger; the popped VP-full task must stay consumed.

test(concurrent): serve range payload in prewarm recorder

f326da5

test: drain DefaultNetworkPool in scheduler and concurrent TestMain

875202d

test(concurrent): drop duplicate pause completion finalization

0357b9d

test(concurrent): drop duplicate pause completion boundary

b816399

perf(progress): add cache line padding to ByteTracker to eliminate fa…

50e6735

…lse sharing

SuperCoolPencil force-pushed the feat/verified-progress branch from 711530c to 50e6735 Compare September 18, 2026 23:58

Navigation Menu

Uh oh!

refactor(concurrent): introduce authoritative VerifiedProgress and harden completion lifecycle - #633

refactor(concurrent): introduce authoritative VerifiedProgress and harden completion lifecycle#633
superGekFordJ wants to merge 12 commits into
SurgeDM:mainfrom
superGekFordJ:feat/verified-progress

superGekFordJ commented Aug 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 26, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

superGekFordJ commented Sep 9, 2026

Uh oh!

SuperCoolPencil commented Sep 9, 2026

Uh oh!

superGekFordJ commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

superGekFordJ commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Side note on future ideas

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis

Uh oh!

coderabbitai Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

superGekFordJ commented Sep 9, 2026

Uh oh!

SuperCoolPencil commented Sep 9, 2026

Uh oh!

superGekFordJ commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

superGekFordJ commented Aug 26, 2026 •

edited

Loading

github-actions Bot commented Aug 26, 2026 •

edited

Loading

coderabbitai Bot commented Aug 26, 2026 •

edited

Loading