Repository navigation
fix(preheat): batch paths-info when sizing a snapshot - #127
Merged
mangoknight merged 1 commit intoOct 13, 2026
Merged
Conversation
preheatMetadata posted one paths-info request per file, serially, inside the cache-job create request. Through a regional proxy each call takes ~0.75s, so a dataset with 167 files (CASIA-LM/ChineseWebText2.0) needed ~2 minutes while the scheduler cancels after 10s and preparation is bounded at 45s: create always failed with "context canceled" for any repository with more than a dozen files. Introduced in 5bcef49. Query sizes with one paths-info request per 128 paths (same batching as remote_sync) and reject responses that miss or add paths. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cooocy
approved these changes
Oct 13, 2026
mangoknight
added a commit
to mangoknight/dingospeed
that referenced
this pull request
Oct 13, 2026
Creating a preheat job measured every file of the snapshot before answering, inside the scheduler's 10s request. Batching paths-info (dingodb#127) fixed repositories with a few hundred files, but uonlp/CulturaX has 9059: 71 batches at 1-4s each through the regional exit, so create still failed with "context canceled". - Create only resolves the commit and file list (one request) and registers the job with the upstream usedStorage as an estimate. - The task measures the snapshot first (Prepare), records the exact size locally and reports it to the scheduler, then transfers. A sizing failure now fails the job with its reason instead of the create call. - Sizes come from one revision?blobs=true request (9059 files: 1.7s direct), falling back to batched paths-info when sizes are missing. - Resume reports "running" with the size once preparation is done, so the scheduler no longer shows a resumed job as interrupted. - Completed-job reuse checks against the size that job recorded, since a create request now only carries the estimate. - proto: UpdateCacheJobStatusReq.usedStorage (field 8). The running report is best effort and never queued for replay, so it cannot land after a terminal status. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
preheatMetadatasized the snapshot by issuing one paths-info request per file, serially, inside the create request. Repositories with more than a few dozen files exceeded the caller's timeout and job creation failed withcontext canceled.Fix
FileDao.GetPathsSizesContextsends paths-info in batches of 128 (asremote_syncdoes) and errors if the response is missing or adds paths.preheatMetadatauses it.Tests
TestPreheatMetadataBatchesPathsInfo: 200 files take 2 requests with correct total size; a missing path is an error. The old code issued 200 requests.🤖 Generated with Claude Code