Repository navigation
fix(preheat): async snapshot sizing and fail-fast scheduler calls - #128
Merged
mangoknight merged 2 commits intoOct 13, 2026
Merged
Conversation
Peer lookup (SchedulerFile) and the cached-file report both run inside the download path and both already have a fallback: download from upstream, or queue the report locally for the hourly replay. They still waited the full 300s RpcRequestTimeout. On 2026-10-13 a DingoDB region without a leader made every scheduler lookup take 60s, so each file of every cache job in every region waited 60s+ before downloading. - SchedulerFile and the new ReportCachedFile use a 5s deadline. - After any failure both skip the scheduler for 30s, so a stalled scheduler costs at most one timeout per cooldown, not one per file. - Other scheduler RPCs (registration, local-queue replay) keep the long timeout. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Creating a preheat job measured every file of the snapshot before answering, inside the scheduler's 10s request. Batching paths-info (dingodb#127) fixed repositories with a few hundred files, but uonlp/CulturaX has 9059: 71 batches at 1-4s each through the regional exit, so create still failed with "context canceled". - Create only resolves the commit and file list (one request) and registers the job with the upstream usedStorage as an estimate. - The task measures the snapshot first (Prepare), records the exact size locally and reports it to the scheduler, then transfers. A sizing failure now fails the job with its reason instead of the create call. - Sizes come from one revision?blobs=true request (9059 files: 1.7s direct), falling back to batched paths-info when sizes are missing. - Resume reports "running" with the size once preparation is done, so the scheduler no longer shows a resumed job as interrupted. - Completed-job reuse checks against the size that job recorded, since a create request now only carries the estimate. - proto: UpdateCacheJobStatusReq.usedStorage (field 8). The running report is best effort and never queued for replay, so it cannot land after a terminal status. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mangoknight
changed the title
cooocy
self-requested a review
October 13, 2026 09:56
cooocy
approved these changes
Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent commits.
1. Fail fast on download-path scheduler calls
SchedulerFileruns before every file download with a 300s timeout. When the scheduler's database stalls, every file waits for it even though both callers already have fallbacks (fetch upstream; queue locally and replay hourly).SchedulerFileand the newReportCachedFileuse a 5s timeout; after a failure, skip the scheduler for 30s.2. Size the snapshot inside the job
Exact sizing ran synchronously in the create request, so repositories with thousands of files could not be created.
usedStorageestimate, and returns.PreheatCacheTask.Preparecomputes the exact size (onerevision?blobs=truerequest, falling back to batched paths-info), stores it and reports it before downloading.UpdateCacheJobStatusReq.usedStorage(field 8); the running report is best-effort and never queued for replay.Requires dingodb/dingoscheduler#30 for the new field; either side can be upgraded first.
Tests
TestSchedulerFastCallsTimeOutAndCoolDown,TestCreatePreheatReturnsBeforeSnapshotIsSized,TestPreheatSnapshotSizeUsesOneBlobsRequest,TestPreheatMetadataBatchesPathsInfo.go test ./internal/...passes, create path also under-race.🤖 Generated with Claude Code