Training control: pause/resume, persistent queue, resume-display fixes - #65
Training control: pause/resume, persistent queue, resume-display fixes#65scragnog wants to merge 4 commits into
Conversation
… offset - load_run_config now merges sidestep_adapter_config.json from final/, the run root, or the highest epoch_N checkpoint so the GUI resume dialog shows real rank/alpha (maps PEFT 'r' -> 'rank') - training chart: xOffset support so a resumed run's X-axis labels continue from the original epoch instead of restarting at 0; session-relative step timing in the stats row - (the adapter weight-loading half of the original fix is superseded by the upstream resume strategy ladder from koda-dernet#46 and is not included) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pause captures a complete resumable checkpoint (adapter, optimizer, scheduler, RNG state, EMA shadow) and stops cleanly; resume restarts from it via the normal resume path. GUI gains pause/resume controls with progress-file handshake. (from 4ef9343; patch1.diff/patch2.diff scratch files excluded) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…art, and PC reboot - Backend: GET/POST /api/train/queue endpoints store queue state to .sidestep/training_queue.json - Frontend: debounced _saveQueueState() on every queue mutation (enqueue, pause, resume, remove, clear) - Frontend: _loadQueueState() in init() restores queue on page load - Running entries reclassified as 'interrupted' on reload (resumable like paused) - Queue panel now shows paused/interrupted entries with inline resume buttons
Advanced page: - 'Use dataset name as run name' checkbox: when checked, run_name is automatically set to the dataset folder basename (e.g. 'my_band') instead of 'lokr_turbo'. Updates on dataset change. - 'Batch mode' checkbox: reveals a multi-select checklist of all available datasets below the dataset picker. - 'Queue Selected' button: generates one training config per checked dataset, using the dataset name as run_name, and enqueues them all in one click. All/None selection helpers included. - Single-dataset workflow unchanged — existing Start Training button and single select still work as before.
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (12)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
Training control: pause/resume, persistent queue, resume-display fixes
Pause/resume. Pause (GUI button → file-based handshake) captures a complete resumable checkpoint — adapter, optimizer, scheduler, RNG state, EMA shadow, chunk-coverage and adaptive-sampler state — then exits cleanly; resume restarts from it through the normal resume path.
Persistent training queue. The multi-dataset training queue is persisted server-side, so it survives browser close, server restart, and reboot. Run names auto-derive from the dataset; batch-queueing several datasets is one action.
Resume display fixes.
load_run_configmergessidestep_adapter_config.jsonfromfinal/, the run root, or the highestepoch_Ncheckpoint (mapping PEFTr→rank).xOffsetso a resumed run's X-axis continues from the original epoch instead of restarting at 0; stats row uses session-relative step timing.Note: an earlier version of this work also patched adapter weight loading on resume; that half is superseded by the #46 resume strategy ladder and is not included here — this PR composes with it.
🤖 Generated with Claude Code