Optimizers: SCAO, Rose, Automagic + per-layer LR scaling - #69
Open
scragnog wants to merge 22 commits into
Open
Conversation
Preprocessing now encodes BOTH the caption and the genre-tag text variant into each .pt (backward-compatible optional keys); training selects caption vs genre per sample per step according to genre_ratio (auto-detected from preprocess_meta.json when unset) instead of the old static per-sample choice at preprocess time. Old tensors keep working. Co-authored-by: Gabriel <gadna166@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… GUI conditioning_type flows from the dataset through collate into ProgressWriter events; the training monitor shows live caption-vs-genre usage. Also ports target-loss scale/EMA telemetry to the Fabric trainer path (previously basic-loop only). Co-authored-by: Gabriel <gadna166@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hannel balancing, latent noise, t_bias (from 5e400f6; scripts/ debug artifacts excluded) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rmup ramp - MSE guarantees convergence to conditional expectation (required by flow matching) - Huber converges to median, under-penalizes extreme flow vectors -> clipped highend - Huber still available as option, just not default - EMA warmup: effective_decay = min(target, (1+step)/(10+step)) so shadow catches up fast early
- ema_warmup_steps (default 2000): prevents EMA shadow from lagging early training - At step 0 decay=0 (shadow=current weights), linearly ramps to target decay - Wired through: defaults, configs, config_factory, CLI, wizard UI, frontend - GUI: warmup field auto-hides when EMA decay=0
- No tracking until start_step, then snapshot current weights as shadow - Mathematically clean: shadow is always a proper exponential average at full target decay - Linear ramp was creating a Frankenstein blend of different averaging rates - Renamed ema_warmup_steps -> ema_start_step across all 16 touchpoints - Default 2000 steps: model learns basic structure before EMA locks in
… distribution - training_step now stashes auxiliary metrics: raw/weighted loss, SNR weight stats, timestep mean, per-channel loss breakdown (max/min/ratio) - drain_step_metrics() returns and clears the dict at log_every intervals - Both Fabric and basic training loops log to TensorBoard: fidelity/*, ema/* - Fixed timestep histogram to fire for flow_snr too (was min_snr only) - Pipe fidelity dict + ema_active through progress writer for real-time GUI - Frontend: unpack fidelity dict from progress messages into _tbScalars - Added curated labels/colors for all new tags in mini-chart registry
- dynamic_channel_balance (opt-in toggle): blends 50/50 static VAE prior with running EMA of per-channel loss, so struggling channels get more gradient - Wired through all config touchpoints: defaults, configs, factory, CLI, GUI - GUI: checkbox auto-hides when channel_balance is off or legacy mode - Fix: SNR weight charts were always 1.0 (logged post-normalization); now logs pre-normalization values + spread ratio - New ema/effective_decay metric (0.0 when inactive, target decay when active) - Added curated mini-chart labels for snr_weight_spread, ch_dynamic_blend_range, ema/effective_decay
…tric telemetry - Add 3 new loss_fn options: pseudo_huber, x0_mse, x0_pseudo_huber - x0_mse implements paper's intended formulation (implicit t² weighting) - pseudo_huber: δ²(√(1+(e/δ)²) - 1) smooth Huber approximation - Log fidelity/arrangement_loss (t>0.5) and fidelity/detail_loss (t≤0.5) - Update config validation to accept all 5 loss_fn values
…prompts - Expand --loss-fn choices to include pseudo_huber, x0_mse, x0_pseudo_huber - Wizard shows delta prompt for all delta-using modes (huber, pseudo_huber, x0_pseudo_huber) - Update review summary label to Huber/Pseudo-Huber delta
…art tags - Add pseudo_huber, x0_mse, x0_pseudo_huber to loss function dropdown - Update help panel with descriptions for all 5 modes - Show huber delta input for all delta-using loss modes - Add Arrangement Loss and Detail Loss to chart tag metadata
- build_train_namespace passes dynamic_channel_balance through (was silently reset to default for wizard and preset runs) - presets recognise dynamic_channel_balance - GUI CLI builder emits --loss-weighting unconditionally (backend default is flow_snr, so omitting 'none' trained with flow_snr), plus --loss-fn/--huber-delta/--t-bias/--latent-noise/ --channel-balance/--dynamic-channel-balance/--vae-channel-prior/ --legacy-loss/--ema-start-step, and snr-gamma for flow_snr too - preset loader maps dynamic_channel_balance into the form Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e 'All The Levers submenu' on the GUI and Wizard
- Vendor Rose optimizer (stateless, zero VRAM) into sidestep_engine/vendor/rose.py - Add SCAO (sparse curvature-aware) + Rose to build_optimizer() factory - SCAO: requires pip install scao>=0.1.0, falls back to AdamW if missing - Rose: vendored, always available, overrides compute_dtype to fp32 - Update _OPTIM_BPP: SCAO=14.0, Rose=0.5 (stateless) - Add to CLI --optimizer-type choices - Add to wizard menu with Rose LR guidance (0.01-0.1) - Add to GUI frontend: both optimizer dropdowns, help panel, LR hints - JS reactivity: Rose LR auto-switch (0.01) + hint display - Add SCAO to dependency_check.py (Rose needs no dep check)
- Vendor Automagic from ostris/ai-toolkit (Apache 2.0) into vendor/automagic.py - Merged optimizer_utils.py inline (Auto8bitTensor, copy_stochastic, etc.) - Made QBytesTensor (optimum.quanto) optional -- not needed for bf16 LoRA - Add to build_optimizer() factory, VRAM estimation (8.0 BPP), CLI, config - GUI: both optimizer dropdowns, help panel, automagic-lr-hint div - JS reactivity: auto-switch LR to 1e-6 + hint display - Wizard: added to optimizer menu - Verified: imports, factory, and 2-step optimization all pass
…ix Rose OOM Root cause: previous presets changed 7+ variables from the proven config, making optimizer comparison meaningless. All 4 runs converged to ~0.71 (vs 0.41 for good run) due to GA=4, Huber loss, SNR weighting, dim=256, and 100-step patience — not optimizer issues. Changes: - All presets now use XL_SFT_Turbo base (GA=12, MSE, dim=512, 1000ep, patience=250) - Only optimizer-specific settings differ: LR and scheduler - Rose: gradient_checkpointing=true (was false, causing OOM on 32GB) - SCAO: install_spec updated to git URL (not on PyPI) - SCAO: installed via 'uv pip install git+https://github.com/whispering3/scao.git' - Help text updated with correct SCAO install command
(from bedf53c; the four experimental Style_*/Optimised_* presets were net-deleted later on the source branch and are not included) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(from 694b1cd, frontend logic only) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lr_scale_self_attn/cross_attn/mlp were absent from build_train_namespace, preset field maps, and the GUI CLI builder, so wizard and preset values silently fell back to 1.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (32)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Jul 29, 2026
scragnog
marked this pull request as ready for review
July 29, 2026 10:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Optimizers: SCAO, Rose, Automagic + per-layer LR scaling
Part 3 of the training-core series — stacked on #68; until that merges this diff includes its commits.
Per-layer-type LR scaling.
lr_scale_self_attn/lr_scale_cross_attn/lr_scale_mlpmultiply the base LR per parameter group, surfaced in GUI ("All The Levers"), wizard, and CLI. Useful for style-vs-content balance without separate runs.Three optimizers integrated (all optional, none default):
dependency_checkhandles it), surfaced with VRAM estimationPreset. Ships one starter preset,
XL_SFT_Turbo.json(force-added past the presets .gitignore) — our proven baseline for XL SFT turbo runs. Also fixes the GUI "smart optimizer reactivity" silently overriding a preset's explicit LR.Note: the vendored files carry their upstream Apache-2.0 attribution headers; happy to restructure (submodule/extra dep) if you prefer a different vendoring policy.
🤖 Generated with Claude Code