Enable web-based LoRA/full-model training with live output - #1
Open
KhryptorGraphics wants to merge 86 commits into
Open
Enable web-based LoRA/full-model training with live output#1KhryptorGraphics wants to merge 86 commits into
KhryptorGraphics wants to merge 86 commits into
Conversation
added 4 commits
July 6, 2026 11:04
… local artifacts - analyze_spectrum(.py/2.py): spectral comparison diagnostics used to chase the fast-note-change 'autotune' artifact and detection-percentage tuning - create_svcfork_checkpoint.py: build svc-fork checkpoint metadata - update_connor_profile.py: profile json updater for the Conor voice - .gitignore: keep local debug launchers (inline flask secret), ruvector.db, .codegraph/, playwright live reports, and .claude/settings.local.json out of the repo
54 agent definitions, command templates, and hook helpers referenced by CLAUDE.md so the tooling setup is reproducible across checkouts. Machine-local settings (.claude/settings.local.json) stay untracked.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
|
Too many files changed for review. ( Bypass the limit by tagging |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
added 2 commits
July 12, 2026 18:23
…c device; add local-audio ingestion + V100 launchers - trainer: DataLoader uses spawn context (16 workers) to avoid fork+CUDA deadlock after GPU ContentVec precompute; warmup scales from immutable scheduler base_lrs (was get_last_lr(), which compounds ExponentialLR and collapsed lr to ~0 on small datasets) - encoder: ContentVecEncoder.encode aligns the HF model to the input device (latent CPU/CUDA mismatch surfaced by multi-sample on-the-fly extraction) - scripts/ingest_local_audio_profile.py: reusable video/audio -> profile ingestion via pretrained htdemucs_ft, optional --slice-seconds clipping - scripts/serve_local_threading.py: threading-mode server (no eventlet) - scripts/start_v100.sh: V100 launcher, UUID GPU pin + sm70 assertion
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
- serialize concurrent log fetches (in-flight guard) so the immediate poll + interval + React StrictMode double-invoke can't all fetch at offset 0 and append the first lines twice - discard a log response whose jobId switched mid-flight (cancel -> start new run) so the old job's lines don't append into the new job's log
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…f16 gate - voice_profiles.list_training_samples re-anchors stored absolute sample paths to the store's actual location, so profiles survive data-dir relocation / cross-machine sync (AV-ksek) - VocalSeparator.separate clamps a too-large segment to the model's max (min across sub-models for htdemucs_ft), preventing the >7.8s htdemucs reshape crash for every caller; api_profiles no longer passes 10.0 (AV-e4p7) - trainer.resolve_precision downgrades bf16 -> fp16 on pre-Ampere GPUs (capability < sm80, e.g. V100 sm70), fail-safe on unknown capability (AV-ua0w) - add tests/test_bugfix_paths_segment_precision.py
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
ConversionHistoryPage crashed with 'Cannot read properties of null (reading toFixed)' when a conversion record had rtf/processing_time_seconds === null (the !== undefined guard let null through). Guard with typeof === 'number' so the history page renders records missing those metrics.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…training UI Fixes crash 'Cannot read properties of undefined (reading toFixed)' on checkpoint.final_loss (LoRA-only checkpoints omit web metrics). Swept the same class on the training/profile/history path: TrainingJobQueue (null-passing !== undefined guard), GPUMonitor, VoiceProfilePage/ConversionWorkflowPage sample durations, and QualityMetricsPanel gauges. QualityMetrics leaves the async metrics path (get_job_metrics -> _calculate_metrics) omits — mean_error_cents, embedding_distance, spectral_distortion — marked optional; GaugeProps.value widened + NaN bar-width clamped.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…); file 3 follow-ups (api.ts split, ConversionHistoryPage guard, fp32 mem watch)
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…al-decoder <-> svc-fork-integration divergence)
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…d, fp32 mem); record AV-owvr divergence assessment
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…n unique commits cherry-picked into hq-lora-universal-decoder)
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
….35 render already exists fork_hq_stereo_width is explicitly 0.0 in data/app_state/app_settings.json and is a documented runtime setting accepting 0.0-1.0 (runtime_contract.py:86, api_runtime.py:482), so Loss 1b is configuration, not a model limit. The pipeline's widening is real model decorrelation - L and R converted separately, difference becomes side, scaled by stereo_width (singing_conversion_pipeline.py:1648-1676) - not a pseudo-stereo trick. A render of these exact ep235 weights at width 0.35 already exists in History (herofix-d9e90fc4, 'ep235 + FIXES (edges kept, stereo 0.35)'), and its own note says it was built 'for your complaints' and asks for a comparison against the serving render. Measured side/mid: original -11.9, served -13.7, width-0.35 -10.9. That render also used db -35 and a different instrumental, so width is not its only difference - noted in the doc and the record title. Also exonerates fork_hq_match_source_bandwidth: it defaults on and low-passes the converted vocal to the source's measured wall, but _detect_bandwidth_hz returns 20000 Hz here, above the render's own rolloff, so it is inert on this material. The 17.9-18k ceiling is the decoder. Listening set is now six entries: the offline air/halo variants are labelled as approximations to isolate the variable, and the shippable spatial variant is the real pipeline width 0.35. Nothing changed in the registry or app settings. detect_changes: no indexed symbols affected.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…x, not the stem The exoneration of fork_hq_match_source_bandwidth was measured on the separated stem, which is precisely the signal the pipeline does NOT use: singing_conversion_pipeline.py:1689-1694 measures the original mix and warns that a stem reads as full-band because the separator invents energy above the source's wall. Re-measured correctly on the original mix: 20000 Hz, identical at 48k native and resampled to 44.1k. So the filter DOES fire (20000 < the 20947 Hz threshold) and applies a 20 kHz low-pass - but 20 kHz sits above the decoder's own 17.9-18k rolloff, so it still cannot be what creates the fmax gap. Conclusion unchanged, provenance now correct; the stem happens to return the same number on this song, which is a coincidence and not a basis to reason from. detect_changes: no indexed symbols affected.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…ue is 0.16, not 0.35 singing_conversion_pipeline.py:1666-1668 scales side by stereo_width * mid_rms / side_rms, so the vocal's side/mid ratio equals stereo_width exactly. That makes the value arithmetic: 0.35 = -9.1 dB, which is 6.6 dB WIDER than Mariah's own stem (-15.7 dB); the matched value is 0.164. If the halo is the missing thing, ship 0.16, not the 0.35 that happens to exist. Higher width also means more of the chorus/decorrelation artefact the pipeline's comments warn about, since side is the difference of two stochastic conversions. Resolved the comparator question empirically: the htdemucs and BS-RoFormer remixes of the same width-0.35 render both measure -10.9 dB side/mid, so the instrumental is not a confound on this axis. Its TONE still cannot be judged as width-only (also used db -35; 1.9 dB darker at 8-12k, 4.6 at 12-16k, walls 4.1 kHz lower). Noted that fork_hq_stereo_width is a global app setting and 0.0 is the documented recipe behind every approved OLT render - user-approved state, so only a listening verdict moves it. Also noted both fix renders predate the complaint (09-04 vs 09-05): if that render was already heard and the complaint stands, the spatial hypothesis is largely dead. Listening set now seven entries with distinct stated roles: 4of7 isolates width alone (spectrum bit-identical to served), 6of7 proves the pipeline's own knob reaches the target. detect_changes: no indexed symbols affected.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
… 0.0 was a later choice Measured the two width-0.35 remixes on vocal-silent frames (instrumental alone): served #5 8-12k -20.9 12-16k -21.6 fmax 22.1k d9e90fc4 htdemucs -30.1 -34.7 17.9k c5e8decd RoFormer -20.8 -21.5 19.6k So the darkness and the 18k wall in d9e90fc4 are its instrumental, not the vocal and not the width - which reverses my earlier 'the instrumental is not a confound' note: it is moot for side/mid (-10.9 both) but decisive for tone. c5e8decd is therefore the only one of the two comparable to serving on tone, and the confounded htdemucs clip is dropped from the listening set (now six entries, relabelled). Corrected the ordering claim: the 0.35 renders came FIRST (09-04 11:19, 13:22) and the served width-0.0 recipe was built later at 18:03, so width 0.0 was a subsequent choice, not an accidental regression - the doc records it as the recipe behind every approved OLT render. Whether those renders were ever auditioned cannot be inferred from the store: isFavorite, notes and user tags are unset on both, and exactly one record in 236 has isFavorite set at all (an unrelated 09-02 render). Recorded as a question that must be asked. register_render.py: skip the copy when src resolves to the destination, so re-registering an already-registered clip to fix its title no longer raises SameFileError. detect_changes: no indexed symbols affected.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
… render The fork lane was bolted onto predicates that only knew about `full_model`, so routing learned about forks and labelling did not. Two consequences, one cause. VoiceProfileStore derives active_model_type from artifacts under trained_models_dir. A fork checkpoint lives in data/fork_models/<id>.json, so no has_* flag can see it and the derivation downgraded a fork voice to "adapter" - the profile then advertised an engine it does not use. save() compounded it by dropping anything off its allow-list, so the correct value could only ever be destroyed, never kept: deleting Brandy's legacy artifacts on 09-04 silently rewrote her to "adapter" on the next save, while Conor kept "svc_fork" only because his profile had not been re-saved since. The conversion runtime_backend label tested `active_model_type == 'full_model'` while the routing line beside it had been widened to accept `fork_backed` too (5388859). A fork profile cannot satisfy the former, so every fork conversion was stamped plain 'pytorch' and the history read as though so-vits-svc-fork had never run. Both routes now emit 'so_vits_svc_fork'. The label is truthful: convert_song() checks the fork bridge first and returns, short-circuiting before the legacy artifact loader, so a fork-registered profile genuinely is fork-served whatever else sits on disk. _normalize_profile is CRITICAL by impact (63 dependents across training, web and inference), so the change is purely additive - one branch preserving an explicit svc_fork, no new filesystem probes in that hot path - and a guard test pins that full_model / adapter / base derivation is unchanged. Co-Authored-By: claude-flow <ruv@ruv.net>
… with _PRESERVED_INFERENCE_KEYS carries requires_uv_contract and crepe_uv_threshold from the outgoing registry entry onto the new one, so a retrain of a uv-contract voice is *served* with the contract on. The trainer never exported the matching env vars, so the retrain itself ran with crepe's uv==1 everywhere: the model learned no unvoiced frames and was then served as though it had. That is exactly the train/serve mismatch those keys exist to prevent, inverted. Not theoretical - Brandy is serving with requires_uv_contract + crepe_uv_threshold 0.3 right now, so pressing Train in the UI would have produced a silently broken model with no error anywhere. train_svc_fork now reads the outgoing registry entry before training instead of only at promotion, and derives the training env from the same `previous` dict the preservation reads, so the two agree by construction. pre-hubert (where crepe f0 and therefore uv is computed) and train both run under it; pre-split needs no contract as it only cuts audio. _clean_env also scrubs any SVCFORK_UV_CONTRACT / SVCFORK_CREPE_UV_THRESHOLD inherited from whatever launched the service, so the value can only come from the registry entry - a contract leaking in from the environment would reintroduce the same mismatch from the other direction. Co-Authored-By: claude-flow <ruv@ruv.net>
test_config_options_advertises_architectures expected three architectures while the enum ships four. 'como' is the legacy L1-regression CoMoSVC decoder, still reachable from training/job_manager and surfaced in the training UI with its own description, so the code is right and the expectation was stale. Verified the failure predates this session's changes by re-running it against a stashed tree. Co-Authored-By: claude-flow <ruv@ruv.net>
The trainer's subprocess helpers gained an `env_extra` parameter (c81f08a) and this suite's doubles did not, so the happy path and the no-checkpoint case both died on TypeError - caught by a full-suite sweep, not by the targeted runs. The doubles now accept it and record what each stage received, so the contract is asserted where it actually matters: that a registry entry carrying requires_uv_contract / crepe_uv_threshold reaches pre-hubert and train, and that a plain voice gets nothing. _uv_contract_env alone could not catch a wiring mistake between resolving the contract and handing it to the subprocess. The fixture still returns a list, so the four existing callers are untouched; the recorded env hangs off it as `.env`. Co-Authored-By: claude-flow <ruv@ruv.net>
…load Every full fine-tune of this ~80 min corpus degraded after ~epoch 150 - four runs on 2026-09-04/05, all peaking early and losing the top octave late. That is overfitting, and constraining the update is the standard remedy. The fork had no LoRA path: `inject_lora` in training/job_manager only ever runs on CoMoSVCDecoder, a different architecture entirely. inject_lora() attaches a parallel A -> B Conv1d path by forward HOOK rather than replacing the module, so every base parameter keeps its exact state_dict key. Verified: 751 -> 1109 keys with all 751 originals intact, and safe_load of an ordinary non-LoRA checkpoint into a LoRA model succeeds. lora_B is zero-init, so training starts bit-identical to the base and the delta is provably live once B moves (max output change 0.53). Targets dec./flow./enc_p. only. enc_q is excluded on purpose: it runs during training but never in SynthesizerTrn.infer, so adapting it would spend capacity the served model cannot use - 8.9M params, 19.5% of the generator. ConvTranspose1d upsamplers are skipped; their (in,out,k) layout needs a different delta construction and a wrong one is worse than none. train.py freezes the base and narrows optim_g to the deltas plus emb_g, which must move for the voice to change at all: 2.39M of 48.0M (5.0%) at rank 16. Env-gated on SVCFORK_LORA_RANK, so serving and existing training are untouched. Injection collects its targets before mutating - adding child modules while walking named_modules() makes the walk descend into what it just added and recurse without bound. Co-Authored-By: claude-flow <ruv@ruv.net>
inference/core.py built SynthesizerTrn and loaded the checkpoint without ever
injecting the LoRA side-paths. A LoRA checkpoint carries 358 extra `_lora_*`
tensors and safe_load copies only matching keys, so every delta was dropped on
load and the model served as the unadapted base - the LoRA run would have
measured as an exact no-op, and the cause would have looked like "LoRA does not
work" rather than "LoRA was never loaded". Caught before scoring the first run,
not after.
inject_lora now runs BEFORE load_checkpoint, gated on SVCFORK_LORA_RANK, and a
test pins that ordering: swapping the two lines reintroduces the bug silently.
The bridge maps registry `lora_rank` / `lora_alpha` onto that env, the same
per-model pattern as requires_uv_contract and crepe_uv_threshold, and both keys
join _PRESERVED_INFERENCE_KEYS so a retrain cannot drop them - serving a LoRA
checkpoint with the wrong rank rebuilds the wrong side-paths.
Verified end to end: the bridge hands svc
{UV_CONTRACT=1, CREPE_UV_THRESHOLD=0.3, LORA_RANK=16, LORA_ALPHA=32.0} for the
promoted entry, and the rendered conversion reports runtime_backend
so_vits_svc_fork.
Co-Authored-By: claude-flow <ruv@ruv.net>
…pans to backing Cluster-level spectral centroid gate fails on dense orchestral duets with overlapping registers (Beauty and the Beast: male 2376-2532 Hz vs female 2607 Hz) — male spans stayed in the lead and were re-sung by the female model (male-frame f0 corr 0.072). - SpeakerDiarizer.extract_span_embeddings: per-span wavlm embeddings from the original mix, profile-compatible 256-dim, L2-normalized - _select_speaker_spans: after the cluster gate, second pass on primary_spans; windows embedding closer to the preserved-cluster centroid than the primary centroid by margin move to backing - config: multi_speaker_embedding_reassign (default true), multi_speaker_embedding_margin (0.9), window 4.0s - tests: TestEmbeddingReassignment (move/no-move/disabled), TestTimeRangePreserve (carve/clip/token)
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…tion is verified end to end Supersedes docs/handover-fork-output-chain.md (partial, pre-calibration). The verified result: 7 lead span(s) moved to preserved backing by embedding reassignment on the real stems; backing male-span f0 corr vs source vocal A +0.576/B +0.516/C +0.563 (run4 baseline was negative on all three); the lead male-frame corr medians moved from 0.072 toward the calibrated target. Also fixes the runbook refs that the verification superseded.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…h-data 2x2, replication, averaging, env pinning
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…eded by replication + averaging
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
…5 beats the seed on conor (+1.6 dB 6-8k, fmax 22.1k, identity above all challengers)
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
training_mode(lora|full) andarchitecture(como|diffusion_mel|mel_gan|svc_fork).force: trueoverride.TrainingJobManager._save_trained_adapter()now returns/persistsadapter_model/serving_model_path, and_update_profile_training_state()prefers the self-contained serving artifact over the deltas-only adapter file.GET /api/v1/training/jobs/<job_id>/logs?offset=Nendpoint backed byTrainingJobManager.append_job_log()/get_job_logs().TrainingConfigPanelexposes LoRA vs Full-model mode and the CoMoSVC architecture;ConversionWorkflowPagesendsforce: truefor early full-model starts and surfaces returned warnings;LiveTrainingMonitorpolls the new logs endpoint and renders a live output panel.Verification
python -m py_compile src/auto_voice/web/api_training.py src/auto_voice/training/job_manager.py src/auto_voice/cli.py— clean.pytest -q tests/test_training_ui_routes.py— 9 passed (4 new: config-options architecture, unknown-QA warning path, force-override full training, job logs buffering).cd frontend && npm run build— clean production build.cd frontend && npm run test:e2e -- training-ui.smoke.spec.ts— 3 passed.POST /training/jobs(lora, force=true) -> 201;GET .../logs-> real buffered lines;POST .../cancel-> 200.detect_changestimed out on this box;code-review-graphequivalent used as the scoped fallback).Commits
3564457cEnable web training mode selection331b1027Refresh agent helper metadata (unrelated.claude/helpers/*+agentdb.rvf*refresh, split out per session notes)