Repository navigation
feat(sound): expand Simple prompts with chat - #12772
Merged
Merged
Conversation
Allow callers to select an installed chat model for sound prompt expansion. Validate its JSON before audio generation and preserve explicit instrumental and language controls. Keep backend defaults when expansion is omitted, including native Python ACE-Step sampling. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Map neutral expanded fields where the resolved audio family is known. Keep unmarked requests unchanged and reject unsupported expansion before inference. Clear YuE2 and HeartMuLa transcripts to prevent lyric fallback. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Require an explicit Simple choice and remember it separately from chat. Clear unavailable choices instead of silently switching language models. Preserve the choice in history and restore it when editing a past result. Keep Backend default and Advanced requests compatible with existing APIs. Document family support and align the published optional request schema. Verify selection, request fields, persistence and history in the browser. Assisted-by: Codex:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Check the resolved audio model and explicit expansion model before loading either. Permission lookup failures stop generation. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Constrain expansion output with the existing JSON grammar before decoding. Require a llama-cpp chat model so unsupported backends cannot ignore it. Keep strict field validation and fail before generating audio. Assisted-by: Codex:gpt-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Simple Sound can expand a description with an installed chat model before audio generation. The page remembers an explicit choice separately from Chat, including Backend default. Re-running a saved result restores its original choice; unavailable models require a new selection.
API callers opt in with
expansion_model, selecting an installed llama.cpp chat model (including its supported aliases). The server enforces a JSON object grammar during generation, then separately validates field types, allowed fields, and bounds. Other chat backends are rejected before inference because their decoding paths may ignore this grammar. The server preserves explicit instrumental and language controls and authorizes both models before loading either. Failed or cancelled expansion never starts audio generation.Audio.cpp maps the neutral result using the loaded family: YuE2, HeartMuLa, ACE-Step, or Stable Audio. Unsupported families reject expanded requests; HeartMuLa requires lyrics. Omitting
expansion_modelpreserves backend defaults, including native Python ACE-Step expansion. Advanced requests keep their existing fields.Notes for Reviewers
expandedmarker.swagwas unavailable. A test checks the optional string field across JSON, YAML, and Go output.Validation:
make lint LINT_NEW_FROM=origin/masterpassed with zero issues.golangci-lint run --build-tags=auth --new-from-merge-base= --new-from-rev=2ceb6be31 ./core/backend ./core/http/endpoints/elevenlabspasses with zero issues. Full auth-tagged Sound HTTP tests and normal HTTP/backend Sound tests pass.507495611b22dc4590e0e5609e05e8096384e33f; all 24 standalone C++ test files pass.Useful focused commands:
From
core/http/react-ui, build the app and runnpx playwright test e2e/sound-expansion.spec.jsagainst the current UI test server.Signed commits
docs/content/for user-facing changes.