Skip to content

feat(sound): expand Simple prompts with chat - #12772

Merged
mudler merged 5 commits into
masterfrom
feat/sound-prompt-expansion
Oct 19, 2026
Merged

mudler merged 5 commits into
masterfrom
feat/sound-prompt-expansion

Conversation

@mudler-agent

@mudler-agent mudler-agent commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Simple Sound can expand a description with an installed chat model before audio generation. The page remembers an explicit choice separately from Chat, including Backend default. Re-running a saved result restores its original choice; unavailable models require a new selection.

API callers opt in with expansion_model, selecting an installed llama.cpp chat model (including its supported aliases). The server enforces a JSON object grammar during generation, then separately validates field types, allowed fields, and bounds. Other chat backends are rejected before inference because their decoding paths may ignore this grammar. The server preserves explicit instrumental and language controls and authorizes both models before loading either. Failed or cancelled expansion never starts audio generation.

Audio.cpp maps the neutral result using the loaded family: YuE2, HeartMuLa, ACE-Step, or Stable Audio. Unsupported families reject expanded requests; HeartMuLa requires lyrics. Omitting expansion_model preserves backend defaults, including native Python ACE-Step expansion. Advanced requests keep their existing fields.

Notes for Reviewers

  • Deploy matching updated server and audio backend builds for explicit expansion; the protocol adds an optional expanded marker.
  • No actual LLM/audio inference, deployment, or full engine-linked build ran. Tests use mocked inference and the real C++ request builder against the pinned audio.cpp headers.
  • The three tracked Swagger schemas receive minimal aligned edits because swag was unavailable. A test checks the optional string field across JSON, YAML, and Go output.

Validation:

  • Go expansion/schema tests, mocked HTTP requests, cancellation/error paths, and protocol defaults pass. Regressions verify generation-time grammar reaches the backend with rendered and tokenizer templates, preserves installed configuration, and rejects unsupported backends before inference/audio.
  • Auth-tagged tests cover denied primary/secondary models, lookup failures, admin/no-auth access, and registered routes. Denied requests never load either model.
  • Initial implementation: matched-toolchain make lint LINT_NEW_FROM=origin/master passed with zero issues.
  • Structured-output follow-up: Go 1.26.8 golangci-lint run --build-tags=auth --new-from-merge-base= --new-from-rev=2ceb6be31 ./core/backend ./core/http/endpoints/elevenlabs passes with zero issues. Full auth-tagged Sound HTTP tests and normal HTTP/backend Sound tests pass.
  • Python ACE-Step native/structured adapter tests pass.
  • The production C++ request-builder tests pass against audio.cpp 507495611b22dc4590e0e5609e05e8096384e33f; all 24 standalone C++ test files pass.
  • UI build and 10 focused browser/schema tests pass, including persistence, stale choices, history, and Advanced serialization. Five existing Sound compatibility cases pass.

Useful focused commands:

go test ./core/backend ./core/schema -count=1 -ginkgo.focus='Sound|Expansion'
go test -tags auth ./core/http/endpoints/elevenlabs -run Sound -count=1
go test ./pkg/grpc/proto -run Sound -count=1
python3 backend/python/ace-step/test_sound_expansion.py
bash backend/cpp/run-unit-tests.sh

From core/http/react-ui, build the app and run npx playwright test e2e/sound-expansion.spec.js against the current UI test server.

Signed commits

  • Yes, all commits include the maintainer's DCO sign-off and AI assistance attribution.
  • Documentation updated under docs/content/ for user-facing changes.

Allow callers to select an installed chat model for sound prompt
expansion. Validate its JSON before audio generation and preserve
explicit instrumental and language controls. Keep backend defaults
when expansion is omitted, including native Python ACE-Step sampling.

Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Map neutral expanded fields where the resolved audio family is known.
Keep unmarked requests unchanged and reject unsupported expansion before
inference. Clear YuE2 and HeartMuLa transcripts to prevent lyric fallback.

Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Require an explicit Simple choice and remember it separately from chat.
Clear unavailable choices instead of silently switching language models.
Preserve the choice in history and restore it when editing a past result.
Keep Backend default and Advanced requests compatible with existing APIs.

Document family support and align the published optional request schema.
Verify selection, request fields, persistence and history in the browser.

Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Check the resolved audio model and explicit expansion model before
loading either. Permission lookup failures stop generation.

Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Constrain expansion output with the existing JSON grammar before decoding.
Require a llama-cpp chat model so unsupported backends cannot ignore it.
Keep strict field validation and fail before generating audio.

Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
mudler merged commit ae8092e into master Oct 19, 2026
73 of 76 checks passed
@mudler
mudler deleted the feat/sound-prompt-expansion branch October 19, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants