Missing "Audio Reference" upload field in Music and sound generation?

[Pinucci0]

Hi everyone,

I am currently exploring the audio generation capabilities in LocalAI and I've noticed a potential limitation in the current Web UI regarding the ACE-STEP-TURBO model.

The ACE-STEP-TURBO model supports a "cover mode", which inherently requires an "Audio Reference" to function correctly. However, looking at the audio generation section in the UI, the available fields are:

Model
Caption
Text (optional)
BPM
Duration (s)
Key/Scale
Language
Time Signature

It appears there is no option or input field to upload or provide a path to an audio file to act as the "Audio Reference".

My questions are:

1. Is it correct that the current Web UI does not yet support uploading an audio reference for models that require it?
   

2. Is there a workaround to enable "Cover Mode" (for example, via direct API calls or by modifying a config file)?
   

3. If this is a UI limitation, would it be possible to add a file upload field (or a way to specify a file path) specifically for models that support an audio reference?
   

Thank you for your hard work on this amazing project!

[aashish254]

you are right, the web ui does not have the field. i checked the source and the music page posts to POST /v1/sound-generation. the request schema for that endpoint has no audio reference parameter at all (core/schema/elevenlabs.go). the handler also hardcodes nil for the source file when it calls the sound generation backend (core/http/endpoints/elevenlabs/soundgeneration.go). the backend protocol itself does support it through the src field in backend.proto. so the ui simply cannot reach cover mode today.

the workaround that works now is the cli:

local-ai sound-generation -b ace-step -m -i reference.wav -d 120 "your prompt"

the -i flag is the input file to condition generation on and it is passed straight to the backend as the source file. there is also -f if you only want the first portion of the sample used.

so yes to your first question, it is a ui and api gap. and your third question is the real fix: the ui needs a file upload field and the endpoint needs to pass the source file through instead of nil.

[Pinucci0]

Thank you so much @aashish254
I opened a issue.