audio-to-audio: is it possible to use more than one samples?

[buscon]

for sound design purposes, it would be interesting to use the audio-to-audio feature with multiple samples

is that possible?
if not, any hint on how I could add such feature?

[ivcylc]

can you describe your request in detail? maybe i can help implement it

[buscon]

thanks!

This is what I imagine:

1. the user provide as a prompt different samples, the output will be a combination of these samples. it is not a simple mix of different sounds, but more each sound merges and influence the other.
2. the prompt should include how much influence each prompt sample have, something like audioldm --mode "transfer" --file_path trumpet.wav 70% cello.wav 30% -t "Children Singing"
   if no percentage is given, each prompt sample should have the same influence on the output. I think you are doing something similar, mixing the influence of the audio prompt together with the influence of the text prompt
3. as an extra feature, it'd be useful to have some extra audio parameters, similar to what you have in synthetizers, like filters, eqs, effects to influence the output. But this is something for later on.

[ivcylc]

i have a lot of ideas to implement your idea , wait my arXiv paper this year (i am doing something other currently)

[Tortoise17]

@buscon does transfer learning from the input audio sample for human spoken to generate a specific human voice with model with giga_speech already works?

[buscon]

@buscon does transfer learning from the input audio sample for human spoken to generate a specific human voice with model with giga_speech already works?

I think it does, though I tested it long time ago and cannot remember right now.
I will try it again soon and report back here.

[Tortoise17]

@buscon did you manage to find the way? or any success?

[buscon]

@buscon did you manage to find the way? or any success?

not yet, I cannot install audioldm with pip anymore. I think it's related to the overall upgrade of python 3.12.
I will answer again when I figured that out.