Discussions! Any way to tie this to Ooba or Silly Tavern?

[St33lMouse]

Is there any way to get Ooba or Silly Tavern to use voices from this repo? The Bark voice cloning here is excellent. If this repo can act as the voice output for Ooba or Silly Tavern, we'd finally have the ability to output tts in Ooba and have the AI read to you in any voice you want quickly and at high quality. Same goes for Silly Tavern. You could give any Silly Tavern character the voice of your choosing.

All other efforts have been limited so far. You can't clone voices in Silero, Tortoise is slow, etc. This repo is the magic ingredient to making characters come alive.

[rsxdalv]

It should be possible, just need to connect the APIs. I haven't used ooba recently, do you know if TTS is built in or is it a plugin? Also, probably the settings might need to be changed such as CPU offloading, assuming that ooba eats up most of the VRAM. Just a quick mention: tortoise can be run quicker. Using the new options if you set samples to 4 and iterations to 4-16 you'll get much faster results. (Iterations improve output quality more cheaply than samples). The quality does drop though, and bark, when it behaves properly, is more impressive.

…

On Sat, Jul 1, 2023, 6:37 AM St33lMouse ***@***.***> wrote: Is there any way to get Ooba or Silly Tavern to use voices from this repo? The Bark voice cloning here is excellent. If this repo can act as the voice output for Ooba or Silly Tavern, we'd finally have the ability to output tts in Ooba and have the AI read to you in any voice you want quickly and at high quality. Same goes for Silly Tavern. You could give any Silly Tavern character the voice of your choosing. All other efforts have been limited so far. You can't clone voices in Silero, Tortoise is slow, etc. This repo is the magic ingredient to making characters come alive. — Reply to this email directly, view it on GitHub <#57>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTRXIYDN7WVACI3IAIFLHTXN6LN5ANCNFSM6AAAAAAZ2Q3WCI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[St33lMouse]

both Ooba and Silly Tavern have voice. Ooba has Silero, eleven labs, and a Bark extension here:

https://github.com/wsippel/bark_tts

I don't know if the extension is using the same version of Bark you are using. Quality seems different, but maybe that's just me. There are other ooba voice projects underway, including a coquoi and tortoise project. I don't think it has been merged to main yet, and it's been awhile.

Silly Tavern has Silero and maybe a couple of other choices, but not Bark. Bark is clearly the strongest. An API to Silly Tavern would be most welcome.

I'll play around with Tortoise on your repo. There is a repo that made voice cloning possible with tortoise by finetuning the voice models here:

https://git.ecker.tech/mrq/ai-voice-cloning

Quality was quite good, strong resemblance, but slower than Bark.

[rsxdalv]

It should be all means be the "same version". So if you can find a sweet spot using my UI, like temperature values and the "voice", it should be compatible with any other bark implementation. One difference is the long form generation, which might differ as it's more custom.

[rsxdalv]

As for the speed of tortoise, apparently you can get something "passable" with just 4 samples and ~30-50 iterations. But hopefully things should speed up with bark switching to HF's transformers.

[rsxdalv]

Ok so I took a look at bark_tts plugin, and it seems to do several things specifically to enhance the quality when running with oobabooga. What I'd say is that for this moment you could find the "optimal" parameters using tts-generation-webui, and then just dial them into the plugin.
For me it's tricky to work with that code since it's an AGPL repo, but I will see what I can do once I fix the installation issues.

[78Alpha]

If quality is the focus, using a tts to get the words (maybe emotion with the gpt backed versions), and then throwing RVC on top might make it sound better. While it has artifacts for sounds that are too different, I have found that training a tortoise model, cleaning it, and then training RVC makes a very high quality and adaptable voice. I do state tortoise because bark training is... unreliable to say the least, making for artifacts on the RVC part.

Whether Ooba uses all the VRAM is up to the user. Loading a 13B model will fill most of the VRAM of a 24 GB card. I usually have 4 GB spare. Loading it in 4bit mode will leave about 10 GB to spare. Most users on 8 GB will be running 7 B and near the limit of their VRAM capacity.

[rsxdalv]

Thanks for the feedback, I have seen impressive results on that route!

As for running it with oobabooga, I think bark could be better because you can run it with 2gb active VRAM and near 0 passive one (by moving it to the RAM with CPU offloading option), perhaps even enabling a triplet:
oobabooga(4bit 6-8B parameter model) + stable diffusion (1.5:with -medvram or -lowvram) + bark (CPU offloading, small) on a 8GB graphics card.