Status: Archive (code is provided as-is, no updates expected)
We are trying to clone voices for speakers which is content independent. This means that we have to encapture the identity of the speaker rather than the content they speak. We try to do this by making a speaker embedding space for different speakers.
The speaker embeddings try to represent the identity of the speaker(various aspects of the voice like pitch, accent, etc of the speaker), you can consider this as the voice fingerprint of the speaker.
We are right now referring to the following paper for our Implementation:-
The architecture for the Multi-Speaker Generative and Speaker Encoder Model have been built.
Multi-Speaker Generative model has been trained for speaker adaptation for 84 speakers using VCTK-dataset has been completed on NVIDIA - V100 GPU for 190000 epochs.
VCTK dataset was split for training and testing: 84 speakers are used for training the multi-speaker model, 8 speakers for validation, and 16 speakers for cloning.
The following will train the model on the first 84 speakers in the dataset.
python speaker_adaptation.py --data-root=<path_of_vctk_dataset> --checkpoint-dir=<path> --checkpoint-interval=<int>
This can take upto 20 hours using a GPU.
To adapt the model to a particular speaker after the initial training
python speaker_adaptation.py --data-root=<path_of_vctk_dataset> --restore-parts=<path_of_checkpoint> --checkpoint-dir=<path> --checkpoint-interval=<int>
This will take on an average of 10 to 20 minutes.
So far some of the coned voices we have got using speaker adaptation LINK
-
The implementation of Multi-Speaker Generative model was inspired from https://github.com/r9y9/deepvoice3_pytorch
If you find the code in the repository useful, please cite it using:
@misc{chitlangia2021voicecloning,
author = {Chitlangia, Sharad and Rastogi, Mehul and Ganguly, Rijul},
title = {An Open Source Implementation of Neural Voice Cloning With Few Samples},
year = {2021},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {Available at \url{https://github.com/SforAiDl/Neural-Voice-Cloning-With-Few-Samples/} or \url{https://github.com/Sharad24/Neural-Voice-Cloning-With-Few-Samples/}},
}