Skip to content

⏩ Generating speech in a single forward pass without any attention!

License

Notifications You must be signed in to change notification settings

sbuser/ForwardTacotron

 
 

Repository files navigation

⏩ ForwardTacotron

Inspired by Microsoft's FastSpeech we modified Tacotron (Fork from fatchord's WaveRNN) to generate speech in a single forward pass using a duration predictor to align text and generated mel spectrograms. Hence, we call the model ForwardTacotron (see Figure 1).

Figure 1: Model Architecture.

The model has following advantages:

  • Robustness: No repeats and failed attention modes for challenging sentences.
  • Speed: The generation of a mel spectogram takes about 0.04s on a GeForce RTX 2080.
  • Controllability: It is possible to control the speed of the generated utterance.
  • Efficiency: In contrast to FastSpeech and Tacotron, the model of ForwardTacotron does not use any attention. Hence, the required memory grows linearly with text size, which makes it possible to synthesize large articles at once.

MAJOR UPDATE V1 --> V2 (10.05.2020)

  • Added optional energy conditioning similar to the one in FastSpeech2
  • Replaced hparams.py with config.yaml that is now stored in the model and loaded automatically
  • Major refactoring, added tests etc.

Check out the latest audio samples (ForwardTacotron + HiFiGAN)!

Energy conditioning reduces mel validation loss:

🔈 Samples

Can be found here.

The samples are generated with a model trained on LJSpeech and vocoded with WaveRNN, MelGAN, or HiFiGAN. You can try out the latest pretrained model with the following notebook:

Open In Colab

⚙️ Installation

Make sure you have:

  • Python >= 3.6

Install espeak as phonemizer backend (for macOS use brew):

sudo apt-get install espeak

Then install the rest with pip:

pip install -r requirements.txt

🚀 Training your own Model

Change the params in the config.yaml according to your needs and follow the steps below:

(1) Download and preprocess the LJSpeech dataset:

python preprocess.py --path /path/to/ljspeech

(2) Train Tacotron with:

python train_tacotron.py

Once the training is finished, the model will automatically extract the alignment features from the dataset. In case you stopped the training early, you can use the latest checkpoint to manually run the process with:

python train_tacotron.py --force_align

(3) Train ForwardTacotron with:

python train_forward.py

(4) Generate Sentences with Griffin-Lim vocoder:

python gen_forward.py --alpha 1 --input_text 'this is whatever you want it to be' griffinlim

If you want to use the MelGAN or HiFiGAN vocoder, you can produce .mel files with:

python gen_forward.py --input_text 'this is whatever you want it to be' melgan

To vocode the resulting .mel files use the inference.py script from the MelGAN or HiFiGAN repo and point to the model output folder.

As in the original repo you can also use a trained WaveRNN vocoder:

python gen_forward.py --input_text 'this is whatever you want it to be' wavernn

For training the model on your own dataset just bring it to the LJSpeech-like format:

|- dataset_folder/
|   |- metadata.csv
|   |- wav/
|       |- file1.wav
|       |- ...

For languages other than English, change the language and cleaners params in the hparams.py, e.g. for French:

language = 'fr'
tts_cleaner_name = 'basic_cleaners'

You can monitor the training processes for Tacotron and ForwardTacotron with

tensorboard --logdir checkpoints

Here is what the ForwardTacotron tensorboard looks like:

Figure 2: Tensorboard example for training a ForwardTacotron model.

Pretrained Models

Model Dataset Commit
forward_tacotron ljspeech latest
wavernn ljspeech latest

Our pre-trained LJSpeech model is compatible with the pre-trained vocoders:

After downloading the models you can synthesize text using the pretrained models with

python gen_forward.py --input_text 'Hi there!' --checkpoint forward_step90k.pt wavernn --voc_checkpoint wave_step_575k.pt

Tips for training a WaveRNN model

  • From experience I recommend starting with the standard params (RAW mode with 9 bit), which should start to sound good after about 300k steps.
  • Sound quality of the models varies quite a bit, so it is important to cherry-pick the best one.
  • For cherry-picking it is useful to listen to the validation sound samples in tensorboard. The sound quality of the samples is measured by an additional metric (L1 distance of mel specs).
  • The top k models according to the above metric are constantly monitored and checkpointed under path/to/checkpoint/top_k_models.

Here is what the WaveRNN tensorboard looks like:

Figure 3: Tensorboard example for training a WaveRNN model.

References

Acknowlegements

Maintainers

Copyright

See LICENSE for details.

About

⏩ Generating speech in a single forward pass without any attention!

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 100.0%