MultiSpeech: Multi-Speaker Text to Speech with Transformer

Abstract

Transformer-based text to speech (TTS) model (e.g., Transformer TTS~\cite{li2019neural}, FastSpeech~\cite{ren2019fastspeech}) has shown the advantages of training and inference efficiency over RNN-based model (e.g., Tacotron~\cite{shen2018natural}) due to its parallel computation in training and/or inference. However, the parallel computation increases the difficulty while learning the alignment between text and speech in Transformer, which is further magnified in the multi-speaker scenario with noisy data and diverse speakers, and hinders the applicability of Transformer for multi-speaker TTS. In this paper, we develop a robust and high-quality multi-speaker Transformer TTS system called MultiSpeech, with several specially designed components/techniques to improve text-to-speech alignment: 1) a diagonal constraint on the weight matrix of encoder-decoder attention in both training and inference; 2) layer normalization on phoneme embedding in encoder to better preserve position information; 3) a bottleneck in decoder pre-net to prevent copy between consecutive speech frames. Experiments on VCTK and LibriTTS multi-speaker datasets demonstrate the effectiveness of MultiSpeech: 1) it synthesizes more robust and better quality multi-speaker voice than naive Transformer based TTS; 2) with a MutiSpeech model as the teacher, we obtain a strong multi-speaker FastSpeech model with almost zero quality degradation while enjoying extremely fast inference speed.

References in corpus (9)

• Attention-Based Models for Speech Recognition
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
• Tacotron: Towards End-to-End Speech Synthesis
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesis
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion
• Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments

Cited by in corpus (13)

• A Survey on Neural Speech Synthesis
• ESPnet2-TTS: Extending the Edge of TTS Research
• Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis
• Hi-Fi Multi-Speaker English TTS Dataset
• fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
• Neural Dubber: Dubbing for Videos According to Scripts
• WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution
• Federated Learning with Dynamic Transformer for Text to Speech
• GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis
• EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model
• AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data
• Speech BERT Embedding For Improving Prosody in Neural TTS
• Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representations