Neural Speech Synthesis with Transformer Network

Abstract

Although end-to-end neural text-to-speech (TTS) methods (such as Tacotron2) are proposed and achieve state-of-the-art performance, they still suffer from two problems: 1) low efficiency during training and inference; 2) hard to model long dependency using current recurrent neural networks (RNNs). Inspired by the success of Transformer network in neural machine translation (NMT), in this paper, we introduce and adapt the multi-head attention mechanism to replace the RNN structures and also the original attention mechanism in Tacotron2. With the help of multi-head self-attention, the hidden states in the encoder and decoder are constructed in parallel, which improves the training efficiency. Meanwhile, any two inputs at different times are connected directly by self-attention mechanism, which solves the long range dependency problem effectively. Using phoneme sequences as input, our Transformer TTS network generates mel spectrograms, followed by a WaveNet vocoder to output the final audio results. Experiments are conducted to test the efficiency and performance of our new network. For the efficiency, our Transformer TTS network can speed up the training about 4.25 times faster compared with Tacotron2. For the performance, rigorous human tests show that our proposed model achieves state-of-the-art performance (outperforms Tacotron2 with a gap of 0.048) and is very close to human quality (4.39 vs 4.44 in MOS).

Cited by in corpus (19)

• A Comparative Study on Transformer vs RNN in Speech Applications
• WaveFlow: A Compact Flow-based Model for Raw Audio
• Text-Conditioned Transformer for Automatic Pronunciation Error Detection
• Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models
• Learning pronunciation from a foreign language in speech synthesis networks
• MOTR: End-to-End Multiple-Object Tracking with Transformer
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
• TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation
• AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignment
• Improved parallel WaveGAN vocoder with perceptually weighted spectrogram loss
• MelGlow: Efficient Waveform Generative Network Based on Location-Variable Convolution
• Fast DCTTS: Efficient Deep Convolutional Text-to-Speech
• OadTR: Online Action Detection with Transformers
• Prosody Learning Mechanism for Speech Synthesis System Without Text Length Limit
• Phrase break prediction with bidirectional encoder representations in Japanese text-to-speech synthesis
• Joint Intent Detection And Slot Filling Based on Continual Learning Model
• Neural text-to-speech with a modeling-by-generation excitation vocoder