Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
arXiv:1712.05884
Abstract
This paper describes Tacotron 2, a neural network architecture for speech synthesis directly from text. The system is composed of a recurrent sequence-to-sequence feature prediction network that maps character embeddings to mel-scale spectrograms, followed by a modified WaveNet model acting as a vocoder to synthesize timedomain waveforms from those spectrograms. Our model achieves a mean opinion score (MOS) of comparable to a MOS of for professionally recorded speech. To validate our design choices, we present ablation studies of key components of our system and evaluate the impact of using mel spectrograms as the input to WaveNet instead of linguistic, duration, and features. We further demonstrate that using a compact acoustic intermediate representation enables significant simplification of the WaveNet architecture.
Accepted to ICASSP 2018
References in corpus (4)
Cited by in corpus (37)
- Machine learning in acoustics: theory and applications
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
- Neural Voice Cloning with a Few Samples
- Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
- Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
- The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems
- Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
- Synthesizing Speech from Intracranial Depth Electrodes using an Encoder-Decoder Framework
- An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
- Towards Robust Neural Vocoding for Speech Generation: A Survey
- Direct speech-to-speech translation with a sequence-to-sequence model
- Fitting New Speakers Based on a Short Untranscribed Sample
- VSMask: Defending Against Voice Synthesis Attack via Real-Time Predictive Perturbation
- Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
- ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
- Adversarial Generation of Time-Frequency Features with application in audio synthesis
- Controllable neural text-to-speech synthesis using intuitive prosodic features
- HooliGAN: Robust, High Quality Neural Vocoding
- Towards achieving robust universal neural vocoding
- Training Keyword Spotters with Limited and Synthesized Speech Data
- Simulation of machine learning-based 6G systems in virtual worlds
- Hierarchical Sequence to Sequence Voice Conversion with Limited Data
- MoBoAligner: a Neural Alignment Model for Non-autoregressive TTS with Monotonic Boundary Search
- Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
- Voice Imitating Text-to-Speech Neural Networks
- Speech-to-Singing Conversion based on Boundary Equilibrium GAN
- Cycle-consistency training for end-to-end speech recognition
- Deepfake audio as a data augmentation technique for training automatic speech to text transcription models
- Wasserstein GAN and Waveform Loss-based Acoustic Model Training for Multi-speaker Text-to-Speech Synthesis Systems Using a WaveNet Vocoder
- From the perspective of perceptual speech quality: The robustness of frequency bands to noise
- Investigating context features hidden in End-to-End TTS
- Daily Turking: Designing Longitudinal Daily-task Studies on Mechanical Turk
- A Unified Neural Architecture for Instrumental Audio Tasks
- Do WaveNets Dream of Acoustic Waves?
- Modeling of Rakugo Speech and Its Limitations: Toward Speech Synthesis That Entertains Audiences
- Mathematical Vocoder Algorithm : Modified Spectral Inversion for Efficient Neural Speech Synthesis