Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis

Abstract

In this paper we propose Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis with control over speech variation and style transfer. Flowtron borrows insights from IAF and revamps Tacotron in order to provide high-quality and expressive mel-spectrogram synthesis. Flowtron is optimized by maximizing the likelihood of the training data, which makes training simple and stable. Flowtron learns an invertible mapping of data to a latent space that can be manipulated to control many aspects of speech synthesis (pitch, tone, speech rate, cadence, accent). Our mean opinion scores (MOS) show that Flowtron matches state-of-the-art TTS models in terms of speech quality. In addition, we provide results on control of speech variation, interpolation between samples and style transfer between speakers seen and unseen during training. Code and pre-trained models will be made publicly available at https://github.com/NVIDIA/flowtron

10 pages, 7 pictures

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Deep Voice: Real-time Neural Text-to-Speech
• Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech
• Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
• Tacotron: Towards End-to-End Speech Synthesis
• High Fidelity Speech Synthesis with Adversarial Networks
• Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning
• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Transflow Learning: Repurposing Flow Models Without Retraining

Cited by in corpus (21)

• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• A Survey on Neural Speech Synthesis
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• End-to-End Adversarial Text-to-Speech
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
• Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?
• Review of end-to-end speech synthesis technology based on deep learning
• TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention
• Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition
• Creating New Voices using Normalizing Flows
• Exploring emotional prototypes in a high dimensional TTS latent space
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
• OverFlow: Putting flows on top of neural transducers for better TTS
• EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecture
• Speech Synthesis and Control Using Differentiable DSP
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow
• Text-to-speech for the hearing impaired
• Sprachsynthese -- State-of-the-Art in englischer und deutscher Sprache
• GANtron: Emotional Speech Synthesis with Generative Adversarial Networks