Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

Abstract

Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.

arXiv commentICML 2021

References in corpus (9)

• Auto-Encoding Variational Bayes
• Decoupled Weight Decay Regularization
• Neural Discrete Representation Learning
• Variational Inference with Normalizing Flows
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• FastSpeech: Fast, Robust and Controllable Text to Speech
• Variational Lossy Autoencoder
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Latent Normalizing Flows for Discrete Sequences

Cited by in corpus (11)

• A Survey on Neural Speech Synthesis
• ESPnet2-TTS: Extending the Edge of TTS Research
• Autovocoder: Fast Waveform Generation from a Learned Speech Representation using Differentiable Digital Signal Processing
• MuSE-SVS: Multi-Singer Emotional Singing Voice Synthesizer that Controls Emotional Intensity
• SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech
• PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping
• Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAE
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Speech-Forensics: Towards Comprehensive Synthetic Speech Dataset Establishment and Analysis
• Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow