End-to-End Adversarial Text-to-Speech

Abstract

Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.

arXiv comment23 pages. In proceedings of ICLR 2021

References in corpus (28)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• WaveNet: A Generative Model for Raw Audio
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Convolutional Sequence to Sequence Learning
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Sequence Transduction with Recurrent Neural Networks
• A Learned Representation For Artistic Style
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Deep Voice: Real-time Neural Text-to-Speech
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Modulating early visual processing by language
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks
• MelNet: A Generative Model for Audio in the Frequency Domain
• Adversarial Audio Synthesis
• Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesis
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• DDSP: Differentiable Digital Signal Processing
• WaveFlow: A Compact Flow-based Model for Raw Audio
• Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
• TF-Replicator: Distributed Machine Learning for Researchers
• Probability density distillation with generative adversarial networks for high-quality parallel waveform generation
• VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop

Cited by in corpus (20)

• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling
• Parallel Tacotron: Non-Autoregressive and Controllable TTS
• Differentiable Divergences Between Time Series
• Review of end-to-end speech synthesis technology based on deep learning
• VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention
• Neural HMMs are all you need (for high-quality attention-free TTS)
• Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
• EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecture
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• PortaSpeech: Portable and High-Quality Generative Text-to-Speech
• WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis
• Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
• Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis
• Attention Forcing for Machine Translation
• The NTU-AISG Text-to-speech System for Blizzard Challenge 2020
• AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person