ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech

Abstract

In this work, we propose a new solution for parallel wave generation by WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet by minimizing a regularized KL divergence between their highly-peaked output distributions. Our method computes the KL divergence in closed-form, which simplifies the training algorithm and provides very efficient distillation. In addition, we introduce the first text-to-wave neural architecture for speech synthesis, which is fully convolutional and enables fast end-to-end training from scratch. It significantly outperforms the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet (Ping et al., 2018). We also successfully distill a parallel waveform synthesizer conditioned on the hidden representation in this end-to-end model.

arXiv commentPublished at ICLR 2019. (v3: add important details & discussion in Appendix A)

References in corpus (15)

• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• On the difficulty of training Recurrent Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Convolutional Sequence to Sequence Learning
• Conditional Image Generation with PixelCNN Decoders
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• A Recurrent Latent Variable Model for Sequential Data
• Non-Autoregressive Neural Machine Translation
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Improving Variational Inference with Inverse Autoregressive Flow
• Fast Decoding in Sequence Models using Discrete Latent Variables
• RNADE: The real-valued neural autoregressive density-estimator
• Theory and Experiments on Vector Quantized Autoencoders

Cited by in corpus (36)

• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• MelNet: A Generative Model for Audio in the Frequency Domain
• Audeo: Audio Generation for a Silent Performance Video
• CaloFlow II: Even Faster and Still Accurate Generation of Calorimeter Showers with Normalizing Flows
• FloWaveNet : A Generative Flow for Raw Audio
• NU-GAN: High resolution neural upsampling with GAN
• TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis
• ConvS2S-VC: Fully convolutional sequence-to-sequence voice conversion
• A Spectral Energy Distance for Parallel Speech Synthesis
• WaveCycleGAN2: Time-domain Neural Post-filter for Speech Waveform Generation
• TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese
• Feature reinforcement with word embedding and parsing information in neural TTS
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics
• FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction
• Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
• Towards achieving robust universal neural vocoding
• DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech
• Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding
• WaveNODE: A Continuous Normalizing Flow for Speech Synthesis
• A New GAN-based End-to-End TTS Training Algorithm
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework
• Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis
• Speaking style adaptation in Text-To-Speech synthesis using Sequence-to-sequence models with attention
• Nonparallel Voice Conversion with Augmented Classifier Star Generative Adversarial Networks
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation
• Reverberation Modeling for Source-Filter-based Neural Vocoder
• Training a Neural Speech Waveform Model using Spectral Losses of Short-Time Fourier Transform and Continuous Wavelet Transform
• PeriodNet: A non-autoregressive waveform generation model with a structure separating periodic and aperiodic components
• Improve GAN-based Neural Vocoder using Pointwise Relativistic LeastSquare GAN
• AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person
• Generative Adversarial Network based Speaker Adaptation for High Fidelity WaveNet Vocoder
• Empirical Evaluation of Deep Learning Model Compression Techniques on the WaveNet Vocoder