WaveFlow: A Compact Flow-based Model for Raw Audio

Abstract

In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dilated 2-D convolutional architecture, while modeling the local variations using expressive autoregressive functions. WaveFlow provides a unified view of likelihood-based models for 1-D data, including WaveNet and WaveGlow as special cases. It generates high-fidelity speech as WaveNet, while synthesizing several orders of magnitude faster as it only requires a few sequential steps to generate very long waveforms with hundreds of thousands of time-steps. Furthermore, it can significantly reduce the likelihood gap that has existed between autoregressive models and flow-based models for efficient synthesis. Finally, our small-footprint WaveFlow has only 5.91M parameters, which is 15$\times$ smaller than WaveGlow. It can generate 22.05 kHz high-fidelity audio 42.6$\times$ faster than real-time (at a rate of 939.3 kHz) on a V100 GPU without engineered inference kernels.

arXiv commentPublished at ICML 2020. Code and pre-trained models: https://github.com/PaddlePaddle/Parakeet

References in corpus (21)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• WaveNet: A Generative Model for Raw Audio
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Conditional Image Generation with PixelCNN Decoders
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• FastSpeech: Fast, Robust and Controllable Text to Speech
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Improving Variational Inference with Inverse Autoregressive Flow
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• The challenge of realistic music generation: modelling raw audio at scale
• Sylvester Normalizing Flows for Variational Inference
• High Fidelity Speech Synthesis with Adversarial Networks
• WaveFlow: A Compact Flow-based Model for Raw Audio
• Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
• Discrete Flows: Invertible Generative Models of Discrete Data
• Neural Speech Synthesis with Transformer Network
• Fast Wavenet Generation Algorithm
• Probability density distillation with generative adversarial networks for high-quality parallel waveform generation

Cited by in corpus (17)

• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling
• WaveFlow: A Compact Flow-based Model for Raw Audio
• End-to-End Adversarial Text-to-Speech
• TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis
• A Spectral Energy Distance for Parallel Speech Synthesis
• Restoring degraded speech via a modified diffusion model
• A Contrastive Learning Approach for Training Variational Autoencoder Priors
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Parallel Synthesis for Autoregressive Speech Generation
• ConVoice: Real-Time Zero-Shot Voice Style Transfer with Convolutional Network
• LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation
• NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity
• Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics
• Empirical Evaluation of Deep Learning Model Compression Techniques on the WaveNet Vocoder