DiffWave: A Versatile Diffusion Model for Audio Synthesis

Abstract

In this work, we propose DiffWave, a versatile diffusion probabilistic model for conditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.

ICLR 2021 (oral)

References in corpus (21)

• Auto-Encoding Variational Bayes
• Denoising Diffusion Probabilistic Models
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Neural Discrete Representation Learning
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Generative Modeling by Estimating Gradients of the Data Distribution
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• FastSpeech: Fast, Robust and Controllable Text to Speech
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• GANSynth: Adversarial Neural Audio Synthesis
• Improved Techniques for Training Score-Based Generative Models
• MelNet: A Generative Model for Audio in the Frequency Domain
• Adversarial Audio Synthesis
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• On GANs and GMMs
• WaveFlow: A Compact Flow-based Model for Raw Audio
• Activation Maximization Generative Adversarial Nets
• End-to-End Adversarial Text-to-Speech
• Non-Autoregressive Neural Text-to-Speech

Cited by in corpus (36)

• Diffusion Models Beat GANs on Image Synthesis
• Score-Based Generative Modeling through Stochastic Differential Equations
• Improved Denoising Diffusion Probabilistic Models
• Variational Diffusion Models
• Structured Denoising Diffusion Models in Discrete State-Spaces
• How to Train Your Energy-Based Models
• A Diffusion model for POI recommendation
• On Fast Sampling of Diffusion Probabilistic Models
• Learning to Efficiently Sample from Diffusion Probabilistic Models
• Maximum Likelihood Training of Score-Based Diffusion Models
• WaveGrad: Estimating Gradients for Waveform Generation
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• Noise Estimation for Generative Diffusion Models
• SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models
• Review of end-to-end speech synthesis technology based on deep learning
• A Study on Speech Enhancement Based on Diffusion Probabilistic Model
• Gotta Go Fast When Generating Data with Score-Based Models
• SafeGen: Mitigating Sexually Explicit Content Generation in Text-to-Image Models
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models
• Non Gaussian Denoising Diffusion Models
• 4D Facial Expression Diffusion Model
• Restoring degraded speech via a modified diffusion model
• ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models
• SegDiff: Image Segmentation with Diffusion Probabilistic Models
• Score-based Generative Modeling in Latent Space
• Bilateral Denoising Diffusion Models
• Diffusion-Based Audio Inpainting
• Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Denoising Diffusion Gamma Models
• LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation
• Dreaming of Electrical Waves: Generative Modeling of Cardiac Excitation Waves using Diffusion Models
• DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion
• SpecSinGAN: Sound Effect Variation Synthesis Using Single-Image GANs
• On the Generative Utility of Cyclic Conditionals
• Diffusion Normalizing Flow