NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

Abstract

In this work, we introduce NU-Wave, the first neural audio upsampling model to produce waveforms of sampling rate 48kHz from coarse 16kHz or 24kHz inputs, while prior works could generate only up to 16kHz. NU-Wave is the first diffusion probabilistic model for audio super-resolution which is engineered based on neural vocoders. NU-Wave generates high-quality audio that achieves high performance in terms of signal-to-noise ratio (SNR), log-spectral distance (LSD), and accuracy of the ABX test. In all cases, NU-Wave outperforms the baseline models despite the substantially smaller model capacity (3.0M parameters) than baselines (5.4-21%). The audio samples of our model are available at https://mindslab-ai.github.io/nuwave, and the code will be made available soon.

arXiv commentAccepted to Interspeech 2021

References in corpus (9)

• Auto-Encoding Variational Bayes
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Generative Modeling by Estimating Gradients of the Data Distribution
• Improved Techniques for Training Score-Based Generative Models
• WaveFlow: A Compact Flow-based Model for Raw Audio
• Adversarial score matching and improved sampling for image generation
• Non-Autoregressive Neural Text-to-Speech
• Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations
• Bandwidth Extension on Raw Audio via Generative Adversarial Networks

Cited by in corpus (5)

• On Fast Sampling of Diffusion Probabilistic Models
• TUNet: A Block-online Bandwidth Extension Model based on Transformers and Self-supervised Pretraining
• VoiceFixer: Toward General Speech Restoration with Neural Vocoder
• Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution
• PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping