NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates

Abstract

Conventionally, audio super-resolution models fixed the initial and the target sampling rates, which necessitate the model to be trained for each pair of sampling rates. We introduce NU-Wave 2, a diffusion model for neural audio upsampling that enables the generation of 48 kHz audio signals from inputs of various sampling rates with a single model. Based on the architecture of NU-Wave, NU-Wave 2 uses short-time Fourier convolution (STFC) to generate harmonics to resolve the main failure modes of NU-Wave, and incorporates bandwidth spectral feature transform (BSFT) to condition the bandwidths of inputs in the frequency domain. We experimentally demonstrate that NU-Wave 2 produces high-resolution audio regardless of the sampling rate of input while requiring fewer parameters than other models. The official code and the audio samples are available at https://mindslab-ai.github.io/nuwave2.

arXiv commentAccepted to Interspeech 2022

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Denoising Diffusion Probabilistic Models
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Score-Based Generative Modeling through Stochastic Differential Equations
• Generative Modeling by Estimating Gradients of the Data Distribution
• Classifier-Free Diffusion Guidance
• Variational Diffusion Models
• Progressive Distillation for Fast Sampling of Diffusion Models
• TUNet: A Block-online Bandwidth Extension Model based on Transformers and Self-supervised Pretraining
• VoiceFixer: Toward General Speech Restoration with Neural Vocoder
• Bandwidth Extension on Raw Audio via Generative Adversarial Networks

Cited by in corpus (5)

• StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
• Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution
• PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping
• FLowHigh: Towards Efficient and High-Quality Audio Super-Resolution with Single-Step Flow Matching
• An empirical study on speech restoration guided by self supervised speech representation