Stochastic Restoration of Heavily Compressed Musical Audio using Generative Adversarial Networks

Abstract

Lossy audio codecs compress (and decompress) digital audio streams by removing information that tends to be inaudible in human perception. Under high compression rates, such codecs may introduce a variety of impairments in the audio signal. Many works have tackled the problem of audio enhancement and compression artifact removal using deep learning techniques. However, only a few works tackle the restoration of heavily compressed audio signals in the musical domain. In such a scenario, there is no unique solution for the restoration of the original signal. Therefore, in this study, we test a stochastic generator of a Generative Adversarial Network (GAN) architecture for this task. Such a stochastic generator, conditioned on highly compressed musical audio signals, could one day generate outputs indistinguishable from high-quality releases. Therefore, the present study may yield insights into more efficient musical data storage and transmission. We train stochastic and deterministic generators on MP3-compressed audio signals with 16, 32, and 64 kbit/s. We perform an extensive evaluation of the different experiments utilizing objective metrics and listening tests. We find that the models can improve the quality of the audio signals over the MP3 versions for 16 and 32 kbit/s and that the stochastic generators are capable of generating outputs that are closer to the original signals than those of the deterministic generators.

arXiv comment21 pages, 5 figures, published in MDPI Electronics Special Issue "Machine Learning Applied to Music/Audio Signal Processing"

References in corpus (15)

• Improved Training of Wasserstein GANs
• Language Modeling with Gated Convolutional Networks
• Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification
• GANSynth: Adversarial Neural Audio Synthesis
• Deep Complex Networks
• Improving GANs for Speech Enhancement
• MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement
• Jukebox: A Generative Model for Music
• Waveform Modeling and Generation Using Hierarchical Recurrent Neural Networks for Speech Bandwidth Extension
• DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement
• Audio Super Resolution using Neural Networks
• NU-GAN: High resolution neural upsampling with GAN
• HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks
• DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks
• Monaural Speech Enhancement with Complex Convolutional Block Attention Module and Joint Time Frequency Losses