Deep Learning for Audio Signal Processing

Abstract

Given the recent surge in developments of deep learning, this article provides a review of the state-of-the-art deep learning techniques for audio signal processing. Speech, music, and environmental sound processing are considered side-by-side, in order to point out similarities and differences between the domains, highlighting general methods, problems, key references, and potential for cross-fertilization between areas. The dominant feature representations (in particular, log-mel spectra and raw waveform) and deep learning models are reviewed, including convolutional neural networks, variants of the long short-term memory architecture, as well as more audio-specific neural network models. Subsequently, prominent deep learning application areas are covered, i.e. audio recognition (automatic speech recognition, music information retrieval, environmental sound detection, localization and tracking) and synthesis and transformation (source separation, audio enhancement, generative models for speech, sound, and music synthesis). Finally, key issues and future questions regarding deep learning applied to audio signal processing are identified.

15 pages, 2 pdf figures

References in corpus (26)

• Neural Machine Translation by Jointly Learning to Align and Translate
• WaveNet: A Generative Model for Raw Audio
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Attention-Based Models for Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• Joint Optimization of Masks and Deep Recurrent Neural Networks for Monaural Source Separation
• Deep attractor network for single-microphone speaker separation
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• Recurrent Neural Networks for Polyphonic Sound Event Detection in Real Life Recordings
• Listen, Attend and Spell
• Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification
• Automatic tagging using deep convolutional neural networks
• Learning from Between-class Examples for Deep Sound Recognition
• Deep Complex Networks
• Learning Features of Music from Scratch
• Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms
• Advances in All-Neural Speech Recognition
• Feature Learning for Chord Recognition: The Deep Chroma Extractor
• Supervised Speech Separation Based on Deep Learning: An Overview
• Dense Prediction on Sequences with Time-Dilated Convolutions for Speech Recognition
• Multi-Speaker Localization Using Convolutional Neural Network Trained with Noise
• Deep Learning for Music
• Monaural Speech Enhancement using Deep Neural Networks by Maximizing a Short-Time Objective Intelligibility Measure
• Piano Genie

Cited by in corpus (56)

• Machine learning in acoustics: theory and applications
• Wireless Image Transmission Using Deep Source Channel Coding With Attention Modules
• A Survey of Sound Source Localization with Deep Learning Methods
• Combining a Convolutional Neural Network with Autoencoders to Predict the Survival Chance of COVID-19 Patients
• CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement
• Environmental Sound Classification on the Edge: A Pipeline for Deep Acoustic Networks on Extremely Resource-Constrained Devices
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Deep Feature Space: A Geometrical Perspective
• Piecewise Linear Neural Networks and Deep Learning
• How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
• The Ethical Implications of Generative Audio Models: A Systematic Literature Review
• Machine Learning Assisted Inverse Design of Microresonators
• Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines
• Interpreting glottal flow dynamics for detecting COVID-19 from voice
• Semi-supervised source localization with deep generative modeling
• Listening to Sounds of Silence for Speech Denoising
• Audio Tagging by Cross Filtering Noisy Labels
• On the computation of counterfactual explanations -- A survey
• ElectrodeNet -- A Deep Learning Based Sound Coding Strategy for Cochlear Implants
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• Synthetic data generation method for data-free knowledge distillation in regression neural networks
• Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond
• Approximation in shift-invariant spaces with deep ReLU neural networks
• Conditional Generative Data Augmentation for Clinical Audio Datasets
• Better Modelling Out-of-Distribution Regression on Distributed Acoustic Sensor Data Using Anchored Hidden State Mixup
• An unsupervised deep learning framework for medical image denoising
• JaCappella Corpus: A Japanese a Cappella Vocal Ensemble Corpus
• A Systematic Evaluation of Adversarial Attacks against Speech Emotion Recognition Models
• Deep Feature Learning for Medical Acoustics
• On Neural Architectures for Deep Learning-based Source Separation of Co-Channel OFDM Signals
• Recent Advances and Challenges in Deep Audio-Visual Correlation Learning
• Spatial mixup: Directional loudness modification as data augmentation for sound event localization and detection
• Detection of Audio-Video Synchronization Errors Via Event Detection
• A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences
• Low-complexity Attention-based Unsupervised Anomalous Sound Detection exploiting Separable Convolutions and Angular Loss
• nnAudio: An on-the-fly GPU Audio to Spectrogram Conversion Toolbox Using 1D Convolution Neural Networks
• A new sensing paradigm for the vibroacoustic detection of pedicle screw loosening
• Randomized Overdrive Neural Networks
• A t-distribution based operator for enhancing out of distribution robustness of neural network classifiers
• Leveraging Voltage-Controlled Magnetic Anisotropy to Solve Sneak Path Issues in Crossbar Arrays
• The Impact of Feature Representation on the Accuracy of Photonic Neural Networks
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• Neural network for multi-exponential sound energy decay analysis
• AudioSet-R: A Refined AudioSet with Multi-Stage LLM Label Reannotation
• Variational Information Bottleneck for Effective Low-resource Audio Classification
• Improved Techniques for the Conditional Generative Augmentation of Clinical Audio Data
• Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression
• Phase-Based Signal Representations for Scattering
• A GAN-based Approach for Mitigating Inference Attacks in Smart Home Environment
• Adopting State-of-the-Art Pretrained Audio Representations for Music Recommender Systems
• Cascaded all-pass filters with randomized center frequencies and phase polarity for acoustic and speech measurement and data augmentation
• End-To-End Dilated Variational Autoencoder with Bottleneck Discriminative Loss for Sound Morphing -- A Preliminary Study
• Canonical Polyadic Decomposition and Deep Learning for Machine Fault Detection
• PERSA+: A Deep Learning Front-End for Context-Agnostic Audio Classification
• Event2Audio: Event-Based Optical Vibration Sensing
• Interpreting deep urban sound classification using Layer-wise Relevance Propagation