Compute and memory efficient universal sound source separation

Abstract

Recent progress in audio source separation lead by deep learning has enabled many neural network models to provide robust solutions to this fundamental estimation problem. In this study, we provide a family of efficient neural network architectures for general purpose audio source separation while focusing on multiple computational aspects that hinder the application of neural networks in real-world scenarios. The backbone structure of this convolutional network is the SUccessive DOwnsampling and Resampling of Multi-Resolution Features (SuDoRM-RF) as well as their aggregation which is performed through simple one-dimensional convolutions. This mechanism enables our models to obtain high fidelity signal separation in a wide variety of settings where variable number of sources are present and with limited computational resources (e.g. floating point operations, memory footprint, number of parameters and latency). Our experiments show that SuDoRM-RF models perform comparably and even surpass several state-of-the-art benchmarks with significantly higher computational resource requirements. The causal variation of SuDoRM-RF is able to obtain competitive performance in real-time speech separation of around 10dB scale-invariant signal-to-distortion ratio improvement (SI-SDRi) while remaining up to 20 times faster than real-time on a laptop device.

arXiv commentAccepted to Journal of Signal Processing Systems https://www.springer.com/journal/11265. arXiv admin note: substantial text overlap with arXiv:2007.06833

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Music Source Separation in the Waveform Domain
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Improving Universal Sound Separation Using Sound Classification
• Two-Step Sound Source Separation: Training on Learned Latent Targets
• Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separation
• Ultra-Lightweight Speech Separation via Group Communication

Cited by in corpus (6)

• The CHiME-7 UDASE task: Unsupervised domain adaptation for conversational speech enhancement
• Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data
• Continual self-training with bootstrapped remixing for speech enhancement
• RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing
• Latent Iterative Refinement for Modular Source Separation
• Few-shot learning of new sound classes for target sound extraction