Improving Universal Sound Separation Using Sound Classification

Abstract

Deep learning approaches have recently achieved impressive performance on both audio source separation and sound classification. Most audio source separation approaches focus only on separating sources belonging to a restricted domain of source classes, such as speech and music. However, recent work has demonstrated the possibility of "universal sound separation", which aims to separate acoustic sources from an open domain, regardless of their class. In this paper, we utilize the semantic information learned by sound classifier networks trained on a vast amount of diverse sounds to improve universal sound separation. In particular, we show that semantic embeddings extracted from a sound classifier can be used to condition a separation network, providing it with useful additional information. This approach is especially useful in an iterative setup, where source estimates from an initial separation stage and their corresponding classifier-derived embeddings are fed to a second separation network. By performing a thorough hyperparameter search consisting of over a thousand experiments, we find that classifier embeddings from clean sources provide nearly one dB of SNR gain, and our best iterative models achieve a significant fraction of this oracle performance, establishing a new state-of-the-art for universal sound separation.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Conditional Generative Adversarial Nets
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Conditional Image Generation with PixelCNN Decoders
• Speaker-independent Speech Separation with Deep Attractor Network
• Exponential Family Embeddings
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• Speaker Diarization using Deep Recurrent Convolutional Neural Networks for Speaker Embeddings
• Audio query-based music source separation

Cited by in corpus (18)

• Sudo rm -rf: Efficient Networks for Universal Audio Source Separation
• Unsupervised Sound Separation Using Mixture Invariant Training
• Compute and memory efficient universal sound source separation
• Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds
• Conditioned Source Separation for Music Instrument Performances
• Signal-Aware Direction-of-Arrival Estimation Using Attention Mechanisms
• Stepwise-Refining Speech Separation Network via Fine-Grained Encoding in High-order Latent Domain
• DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classification
• Improving Sound Event Detection In Domestic Environments Using Sound Separation
• Content based singing voice source separation via strong conditioning using aligned phonemes
• Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention
• Unified Gradient Reweighting for Model Biasing with Applications to Source Separation
• SADDEL: Joint Speech Separation and Denoising Model based on Multitask Learning
• Multi-Task Audio Source Separation
• Far-Field Automatic Speech Recognition
• On permutation invariant training for speech source separation
• Informed Source Extraction With Application to Acoustic Echo Reduction
• Complete and separate: Conditional separation with missing target source attribute completion