A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation

Abstract

Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue, music, and effects stems from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psychoacoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.

arXiv commentAccepted to the IEEE Open Journal of Signal Processing (ICASSP 2024 Track)

References in corpus (8)

• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Music Source Separation in the Waveform Domain
• Equivariance Through Parameter-Sharing
• Hybrid Spectrogram and Waveform Source Separation
• KUIELab-MDX-Net: A Two-Stream Neural Network for Music Demixing
• Learning to Denoise Historical Music
• End-to-End Complex-Valued Multidilated Convolutional Neural Network for Joint Acoustic Echo Cancellation and Noise Suppression
• DESNet: A Multi-channel Network for Simultaneous Speech Dereverberation, Enhancement and Separation