Unsupervised Sound Separation Using Mixture Invariant Training

Abstract

In recent years, rapid progress has been made on the problem of single-channel sound separation using supervised training of deep neural networks. In such supervised approaches, a model is trained to predict the component sources from synthetic mixtures created by adding up isolated ground-truth sources. Reliance on this synthetic training data is problematic because good performance depends upon the degree of match between the training data and real-world audio, especially in terms of the acoustic conditions and distribution of sources. The acoustic properties can be challenging to accurately simulate, and the distribution of sound types may be hard to replicate. In this paper, we propose a completely unsupervised method, mixture invariant training (MixIT), that requires only single-channel acoustic mixtures. In MixIT, training examples are constructed by mixing together existing mixtures, and the model separates them into a variable number of latent sources, such that the separated sources can be remixed to approximate the original mixtures. We show that MixIT can achieve competitive performance compared to supervised methods on speech separation. Using MixIT in a semi-supervised learning setting enables unsupervised domain adaptation and learning from large amounts of real world data without ground-truth source waveforms. In particular, we significantly improve reverberant speech separation performance by incorporating reverberant mixtures, train a speech enhancement system from noisy mixtures, and improve universal sound separation by incorporating a large amount of in-the-wild data.

Accepted for spotlight presentation at NeurIPS 2020

References in corpus (9)

• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• YFCC100M: The New Data in Multimedia Research
• Domain Separation Networks
• LibriMix: An Open-Source Dataset for Generalizable Speech Separation
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Voice Separation with an Unknown Number of Multiple Speakers
• Improving Universal Sound Separation Using Sound Classification
• Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures using Spatial Information
• Building Corpora for Single-Channel Speech Separation Across Multiple Domains

Cited by in corpus (11)

• Elephants and Algorithms: A Review of the Current and Future Role of AI in Elephant Monitoring
• Efficient Personalized Speech Enhancement through Self-Supervised Learning
• The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track
• PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings
• Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data
• Continual self-training with bootstrapped remixing for speech enhancement
• Self-Supervised Learning from Contrastive Mixtures for Personalized Speech Enhancement
• Improving Sound Event Detection In Domestic Environments Using Sound Separation
• On monoaural speech enhancement for automatic recognition of real noisy speech using mixture invariant training
• Distortion-controlled Training for End-to-end Reverberant Speech Separation with Auxiliary Autoencoding Loss
• Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation