Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data

Abstract

We propose FEDENHANCE, an unsupervised federated learning (FL) approach for speech enhancement and separation with non-IID distributed data across multiple clients. We simulate a real-world scenario where each client only has access to a few noisy recordings from a limited and disjoint number of speakers (hence non-IID). Each client trains their model in isolation using mixture invariant training while periodically providing updates to a central server. Our experiments show that our approach achieves competitive enhancement performance compared to IID training on a single device and that we can further facilitate the convergence speed and the overall performance using transfer learning on the server-side. Moreover, we show that we can effectively combine updates from clients trained locally with supervised and unsupervised losses. We also release a new dataset LibriFSD50K and its creation recipe in order to facilitate FL research for source separation problems.

Accepted to WASPAA 21

References in corpus (6)

• Sudo rm -rf: Efficient Networks for Universal Audio Source Separation
• Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework
• D3Net: Densely connected multidilated DenseNet for music source separation
• Compute and memory efficient universal sound source separation
• Speech enhancement with weakly labelled data from AudioSet
• Personalized Speech Enhancement through Self-Supervised Data Augmentation and Purification

Cited by in corpus (3)

• Continual self-training with bootstrapped remixing for speech enhancement
• Leveraging Low-Distortion Target Estimates for Improved Speech Enhancement
• RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing