Towards Listening to 10 People Simultaneously: An Efficient Permutation Invariant Training of Audio Source Separation Using Sinkhorn's Algorithm

Abstract

In neural network-based monaural speech separation techniques, it has been recently common to evaluate the loss using the permutation invariant training (PIT) loss. However, the ordinary PIT requires to try all $N!$ permutations between $N$ ground truths and $N$ estimates. Since the factorial complexity explodes very rapidly as $N$ increases, a PIT-based training works only when the number of source signals is small, such as $N=2$ or $3$. To overcome this limitation, this paper proposes a SinkPIT, a novel variant of the PIT losses, which is much more efficient than the ordinary PIT loss when $N$ is large. The SinkPIT is based on Sinkhorn's matrix balancing algorithm, which efficiently finds a doubly stochastic matrix which approximates the best permutation in a differentiable manner. The author conducted an experiment to train a neural network model to decompose a single-channel mixture into 10 sources using the SinkPIT, and obtained promising results.

arXiv comment5 pages, 8 figures, IEEE ICASSP 2021

References in corpus (7)

• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration
• LibriMix: An Open-Source Dataset for Generalizable Speech Separation
• Learning Latent Permutations with Gumbel-Sinkhorn Networks
• Voice Separation with an Unknown Number of Multiple Speakers
• Asteroid: the PyTorch-based audio source separation toolkit for researchers
• JVS corpus: free Japanese multi-speaker voice corpus