Attention is All You Need in Speech Separation

Abstract

Recurrent Neural Networks (RNNs) have long been the dominant architecture in sequence-to-sequence learning. RNNs, however, are inherently sequential models that do not allow parallelization of their computations. Transformers are emerging as a natural alternative to standard RNNs, replacing recurrent computations with a multi-head attention mechanism. In this paper, we propose the SepFormer, a novel RNN-free Transformer-based neural network for speech separation. The SepFormer learns short and long-term dependencies with a multi-scale approach that employs transformers. The proposed model achieves state-of-the-art (SOTA) performance on the standard WSJ0-2/3mix datasets. It reaches an SI-SNRi of 22.3 dB on WSJ0-2mix and an SI-SNRi of 19.5 dB on WSJ0-3mix. The SepFormer inherits the parallelization advantages of Transformers and achieves a competitive performance even when downsampling the encoded representation by a factor of 8. It is thus significantly faster and it is less memory-demanding than the latest speech separation systems with comparable performance.

arXiv commentAccepted to ICASSP 2021

References in corpus (9)

• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• A Comparative Study on Transformer vs RNN in Speech Applications
• Light Gated Recurrent Units for Speech Recognition
• Voice Separation with an Unknown Number of Multiple Speakers
• Two-Step Sound Source Separation: Training on Learned Latent Targets
• TasNet: time-domain audio separation network for real-time, single-channel speech separation
• Discriminative Neural Clustering for Speaker Diarisation
• Untangling tradeoffs between recurrence and self-attention in neural networks
• FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks

Cited by in corpus (4)

• Phase-Aware Deep Speech Enhancement: It's All About The Frame Length
• Stepwise-Refining Speech Separation Network via Fine-Grained Encoding in High-order Latent Domain
• TransMask: A Compact and Fast Speech Separation Model Based on Transformer
• MIMO Self-attentive RNN Beamformer for Multi-speaker Speech Separation