Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks
arXiv:2101.05014
Abstract
Recent research on the time-domain audio separation networks (TasNets) has brought great success to speech separation. Nevertheless, conventional TasNets struggle to satisfy the memory and latency constraints in industrial applications. In this regard, we design a low-cost high-performance architecture, namely, globally attentive locally recurrent (GALR) network. Alike the dual-path RNN (DPRNN), we first split a feature sequence into 2D segments and then process the sequence along both the intra- and inter-segment dimensions. Our main innovation lies in that, on top of features recurrently processed along the inter-segment dimensions, GALR applies a self-attention mechanism to the sequence along the inter-segment dimension, which aggregates context-aware information and also enables parallelization. Our experiments suggest that GALR is a notably more effective network than the prior work. On one hand, with only 1.5M parameters, it has achieved comparable separation performance at a much lower cost with 36.1% less runtime memory and 49.4% fewer computational operations, relative to the DPRNN. On the other hand, in a comparable model size with DPRNN, GALR has consistently outperformed DPRNN in three datasets, in particular, with a substantial margin of 2.4dB absolute improvement of SI-SNRi in the benchmark WSJ0-2mix task.
Accepted in IEEE SLT 2021
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
- Light Gated Recurrent Units for Speech Recognition
- R-Transformer: Recurrent Neural Network Enhanced Transformer
- Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
- Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments
- Experiments on Parallel Training of Deep Neural Network using Model Averaging
- Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation
- Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation
- End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction
- FaSNet: Low-latency Adaptive Beamforming for Multi-microphone Audio Processing
- Audio-visual Recognition of Overlapped speech for the LRS2 dataset
- Improved Speech Separation with Time-and-Frequency Cross-domain Joint Embedding and Clustering