End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction

Abstract

This paper proposes an end-to-end approach for single-channel speaker-independent multi-speaker speech separation, where time-frequency (T-F) masking, the short-time Fourier transform (STFT), and its inverse are represented as layers within a deep network. Previous approaches, rather than computing a loss on the reconstructed signal, used a surrogate loss based on the target STFT magnitudes. This ignores reconstruction error introduced by phase inconsistency. In our approach, the loss function is directly defined on the reconstructed signals, which are optimized for best separation. In addition, we train through unfolded iterations of a phase reconstruction algorithm, represented as a series of STFT and inverse STFT layers. While mask values are typically limited to lie between zero and one for approaches using the mixture phase for reconstruction, this limitation is less relevant if the estimated magnitudes are to be used together with phase reconstruction. We thus propose several novel activation functions for the output layer of the T-F masking, to allow mask values beyond one. On the publicly-available wsj0-2mix dataset, our approach achieves state-of-the-art 12.6 dB scale-invariant signal-to-distortion ratio (SI-SDR) and 13.1 dB SDR, revealing new possibilities for deep learning based phase reconstruction and representing a fundamental progress towards solving the notoriously-hard cocktail party problem.

Submitted to Interspeech 2018

References in corpus (6)

• Deep attractor network for single-microphone speaker separation
• Speaker-independent Speech Separation with Deep Attractor Network
• Deep Clustering and Conventional Networks for Music Separation: Stronger Together
• Supervised Speech Separation Based on Deep Learning: An Overview
• Raw Waveform-based Speech Enhancement by Fully Convolutional Networks
• End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networks

Cited by in corpus (15)

• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Music Source Separation in the Waveform Domain
• Learning to Optimize: A Primer and A Benchmark
• End-to-End Multi-Channel Speech Separation
• Learning with Learned Loss Function: Speech Enhancement with Quality-Net to Improve Perceptual Evaluation of Speech Quality
• Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation
• SpEx+: A Complete Time Domain Speaker Extraction Network
• Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks
• End-to-end Networks for Supervised Single-channel Speech Separation
• Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective
• A Unified Framework for Speech Separation
• Improved Speech Separation with Time-and-Frequency Cross-domain Joint Embedding and Clustering
• Mixup-breakdown: a consistency training method for improving generalization of speech separation models
• CLC: Complex Linear Coding for the DNS 2020 Challenge
• Mask-dependent Phase Estimation for Monaural Speaker Separation