End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networks

Abstract

Speech enhancement model is used to map a noisy speech to a clean speech. In the training stage, an objective function is often adopted to optimize the model parameters. However, in most studies, there is an inconsistency between the model optimization criterion and the evaluation criterion on the enhanced speech. For example, in measuring speech intelligibility, most of the evaluation metric is based on a short-time objective intelligibility (STOI) measure, while the frame based minimum mean square error (MMSE) between estimated and clean speech is widely used in optimizing the model. Due to the inconsistency, there is no guarantee that the trained model can provide optimal performance in applications. In this study, we propose an end-to-end utterance-based speech enhancement framework using fully convolutional neural networks (FCN) to reduce the gap between the model optimization and evaluation criterion. Because of the utterance-based optimization, temporal correlation information of long speech segments, or even at the entire utterance level, can be considered when perception-based objective functions are used for the direct optimization. As an example, we implement the proposed FCN enhancement framework to optimize the STOI measure. Experimental results show that the STOI of test speech is better than conventional MMSE-optimized speech due to the consistency between the training and evaluation target. Moreover, by integrating the STOI in model optimization, the intelligibility of human subjects and automatic speech recognition (ASR) system on the enhanced speech is also substantially improved compared to those generated by the MMSE criterion.

arXiv commentAccepted in IEEE Transactions on Audio, Speech and Language Processing (TASLP)

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• WaveNet: A Generative Model for Raw Audio
• Joint Optimization of Masks and Deep Recurrent Neural Networks for Monaural Source Separation
• Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification
• Supervised Speech Separation Based on Deep Learning: An Overview
• Raw Waveform-based Speech Enhancement by Fully Convolutional Networks
• Monaural Speech Enhancement using Deep Neural Networks by Maximizing a Short-Time Objective Intelligibility Measure
• Single Channel Audio Source Separation using Convolutional Denoising Autoencoders
• Multi-Resolution Fully Convolutional Neural Networks for Monaural Audio Source Separation

Cited by in corpus (4)

• End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction
• Multi-Resolution Fully Convolutional Neural Networks for Monaural Audio Source Separation
• Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks
• Convolutional Neural Networks to Enhance Coded Speech