Cycle-Consistent Speech Enhancement

Abstract

Feature mapping using deep neural networks is an effective approach for single-channel speech enhancement. Noisy features are transformed to the enhanced ones through a mapping network and the mean square errors between the enhanced and clean features are minimized. In this paper, we propose a cycle-consistent speech enhancement (CSE) in which an additional inverse mapping network is introduced to reconstruct the noisy features from the enhanced ones. A cycle-consistent constraint is enforced to minimize the reconstruction loss. Similarly, a backward cycle of mappings is performed in the opposite direction with the same networks and losses. With cycle-consistency, the speech structure is well preserved in the enhanced features while noise is effectively reduced such that the feature-mapping network generalizes better to unseen data. In cases where only unparalleled noisy and clean data is available for training, two discriminator networks are used to distinguish the enhanced and noised features from the clean and noisy ones. The discrimination losses are jointly optimized with reconstruction losses through adversarial multi-task learning. Evaluated on the CHiME-3 dataset, the proposed CSE achieves 19.60% and 6.69% relative word error rate improvements respectively when using or without using parallel clean and noisy speech data.

arXiv comment5 pages, 2 figures. Interspeech 2018. arXiv admin note: text overlap with arXiv:1809.02251

References in corpus (10)

• Unsupervised Domain Adaptation by Backpropagation
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Wasserstein GAN
• Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks
• Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
• Speaker-Invariant Training via Adversarial Learning
• Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition
• Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation
• Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
• Adversarial Feature-Mapping for Speech Enhancement

Cited by in corpus (18)

• Conditional Teacher-Student Learning
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Adversarial Speaker Verification
• Adversarial Feature-Mapping for Speech Enhancement
• Speaker Adaptation for Attention-Based End-to-End Speech Recognition
• Adversarial Speaker Adaptation
• An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning
• CycleGAN-based Non-parallel Speech Enhancement with an Adaptive Attention-in-attention Mechanism
• Noise Adaptive Speech Enhancement using Domain Adversarial Training
• Attentive Adversarial Learning for Domain-Invariant Training
• On monoaural speech enhancement for automatic recognition of real noisy speech using mixture invariant training
• A Two-stage Complex Network using Cycle-consistent Generative Adversarial Networks for Speech Enhancement
• Deep Noise Suppression With Non-Intrusive PESQNet Supervision Enabling the Use of Real Training Data
• Single Channel Far Field Feature Enhancement For Speaker Verification In The Wild
• Speech Enhancement Based on Cyclegan with Noise-informed Training
• L-Vector: Neural Label Embedding for Domain Adaptation
• Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition
• Reinforcement Learning Based Speech Enhancement for Robust Speech Recognition