Speaker-Invariant Training via Adversarial Learning

Abstract

We propose a novel adversarial multi-task learning scheme, aiming at actively curtailing the inter-talker feature variability while maximizing its senone discriminability so as to enhance the performance of a deep neural network (DNN) based ASR system. We call the scheme speaker-invariant training (SIT). In SIT, a DNN acoustic model and a speaker classifier network are jointly optimized to minimize the senone (tied triphone state) classification loss, and simultaneously mini-maximize the speaker classification loss. A speaker-invariant and senone-discriminative deep feature is learned through this adversarial multi-task learning. With SIT, a canonical DNN acoustic model with significantly reduced variance in its output probabilities is learned with no explicit speaker-independent (SI) transformations or speaker-specific representations used in training or testing. Evaluated on the CHiME-3 dataset, the SIT achieves 4.99% relative word error rate (WER) improvement over the conventional SI acoustic model. With additional unsupervised speaker adaptation, the speaker-adapted (SA) SIT model achieves 4.86% relative WER gain over the SA SI acoustic model.

arXiv comment5 pages, 3 figures, ICASSP 2018

References in corpus (8)

• Unsupervised Domain Adaptation by Backpropagation
• Domain Separation Networks
• Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation
• Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition
• Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation
• Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
• Invariant Representations for Noisy Speech Recognition
• Maximum a Posteriori Adaptation of Network Parameters in Deep Models

Cited by in corpus (32)

• Conditional Teacher-Student Learning
• Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Adversarial Speaker Verification
• Cycle-Consistent Speech Enhancement
• Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion?
• Adversarial Feature-Mapping for Speech Enhancement
• Speaker Adaptation for Attention-Based End-to-End Speech Recognition
• Adversarial Speaker Adaptation
• Unsupervised Representation Disentanglement using Cross Domain Features and Adversarial Learning in Variational Autoencoder based Voice Conversion
• Bayesian Learning for Deep Neural Network Adaptation
• Combining Adversarial Training and Disentangled Speech Representation for Robust Zero-Resource Subword Modeling
• Towards adversarial learning of speaker-invariant representation for speech emotion recognition
• Noise Adaptive Speech Enhancement using Domain Adversarial Training
• Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning
• Unsupervised Domain Adaptation by Adversarial Learning for Robust Speech Recognition
• Attentive Adversarial Learning for Domain-Invariant Training
• Adversarial Representation Learning for Robust Privacy Preservation in Audio
• Adversarial Training for Multilingual Acoustic Modeling
• Supervised Speech Representation Learning for Parkinson's Disease Classification
• Enhancing and Adversarial: Improve ASR with Speaker Labels
• Pre-Finetuning for Few-Shot Emotional Speech Recognition
• Speaker De-identification System using Autoencoders and Adversarial Training
• Recent Progresses in Deep Learning based Acoustic Models (Updated)
• Active Voice Authentication
• Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition
• L-Vector: Neural Label Embedding for Domain Adaptation
• Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition
• Senone-aware Adversarial Multi-task Training for Unsupervised Child to Adult Speech Adaptation
• Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition
• Human-Imitating Metrics for Training and Evaluating Privacy Preserving Emotion Recognition Models Using Sociolinguistic Knowledge
• Exploiting Cross-Lingual Knowledge in Unsupervised Acoustic Modeling for Low-Resource Languages