Conditional Teacher-Student Learning

Abstract

The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student model towards a suboptimal performance. To overcome this problem, we propose a conditional T/S learning scheme, in which a "smart" student model selectively chooses to learn from either the teacher model or the ground truth labels conditioned on whether the teacher can correctly predict the ground truth. Unlike a naive linear combination of the two knowledge sources, the conditional learning is exclusively engaged with the teacher model when the teacher model's prediction is correct, and otherwise backs off to the ground truth. Thus, the student model is able to learn effectively from the teacher and even potentially surpass the teacher. We examine the proposed learning scheme on two tasks: domain adaptation on CHiME-3 dataset and speaker adaptation on Microsoft short message dictation dataset. The proposed method achieves 9.8% and 12.8% relative word error rate reductions, respectively, over T/S learning for environment adaptation and speaker-independent model for speaker adaptation.

5 pages, 1 figure, ICASSP 2019

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation
• Speaker-Invariant Training via Adversarial Learning
• Recurrent Neural Network Training with Dark Knowledge Transfer
• Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition
• Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation
• Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
• Cycle-Consistent Speech Enhancement
• Adversarial Feature-Mapping for Speech Enhancement
• Adversarial Speaker Adaptation
• Attentive Adversarial Learning for Domain-Invariant Training
• Developing Far-Field Speaker System Via Teacher-Student Learning

Cited by in corpus (16)

• Knowledge Distillation: A Survey
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Speaker Adaptation for Attention-Based End-to-End Speech Recognition
• Adversarial Speaker Adaptation
• Bridging the Gap Between Patient-specific and Patient-independent Seizure Prediction via Knowledge Distillation
• Convolutional Neural Networks for Speech Controlled Prosthetic Hands
• Attentive Adversarial Learning for Domain-Invariant Training
• Balanced Knowledge Distillation for Long-tailed Learning
• Knowledge Distillation on Spatial-Temporal Graph Convolutional Network for Traffic Prediction
• A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation
• ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression
• Domain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition
• L-Vector: Neural Label Embedding for Domain Adaptation
• Confidence Conditioned Knowledge Distillation
• Sequence-level self-learning with multiple hypotheses
• Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition