Recurrent Neural Network Training with Dark Knowledge Transfer

Abstract

Recurrent neural networks (RNNs), particularly long short-term memory (LSTM), have gained much attention in automatic speech recognition (ASR). Although some successful stories have been reported, training RNNs remains highly challenging, especially with limited training data. Recent research found that a well-trained model can be used as a teacher to train other child models, by using the predictions generated by the teacher model as supervision. This knowledge transfer learning has been employed to train simple neural nets with a complex one, so that the final performance can reach a level that is infeasible to obtain by regular training. In this paper, we employ the knowledge transfer learning approach to train RNNs (precisely LSTM) using a deep neural network (DNN) model as the teacher. This is different from most of the existing research on knowledge transfer learning, since the teacher (DNN) is assumed to be weaker than the child (RNN); however, our experiments on an ASR task showed that it works fairly well: without applying any tricks on the learning scheme, this approach can train RNNs successfully even with limited training data.

ICASSP 2016

References in corpus (5)

• Deep Learning in Neural Networks: An Overview
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Transferring Knowledge from a RNN to a DNN

Cited by in corpus (24)

• Compressing Recurrent Neural Network with Tensor Train
• Conditional Teacher-Student Learning
• Transfer Learning for Speech and Language Processing
• Bolt: Accelerated Data Mining with Fast Vector Compression
• Multi-Label Knowledge Distillation
• Copycat CNN: Are Random Non-Labeled Data Enough to Steal Knowledge from Black-box Models?
• Why distillation helps: a statistical perspective
• Distilling Word Embeddings: An Encoding Approach
• A Model Compression Method with Matrix Product Operators for Speech Enhancement
• Tracking-by-Trackers with a Distilled and Reinforced Model
• Learning Deep Representations with Probabilistic Knowledge Transfer
• Generative Knowledge Transfer for Neural Language Models
• Blending LSTMs into CNNs
• Tensor Decomposition for Compressing Recurrent Neural Network
• Training Recurrent Neural Networks via Dynamical Trajectory-Based Optimization
• Knowledge distillation for optimization of quantized deep neural networks
• Relation Clustering in Narrative Knowledge Graphs
• DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs
• Spectral Pruning for Recurrent Neural Networks
• LabelEnc: A New Intermediate Supervision Method for Object Detection
• The Pupil Has Become the Master: Teacher-Student Model-Based Word Embedding Distillation with Ensemble Learning
• Neuroevolutionary Transfer Learning of Deep Recurrent Neural Networks through Network-Aware Adaptation
• Filter Grafting for Deep Neural Networks: Reason, Method, and Cultivation
• LoANs: Weakly Supervised Object Detection with Localizer Assessor Networks