Transferring Knowledge from a RNN to a DNN

Abstract

Deep Neural Network (DNN) acoustic models have yielded many state-of-the-art results in Automatic Speech Recognition (ASR) tasks. More recently, Recurrent Neural Network (RNN) models have been shown to outperform DNNs counterparts. However, state-of-the-art DNN and RNN models tend to be impractical to deploy on embedded systems with limited computational capacity. Traditionally, the approach for embedded platforms is to either train a small DNN directly, or to train a small DNN that learns the output distribution of a large DNN. In this paper, we utilize a state-of-the-art RNN to transfer knowledge to small DNN. We use the RNN model to generate soft alignments and minimize the Kullback-Leibler divergence against the small DNN. The small DNN trained on the soft RNN alignments achieved a 3.93 WER on the Wall Street Journal (WSJ) eval92 task compared to a baseline 4.54 WER or more than 13% relative improvement.

References in corpus (3)

• Distilling the Knowledge in a Neural Network
• Show and Tell: A Neural Image Caption Generator
• Deep Recurrent Neural Networks for Acoustic Modelling

Cited by in corpus (18)

• Knowledge Distillation: A Survey
• Recurrent Neural Network Training with Dark Knowledge Transfer
• Sequence-Level Knowledge Distillation
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Capitalization and Punctuation Restoration: a Survey
• Transfer Learning for Speech and Language Processing
• Distilling Word Embeddings: An Encoding Approach
• Knowledge Transfer Pre-training
• Diving deeper into mentee networks
• Learning Deep Representations with Probabilistic Knowledge Transfer
• Improved training for online end-to-end speech recognition systems
• Generative Knowledge Transfer for Neural Language Models
• Blending LSTMs into CNNs
• Low-rank and Sparse Soft Targets to Learn Better DNN Acoustic Models
• Developing Far-Field Speaker System Via Teacher-Student Learning
• Distilling Knowledge Using Parallel Data for Far-field Speech Recognition
• Recent Progresses in Deep Learning based Acoustic Models (Updated)
• The Pupil Has Become the Master: Teacher-Student Model-Based Word Embedding Distillation with Ensemble Learning