Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition

Abstract

The smaller memory bandwidth in smart devices prompts development of smaller Automatic Speech Recognition (ASR) models. To obtain a smaller model, one can employ the model compression techniques. Knowledge distillation (KD) is a popular model compression approach that has shown to achieve smaller model size with relatively lesser degradation in the model performance. In this approach, knowledge is distilled from a trained large size teacher model to a smaller size student model. Also, the transducer based models have recently shown to perform well for on-device streaming ASR task, while the conformer models are efficient in handling long term dependencies. Hence in this work we employ a streaming transducer architecture with conformer as the encoder. We propose a multi-stage progressive approach to compress the conformer transducer model using KD. We progressively update our teacher model with the distilled student model in a multi-stage setup. On standard LibriSpeech dataset, our experimental results have successfully achieved compression rates greater than 60% without significant degradation in the performance compared to the larger teacher model.

arXiv commentPublished in INTERSPEECH 2022

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Knowledge Distillation: A Survey
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling