Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models
arXiv:2106.06126
Abstract
We present results from Alexa speech teams on semi-supervised learning (SSL) of acoustic models (AM) with experiments spanning over 3000 hours of GPU time, making our study one of the largest of its kind. We discuss SSL for AMs in a small footprint setting, showing that a smaller capacity model trained with 1 million hours of unsupervised data can outperform a baseline supervised system by 14.3% word error rate reduction (WERR). When increasing the supervised data to seven-fold, our gains diminish to 7.1% WERR; to improve SSL efficiency at larger supervised data regimes, we employ a step-wise distillation into a smaller model, obtaining a WERR of 14.4%. We then switch to SSL using larger student models in low data regimes; while learning efficiency with unsupervised data is higher, student models may outperform teacher models in such a setting. We develop a theoretical sketch to explain this behavior.
arXiv commentTSD2021
References in corpus (10)
- Distilling the Knowledge in a Neural Network
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- On Calibration of Modern Neural Networks
- Unsupervised Data Augmentation for Consistency Training
- Self-Training for End-to-End Speech Recognition
- Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
- Streaming End-to-end Speech Recognition For Mobile Devices
- Semi-supervised ASR by End-to-end Self-training
- Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation
- Semi-Supervised Learning with Data Augmentation for End-to-End ASR