Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion

Abstract

Grapheme-to-phoneme (G2P) conversion is an important task in automatic speech recognition and text-to-speech systems. Recently, G2P conversion is viewed as a sequence to sequence task and modeled by RNN or CNN based encoder-decoder framework. However, previous works do not consider the practical issues when deploying G2P model in the production system, such as how to leverage additional unlabeled data to boost the accuracy, as well as reduce model size for online deployment. In this work, we propose token-level ensemble distillation for G2P conversion, which can (1) boost the accuracy by distilling the knowledge from additional unlabeled data, and (2) reduce the model size but maintain the high accuracy, both of which are very practical and helpful in the online production system. We use token-level knowledge distillation, which results in better accuracy than the sequence-level counterpart. What is more, we adopt the Transformer instead of RNN or CNN based models to further boost the accuracy of G2P conversion. Experiments on the publicly available CMUDict dataset and an internal English dataset demonstrate the effectiveness of our proposed method. Particularly, our method achieves 19.88% WER on CMUDict dataset, outperforming the previous works by more than 4.22% WER, and setting the new state-of-the-art results.

arXiv comment5 pages, accepted by interspeech 2019

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• FastSpeech: Fast, Robust and Controllable Text to Speech
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Achieving Human Parity on Automatic Chinese to English News Translation
• Born Again Neural Networks
• The Evolved Transformer
• Large scale distributed neural network training through online distillation
• Multilingual Neural Machine Translation with Knowledge Distillation
• Sequence-Level Knowledge Distillation
• Almost Unsupervised Text to Speech and Automatic Speech Recognition
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion
• Ensemble Distillation for Neural Machine Translation
• Attending to Mathematical Language with Transformers
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input

Cited by in corpus (7)

• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• A Survey on Neural Speech Synthesis
• AdaSpeech: Adaptive Text to Speech for Custom Voice
• HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis
• MultiSpeech: Multi-Speaker Text to Speech with Transformer
• Low-Rank Bottleneck in Multi-head Attention Models
• Emotional Prosody Control for Speech Generation