On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition

Abstract

In conventional speech recognition, phoneme-based models outperform grapheme-based models for non-phonetic languages such as English. The performance gap between the two typically reduces as the amount of training data is increased. In this work, we examine the impact of the choice of modeling unit for attention-based encoder-decoder models. We conduct experiments on the LibriSpeech 100hr, 460hr, and 960hr tasks, using various target units (phoneme, grapheme, and word-piece); across all tasks, we find that grapheme or word-piece models consistently outperform phoneme-based models, even though they are evaluated without a lexicon or an external language model. We also investigate model complementarity: we find that we can improve WERs by up to 9% relative by rescoring N-best lists generated from a strong word-piece based baseline with either the phoneme or the grapheme model. Rescoring an N-best list generated by the phonemic system, however, provides limited improvements. Further analysis shows that the word-piece-based models produce more diverse N-best hypotheses, and thus lower oracle WERs, than phonemic models.

arXiv commentTo appear in the proceedings of INTERSPEECH 2019

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• Improved training of end-to-end attention models for speech recognition
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• The CAPIO 2017 Conversational Speech Recognition System
• Exploring Neural Transducers for End-to-End Speech Recognition
• A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese

Cited by in corpus (13)

• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Language Modeling with Deep Transformers
• Effectiveness of self-supervised pre-training for speech recognition
• Direct speech-to-speech translation with a sequence-to-sequence model
• Binary Black-box Evasion Attacks Against Deep Learning-based Static Malware Detectors with Adversarial Byte-Level Language Model
• Multi-head Monotonic Chunkwise Attention For Online Speech Recognition
• Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
• Bimodal Speech Emotion Recognition Using Pre-Trained Language Models
• Phoneme-Based Contextualization for Cross-Lingual Speech Recognition in End-to-End Models
• Improving RNN transducer with normalized jointer network
• Phoneme Based Neural Transducer for Large Vocabulary Speech Recognition
• Quantifying and Maximizing the Benefits of Back-End Noise Adaption on Attention-Based Speech Recognition Models
• Exploring Lexicon-Free Modeling Units for End-to-End Korean and Korean-English Code-Switching Speech Recognition