Scaling End-to-End Models for Large-Scale Multilingual ASR

Abstract

Building ASR models across many languages is a challenging multi-task learning problem due to large variations and heavily unbalanced data. Existing work has shown positive transfer from high resource to low resource languages. However, degradations on high resource languages are commonly observed due to interference from the heterogeneous multilingual data and reduction in per-language capacity. We conduct a capacity study on a 15-language task, with the amount of data per language varying from 7.6K to 53.5K hours. We adopt GShard [1] to efficiently scale up to 10B parameters. Empirically, we find that (1) scaling the number of model parameters is an effective way to solve the capacity bottleneck - our 500M-param model already outperforms monolingual baselines and scaling it to 1B and 10B brought further quality gains; (2) larger models are not only more data efficient, but also more efficient in terms of training cost as measured in TPU days - the 1B-param model reaches the same accuracy at 34% of training time as the 500M-param model; (3) given a fixed capacity budget, adding depth works better than width and large encoders do better than large decoders; (4) with continuous training, they can be adapted to new languages and domains.

arXiv commentASRU 2021

References in corpus (17)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Sequence Transduction with Recurrent Neural Networks
• Conformer: Convolution-augmented Transformer for Speech Recognition
• MLS: A Large-Scale Multilingual Dataset for Speech Research
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Listen, Attend and Spell
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languages
• On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
• Balancing Training for Multilingual Neural Machine Translation
• Knowledge Distillation for Multi-task Learning

Cited by in corpus (4)

• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• A Study of Multilingual End-to-End Speech Recognition for Kazakh, Russian, and English