UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset

Abstract

Recently, there has been a vast interest in self-supervised learning (SSL) where the model is pre-trained on large scale unlabeled data and then fine-tuned on a small labeled dataset. The common wisdom is that SSL helps resource-limited tasks in which only a limited amount of labeled data is available. The benefit of SSL keeps diminishing when the labeled training data amount increases. To our best knowledge, at most a few thousand hours of labeled data was used in the study of SSL. In contrast, the industry usually uses tens of thousands of hours of labeled data to build high-accuracy speech recognition (ASR) systems for resource-rich languages. In this study, we take the challenge to investigate whether and how SSL can improve the ASR accuracy of a state-of-the-art production-scale Transformer-Transducer model, which was built with 65 thousand hours of anonymized labeled EN-US data.

References in corpus (10)

• Sequence Transduction with Recurrent Neural Networks
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• On Layer Normalization in the Transformer Architecture
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
• Transformer-Transducer: End-to-End Speech Recognition with Self-Attention
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
• BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition
• Unsupervised Speech Recognition

Cited by in corpus (2)

• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Guided contrastive self-supervised pre-training for automatic speech recognition