Large-Scale Self- and Semi-Supervised Learning for Speech Translation
arXiv:2104.06678
Abstract
In this paper, we improve speech translation (ST) through effectively leveraging large quantities of unlabeled speech and text data in different and complementary ways. We explore both pretraining and self-training by using the large Libri-Light speech audio corpus and language modeling with CommonCrawl. Our experiments improve over the previous state of the art by 2.6 BLEU on average on all four considered CoVoST 2 language pairs via a simple recipe of combining wav2vec 2.0 pretraining, a single iteration of self-training and decoding with a language model. Different to existing work, our approach does not leverage any other supervision than ST data. Code and models will be publicly released.
References in corpus (14)
- Sequence to Sequence Learning with Neural Networks
- A Simple Framework for Contrastive Learning of Visual Representations
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Attention-Based Models for Speech Recognition
- On Using Monolingual Corpora in Neural Machine Translation
- Libri-Light: A Benchmark for ASR with Limited or No Supervision
- Improved Noisy Student Training for Automatic Speech Recognition
- Self-Training for End-to-End Speech Recognition
- Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
- End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
- Self-training Improves Pre-training for Natural Language Understanding
- CoVoST 2 and Massively Multilingual Speech-to-Text Translation
- Iterative Pseudo-Labeling for Speech Recognition
- Multilingual Speech Translation with Efficient Finetuning of Pretrained Models