End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures

Abstract

We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard LibriSpeech dataset, and leverage additional unlabeled data from LibriVox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.

arXiv commentPublished at the workshop on Self-supervision in Audio and Speech (SAS) at the 37th International Conference on Machine Learning (ICML 2020), Vienna, Austria

References in corpus (21)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• A Comparative Study on Transformer vs RNN in Speech Applications
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• The Microsoft 2016 Conversational Speech Recognition System
• Reducing Transformer Depth on Demand with Structured Dropout
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Self-Training for End-to-End Speech Recognition
• wav2letter++: The Fastest Open-source Speech Recognition System
• Transformers with convolutional context for ASR
• Fully Convolutional Speech Recognition
• The CAPIO 2017 Conversational Speech Recognition System
• Efficient Neural Audio Synthesis
• Residual Convolutional CTC Networks for Automatic Speech Recognition
• Towards better decoding and language model integration in sequence to sequence models
• Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
• Who Needs Words? Lexicon-Free Speech Recognition
• A Fully Differentiable Beam Search Decoder

Cited by in corpus (46)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• MLS: A Large-Scale Multilingual Dataset for Speech Research
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
• Unsupervised Automatic Speech Recognition: A Review
• TrTr: Visual Tracking with Transformer
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking
• Self-training Improves Pre-training for Natural Language Understanding
• CoVoST 2 and Massively Multilingual Speech-to-Text Translation
• Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognition
• LEAF: A Learnable Frontend for Audio Classification
• Iterative Pseudo-Labeling for Speech Recognition
• PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers
• Semantic Mask for Transformer based End-to-End Speech Recognition
• Unsupervised Speech Recognition
• Towards a Competitive End-to-End Speech Recognition for CHiME-6 Dinner Party Transcription
• Associating Objects with Transformers for Video Object Segmentation
• Transformer Lesion Tracker
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Learning Robust and Multilingual Speech Representations
• Self-training and Pre-training are Complementary for Speech Recognition
• Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
• Attention based end to end Speech Recognition for Voice Search in Hindi and English
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• Self-Training for End-to-End Speech Translation
• Boosting Active Learning for Speech Recognition with Noisy Pseudo-labeled Samples
• Semi-Supervised Learning with Data Augmentation for End-to-End ASR
• ASAPP-ASR: Multistream CNN and Self-Attentive SRU for SOTA Speech Recognition
• Explaining Natural Language Processing Classifiers with Occlusion and Language Modeling
• Low Resource German ASR with Untranscribed Data Spoken by Non-native Children -- INTERSPEECH 2021 Shared Task SPAPL System
• Online Visual Tracking with One-Shot Context-Aware Domain Adaptation
• Faster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces
• KoSpeech: Open-Source Toolkit for End-to-End Korean Speech Recognition
• Contrastive Semi-supervised Learning for ASR
• CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription
• Label-Synchronous Speech-to-Text Alignment for ASR Using Forward and Backward Transformers
• Effective Decoder Masking for Transformer Based End-to-End Speech Recognition
• Population Based Training for Data Augmentation and Regularization in Speech Recognition
• A Parallelizable Lattice Rescoring Strategy with Neural Language Models
• Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning
• Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems