Effectiveness of self-supervised pre-training for speech recognition

Abstract

We compare self-supervised representation learning algorithms which either explicitly quantize the audio data or learn representations without quantization. We find the former to be more accurate since it builds a good vocabulary of the data through vq-wav2vec [1] to enable learning of effective representations in subsequent BERT training. Different to previous work, we directly fine-tune the pre-trained BERT models on transcribed speech using a Connectionist Temporal Classification (CTC) loss instead of feeding the representations into a task-specific model. We also propose a BERT-style model learning directly from the continuous audio data and compare pre-training on raw audio to spectral features. Fine-tuning a BERT model on 10 hour of labeled Librispeech data with a vq-wav2vec vocabulary is almost as good as the best known reported system trained on 100 hours of labeled data on testclean, while achieving a 25% WER reduction on test-other. When using only 10 minutes of labeled data, WER is 25.2 on test-other and 16.3 on test-clean. This demonstrates that self-supervision can enable speech recognition systems trained on a near-zero amount of transcribed data.

References in corpus (15)

• Distributed Representations of Words and Phrases and their Compositionality
• Representation Learning with Contrastive Predictive Coding
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Neural Discrete Representation Learning
• Learning Representations by Maximizing Mutual Information Across Views
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Self-Training for End-to-End Speech Recognition
• wav2letter++: The Fastest Open-source Speech Recognition System
• Transformers with convolutional context for ASR
• Representations of language in a model of visually grounded speech signal
• On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
• Multi-view Recurrent Neural Acoustic Word Embeddings

Cited by in corpus (29)

• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Unsupervised Automatic Speech Recognition: A Review
• TUNet: A Block-online Bandwidth Extension Model based on Transformers and Self-supervised Pretraining
• Towards Better Domain Adaptation for Self-supervised Models: A Case Study of Child ASR
• Iterative Pseudo-Labeling for Speech Recognition
• Codified audio language modeling learns useful representations for music information retrieval
• Unsupervised Speech Recognition
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
• Self-training and Pre-training are Complementary for Speech Recognition
• LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Indonesian Automatic Speech Recognition with XLSR-53
• Towards Semi-Supervised Semantics Understanding from Speech
• Vector-Quantized Autoregressive Predictive Coding
• A Brief Overview of Unsupervised Neural Speech Representation Learning
• OkwuGbé: End-to-End Speech Recognition for Fon and Igbo
• Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
• Understanding Self-Attention of Self-Supervised Audio Transformers
• Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutch
• Contrastive Semi-supervised Learning for ASR
• LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR
• ASMDD: Arabic Speech Mispronunciation Detection Dataset
• Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers