Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition

Abstract

We propose a novel approach to semi-supervised automatic speech recognition (ASR). We first exploit a large amount of unlabeled audio data via representation learning, where we reconstruct a temporal slice of filterbank features from past and future context frames. The resulting deep contextualized acoustic representations (DeCoAR) are then used to train a CTC-based end-to-end ASR system using a smaller amount of labeled audio data. In our experiments, we show that systems trained on DeCoAR consistently outperform ones trained on conventional filterbank features, giving 42% and 19% relative improvement over the baseline on WSJ eval92 and LibriSpeech test-clean, respectively. Our approach can drastically reduce the amount of labeled data required; unsupervised training on LibriSpeech then supervision with 100 hours of labeled data achieves performance on par with training on all 960 hours directly. Pre-trained models and code will be released online.

Accepted to ICASSP 2020 (oral)

References in corpus (5)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Unsupervised speech representation learning using WaveNet autoencoders
• RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentation

Cited by in corpus (41)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Self-Supervised Speech Representation Learning: A Review
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Improved Noisy Student Training for Automatic Speech Recognition
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition
• Towards Better Domain Adaptation for Self-supervised Models: A Case Study of Child ASR
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
• BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition
• Iterative Pseudo-Labeling for Speech Recognition
• Augmented Datasheets for Speech Datasets and Ethical Decision-Making
• A Comparative Study of Self-supervised Speech Representation Based Voice Conversion
• DECAR: Deep Clustering for learning general-purpose Audio Representations
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
• MusiCoder: A Universal Music-Acoustic Encoder Based on Transformers
• Speech Separation with Pretrained Frontend to Minimize Domain Mismatch
• Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks
• UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
• General-Purpose Speech Representation Learning through a Self-Supervised Multi-Granularity Framework
• SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks
• ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale
• Masked Pre-trained Encoder base on Joint CTC-Transformer
• Vector-Quantized Autoregressive Predictive Coding
• A Brief Overview of Unsupervised Neural Speech Representation Learning
• Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
• Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutch
• Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning
• Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization
• Semi-Supervised Learning with Data Augmentation for End-to-End ASR
• Probing Acoustic Representations for Phonetic Properties
• Contrastive Semi-supervised Learning for ASR
• Similarity Analysis of Self-Supervised Speech Representations
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding
• Guided contrastive self-supervised pre-training for automatic speech recognition
• Fusion of Embeddings Networks for Robust Combination of Text Dependent and Independent Speaker Recognition