DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization

Abstract

Recent success in speech representation learning enables a new way to leverage unlabeled data to train speech recognition model. In speech representation learning, a large amount of unlabeled data is used in a self-supervised manner to learn a feature representation. Then a smaller amount of labeled data is used to train a downstream ASR system using the new feature representations. Based on our previous work DeCoAR and inspirations from other speech representation learning, we propose DeCoAR 2.0, a Deep Contextualized Acoustic Representation with vector quantization. We introduce several modifications over the DeCoAR: first, we use Transformers in encoding module instead of LSTMs; second, we introduce a vector quantization layer between encoder and reconstruction modules; third, we propose an objective that combines the reconstructive loss with vector quantization diversity loss to train speech representations. Our experiments show consistent improvements over other speech representations in different data-sparse scenarios. Without fine-tuning, a light-weight ASR model trained on 10 hours of LibriSpeech labeled data with DeCoAR 2.0 features outperforms the model trained on the full 960-hour dataset with filterbank features.

Submitted to ICASSP 2021

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Transformers with convolutional context for ASR
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation

Cited by in corpus (14)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• SUPERB: Speech processing Universal PERformance Benchmark
• Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling
• Text-Free Prosody-Aware Generative Spoken Language Modeling
• Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• How to Estimate Model Transferability of Pre-Trained Speech Models?
• A Brief Overview of Unsupervised Neural Speech Representation Learning
• Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning
• Probing Acoustic Representations for Phonetic Properties
• Phone and speaker spatial organization in self-supervised speech representations
• Auto-KWS 2021 Challenge: Task, Datasets, and Baselines