Self-Supervised Speech Representation Learning: A Review

Abstract

Although supervised deep learning has revolutionized speech and audio processing, it has necessitated the building of specialist models for individual tasks and application scenarios. It is likewise difficult to apply this to dialects and languages for which only limited labeled data is available. Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Such methods have shown success in natural language processing and computer vision domains, achieving new levels of performance while reducing the number of labels required for many downstream scenarios. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. Other approaches rely on multi-modal data for pre-training, mixing text or visual data streams with speech. Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years. This review presents approaches for self-supervised speech representation learning and their connection to other research areas. Since many current methods focus solely on automatic speech recognition as a downstream task, we review recent efforts on benchmarking learned representations to extend the application beyond speech recognition.

References in corpus (47)

• Representation Learning with Contrastive Predictive Coding
• On the Opportunities and Risks of Foundation Models
• VoxCeleb: a large-scale speaker identification dataset
• Improved Baselines with Momentum Contrastive Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Pre-trained Models for Natural Language Processing: A Survey
• Self-supervised Learning: Generative or Contrastive
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
• On Using Monolingual Corpora in Neural Machine Translation
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge
• MLS: A Large-Scale Multilingual Dataset for Speech Research
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• Improved Noisy Student Training for Automatic Speech Recognition
• GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio
• Unsupervised speech representation learning using WaveNet autoencoders
• Dynamical Variational Autoencoders: A Comprehensive Review
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Representations of language in a model of visually grounded speech signal
• Towards Learning a Universal Non-Semantic Representation of Speech
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• Deep Variational Canonical Correlation Analysis
• A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding
• Unified Hypersphere Embedding for Speaker Recognition
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Language learning using Speech to Image retrieval
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
• The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling
• Training Neural Speech Recognition Systems with Synthetic Speech Augmentation
• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation
• A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition
• BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition
• Self-supervised audio representation learning for mobile devices
• DiscreTalk: Text-to-Speech as a Machine Translation Problem
• Are discrete units necessary for Spoken Language Modeling?
• Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem
• Learning audio representations via phase prediction
• Masked Pre-trained Encoder base on Joint CTC-Transformer

Cited by in corpus (21)

• Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
• VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
• SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model
• Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
• BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models
• Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation
• Distribution-based Emotion Recognition in Conversation
• SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks
• Generic Speech Enhancement with Self-Supervised Representation Space Loss
• Experimenting with Additive Margins for Contrastive Self-Supervised Speaker Verification
• Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalization
• Convexity-based Pruning of Speech Representation Models
• UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models
• Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition
• Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget
• AC-Mix: Self-Supervised Adaptation for Low-Resource Automatic Speech Recognition using Agnostic Contrastive Mixup
• Navigating human-robot interaction challenges in private and residential aged care homes
• Cross-Corpora Spoken Language Identification with Domain Diversification and Generalization
• Advancing automatic speech recognition using feature fusion with self-supervised learning features: A case study on Fearless Steps Apollo corpus
• RECA-PD: A Robust Explainable Cross-Attention Method for Speech-based Parkinson's Disease Classification
• Efficient Extraction of Noise-Robust Discrete Units from Self-Supervised Speech Models