vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

Abstract

We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition.

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Cross-lingual Language Model Pretraining
• Light Gated Recurrent Units for Speech Recognition
• Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
• Transformers with convolutional context for ASR
• On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
• Who Needs Words? Lexicon-Free Speech Recognition
• Pre-trained Language Model Representations for Language Generation
• VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019

Cited by in corpus (81)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Contrastive Representation Learning: A Framework and Review
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• Improved Noisy Student Training for Automatic Speech Recognition
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings
• Effectiveness of self-supervised pre-training for speech recognition
• A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• SUPERB: Speech processing Universal PERformance Benchmark
• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature
• CiwGAN and fiwGAN: Encoding information in acoustic data to model lexical learning with Generative Adversarial Networks
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• An Overview of Indian Spoken Language Recognition from Machine Learning Perspective
• A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition
• Audio Transformers
• Understanding Self-supervised Learning with Dual Deep Networks
• Privacy-preserving Voice Analysis via Disentangled Representations
• Towards Better Domain Adaptation for Self-supervised Models: A Case Study of Child ASR
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
• Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretraining
• Contrastive Learning Inverts the Data Generating Process
• BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• Towards Data Distillation for End-to-end Spoken Conversational Question Answering
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
• Latent-Domain Predictive Neural Speech Coding
• Unsupervised Speech Recognition
• DiscreTalk: Text-to-Speech as a Machine Translation Problem
• Multistage linguistic conditioning of convolutional layers for speech emotion recognition
• Investigation of Ensemble features of Self-Supervised Pretrained Models for Automatic Speech Recognition
• The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units
• Supervised and Self-supervised Pretraining Based COVID-19 Detection Using Acoustic Breathing/Cough/Speech Signals
• ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification
• Probing phoneme, language and speaker information in unsupervised speech representations
• Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
• PMVC: Data Augmentation-Based Prosody Modeling for Expressive Voice Conversion
• Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
• MusiCoder: A Universal Music-Acoustic Encoder Based on Transformers
• Self-training and Pre-training are Complementary for Speech Recognition
• Learning Robust and Multilingual Speech Representations
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization
• Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks
• Indonesian Automatic Speech Recognition with XLSR-53
• End-to-end Speech Translation via Cross-modal Progressive Training
• A Generic Self-Supervised Framework of Learning Invariant Discriminative Features
• End2End Acoustic to Semantic Transduction
• What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure
• Towards Semi-Supervised Semantics Understanding from Speech
• Interpreting intermediate convolutional layers in unsupervised acoustic word classification
• A Brief Overview of Unsupervised Neural Speech Representation Learning
• A Hierarchical Subspace Model for Language-Attuned Acoustic Unit Discovery
• Word-level Embeddings for Cross-Task Transfer Learning in Speech Processing
• Identity-Based Patterns in Deep Convolutional Networks: Generative Adversarial Phonology and Reduplication
• Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition
• fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit
• Audio MFCC-gram Transformers for respiratory insufficiency detection in COVID-19
• Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data
• Self-Supervised Representations Improve End-to-End Speech Translation
• Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutch
• Probing Acoustic Representations for Phonetic Properties
• UWSpeech: Speech to Speech Translation for Unwritten Languages
• Non-Intrusive Binaural Speech Intelligibility Prediction from Discrete Latent Representations
• Multi-Speaker Multi-Lingual VQTTS System for LIMMITS 2023 Challenge
• Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations
• Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding
• Multitask Training with Text Data for End-to-End Speech Recognition
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding
• Self-supervised pre-training with acoustic configurations for replay spoofing detection
• Unsupervised Word Segmentation from Discrete Speech Units in Low-Resource Settings
• Improving speech recognition models with small samples for air traffic control systems
• Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistency
• A Comparison of Discrete Latent Variable Models for Speech Representation Learning
• Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners