PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition

Abstract

Self-supervised speech representation learning (speech SSL) has demonstrated the benefit of scale in learning rich representations for Automatic Speech Recognition (ASR) with limited paired data, such as wav2vec 2.0. We investigate the existence of sparse subnetworks in pre-trained speech SSL models that achieve even better low-resource ASR results. However, directly applying widely adopted pruning methods such as the Lottery Ticket Hypothesis (LTH) is suboptimal in the computational cost needed. Moreover, we show that the discovered subnetworks yield minimal performance gain compared to the original dense network. We present Prune-Adjust-Re-Prune (PARP), which discovers and finetunes subnetworks for much better performance, while only requiring a single downstream ASR finetuning run. PARP is inspired by our surprising observation that subnetworks pruned for pre-training tasks need merely a slight adjustment to achieve a sizeable performance boost in downstream ASR tasks. Extensive experiments on low-resource ASR verify (1) sparse subnetworks exist in mono-lingual/multi-lingual pre-trained speech SSL, and (2) the computational advantage and performance gain of PARP over baseline pruning methods. In particular, on the 10min Librispeech split without LM decoding, PARP discovers subnetworks from wav2vec 2.0 with an absolute 10.9%/12.6% WER decrease compared to the full model. We further demonstrate the effectiveness of PARP via: cross-lingual pruning without any phone recognition degradation, the discovery of a multi-lingual subnetwork for 10 spoken languages in 1 finetuning run, and its applicability to pre-trained BERT/XLNet for natural language tasks.

References in corpus (34)

• The State of Sparsity in Deep Neural Networks
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Deep Rewiring: Training very sparse deep networks
• Exploring Sparsity in Recurrent Neural Networks
• Effectiveness of self-supervised pre-training for speech recognition
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• Optimizing Speech Recognition For The Edge
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• SUPERB: Speech processing Universal PERformance Benchmark
• When BERT Plays the Lottery, All Tickets Are Winning
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Unsupervised Speech Recognition
• Contrastive Predictive Coding Based Feature for Automatic Speaker Verification
• Pushing the Limits of Non-Autoregressive Speech Recognition
• W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
• Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
• Scaling End-to-End Models for Large-Scale Multilingual ASR
• Efficient conformer-based speech recognition with linear attention
• Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
• UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
• Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models
• Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval
• SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts
• Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning
• Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone

Cited by in corpus (2)

• Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget
• Segment-Level Vectorized Beam Search Based on Partially Autoregressive Inference