Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation

Abstract

Transformer-based speech self-supervised learning (SSL) models, such as HuBERT, show surprising performance in various speech processing tasks. However, huge number of parameters in speech SSL models necessitate the compression to a more compact model for wider usage in academia or small companies. In this study, we suggest to reuse attention maps across the Transformer layers, so as to remove key and query parameters while retaining the number of layers. Furthermore, we propose a novel masking distillation strategy to improve the student model's speech representation quality. We extend the distillation loss to utilize both masked and unmasked speech frames to fully leverage the teacher model's high-quality representation. Our universal compression strategy yields the student model that achieves phoneme error rate (PER) of 7.72% and word error rate (WER) of 9.96% on the SUPERB benchmark.

arXiv commentProceedings of Interspeech 2023. Code URL: https://github.com/sungnyun/ARMHuBERT

References in corpus (13)

• Efficient Estimation of Word Representations in Vector Space
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding
• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
• DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT
• FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning

Cited by in corpus (1)

• Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers