GenDistiller: Distilling Pre-trained Language Models based on Generative Models

Yingying Gao, Shilei Zhang, Zihao Cui +3

Self-supervised pre-trained models such as HuBERT and WavLM leverage unlabeled speech data for representation learning and offer significantly improve for numerous downstream tasks…