GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model

Yingying Gao, Shilei Zhang, Chao Deng +1

Pre-trained speech language models such as HuBERT and WavLM leverage unlabeled speech data for self-supervised learning and offer powerful representations for numerous downstream t…