Application of Knowledge Distillation to Multi-task Speech Representation Learning

Mine Kerpicci, Van Nguyen, Shuhua Zhang +1

Model architectures such as wav2vec 2.0 and HuBERT have been proposed to learn speech representations from audio waveforms in a self-supervised manner. When they are combined with…