Adaptive Knowledge Distillation between Text and Speech Pre-trained Models

Jinjie Ni, Yukun Ma, Wen Wang +7

Learning on a massive amount of speech corpus leads to the recent success of many self-supervised speech models. With knowledge distillation, these models may also benefit from the…