Ensemble knowledge distillation of self-supervised speech models

Kuan-Po Huang, Tzu-hsun Feng, Yu-Kuan Fu +5

Distilled self-supervised models have shown competitive performance and efficiency in recent years. However, there is a lack of experience in jointly distilling multiple self-super…