Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach

Mun-Hak Lee, Joon-Hyuk Chang

The remarkable performance of the pre-trained language model (LM) using self-supervised learning has led to a major paradigm shift in the study of natural language processing. In l…