Co-training and Co-distillation for Quality Improvement and Compression of Language Models

Hayeon Lee, Rui Hou, Jongpil Kim +4

Knowledge Distillation (KD) compresses computationally expensive pre-trained language models (PLMs) by transferring their knowledge to smaller models, allowing their use in resourc…