Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method

Shicheng Tan, Weng Lam Tam, Yuanchun Wang +4

The large scale of pre-trained language models poses a challenge for their deployment on various devices, with a growing emphasis on methods to compress these models, particularly…