Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models

Qinyuan Ye, Madian Khabsa, Mike Lewis +3

Distilling state-of-the-art transformer models into lightweight student models is an effective way to reduce computation cost at inference time. The student models are typically co…