Pea-KD: Parameter-efficient and Accurate Knowledge Distillation on BERT

Ikhyun Cho, U Kang

How can we efficiently compress a model while maintaining its performance? Knowledge Distillation (KD) is one of the widely known methods for model compression. In essence, KD trai…