Feature Structure Distillation with Centered Kernel Alignment in BERT Transferring

Hee-Jun Jung, Doyeon Kim, Seung-Hoon Na +1

Knowledge distillation is an approach to transfer information on representations from a teacher to a student by reducing their difference. A challenge of this approach is to reduce…