Knowledge Distillation from Internal Representations

Gustavo Aguilar, Yuan Ling, Yu Zhang +3

Knowledge distillation is typically conducted by training a small model (the student) to mimic a large and cumbersome model (the teacher). The idea is to compress the knowledge fro…