Learning from a Lightweight Teacher for Efficient Knowledge Distillation

Yuang Liu, Wei Zhang, Jun Wang

Knowledge Distillation (KD) is an effective framework for compressing deep learning models, realized by a student-teacher paradigm requiring small student networks to mimic the sof…