Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search

Taehyeon Kim, Heesoo Myeong, Se-Young Yun

Knowledge Distillation (KD) has recently emerged as a popular method for compressing neural networks. In recent studies, generalized distillation methods that find parameters and a…