Improving Diversity in Black-box Few-shot Knowledge Distillation

Tri-Nhan Vo, Dang Nguyen, Kien Do +1

Knowledge distillation (KD) is a well-known technique to effectively compress a large network (teacher) to a smaller network (student) with little sacrifice in performance. However…