Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective

Jinjing Zhu, Songze Li, Lin Wang

Conventional knowledge distillation (KD) approaches are designed for the student model to predict similar output as the teacher model for each sample. Unfortunately, the relationsh…