Learn From the Past: Experience Ensemble Knowledge Distillation

Chaofei Wang, Shaowei Zhang, Shiji Song +1

Traditional knowledge distillation transfers "dark knowledge" of a pre-trained teacher network to a student network, and ignores the knowledge in the training process of the teache…