23 citations · 41 across the 3 of their papers we have counts for
3 papers
cs.CV2023★ 5 cited
From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels
Zhendong Yang, Ailing Zeng, Zhe Li +3
Knowledge Distillation (KD) uses the teacher's prediction logits as soft labels to guide the student, while self-KD does not need a real teacher to require the soft labels. This wo…
cs.CV2022★ 23 cited
ViTKD: Practical Guidelines for ViT feature knowledge distillation
Zhendong Yang, Zhe Li, Ailing Zeng +3
Knowledge Distillation (KD) for Convolutional Neural Network (CNN) is extensively studied as a way to boost the performance of a small model. Recently, Vision Transformer (ViT) has…
cs.CV2022★ 13 cited
Rethinking Knowledge Distillation via Cross-Entropy
Zhendong Yang, Zhe Li, Yuan Gong +4
Knowledge Distillation (KD) has developed extensively and boosted various tasks. The classical KD method adds the KD loss to the original cross-entropy (CE) loss. We try to decompo…