ViTKD: Practical Guidelines for ViT feature knowledge distillation

Zhendong Yang, Zhe Li, Ailing Zeng +3

Knowledge Distillation (KD) for Convolutional Neural Network (CNN) is extensively studied as a way to boost the performance of a small model. Recently, Vision Transformer (ViT) has…