Attention Distillation: self-supervised vision transformer students need more guidance

Kai Wang, Fei Yang, Joost van de Weijer

Self-supervised learning has been widely applied to train high-quality vision transformers. Unleashing their excellent performance on memory and compute constraint devices is there…