ScaleKD: Strong Vision Transformers Could Be Excellent Teachers

Jiawei Fan, Chao Li, Xiaolong Liu +1

In this paper, we question if well pre-trained vision transformer (ViT) models could be used as teachers that exhibit scalable properties to advance cross architecture knowledge di…