On the Surprising Effectiveness of Attention Transfer for Vision Transformers

Alexander C. Li, Yuandong Tian, Beidi Chen +2

Conventional wisdom suggests that pre-training Vision Transformers (ViT) improves downstream performance by learning useful representations. Is this actually true? We investigate t…