What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers

Jesse Ponnock

Vision transformers (ViTs) trained to copy a pretrained teacher's attention maps recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distri…