Structured Initialization for Attention in Vision Transformers

Jianqiao Zheng, Xueqian Li, Simon Lucey

The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge. By contrast, convolutional neural networks (CNNs) have an architectural ind…