Structured Initialization for Vision Transformers

Jianqiao Zheng, Xueqian Li, Hemanth Saratchandran +1

Convolutional Neural Networks (CNNs) inherently encode strong inductive biases, enabling effective generalization on small-scale datasets. In this paper, we propose integrating thi…