Early Convolutions Help Transformers See Better

Tete Xiao, Mannat Singh, Eric Mintun +3

Vision transformer (ViT) models exhibit substandard optimizability. In particular, they are sensitive to the choice of optimizer (AdamW vs. SGD), optimizer hyperparameters, and tra…