How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Abstract

Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation ("AugReg" for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.

Andreas, Alex, Xiaohua and Lucas contributed equally. We release more than 50'000 ViT models trained under diverse settings on various datasets. Available at https://github.com/google-research/big_vision, https://github.com/google-research/vision_transformer and https://github.com/rwightman/pytorch-image-models TMLR review at https://openreview.net/forum?id=4nPswr1KcP

References in corpus (9)

• Bootstrap your own latent: A new approach to self-supervised Learning
• MLP-Mixer: An all-MLP Architecture for Vision
• EfficientNetV2: Smaller Models and Faster Training
• Transformer in Transformer
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Do ImageNet Classifiers Generalize to ImageNet?
• Revisiting ResNets: Improved Training and Scaling Strategies
• CvT: Introducing Convolutions to Vision Transformers
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

Cited by in corpus (12)

• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• Exploring the Limits of Out-of-Distribution Detection
• MEMO: Test Time Robustness via Adaptation and Augmentation
• Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
• Discrete Representations Strengthen Vision Transformer Robustness
• The Benchmark Lottery
• Vision Transformer for Classification of Breast Ultrasound Images
• Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivity
• Pyramid Adversarial Training Improves ViT Performance
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• The Brownian motion in the transformer model
• The Efficiency Misnomer