Revisiting ResNets: Improved Training and Scaling Strategies

Abstract

Novel computer vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies. Our work revisits the canonical ResNet (He et al., 2015) and studies these three aspects in an effort to disentangle them. Perhaps surprisingly, we find that training and scaling strategies may matter more than architectural changes, and further, that the resulting ResNets match recent state-of-the-art models. We show that the best performing scaling strategy depends on the training regime and offer two new scaling strategies: (1) scale model depth in regimes where overfitting can occur (width scaling is preferable otherwise); (2) increase image resolution more slowly than previously recommended (Tan & Le, 2019). Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1.7x - 2.7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet. In a large-scale semi-supervised learning setup, ResNet-RS achieves 86.2% top-1 ImageNet accuracy, while being 4.7x faster than EfficientNet NoisyStudent. The training techniques improve transfer performance on a suite of downstream tasks (rivaling state-of-the-art self-supervised algorithms) and extend to video classification on Kinetics-400. We recommend practitioners use these simple revised ResNets as baselines for future research.

References in corpus (25)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• mixup: Beyond Empirical Risk Minimization
• Language Models are Few-Shot Learners
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Scaling Laws for Neural Language Models
• AutoAugment: Learning Augmentation Policies from Data
• DropBlock: A regularization method for convolutional networks
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Deep Learning Scaling is Predictable, Empirically
• Rethinking Pre-training and Self-training
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• High-Performance Large-Scale Image Recognition Without Normalization
• Stand-Alone Self-Attention in Vision Models
• Identity Mappings in Deep Residual Networks
• Scaling Laws for Autoregressive Generative Modeling
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• Spatiotemporal Contrastive Video Representation Learning
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• Global Self-Attention Networks for Image Recognition

Cited by in corpus (21)

• MLP-Mixer: An all-MLP Architecture for Vision
• EfficientNetV2: Smaller Models and Faster Training
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• Open-Set Recognition: a Good Closed-Set Classifier is All You Need?
• Exploring the Limits of Large Scale Pre-training
• ResNet strikes back: An improved training procedure in timm
• Forensic Dental Age Estimation Using Modified Deep Learning Neural Network
• Bag of Tricks for Long-Tail Visual Recognition of Animal Species in Camera-Trap Images
• Performance of GAN-based augmentation for deep learning COVID-19 image classification
• Revisiting 3D ResNets for Video Recognition
• Recurrence along Depth: Deep Convolutional Neural Networks with Recurrent Layer Aggregation
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• The Benchmark Lottery
• Cosmic topology. Part IVa. Classification of manifolds using machine learning: a case study with small toroidal universes
• Rethinking ResNets: Improved Stacking Strategies With High Order Schemes
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• Semi-supervised Quality Evaluation of Colonoscopy Procedures
• Greedy Network Enlarging
• Harnessing Orthogonality to Train Low-Rank Neural Networks
• Analyze and Design Network Architectures by Recursion Formulas