ResNet strikes back: An improved training procedure in timm
arXiv:2110.00476
Abstract
The influential Residual Networks designed by He et al. remain the gold-standard architecture in numerous scientific publications. They typically serve as the default architecture in studies, or as baselines when new architectures are proposed. Yet there has been significant progress on best practices for training neural networks since the inception of the ResNet architecture in 2015. Novel optimization & data-augmentation have increased the effectiveness of the training recipes. In this paper, we re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates such advances. We share competitive training settings and pre-trained models in the timm open-source library, with the hope that they will serve as better baselines for future work. For instance, with our more demanding training setting, a vanilla ResNet-50 reaches 80.4% top-1 accuracy at resolution 224x224 on ImageNet-val without extra data or distillation. We also report the performance achieved with popular models with our training procedure.
References in corpus (20)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- mixup: Beyond Empirical Risk Minimization
- MLP-Mixer: An all-MLP Architecture for Vision
- AutoAugment: Learning Augmentation Policies from Data
- CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
- AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
- Billion-scale semi-supervised learning for image classification
- High-Performance Large-Scale Image Recognition Without Normalization
- Averaging Weights Leads to Wider Optima and Better Generalization
- Revisiting ResNets: Improved Training and Scaling Strategies
- Supervised Contrastive Learning
- Sharpness-Aware Minimization for Efficiently Improving Generalization
- AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
- Are we done with ImageNet?
- Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network
- MultiGrain: a unified image embedding for classes and instances
- The iNaturalist Species Classification and Detection Dataset
- Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision
- Grafit: Learning fine-grained image representations with coarse labels