High-Performance Large-Scale Image Recognition Without Normalization

Abstract

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets

References in corpus (24)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• mixup: Beyond Empirical Risk Minimization
• On the difficulty of training Recurrent Neural Networks
• Gaussian Error Linear Units (GELUs)
• Self-Attention Generative Adversarial Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Scaling Laws for Neural Language Models
• Large Batch Training of Convolutional Networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• ReZero is All You Need: Fast Convergence at Large Depth
• Fixup Initialization: Residual Learning Without Normalization
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• Four Things Everyone Should Know to Improve Batch Normalization
• ResizeMix: Mixing Data with Preserved Object Information and True Labels
• MaxUp: A Simple Way to Improve Generalization of Neural Network Training
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• On the distance between two neural networks and the stability of learning
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts

Cited by in corpus (56)

• MLP-Mixer: An all-MLP Architecture for Vision
• EfficientNetV2: Smaller Models and Faster Training
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• XCiT: Cross-Covariance Image Transformers
• Compute and Energy Consumption Trends in Deep Learning Inference
• Revisiting ResNets: Improved Training and Scaling Strategies
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• Perceiver: General Perception with Iterative Attention
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Vision Transformers with Patch Diversification
• Refiner: Refining Self-attention for Vision Transformers
• ResNet strikes back: An improved training procedure in timm
• Pay Attention to MLPs
• Efficient Adaptive Ensembling for Image Classification
• Scaling Vision with Sparse Mixture of Experts
• Rethinking "Batch" in BatchNorm
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• VOLO: Vision Outlooker for Visual Recognition
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
• Visual Parser: Representing Part-whole Hierarchies with Transformers
• Homogeneous vector bundles and $G$-equivariant convolutional neural networks
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• Your "Attention" Deserves Attention: A Self-Diversified Multi-Channel Attention for Facial Action Analysis
• Multiplicative update rules for accelerating deep learning training and increasing robustness
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• Generative Art Using Neural Visual Grammars and Dual Encoders
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
• What Is Considered Complete for Visual Recognition?
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error
• Learning from the few: Fine-grained approach to pediatric wrist pathology recognition on a limited dataset
• Ranger21: a synergistic deep learning optimizer
• NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results
• On the Role of Optimization in Double Descent: A Least Squares Study
• Investigating Transfer Learning Capabilities of Vision Transformers and CNNs by Fine-Tuning a Single Trainable Block
• Recognizing bird species in diverse soundscapes under weak supervision
• Parameter Prediction for Unseen Deep Architectures
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• LIGAR: Lightweight General-purpose Action Recognition
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Towards Learning Universal Audio Representations
• Greedy Network Enlarging
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• The Uncanny Similarity of Recurrence and Depth
• Anomaly Detection Based on Selection and Weighting in Latent Space
• Training or Architecture? How to Incorporate Invariance in Neural Networks
• NoFADE: Analyzing Diminishing Returns on CO2 Investment
• Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
• 3rd Place Solution for NeurIPS 2021 Shifts Challenge: Vehicle Motion Prediction