High-Performance Large-Scale Image Recognition Without Normalization

Abstract

Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets

References in corpus (13)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• On the difficulty of training Recurrent Neural Networks
• Scaling Laws for Neural Language Models
• Large Batch Training of Convolutional Networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Fixup Initialization: Residual Learning Without Normalization
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• ResizeMix: Mixing Data with Preserved Object Information and True Labels
• MaxUp: A Simple Way to Improve Generalization of Neural Network Training
• On the Generalization Benefit of Noise in Stochastic Gradient Descent

Cited by in corpus (3)

• Revisiting ResNets: Improved Training and Scaling Strategies
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Anomaly Detection Based on Selection and Weighting in Latent Space