Characterizing signal propagation to close the performance gap in unnormalized ResNets

Abstract

Batch Normalization is a key component in almost all state-of-the-art image classifiers, but it also introduces practical challenges: it breaks the independence between training examples within a batch, can incur compute and memory overhead, and often results in unexpected bugs. Building on recent theoretical analyses of deep ResNets at initialization, we propose a simple set of analysis tools to characterize signal propagation on the forward pass, and leverage these tools to design highly performant ResNets without activation normalization layers. Crucial to our success is an adapted version of the recently proposed Weight Standardization. Our analysis tools show how this technique preserves the signal in networks with ReLU or Swish activation functions by ensuring that the per-channel activation means do not grow with depth. Across a range of FLOP budgets, our networks attain performance competitive with the state-of-the-art EfficientNets on ImageNet.

arXiv commentPublished as a conference paper at ICLR 2021

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Decoupled Weight Decay Regularization
• Gaussian Error Linear Units (GELUs)
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• ReZero is All You Need: Fast Convergence at Large Depth
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Towards Understanding Regularization in Batch Normalization
• MobileDets: Searching for Object Detection Architectures for Mobile Accelerators
• On the Generalization Benefit of Noise in Stochastic Gradient Descent
• Deep Residual Networks and Weight Initialization
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization

Cited by in corpus (10)

• How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
• Swin Transformer V2: Scaling Up Capacity and Resolution
• A newcomer's guide to deep learning for inverse design in nano-photonics
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• A Loss Curvature Perspective on Training Instability in Deep Learning
• Parameter Prediction for Unseen Deep Architectures
• Towards Learning Universal Audio Representations
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization