Fixup Initialization: Residual Learning Without Normalization

Abstract

Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.

Updating reference. Accepted for publication at ICLR 2019; see https://openreview.net/forum?id=H1gsz30cKX

References in corpus (16)

• mixup: Beyond Empirical Risk Minimization
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Wide Residual Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Fractional Max-Pooling
• How Does Batch Normalization Help Optimization?
• L2 Regularization versus Batch and Weight Normalization
• All you need is a good init
• ShakeDrop Regularization for Deep Residual Learning
• Latent Alignment and Variational Attention
• Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
• How to Start Training: The Effect of Initialization and Architecture

Cited by in corpus (62)

• AutoML: A Survey of the State-of-the-Art
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Reducing Transformer Depth on Demand with Structured Dropout
• High-Performance Large-Scale Image Recognition Without Normalization
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• What is being transferred in transfer learning?
• Music Source Separation in the Waveform Domain
• Improving robustness against common corruptions by covariate shift adaptation
• ReZero is All You Need: Fast Convergence at Large Depth
• Optimization for deep learning: theory and algorithms
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Jukebox: A Generative Model for Music
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Measuring the Algorithmic Efficiency of Neural Networks
• Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• FlauBERT: Unsupervised Language Model Pre-training for French
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• Low-Memory Neural Network Training: A Technical Report
• Principled Weight Initialization for Hypernetworks
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• Towards Efficient Training for Neural Network Quantization
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• An Adaptive and Momental Bound Method for Stochastic Learning
• Deep Transformers with Latent Depth
• Bayesian Inference for Large Scale Image Classification
• PowerNorm: Rethinking Batch Normalization in Transformers
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Point Spread Function Estimation for Wide Field Small Aperture Telescopes with Deep Neural Networks and Calibration Data
• Orthogonalizing Convolutional Layers with the Cayley Transform
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Scaling Imitation Learning in Minecraft
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• A Loss Curvature Perspective on Training Instability in Deep Learning
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Deep Connectomics Networks: Neural Network Architectures Inspired by Neuronal Networks
• Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks
• A Deep Conditioning Treatment of Neural Networks
• Parameter Prediction for Unseen Deep Architectures
• Gradient-Coherent Strong Regularization for Deep Neural Networks
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization
• How many winning tickets are there in one DNN?
• Comparing the costs of abstraction for DL frameworks
• Rethinking ResNets: Improved Stacking Strategies With High Order Schemes
• On the Demystification of Knowledge Distillation: A Residual Network Perspective
• On the Expressivity of Neural Networks for Deep Reinforcement Learning
• Free-viewpoint Indoor Neural Relighting from Multi-view Stereo
• Self-Attentive Ensemble Transformer: Representing Ensemble Interactions in Neural Networks for Earth System Models
• Scale Calibrated Training: Improving Generalization of Deep Networks via Scale-Specific Normalization
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Is Feature Diversity Necessary in Neural Network Initialization?
• WeightAlign: Normalizing Activations by Weight Alignment
• On the generalization of bayesian deep nets for multi-class classification
• Farkas layers: don't shift the data, fix the geometry
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks
• AdjointBackMapV2: Precise Reconstruction of Arbitrary CNN Unit's Activation via Adjoint Operators
• Seven Myths in Machine Learning Research
• Sequentially Aggregated Convolutional Networks