How to Start Training: The Effect of Initialization and Architecture

Abstract

We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in and, for ResNets, by correctly weighting the residual modules. We prove that the second failure mode, exponentially large variance of activation length, never occurs in residual nets once the first failure mode is avoided. In contrast, for fully connected nets, we prove that this failure mode can happen and is avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.

arXiv commentFinal Version, 16p, Accepted NIPS 2018

References in corpus (10)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• A Closer Look at Memorization in Deep Networks
• Exponential expressivity in deep neural networks through transient chaos
• How to Escape Saddle Points Efficiently
• No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Deep Residual Networks and Weight Initialization
• A Correspondence Between Random Neural Networks and Statistical Field Theory

Cited by in corpus (16)

• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Fixup Initialization: Residual Learning Without Normalization
• TransReID: Transformer-based Object Re-Identification
• Multiscale Vision Transformers
• A Mean Field Theory of Batch Normalization
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• A Fine-Grained Spectral Perspective on Neural Networks
• TokenPose: Learning Keypoint Tokens for Human Pose Estimation
• Tensor Programs III: Neural Matrix Laws
• The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
• Manipulating SGD with Data Ordering Attacks
• Vision-Based American Sign Language Classification Approach via Deep Learning
• Where Should We Begin? A Low-Level Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks
• Collegial Ensembles
• "Knights": First Place Submission for VIPriors21 Action Recognition Challenge at ICCV 2021
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization