Deep Residual Networks and Weight Initialization

Abstract

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that goodness of ResNet is correlated with the fact that ResNets are relatively insensitive to choice of initial weights. We also demonstrate how batch normalization improves backpropagation of deep ResNets without tuning initial values of weights.

10 pages, 4 figures

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• All you need is a good init

Cited by in corpus (5)

• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• How to Start Training: The Effect of Initialization and Architecture
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• Fast Certified Robust Training with Short Warmup
• Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization