Luck Matters: Understanding Training Dynamics of Deep ReLU Networks

Abstract

We analyze the dynamics of training deep ReLU networks and their implications on generalization capability. Using a teacher-student setting, we discovered a novel relationship between the gradient received by hidden student nodes and the activations of teacher nodes for deep ReLU networks. With this relationship and the assumption of small overlapping teacher node activations, we prove that (1) student nodes whose weights are initialized to be close to teacher nodes converge to them at a faster rate, and (2) in over-parameterized regimes and 2-layer case, while a small set of lucky nodes do converge to the teacher nodes, the fan-out weights of other nodes converge to zero. This framework provides insight into multiple puzzling phenomena in deep learning like over-parameterization, implicit regularization, lottery tickets, etc. We verify our assumption by showing that the majority of BatchNorm biases of pre-trained VGG11/16 models are negative. Experiments on (1) random deep teacher networks with Gaussian inputs, (2) teacher network pre-trained on CIFAR-10 and (3) extensive ablation studies validate our multiple theoretical predictions.

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distributed Representations of Words and Phrases and their Compositionality
• Playing Atari with Deep Reinforcement Learning
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Deep Learning is Robust to Massive Label Noise
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Stabilizing the Lottery Ticket Hypothesis
• Sharp Minima Can Generalize For Deep Nets
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Stuck in a What? Adventures in Weight Space

Cited by in corpus (8)

• Discovering Neural Wirings
• Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning
• Lottery Tickets in Linear Models: An Analysis of Iterative Magnitude Pruning
• Exploring Weight Importance and Hessian Bias in Model Pruning
• Implicit Regularization and Convergence for Weight Normalization
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
• Understanding Robustness in Teacher-Student Setting: A New Perspective
• Directional Convergence Analysis under Spherically Symmetric Distribution