Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension

Abstract

We consider a deep ReLU / Leaky ReLU student network trained from the output of a fixed teacher network of the same depth, with Stochastic Gradient Descent (SGD). The student network is \emph{over-realized}: at each layer $l$, the number $n_l$ of student nodes is more than that ($m_l$) of teacher. Under mild conditions on dataset and teacher network, we prove that when the gradient is small at every data sample, each teacher node is \emph{specialized} by at least one student node \emph{at the lowest layer}. For two-layer network, such specialization can be achieved by training on any dataset of \emph{polynomial} size $\mathcal{O}(K^{5/2}{d}^3{\epsilon }^{-1})$. until the gradient magnitude drops to $\mathcal{O}(\epsilon /K^{3/2}\sqrt{d})$. Here $d$ is the input dimension, $K=m_1+n_1$ is the total number of neurons in the lowest layer of teacher and student. Note that we require a specific form of data augmentation and the sample complexity includes the additional data generated from augmentation. To our best knowledge, we are the first to give polynomial sample complexity for student specialization of training two-layer (Leaky) ReLU networks with finite depth and width in teacher-student setting, and finite complexity for the lowest layer specialization in multi-layer case, without parametric assumption of the input (like Gaussian). Our theory suggests that teacher nodes with large fan-out weights get specialized first when the gradient is still large, while others are specialized with small gradient, which suggests inductive bias in training. This shapes the stage of training as empirically observed in multiple previous works. Experiments on synthetic and CIFAR10 verify our findings. The code is released in https://github.com/facebookresearch/luckmatters.

arXiv commentAccepted in ICML 2020

References in corpus (27)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Essentially No Barriers in Neural Network Energy Landscape
• The loss surface of deep and wide neural networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Stabilizing the Lottery Ticket Hypothesis
• Convergent Learning: Do different neural networks learn the same representations?
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• Learning One-hidden-layer Neural Networks with Landscape Design
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Escaping Saddles with Stochastic Gradients
• Gradient Descent Can Take Exponential Time to Escape Saddle Points
• Complexity of Linear Regions in Deep Networks
• On exponential convergence of SGD in non-convex over-parametrized learning
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Accelerating SGD with momentum for over-parameterized learning
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
• Small nonlinearities in activation functions create bad local minima in neural networks
• Reverse-Engineering Deep ReLU Networks
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• The Multilinear Structure of ReLU Networks

Cited by in corpus (2)

• Understanding Self-supervised Learning with Dual Deep Networks
• Implicit Regularization and Convergence for Weight Normalization