Mathematical Models of Overparameterized Neural Networks
arXiv:2012.13982 · doi:10.1109/JPROC.2020.3048020
Abstract
Deep learning has received considerable empirical successes in recent years. However, while many ad hoc tricks have been discovered by practitioners, until recently, there has been a lack of theoretical understanding for tricks invented in the deep learning literature. Known by practitioners that overparameterized neural networks are easy to learn, in the past few years there have been important theoretical developments in the analysis of overparameterized neural networks. In particular, it was shown that such systems behave like convex systems under various restricted settings, such as for two-layer NNs, and when learning is restricted locally in the so-called neural tangent kernel space around specialized initializations. This paper discusses some of these recent progresses leading to significant better understanding of neural networks. We will focus on the analysis of two-layer neural networks, and explain the key mathematical models, with their algorithmic implications. We will then discuss challenges in understanding deep neural networks and some current research directions.
References in corpus (63)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- Neural Machine Translation by Jointly Learning to Align and Translate
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Visualizing the Loss Landscape of Neural Nets
- Deep Neural Networks as Gaussian Processes
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Prevalence of Neural Collapse during the terminal phase of deep learning training
- Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- On Lazy Training in Differentiable Programming
- SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator
- How to Escape Saddle Points Efficiently
- Three Factors Influencing Minima in SGD
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
- Gradient Descent Finds Global Minima of Deep Neural Networks
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- The loss surface of deep and wide neural networks
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
- Why Deep Neural Networks for Function Approximation?
- No bad local minima: Data independent training error guarantees for multilayer neural networks
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- A Theoretical Analysis of Deep Q-Learning
- Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- What Can ResNet Learn Efficiently, Going Beyond Kernels?
- On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
- Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
- Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
- The Global Landscape of Neural Networks: An Overview
- Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
- Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
- Identity Matters in Deep Learning
- SGD Learns the Conjugate Kernel Class of the Network
- Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy
- Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
- An Improved Analysis of Training Over-parameterized Deep Neural Networks
- A mean-field limit for certain deep neural networks
- Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss
- Critical Points of Neural Networks: Analytical Forms and Landscape Properties
- Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
- The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
- A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
- Topology and Geometry of Half-Rectified Network Optimization
- On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
- Sharp Analysis for Nonconvex SGD Escaping from Saddle Points
- The Deep Weight Prior
- A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks
- Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits
- Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
- The Local Elasticity of Neural Networks
- Towards Understanding Hierarchical Learning: Benefits of Neural Representations
- Lower Bounds for Non-Convex Stochastic Optimization
- Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory
- Function approximation by neural nets in the mean-field regime: Entropic regularization and controlled McKean-Vlasov dynamics
- Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach
- Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate
- Convex Formulation of Overparameterized Deep Neural Networks