Collective evolution of weights in wide neural networks
arXiv:1810.03974
Abstract
We derive a nonlinear integro-differential transport equation describing collective evolution of weights under gradient descent in large-width neural-network-like models. We characterize stationary points of the evolution and analyze several scenarios where the transport equation can be solved approximately. We test our general method in the special case of linear free-knot splines, and find good agreement between theory and experiment in observations of global optima, stability of stationary points, and convergence rates.
arXiv comment18 pages, 5 figures
References in corpus (9)
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Opening the Black Box of Deep Neural Networks via Information
- The Loss Surfaces of Multilayer Networks
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Exponential expressivity in deep neural networks through transient chaos
- The loss surface of deep and wide neural networks
- Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
- An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
- Deep linear neural networks with arbitrary loss: All local minima are global