47 citations · 60 across the 10 of their papers we have counts for
8 papers · 1 filter
Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan +1
In this paper, we first present an explanation regarding the common occurrence of spikes in the training loss when neural networks are trained with stochastic gradient descent (SGD…
Restricted Strong Convexity of Deep Learning Models with Smooth Activations
Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu +1
We consider the problem of optimization of deep learning models with smooth activation functions. While there exist influential results on the problem from the ``near initializatio…
A note on Linear Bottleneck networks and their Transition to Multilinearity
Libin Zhu, Parthe Pandit, Mikhail Belkin
Randomly initialized wide neural networks transition to linear functions of weights as the width grows, in a ball of radius around initialization. A necessary condition for…
Quadratic models for understanding catapult dynamics of neural networks
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan +1
While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models. In this work we…
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
Libin Zhu, Chaoyue Liu, Mikhail Belkin
In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their "width" approaches infinity. The…
Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
Chaoyue Liu, Libin Zhu, Mikhail Belkin
Wide neural networks with linear output layer have been shown to be near-linear, and to have near-constant neural tangent kernel (NTK), in a region containing the optimization path…
On the linearity of large non-linear models: when and why the tangent kernel is constant
Chaoyue Liu, Libin Zhu, Mikhail Belkin
The goal of this work is to shed light on the remarkable phenomenon of transition to linearity of certain neural networks as their width approaches infinity. We show that the trans…
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
Chaoyue Liu, Libin Zhu, Mikhail Belkin
The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. The purpose of this…