47 citations · 60 across the 10 of their papers we have counts for
5 papers · 1 filter
Restricted Strong Convexity of Deep Learning Models with Smooth Activations
Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu +1
We consider the problem of optimization of deep learning models with smooth activation functions. While there exist influential results on the problem from the ``near initializatio…
A note on Linear Bottleneck networks and their Transition to Multilinearity
Libin Zhu, Parthe Pandit, Mikhail Belkin
Randomly initialized wide neural networks transition to linear functions of weights as the width grows, in a ball of radius around initialization. A necessary condition for…
Quadratic models for understanding catapult dynamics of neural networks
Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan +1
While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models. In this work we…
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture
Libin Zhu, Chaoyue Liu, Mikhail Belkin
In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their "width" approaches infinity. The…
Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
Chaoyue Liu, Libin Zhu, Mikhail Belkin
Wide neural networks with linear output layer have been shown to be near-linear, and to have near-constant neural tangent kernel (NTK), in a region containing the optimization path…