536 citations · 4.4k across the 97 of their papers we have counts for
9 papers · 1 filter
Implicit Regularization in ReLU Networks with the Square Loss
Gal Vardi, Ohad Shamir
Understanding the implicit regularization (or implicit bias) of gradient descent has recently been a very active research area. However, the implicit regularization in nonlinear ne…
High-Order Oracle Complexity of Smooth and Strongly Convex Optimization
Guy Kornowski, Ohad Shamir
In this note, we consider the complexity of optimizing a highly smooth (Lipschitz -th order derivative) and strongly convex function, via calls to a -th order oracle which re…
Gradient Methods Never Overfit On Separable Data
Ohad Shamir
A line of recent works established that when training linear predictors over separable data, using gradient methods and exponentially-tailed losses, the predictors asymptotically c…
The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks
Itay Safran, Gilad Yehudai, Ohad Shamir
We study the effects of mild over-parameterization on the optimization landscape of a simple ReLU neural network of the form $\mathbf{x}\mapsto\sum_{i=1}^k\max\{0,\mathbf{w}_i^{\to…
Neural Networks with Small Weights and Depth-Separation Barriers
Gal Vardi, Ohad Shamir
In studying the expressiveness of neural networks, an important question is whether there are functions which can only be approximated by sufficiently deep networks, assuming their…
Proving the Lottery Ticket Hypothesis: Pruning is All You Need
Eran Malach, Gilad Yehudai, Shai Shalev-Shwartz +1
The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete w…