2.2k citations · 6.4k across the 57 of their papers we have counts for
59 papers
The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie +2
We study the type of solutions to which stochastic gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss. Our result…
Continual Learning in Linear Classification on Separable Data
Itay Evron, Edward Moroshko, Gon Buzaglo +4
We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to sol…
DropCompute: simple and more robust distributed synchronous training via compute variance reduction
Niv Giladi, Shahar Gottlieb, Moran Shkolnik +5
Background: Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-R…
Explore to Generalize in Zero-Shot RL
Ev Zisselman, Itai Lavie, Daniel Soudry +1
We study zero-shot generalization in reinforcement learning-optimizing a policy on a set of training tasks to perform well on a similar but unseen test task. To mitigate overfittin…
Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond
Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry +1
Recent research shows that when Gradient Descent (GD) is applied to neural networks, the loss almost never decreases monotonically. Instead, the loss oscillates as gradient descent…
Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations
Hagay Michaeli, Tomer Michaeli, Daniel Soudry
Although CNNs are believed to be invariant to translations, recent works have shown this is not the case, due to aliasing effects that stem from downsampling layers. The existing a…