2.2k citations · 6.4k across the 57 of their papers we have counts for
30 papers · 1 filter
The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks
Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie +2
We study the type of solutions to which stochastic gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss. Our result…
Continual Learning in Linear Classification on Separable Data
Itay Evron, Edward Moroshko, Gon Buzaglo +4
We analyze continual learning on a sequence of separable linear classification tasks with binary labels. We show theoretically that learning with weak regularization reduces to sol…
DropCompute: simple and more robust distributed synchronous training via compute variance reduction
Niv Giladi, Shahar Gottlieb, Moran Shkolnik +5
Background: Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-R…
Explore to Generalize in Zero-Shot RL
Ev Zisselman, Itai Lavie, Daniel Soudry +1
We study zero-shot generalization in reinforcement learning-optimizing a policy on a set of training tasks to perform well on a similar but unseen test task. To mitigate overfittin…
Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond
Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry +1
Recent research shows that when Gradient Descent (GD) is applied to neural networks, the loss almost never decreases monotonically. Instead, the loss oscillates as gradient descent…
The Role of Codeword-to-Class Assignments in Error-Correcting Codes: An Empirical Study
Itay Evron, Ophir Onn, Tamar Weiss Orzech +2
Error-correcting codes (ECC) are used to reduce multiclass classification tasks to multiple binary classification subproblems. In ECC, classes are represented by the rows of a bina…