77 citations · 151 across the 13 of their papers we have counts for
16 papers · 1 filter
On the Implicit Bias of Gradient Descent for Temporal Extrapolation
Edo Cohen-Karlik, Avichai Ben David, Nadav Cohen +1
When using recurrent neural networks (RNNs) it is common practice to apply trained models to sequences longer than those seen in training. This "extrapolating" usage deviates from…
A Theoretical Analysis of Fine-tuning with Linear Teachers
Gal Shachaf, Alon Brutzkus, Amir Globerson
Fine-tuning is a common practice in deep learning, achieving excellent generalization results on downstream tasks using relatively little training data. Although widely used in pra…
On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson +4
Recent work has highlighted the role of initialization scale in determining the structure of the solutions that gradient methods converge to. In particular, it was shown that large…
Towards Understanding Learning in Neural Networks with Linear Teachers
Roei Sarussi, Alon Brutzkus, Amir Globerson
Can a neural network minimizing cross-entropy learn linearly separable data? Despite progress in the theory of deep learning, this question remains unsolved. Here we prove that SGD…
Maximin Optimization for Binary Regression
Nisan Chiprut, Amir Globerson, Ami Wiesel
We consider regression problems with binary weights. Such optimization problems are ubiquitous in quantized learning models and digital communication systems. A natural approach is…
Regularizing Towards Permutation Invariance in Recurrent Models
Edo Cohen-Karlik, Avichai Ben David, Amir Globerson
In many machine learning problems the output should not depend on the order of the input. Such "permutation invariant" functions have been studied extensively recently. Here we arg…