Guy Gur-Ari

activity

20182020

most citedGradient Descent Happens in a Tiny Subspace

108 citations · 222 across the 5 of their papers we have counts for

collaborators

Showing stat.MLShow all

2 papers · 1 filter

stat.ML2020

On the training dynamics of deep networks with $L_{2}$ regularization

Aitor Lewkowycz, Guy Gur-Ari

We study the role of $L_{2}$ regularization in deep learning, and uncover simple relations between the performance of the model, the $L_{2}$ coefficient, the learning rate, and the num…

stat.ML2020★ 58 cited

The large learning rate phase of deep learning: the catapult mechanism

Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer +2

The choice of initial learning rate can have a profound effect on the performance of deep networks. We present a class of neural networks with solvable training dynamics, and confi…

On the training dynamics of deep networks with L2​ regularization

The large learning rate phase of deep learning: the catapult mechanism

On the training dynamics of deep networks with $L_{2}$ regularization