Aitor Lewkowycz

activity

20112025

most citedPaLM: Scaling Language Modeling with Pathways

2.1k citations · 7.4k across the 36 of their papers we have counts for

collaborators

Showing stat.MLShow all

2 papers · 1 filter

stat.ML2020★ 25 cited

On the training dynamics of deep networks with $L_{2}$ regularization

Aitor Lewkowycz, Guy Gur-Ari

We study the role of $L_{2}$ regularization in deep learning, and uncover simple relations between the performance of the model, the $L_{2}$ coefficient, the learning rate, and the num…

stat.ML2020★ 58 cited

The large learning rate phase of deep learning: the catapult mechanism

Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer +2

The choice of initial learning rate can have a profound effect on the performance of deep networks. We present a class of neural networks with solvable training dynamics, and confi…

On the training dynamics of deep networks with L2​ regularization

The large learning rate phase of deep learning: the catapult mechanism

On the training dynamics of deep networks with $L_{2}$ regularization