18 citations · 18 across the 5 of their papers we have counts for
4 papers · 1 filter
A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
Itay Lavie, Clarissa Lauditi, Cengiz Pehlevan
A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain un…
Statistical Properties of Training & Generalization
Itay Lavie, Noam Levi, Yonatan Kahn
Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investiga…
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Itay Lavie, Kirsten Fischer, Andrey Lekov +3
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present…
Demystifying Spectral Bias on Real-World Data
Itay Lavie, Zohar Ringel
Kernel ridge regression (KRR) and Gaussian processes (GPs) are fundamental tools in statistics and machine learning, with recent applications to highly over-parameterized deep neur…