18 citations · 18 across the 5 of their papers we have counts for
6 papers
A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling
Itay Lavie, Clarissa Lauditi, Cengiz Pehlevan
A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain un…
Statistical Properties of Training & Generalization
Itay Lavie, Noam Levi, Yonatan Kahn
Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investiga…
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Itay Lavie, Kirsten Fischer, Andrey Lekov +3
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present…
Demystifying Spectral Bias on Real-World Data
Itay Lavie, Zohar Ringel
Kernel ridge regression (KRR) and Gaussian processes (GPs) are fundamental tools in statistics and machine learning, with recent applications to highly over-parameterized deep neur…
Towards Understanding Inductive Bias in Transformers: A View From Infinity
Itay Lavie, Guy Gur-Ari, Zohar Ringel
We study inductive bias in Transformers in the infinitely over-parameterized Gaussian process limit and argue transformers tend to be biased towards more permutation symmetric func…
Roadmap to Thermal Dark Matter Beyond the WIMP Unitarity Bound
Ronny Frumkin, Eric Kuflik, Itay Lavie +1
We study the general properties of the freezeout of a thermal relic. We give analytic estimates of the relic abundance for an arbitrary freezeout process, showing when instantaneou…