4 papers
The monotonicity of the Franz-Parisi potential is equivalent with Low-degree MMSE lower bounds
Konstantinos Tsirkas, Leda Wang, Ilias Zadik
Over the last decades, two distinct approaches have been instrumental to our understanding of the computational complexity of statistical estimation. The statistical physics litera…
On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
Jianliang He, Leda Wang, Siyu Chen +1
We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the l…
High-dimensional learning dynamics of multi-pass Stochastic Gradient Descent in multi-index models
Zhou Fan, Leda Wang
We study the learning dynamics of a multi-pass, mini-batch Stochastic Gradient Descent (SGD) procedure for empirical risk minimization in high-dimensional multi-index models with i…
Understanding and Enhancing Mask-Based Pretraining towards Universal Representations
Mingze Dong, Leda Wang, Yuval Kluger
Mask-based pretraining has become a cornerstone of modern large-scale models across language, vision, and recently biology. Despite its empirical success, its role and limits in le…