45 citations · 45 across the 6 of their papers we have counts for
6 papers
The Spectral Dynamics and Noise Geometry of Muon
Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso Poggio
Muon replaces a matrix gradient by its polar factor . This keeps the singular directions selected by the gradient, but makes the update spectrum flat. We stud…
Learning Sparse Compositional Functions with Norm-Constrained Neural Networks
Shuo Huang, Lorenzo Fiorito, Lorenzo Rosasco +1
The ability of deep neural networks to learn hierarchical features is widely regarded as a key mechanism underlying their success in high-dimensional learning. Existing theory part…
Hierarchical Reasoning Models: Perspectives and Misconceptions
Renee Ge, Qianli Liao, Tomaso Poggio
Transformers have demonstrated remarkable performance in natural language processing and related domains, as they largely focus on sequential, autoregressive next-token prediction…
A universal compression theory for lottery ticket hypothesis and neural scaling laws
Hong-Yi Wang, Di Luo, Tomaso Poggio +2
When training large-scale models, the performance typically scales with the number of parameters and the dataset size according to a slow power law. A fundamental theoretical and p…
How Biological Synapses Self-Assemble Gradient Learning
Qianli Liao, Liu Ziyin, Yulu Gan +4
Existing models of learning in the brain explain how given circuits learn, but not how biology could assemble those circuits in the first place. To address this gap, we formulate S…
Theory II: Landscape of the Empirical Risk in Deep Learning
Qianli Liao, Tomaso Poggio
Previous theoretical work on deep learning and neural network optimization tend to focus on avoiding saddle points and local minima. However, the practical observation is that, at…