126 citations · 555 across the 57 of their papers we have counts for
69 papers
Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputs
Emmanuel Abbe, Elisabetta Cornacchia, Aryo Lotfi
Experimental results have shown that curriculum learning, i.e., presenting simpler examples before more complex ones, can improve the efficiency of learning. Some recent theoretica…
Transformers learn through gradual rank increase
Enric Boix-Adsera, Etai Littwin, Emmanuel Abbe +2
We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs…
A proof that Reed-Muller codes achieve Shannon capacity on symmetric channels
Emmanuel Abbe, Colin Sandon
Reed-Muller codes were introduced in 1954, with a simple explicit construction based on polynomial evaluations, and have long been conjectured to achieve Shannon capacity on symmet…
SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics
Emmanuel Abbe, Enric Boix-Adsera, Theodor Misiakiewicz
We investigate the time complexity of SGD learning on fully-connected neural networks with isotropic data. We put forward a complexity measure -- the leap -- which measures how "hi…
Generalization on the Unseen, Logic Reasoning and Degree Curriculum
Emmanuel Abbe, Samy Bengio, Aryo Lotfi +1
This paper considers the learning of logical (Boolean) functions with a focus on the generalization on the unseen (GOTU) setting, a strong case of out-of-distribution generalizatio…
On the non-universality of deep learning: quantifying the cost of symmetry
Emmanuel Abbe, Enric Boix-Adsera
We prove limitations on what neural networks trained by noisy gradient descent (GD) can efficiently learn. Our results apply whenever GD training is equivariant, which holds for ma…