1 citations · 1 across the 5 of their papers we have counts for
5 papers
Logarithmic-time Schedules for Scaling Language Models with Momentum
Damien Ferbach, Courtney Paquette, Gauthier Gidel +2
In practice, the hyperparameters and weight-decay in AdamW are typically kept at fixed values. Is there any reason to do otherwise? We show that for large-scale la…
Dimension-adapted Momentum Outscales SGD
Damien Ferbach, Katie Everett, Gauthier Gidel +2
We investigate scaling laws for stochastic momentum algorithms with small batch on the power law random features model, parameterized by data complexity, target complexity, and mod…
Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences
Damien Ferbach, Quentin Bertrand, Avishek Joey Bose +1
The rapid progress in generative models has resulted in impressive leaps in generation quality, blurring the lines between synthetic and real data. Web-scale datasets are now prone…
Proving Linear Mode Connectivity of Neural Networks via Optimal Transport
Damien Ferbach, Baptiste Goujaud, Gauthier Gidel +1
The energy landscape of high-dimensional non-convex optimization problems is crucial to understanding the effectiveness of modern deep neural network architectures. Recent works ha…
A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesis
Damien Ferbach, Christos Tsirigotis, Gauthier Gidel +2
The Strong Lottery Ticket Hypothesis (SLTH) stipulates the existence of a subnetwork within a sufficiently overparameterized (dense) neural network that -- when initialized randoml…