4 citations · 6 across the 6 of their papers we have counts for
4 papers · 1 filter
An analytic theory of creativity in convolutional diffusion models
Mason Kamb, Surya Ganguli
We obtain an analytic, interpretable and predictive theory of creativity in convolutional diffusion models. Indeed, score-matching diffusion models can generate highly original ima…
Features are fate: a theory of transfer learning in high-dimensional regression
Javan Tahir, Surya Ganguli, Grant M. Rotskoff
With the emergence of large-scale pre-trained neural networks, methods to adapt such "foundation" models to data-limited downstream tasks have become a necessity. Fine-tuning, pref…
Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning
Daniel Kunin, Allan Raventós, Clémentine Dominé +4
While the impressive performance of modern neural networks is often attributed to their capacity to efficiently extract task-relevant features from data, the mechanisms underlying…
Geometric Dynamics of Signal Propagation Predict Trainability of Transformers
Aditya Cowsik, Tamra Nebabu, Xiao-Liang Qi +1
We investigate forward signal propagation and gradient back propagation in deep, randomly initialized transformers, yielding simple necessary and sufficient conditions on initializ…