11 citations · 13 across the 4 of their papers we have counts for
4 papers · 1 filter
Deriving Neural Scaling Laws from the statistics of natural language
Francesco Cagnetta, Allan Raventós, Surya Ganguli +1
Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict t…
Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning
Daniel Kunin, Allan Raventós, Clémentine Dominé +4
While the impressive performance of modern neural networks is often attributed to their capacity to efficiently extract task-relevant features from data, the mechanisms underlying…
Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression
Allan Raventós, Mansheej Paul, Feng Chen +1
Pretrained transformers exhibit the remarkable ability of in-context learning (ICL): they can learn tasks from just a few examples provided in the prompt without updating any weigh…
SIXO: Smoothing Inference with Twisted Objectives
Dieterich Lawson, Allan Raventós, Andrew Warrington +1
Sequential Monte Carlo (SMC) is an inference algorithm for state space models that approximates the posterior by sampling from a sequence of target distributions. The target distri…