21 citations · 46 across the 12 of their papers we have counts for
13 papers
In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model
Francesco Cagnetta, Casper L. Christensen
Sequence models can generalise in fundamentally distinct ways: extrapolating structures seen in training, or using their prompt to adapt to novel tasks. The prevailing strategy det…
Deriving Neural Scaling Laws from the statistics of natural language
Francesco Cagnetta, Allan Raventós, Surya Ganguli +1
Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict t…
Deep networks learn to parse uniform-depth context-free languages from local statistics
Jack T. Parley, Francesco Cagnetta, Matthieu Wyart
Understanding how the structure of language can be learned from sentences alone is a central question in both cognitive science and machine learning. Studies of the internal repres…
Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures
Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi +1
How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural networ…
Learning curves theory for hierarchically compositional data with power-law distributed features
Francesco Cagnetta, Hyunmo Kang, Matthieu Wyart
Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data…
How Compositional Generalization and Creativity Improve as Diffusion Models are Trained
Alessandro Favero, Antonio Sclocchi, Francesco Cagnetta +2
Natural data is often organized as a hierarchical composition of features. How many samples do generative models need in order to learn the composition rules, so as to produce a co…
Towards a theory of how the structure of language is acquired by deep neural networks
Francesco Cagnetta, Matthieu Wyart
How much data is required to learn the structure of a language via next-token prediction? We study this question for synthetic datasets generated via a Probabilistic Context-Free G…
Kernels, Data & Physics
Francesco Cagnetta, Deborah Oliveira, Mahalakshmi Sabanayagam +2
Lecture notes from the course given by Professor Julia Kempe at the summer school "Statistical physics of Machine Learning" in Les Houches. The notes discuss the so-called NTK appr…
How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
Francesco Cagnetta, Leonardo Petrini, Umberto M. Tomasini +2
Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabli…
How deep convolutional neural networks lose spatial information with training
Umberto M. Tomasini, Leonardo Petrini, Francesco Cagnetta +1
A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representatio…
What Can Be Learnt With Wide Convolutional Neural Networks?
Francesco Cagnetta, Alessandro Favero, Matthieu Wyart
Understanding how convolutional neural networks (CNNs) can efficiently learn high-dimensional functions remains a fundamental challenge. A popular belief is that these models harne…
Learning sparse features can lead to overfitting in neural networks
Leonardo Petrini, Francesco Cagnetta, Eric Vanden-Eijnden +1
It is widely believed that the success of deep networks lies in their ability to learn a meaningful representation of the features of the data. Yet, understanding when and how this…
Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
Alessandro Favero, Francesco Cagnetta, Matthieu Wyart
Convolutional neural networks perform a local and translationally-invariant treatment of the data: quantifying which of these two aspects is central to their success remains a chal…