21 citations · 46 across the 12 of their papers we have counts for
3 papers · 1 filter
Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures
Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi +1
How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural networ…
Learning curves theory for hierarchically compositional data with power-law distributed features
Francesco Cagnetta, Hyunmo Kang, Matthieu Wyart
Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data…
How Compositional Generalization and Creativity Improve as Diffusion Models are Trained
Alessandro Favero, Antonio Sclocchi, Francesco Cagnetta +2
Natural data is often organized as a hierarchical composition of features. How many samples do generative models need in order to learn the composition rules, so as to produce a co…