4 papers · 1 filter
A Simple Model of Inference Scaling Laws
Noam Levi
Neural scaling laws have garnered significant interest due to their ability to predict model performance as a function of increasing parameters, data, and compute. In this work, we…
Probing the Latent Hierarchical Structure of Data via Diffusion Models
Antonio Sclocchi, Alessandro Favero, Noam Itzhak Levi +1
High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitati…
Grokking at the Edge of Linear Separability
Alon Beck, Noam Levi, Yohai Bar-Sinai
We investigate the phenomenon of grokking -- delayed generalization accompanied by non-monotonic test loss behavior -- in a simple binary logistic classification task, for which "m…
Measuring Sharpness in Grokking
Jack Miller, Patrick Gleeson, Charles O'Neill +2
Neural networks sometimes exhibit grokking, a phenomenon where perfect or near-perfect performance is achieved on a validation set well after the same performance has been obtained…