10 papers
Statistical Properties of Training & Generalization
Itay Lavie, Noam Levi, Yonatan Kahn
Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investiga…
Sampling Data with Chains of Forward-Backward Diffusion Steps
Hyunmo Kang, Noam Itzhak Levi, Corinna Elena Wegner +2
Sampling from learned high-dimensional distributions is a foundational computational problem. We introduce U-turn chains: Markov chains obtained by iterating short forward-backward…
Scale Dependent Data Duplication
Joshua Kazdan, Noam Levi, Rylan Schaeffer +6
Data duplication during pretraining can degrade generalization and lead to memorization, motivating aggressive deduplication pipelines. However, at web scale, it is unclear what co…
The Implicit Bias of Logit Regularization
Alon Beck, Yohai Bar Sinai, Noam Levi
Logit regularization, the addition of a convex penalty directly in logit space, is widely used in modern classifiers, with label smoothing as a prominent example. While such method…
Ascent Fails to Forget
Ioannis Mavrothalassitis, Pol Puigdemont, Noam Itzhak Levi +1
Contrary to common belief, we show that gradient ascent-based unconstrained optimization methods frequently fail to perform machine unlearning, a phenomenon we attribute to the inh…
A Simple Model of Inference Scaling Laws
Noam Levi
Neural scaling laws have garnered significant interest due to their ability to predict model performance as a function of increasing parameters, data, and compute. In this work, we…
Probing the Latent Hierarchical Structure of Data via Diffusion Models
Antonio Sclocchi, Alessandro Favero, Noam Itzhak Levi +1
High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitati…
Grokking at the Edge of Linear Separability
Alon Beck, Noam Levi, Yohai Bar-Sinai
We investigate the phenomenon of grokking -- delayed generalization accompanied by non-monotonic test loss behavior -- in a simple binary logistic classification task, for which "m…
Measuring Sharpness in Grokking
Jack Miller, Patrick Gleeson, Charles O'Neill +2
Neural networks sometimes exhibit grokking, a phenomenon where perfect or near-perfect performance is achieved on a validation set well after the same performance has been obtained…
Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding
Noam Levi, Alon Beck, Yohai Bar-Sinai
Grokking is the intriguing phenomenon where a model learns to generalize long after it has fit the training data. We show both analytically and numerically that grokking can surpri…