7 papers
Statistical Properties of Training & Generalization
Itay Lavie, Noam Levi, Yonatan Kahn
Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investiga…
Sampling Data with Chains of Forward-Backward Diffusion Steps
Hyunmo Kang, Noam Itzhak Levi, Corinna Elena Wegner +2
Sampling from learned high-dimensional distributions is a foundational computational problem. We introduce U-turn chains: Markov chains obtained by iterating short forward-backward…
Scale Dependent Data Duplication
Joshua Kazdan, Noam Levi, Rylan Schaeffer +6
Data duplication during pretraining can degrade generalization and lead to memorization, motivating aggressive deduplication pipelines. However, at web scale, it is unclear what co…
The Implicit Bias of Logit Regularization
Alon Beck, Yohai Bar Sinai, Noam Levi
Logit regularization, the addition of a convex penalty directly in logit space, is widely used in modern classifiers, with label smoothing as a prominent example. While such method…
Ascent Fails to Forget
Ioannis Mavrothalassitis, Pol Puigdemont, Noam Itzhak Levi +1
Contrary to common belief, we show that gradient ascent-based unconstrained optimization methods frequently fail to perform machine unlearning, a phenomenon we attribute to the inh…
Grokking at the Edge of Linear Separability
Alon Beck, Noam Levi, Yohai Bar-Sinai
We investigate the phenomenon of grokking -- delayed generalization accompanied by non-monotonic test loss behavior -- in a simple binary logistic classification task, for which "m…