75 citations · 347 across the 37 of their papers we have counts for
42 papers
Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo
James Thornton, Louis Bethune, Ruixiang Zhang +3
Diffusion models may be formulated as a time-indexed sequence of energy-based models, where the score corresponds to the negative gradient of an energy function. As opposed to lear…
Mechanisms of Projective Composition of Diffusion Models
Arwen Bradley, Preetum Nakkiran, David Berthelot +2
We study the theoretical foundations of composition in diffusion models, with a particular focus on out-of-distribution extrapolation and length-generalization. Prior work has show…
Normalizing Flows are Capable Generative Models
Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran +7
Normalizing Flows (NFs) are likelihood-based models for continuous inputs. They have demonstrated promising results on both density estimation and generative modeling tasks, but ha…
A Formal Framework for Understanding Length Generalization in Transformers
Xinting Huang, Andy Yang, Satwik Bhattamishra +5
A major challenge for transformers is generalizing to sequences longer than those observed during training. While previous works have empirically shown that transformers can either…
Classifier-Free Guidance is a Predictor-Corrector
Arwen Bradley, Preetum Nakkiran
We investigate the theoretical foundations of classifier-free guidance (CFG). CFG is the dominant method of conditional sampling for text-to-image diffusion models, yet unlike othe…
How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
Etai Littwin, Omid Saremi, Madhu Advani +4
Two competing paradigms exist for self-supervised learning of data representations. Joint Embedding Predictive Architecture (JEPA) is a class of architectures in which semantically…
Step-by-Step Diffusion: An Elementary Tutorial
Preetum Nakkiran, Arwen Bradley, Hattie Zhou +1
We present an accessible first course on diffusion models and flow matching for machine learning, aimed at a technical audience with no diffusion experience. We try to simplify the…
When is Multicalibration Post-Processing Necessary?
Dutch Hansen, Siddartha Devic, Preetum Nakkiran +1
Calibration is a well-studied property of predictors which guarantees meaningful uncertainty estimates. Multicalibration is a related notion -- originating in algorithmic fairness…
Perspectives on the State and Future of Deep Learning - 2023
Micah Goldblum, Anima Anandkumar, Richard Baraniuk +7
The goal of this series is to chronicle opinions and issues in the field of machine learning as they stand today and as they change over time. The plan is to host this survey perio…
LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
Vimal Thilak, Chen Huang, Omid Saremi +5
Joint embedding (JE) architectures have emerged as a promising avenue for acquiring transferable data representations. A key obstacle to using JE methods, however, is the inherent…
What Algorithms can Transformers Learn? A Study in Length Generalization
Hattie Zhou, Arwen Bradley, Etai Littwin +5
Large language models exhibit surprising emergent generalization properties, yet also struggle on many simple reasoning tasks such as arithmetic and parity. This raises the questio…
Vanishing Gradients in Reinforcement Finetuning of Language Models
Noam Razin, Hattie Zhou, Omid Saremi +5
Pretrained language models are commonly aligned with human preferences and downstream tasks via reinforcement finetuning (RFT), which refers to maximizing a (possibly learned) rewa…
Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing
Jarosław Błasiok, Preetum Nakkiran
Calibration measures and reliability diagrams are two fundamental tools for measuring and interpreting the calibration of probabilistic predictors. Calibration measures quantify th…
When Does Optimizing a Proper Loss Yield Calibration?
Jarosław Błasiok, Parikshit Gopalan, Lunjia Hu +1
Optimizing proper loss functions is popularly believed to yield predictors with good calibration properties; the intuition being that for such losses, the global optimum is to pred…
Loss Minimization Yields Multicalibration for Large Neural Networks
Jarosław Błasiok, Parikshit Gopalan, Lunjia Hu +2
Multicalibration is a notion of fairness for predictors that requires them to provide calibrated predictions across a large set of protected groups. Multicalibration is known to be…
A Unifying Theory of Distance from Calibration
Jarosław Błasiok, Parikshit Gopalan, Lunjia Hu +1
We study the fundamental question of how to define and measure the distance from calibration for probabilistic predictors. While the notion of perfect calibration is well-understoo…
APE: Aligning Pretrained Encoders to Quickly Learn Aligned Multimodal Representations
Elan Rosenfeld, Preetum Nakkiran, Hadi Pouransari +2
Recent advances in learning aligned multimodal representations have been primarily driven by training large neural networks on massive, noisy paired-modality datasets. In this work…
The Calibration Generalization Gap
A. Michael Carrell, Neil Mallinar, James Lucas +1
Calibration is a fundamental property of a good predictive model: it requires that the model predicts correctly in proportion to its confidence. Modern neural networks, however, pr…
Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
Neil Mallinar, James B. Simon, Amirhesam Abedsoltan +3
The practical success of overparameterized neural networks has motivated the recent scientific study of interpolating methods, which perfectly fit their training data. Certain inte…
Limitations of the NTK for Understanding Generalization in Deep Learning
Nikhil Vyas, Yamini Bansal, Preetum Nakkiran
The ``Neural Tangent Kernel'' (NTK) (Jacot et al 2018), and its empirical variants have been proposed as a proxy to capture certain behaviors of real neural networks. In this work,…
What You See is What You Get: Principled Deep Learning via Distributional Generalization
Bogdan Kulynych, Yao-Yuan Yang, Yaodong Yu +2
Having similar behavior at training time and test time what we call a "What You See Is What You Get" (WYSIWYG) property is desirable in machine learning. Models trained wit…
Knowledge Distillation: Bad Models Can Be Good Role Models
Gal Kaplun, Eran Malach, Preetum Nakkiran +1
Large neural networks trained in the overparameterized regime are able to fit noise to zero train error. Recent work \citep{nakkiran2020distributional} has empirically observed tha…
Limitations of Neural Collapse for Understanding Generalization in Deep Learning
Like Hui, Mikhail Belkin, Preetum Nakkiran
The recent work of Papyan, Han, & Donoho (2020) presented an intriguing "Neural Collapse" phenomenon, showing a structural property of interpolating classifiers in the late stage o…
Deconstructing Distributions: A Pointwise Framework of Learning
Gal Kaplun, Nikhil Ghosh, Saurabh Garg +2
In machine learning, we traditionally evaluate the performance of a single model, averaged over a collection of test inputs. In this work, we propose a new approach: we measure the…