2.3k citations · 2.7k across the 38 of their papers we have counts for
10 papers · 1 filter
An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence
Qizhen Zhang, Ankush Garg, Jakob Foerster +3
Large-scale pretraining datasets drive the success of large language models (LLMs). However, these web-scale corpora inevitably contain large amounts of noisy data due to unregulat…
Compute Optimal Scaling of Skills: Knowledge vs Reasoning
Nicholas Roberts, Niladri Chatterji, Sharan Narang +2
Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count…
Deep Linear Networks can Benignly Overfit when Shallow Ones Do
Niladri S. Chatterji, Philip M. Long
We bound the excess risk of interpolating deep linear networks trained using gradient flow. In a setting previously used to establish risk bounds for the minimum -norm inte…
Undersampling is a Minimax Optimal Robustness Intervention in Nonparametric Classification
Niladri S. Chatterji, Saminul Haque, Tatsunori Hashimoto
While a broad range of techniques have been proposed to tackle distribution shift, the simple baseline of training on an balanced dataset often achieves clo…
Random Feature Amplification: Feature Learning and Generalization in Neural Networks
Spencer Frei, Niladri S. Chatterji, Peter L. Bartlett
In this work, we provide a characterization of the feature-learning process in two-layer ReLU networks trained by gradient descent on the logistic loss following random initializat…
Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data
Spencer Frei, Niladri S. Chatterji, Peter L. Bartlett
Benign overfitting, the phenomenon where interpolating models generalize well in the presence of noisy data, was first observed in neural network models trained with gradient desce…
Is Importance Weighting Incompatible with Interpolating Classifiers?
Ke Alexander Wang, Niladri S. Chatterji, Saminul Haque +1
Importance weighting is a classic technique to handle distribution shifts. However, prior work has presented strong empirical and theoretical evidence demonstrating that importance…
On the Opportunities and Risks of Foundation Models
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli +111
AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.…
On the Theory of Reinforcement Learning with Once-per-Episode Feedback
Niladri S. Chatterji, Aldo Pacchiano, Peter L. Bartlett +1
We study a theory of reinforcement learning (RL) in which the learner receives binary feedback only once at the end of an episode. While this is an extreme test case for theory, it…
The intriguing role of module criticality in the generalization of deep networks
Niladri S. Chatterji, Behnam Neyshabur, Hanie Sedghi
We study the phenomenon that some modules of deep neural networks (DNNs) are more critical than others. Meaning that rewinding their parameter values back to initialization, while…